<2018年5月>
293012345
6789101112
13141516171819
20212223242526
272829303112
3456789

文章分类

导航

订阅

Ubuntu内核调试要点(下)——常见问题

  前两讲我们分别介绍了安装调试符号和源文件,以及设置目标机和开始调试会话,这一讲我们介绍一下大家可能遇到的两个常见问题:架构不匹配和内核地址错位。

   架构不匹配

   如果目标机是32位的,那么一般没有问题,如果是64位的,那么主机端的GDB可能误以为对方是32位的,表现出的症状就是找不到函数的符号(因为错把64位的地址当作32位,只取一半),如果观察寄存器,那么也都不对,比如:

(gdb) info registers     
    eax            0x1b 27
    ecx            0x0 0
    edx            0x67 103
    ebx            0x0 0
    esp            0x6 0x6 <irq_stack_union+6>
    ebp            0x0 0x0 <irq_stack_union>
    esi            0x0 0
    edi            0x0 0
    eip            0x246 0x246 <irq_stack_union+582>
    eflags         0x0 [ ]
    cs             0x3f40dc80 1061215360
    ss             0xffff96dd -26915
    ds             0x1e5be38 31833656
    es             0xffffaf96 -20586
    fs             0x1e5be38 31833656
    gs             0xffffaf96 -20586

  仔细观察上面的寄存器内容,对于熟悉x86架构的读者,很容易看出很多异常,比如段寄存器都是16位,数值应该很小,可上面显示的却很大,再比如程序指针寄存器(eip)的值也太小。这都是因为GDB把64位的寄存器上下文强行按32位来对待了。

  有人说,GDB真的这么傻么?真的如此。为了了解GDB里的原委,老雷还特意启动第二个GDB,以GDB调试GDB,截图如下:


  上图左侧是做内核调试的GDB,右侧便是调试左侧GDB的GDB。 

   当我们执行target remote命令时,GDB就会初始化一个gdbarch实例,而此时还未与目标机建立连接,不知道对方是32位还是64位,所以此时GDB是始终初始化一个32为的gdbarch,过程如下:    

#0  i386_gdbarch_init (info=..., arches=0x1240c00) at /build/gdb-9un5Xp/gdb-7.11.1/gdb/i386-tdep.c:8255
#1  0x00000000005e218f in gdbarch_find_by_info (info=...) at /build/gdb-9un5Xp/gdb-7.11.1/gdb/gdbarch.c:5100
#2  0x00000000005e2b71 in gdbarch_update_p (info=...) at /build/gdb-9un5Xp/gdb-7.11.1/gdb/arch-utils.c:522
#3  0x00000000006bb2c3 in target_clear_description () at /build/gdb-9un5Xp/gdb-7.11.1/gdb/target-descriptions.c:400
#4  0x0000000000600637 in target_pre_inferior (from_tty=from_tty@entry=1) at /build/gdb-9un5Xp/gdb-7.11.1/gdb/target.c:2161
#5  0x0000000000601aad in target_preopen (from_tty=from_tty@entry=1) at /build/gdb-9un5Xp/gdb-7.11.1/gdb/target.c:2220
#6  0x00000000004c6b73 in remote_open_1 (name=0x1082e7e "/dev/ttyS0", from_tty=1, target=0xc44040 <remote_ops>, extended_p=0)
    at /build/gdb-9un5Xp/gdb-7.11.1/gdb/remote.c:4886
#7  0x00000000005f4538 in open_target (args=0x1082e7e "/dev/ttyS0", from_tty=1, command=<optimized out>)
    at /build/gdb-9un5Xp/gdb-7.11.1/gdb/target.c:356
#8  0x000000000069dbc6 in execute_command (p=<optimized out>, p@entry=0x1082e70 "", from_tty=1) at /build/gdb-9un5Xp/gdb-7.11.1/gdb/top.c:475
#9  0x00000000005d490c in command_handler (command=0x1082e70 "") at /build/gdb-9un5Xp/gdb-7.11.1/gdb/event-top.c:491
#10 0x00000000005d4fef in command_line_handler (rl=<optimized out>) at /build/gdb-9un5Xp/gdb-7.11.1/gdb/event-top.c:690
#11 0x00007f9a096586f5 in rl_callback_read_char () from /lib/x86_64-linux-gnu/libreadline.so.6
#12 0x00000000005d4969 in rl_callback_read_char_wrapper (client_data=<optimized out>) at /build/gdb-9un5Xp/gdb-7.11.1/gdb/event-top.c:171
#13 0x00000000005d49b3 in stdin_event_handler (error=<optimized out>, client_data=0x0) at /build/gdb-9un5Xp/gdb-7.11.1/gdb/event-top.c:430
#14 0x00000000005d3795 in gdb_wait_for_event (block=block@entry=1) at /build/gdb-9un5Xp/gdb-7.11.1/gdb/event-loop.c:834
#15 0x00000000005d3939 in gdb_do_one_event () at /build/gdb-9un5Xp/gdb-7.11.1/gdb/event-loop.c:323
#16 0x00000000005d3a7e in start_event_loop () at /build/gdb-9un5Xp/gdb-7.11.1/gdb/event-loop.c:347
#17 0x00000000005cd443 in captured_command_loop (data=data@entry=0x0) at /build/gdb-9un5Xp/gdb-7.11.1/gdb/main.c:318
#18 0x00000000005ca25d in catch_errors (func=func@entry=0x5cd430 <captured_command_loop>, func_args=func_args@entry=0x0, 
    errstring=errstring@entry=0x7abfeb "", mask=mask@entry=RETURN_MASK_ALL) at /build/gdb-9un5Xp/gdb-7.11.1/gdb/exceptions.c:240
#19 0x00000000005ce036 in captured_main (data=data@entry=0x7ffec93a15d0) at /build/gdb-9un5Xp/gdb-7.11.1/gdb/main.c:1157
#20 0x00000000005ca25d in catch_errors (func=func@entry=0x5cd990 <captured_main>, func_args=func_args@entry=0x7ffec93a15d0, 
    errstring=errstring@entry=0x7abfeb "", mask=mask@entry=RETURN_MASK_ALL) at /build/gdb-9un5Xp/gdb-7.11.1/gdb/exceptions.c:240
#21 0x00000000005ce90b in gdb_main (args=args@entry=0x7ffec93a15d0) at /build/gdb-9un5Xp/gdb-7.11.1/gdb/main.c:1165
#22 0x000000000045ecd5 in main (argc=<optimized out>, argv=<optimized out>) at /build/gdb-9un5Xp/gdb-7.11.1/gdb/gdb.c:32

  GDB虽然代码量不小,但是设计逻辑还是很清楚的,一个current_inferior_全局变量用来记录当前的被调试对象。在GDB中,调试对象被统称为inferior,是下属晚辈的意思,老雷将其翻译为“下程”(下属程序之意)。

  在current_inferior_结构体中,有一个gdbarch成员,就是用来记录当前调试目标的架构的(32/64位之类),即:

(gdb) p *current_inferior_
$20 = {next = 0x0, num = 1, pid = 42000, fake_pid_p = 1,
  highest_thread_num = 384, control = {stop_soon = STOP_QUIETLY_REMOTE},
  removable = 0, aspace = 0x22d3fc0, pspace = 0x11992b0, args = 0x0,
  argc = 0, argv = 0x0, terminal = 0x0, environment = 0x11c9780,
  attach_flag = 0, vfork_parent = 0x0, vfork_child = 0x0,
  pending_detach = 0, waiting_for_vfork_done = 0, detaching = 0,
  continuations = 0x0, needs_setup = 0, priv = 0x0, has_exit_code = 0,
  exit_code = 0, symfile_flags = 0, tdesc_info = 0x11f2e30,
  gdbarch = 0x1242b80, registry_data = {data = 0x123d160, num_data = 7}}

  观察gdbarch,可以看到它内部的函数指向的都是32位版本(以i386开头)。

  可以用watch命令设置硬件断点监视gdbarch字段的变化,但是当GDB与目标建立连接是,这个断点并没有命中。至少在老雷调试的GDB中,它不会自动检测目标的架构并作切换。

  怎么办呢?答案是需要执行如下命令手工切换:

(gdb) set architecture i386:x86-64
The target architecture is assumed to be i386:x86-64

  这样切换后,再观察gdbarch,就是64位版本的了。

(gdb) p *gdbarch
$42 = {initialized_p = 1, obstack = 0x12e40b0, bfd_arch_info = 0x952dc0 <bfd_x86_64_arch>, byte_order = BFD_ENDIAN_LITTLE, 
  byte_order_for_code = BFD_ENDIAN_LITTLE, osabi = GDB_OSABI_LINUX, target_desc = 0x0, tdep = 0x12e3f10, dump_tdep = 0x0, nr_data = 26, 
  data = 0x12f9990, bits_big_endian = 0, short_bit = 16, int_bit = 32, long_bit = 64, long_long_bit = 64, long_long_align_bit = 32, half_bit = 16, 
  half_format = 0xc37970 <floatformats_ieee_half>, float_bit = 32, float_format = 0xc37960 <floatformats_ieee_single>, double_bit = 64, 
  double_format = 0xc37950 <floatformats_ieee_double>, long_double_bit = 128, long_double_format = 0xc37930 <floatformats_i387_ext>, ptr_bit = 64, 
  addr_bit = 64, dwarf2_addr_size = 8, char_signed = 1, read_pc = 0x0, write_pc = 0x46bf90 <amd64_linux_write_pc>, 
  virtual_frame_pointer = 0x5e29f0 <legacy_virtual_frame_pointer>, pseudo_register_read = 0x0, 
  pseudo_register_read_value = 0x45fc30 <amd64_pseudo_register_read_value>, pseudo_register_write = 0x45fb20 <amd64_pseudo_register_write>, 
  num_regs = 152, num_pseudo_regs = 52, ax_pseudo_register_collect = 0x0, ax_pseudo_register_push_stack = 0x0, sp_regnum = 7, pc_regnum = 16, 
  ps_regnum = 17, fp0_regnum = 24, stab_reg_to_regnum = 0x45f9c0 <amd64_dwarf_reg_to_regnum>, ecoff_reg_to_regnum = 0x5e2990 <no_op_reg_to_regnum>, 
  sdb_reg_to_regnum = 0x47ef30 <i386_dbx_reg_to_regnum>, dwarf2_reg_to_regnum = 0x45f9c0 <amd64_dwarf_reg_to_regnum>, 
  register_name = 0x6bb050 <tdesc_register_name>, register_type = 0x6bc1a0 <tdesc_register_type>, dummy_id = 0x45f950 <amd64_dummy_id>, 
  deprecated_fp_regnum = -1, push_dummy_call = 0x467330 <amd64_push_dummy_call>, call_dummy_location = 1, 
  push_dummy_code = 0x477b80 <i386_push_dummy_code>, print_registers_info = 0x5aef10 <default_print_registers_info>, 
  print_float_info = 0x488170 <i387_print_float_info>, print_vector_info = 0x0, register_sim_regno = 0x5e2890 <legacy_register_sim_regno>, 
  cannot_fetch_register = 0x5e29e0 <cannot_register_not>, cannot_store_register = 0x5e29e0 <cannot_register_not>, 
  get_longjmp_target = 0x45f400 <amd64_get_longjmp_target>, believe_pcc_promotion = 0, convert_register_p = 0x488c50 <i387_convert_register_p>, 
  register_to_value = 0x488c80 <i387_register_to_value>, value_to_register = 0x488dc0 <i387_value_to_register>, 
  value_from_register = 0x561a10 <default_value_from_register>, pointer_to_address = 0x561790 <unsigned_pointer_to_address>, 
  address_to_pointer = 0x5617f0 <unsigned_address_to_pointer>, integer_to_address = 0x0, return_value = 0x465e20 <amd64_return_value>, 
  return_in_first_hidden_param_p = 0x5e3880 <default_return_in_first_hidden_param_p>, skip_prologue = 0x4674a0 <amd64_skip_prologue>, 
  skip_main_prologue = 0x0, skip_entrypoint = 0x0, inner_than = 0x5e2950 <core_addr_lessthan>, 
  breakpoint_from_pc = 0x477b10 <i386_breakpoint_from_pc>, remote_breakpoint_from_pc = 0x5e3850 <default_remote_breakpoint_from_pc>, 
  adjust_breakpoint_address = 0x0, memory_insert_breakpoint = 0x5f1ba0 <default_memory_insert_breakpoint>, 
  memory_remove_breakpoint = 0x5f1c70 <default_memory_remove_breakpoint>, decr_pc_after_break = 1, deprecated_function_start_offset = 0, 
  remote_register_number = 0x6bb020 <tdesc_remote_register_number>, fetch_tls_load_module_address = 0x493be0 <svr4_fetch_objfile_link_map>, 
  frame_args_skip = 8, unwind_pc = 0x47a250 <i386_unwind_pc>, unwind_sp = 0x0, frame_num_args = 0x0, frame_align = 0x45efd0 <amd64_frame_align>, 
  stabs_argument_has_addr = 0x5e2ac0 <default_stabs_argument_has_addr>, frame_red_zone_size = 128, 
  convert_from_func_ptr_addr = 0x5e2980 <convert_from_func_ptr_addr_identity>, addr_bits_remove = 0x5e2970 <core_addr_identity>, 
  software_single_step = 0x0, single_step_through_delay = 0x0, print_insn = 0x47e170 <i386_print_insn>, 
  skip_trampoline_code = 0x613c30 <find_solib_trampoline_target>, skip_solib_resolver = 0x490380 <glibc_skip_solib_resolver>, 
  in_solib_return_trampoline = 0x5e2930 <generic_in_solib_return_trampoline>, stack_frame_destroyed_p = 0x5e2940 <generic_stack_frame_destroyed_p>, 
  elf_make_msymbol_special = 0x0, coff_make_msymbol_special = 0x5e29a0 <default_coff_make_msymbol_special>, 
  make_symbol_special = 0x5e29b0 <default_make_symbol_special>, adjust_dwarf2_addr = 0x5e29c0 <default_adjust_dwarf2_addr>, 
  adjust_dwarf2_line = 0x5e29d0 <default_adjust_dwarf2_line>, cannot_step_breakpoint = 0, have_nonsteppable_watchpoint = 0, 
  address_class_type_flags = 0x0, address_class_type_flags_to_name = 0x0, address_class_name_to_type_flags = 0x0, 
  register_reggroup_p = 0x474950 <amd64_linux_register_reggroup_p>, fetch_pointer_argument = 0x4796f0 <i386_fetch_pointer_argument>, 
  iterate_over_regset_sections = 0x46b610 <amd64_linux_iterate_over_regset_sections>, make_corefile_notes = 0x496c60 <linux_make_corefile_notes>, 
  elfcore_write_linux_prpsinfo = 0x0, find_memory_regions = 0x4976a0 <linux_find_memory_regions>, core_xfer_shared_libraries = 0x0, 
  core_xfer_shared_libraries_aix = 0x0, core_pid_to_str = 0x495e30 <linux_core_pid_to_str>, core_thread_name = 0x0, gcore_bfd_target = 0x0, 
  vtable_function_descriptors = 0, vbit_in_delta = 0, skip_permanent_breakpoint = 0x5e38c0 <default_skip_permanent_breakpoint>, max_insn_length = 16, 
  displaced_step_copy_insn = 0x467770 <amd64_displaced_step_copy_insn>, 
  displaced_step_hw_singlestep = 0x5e2810 <default_displaced_step_hw_singlestep>, displaced_step_fixup = 0x467b10 <amd64_displaced_step_fixup>, 
  displaced_step_free_closure = 0x5e2800 <simple_displaced_step_free_closure>, displaced_step_location = 0x4981d0 <linux_displaced_step_location>, 
  relocate_instruction = 0x45f0f0 <amd64_relocate_instruction>, overlay_update = 0x0, 
  core_read_description = 0x4748a0 <amd64_linux_core_read_description>, static_transform_name = 0x0, sofun_address_maybe_missing = 0, 
  process_record = 0x480860 <i386_process_record>, process_record_signal = 0x474800 <amd64_linux_record_signal>, 
  gdb_signal_from_target = 0x494280 <linux_gdb_signal_from_target>, gdb_signal_to_target = 0x4944c0 <linux_gdb_signal_to_target>, 
  get_siginfo_type = 0x48b2d0 <x86_linux_get_siginfo_type>, record_special_symbol = 0x0, 
  get_syscall_number = 0x46bf10 <amd64_linux_get_syscall_number>, xml_syscall_file = 0x79fce7 "syscalls/amd64-linux.xml", syscalls_info = 0x0, 
---Type <return> to continue, or q <return> to quit---
  stap_integer_prefixes = 0x79f3b0 <stap_integer_prefixes>, stap_integer_suffixes = 0x0, stap_register_prefixes = 0x79f3a0 <stap_register_prefixes>, 
  stap_register_suffixes = 0x0, stap_register_indirection_prefixes = 0x79f390 <stap_register_indirection_prefixes>, 
  stap_register_indirection_suffixes = 0x79f380 <stap_register_indirection_suffixes>, stap_gdb_register_prefix = 0x0, stap_gdb_register_suffix = 0x0, 
  stap_is_single_operand = 0x478270 <i386_stap_is_single_operand>, stap_parse_special_token = 0x478b80 <i386_stap_parse_special_token>, 
  dtrace_parse_probe_argument = 0x474990 <amd64_dtrace_parse_probe_argument>, dtrace_probe_is_enabled = 0x46c8e0 <amd64_dtrace_probe_is_enabled>, 
  dtrace_enable_probe = 0x46c8c0 <amd64_dtrace_enable_probe>, dtrace_disable_probe = 0x46c8a0 <amd64_dtrace_disable_probe>, has_global_solist = 0, 
  has_global_breakpoints = 0, has_shared_address_space = 0x4981c0 <linux_has_shared_address_space>, 
  fast_tracepoint_valid_at = 0x4795d0 <i386_fast_tracepoint_valid_at>, auto_charset = 0x566ae0 <default_auto_charset>, 
  auto_wide_charset = 0x566af0 <default_auto_wide_charset>, solib_symbols_extension = 0x0, has_dos_based_file_system = 0, 
  gen_return_address = 0x45f090 <amd64_gen_return_address>, info_proc = 0x494f70 <linux_info_proc>, core_info_proc = 0x4976f0 <linux_core_info_proc>, 
  iterate_over_objfiles_in_search_order = 0x60fa60 <default_iterate_over_objfiles_in_search_order>, ravenscar_ops = 0x0, 
  insn_is_call = 0x45f080 <amd64_insn_is_call>, insn_is_ret = 0x45f070 <amd64_insn_is_ret>, insn_is_jump = 0x45f060 <amd64_insn_is_jump>, 
  auxv_parse = 0x0, vsyscall_range = 0x494990 <linux_vsyscall_range>, infcall_mmap = 0x494800 <linux_infcall_mmap>, 
  infcall_munmap = 0x494710 <linux_infcall_munmap>, gcc_target_options = 0x5e3970 <default_gcc_target_options>, 
  gnu_triplet_regexp = 0x477ba0 <i386_gnu_triplet_regexp>, addressable_memory_unit_size = 0x5e39d0 <default_addressable_memory_unit_size>}

  接下来,再观察寄存器,就对了,比如:

(gdb) i r
rax            0x2f    47
rbx            0xffffffff81efadc0    -2114998848
rcx            0xffffffff81e5f568    -2115635864
rdx            0x0    0
rsi            0x246    582
rdi            0x246    582
rbp            0xffffc90000643da8    0xffffc90000643da8
rsp            0xffffc90000643da8    0xffffc90000643da8
r8             0xb57e8    743400
r9             0x207    519
r10            0xd2a81d91    3534232977
r11            0xffffffff822487ed    -2111535123
r12            0xffff880111a2be40    -131936804487616
r13            0x0    0
r14            0x55a9bbc50cf0    94187488087280
r15            0x55a9bbc503c0    94187488084928
rip            0xffffffff81141344    0xffffffff81141344 <kgdb_breakpoint+20>
eflags         0x202    [ IF ]
cs             0x10    16
ss             0x0    0
ds             0x0    0
es             0x0    0
fs             0x0    0
gs             0xb    11

 

 内核地址错位

 第二个常见的问题是,GDB里找不到函数的名字,长话短说是因为在GDB看来,内核的编译地址和运行时的地址是一样的,所以就直接用符号文件中查找到的地址来访问目标机器。但其实,今天的较高版本内核(3.14+)可能启用了KASLR(内核空间地址随机化),把内核也做了重定位。

  不妨做个小实验来理解KASLR,先观察/proc/kallsyms中的vfs_read函数的地址(这是实际运行时的地址),再观察/boot目录下编译时的地址,会发现二者明显不同。

gedu@gedu-VirtualBox:~$ sudo cat /proc/kallsyms | grep " vfs_read"
ffffffff86c31960 T vfs_readf
ffffffff86c32ce0 T vfs_read
ffffffff86c33210 T vfs_readv
gedu@gedu-VirtualBox:~$ sudo cat /boot/System.map-4.8.0-36-generic | grep " vfs_read"
ffffffff81231960 T vfs_readf
ffffffff81232ce0 T vfs_read
ffffffff81233210 T vfs_readv

KASLR是一项安全措施,与黑客捉迷藏。但这样躲躲闪闪,让GDB也蒙了。

  那么如何让告诉GDB内核搬家了呢?今天用的方法大多都是禁止KASLR,也就是在内核的命令行中加入nokaslr,然后重启问题就消除了。 

  读到这里,大家是不觉得陷阱很多啊。是的,新问题,新挑战总是有的。GNU领袖RMS在GDB教程的封面如此写道:“Don't worry if it doesn't work right. If everything did, you'd be out of job.”翻译一下:“别担心它工作的不好。如果一切都工作的好,那么你就没工作了。” 其实,我并不很赞同这句话,因为不够积极。不过RMS如此说,也只是开个玩笑而已,看看他的相册,可以看到他走到哪里就在哪里掏出本子改BUG啊。 :-) 


posted on 2018年3月10日 14:21 由 admin

Powered by Community Server Powered by CnForums.Net