工欲善其事，必先利其器。本文主要介绍linux下crash工具常用命令的功能和使用。

背景知识

crash是redhat的工程师开发的，主要用来离线分析linux内核转存文件，它整合了gdb工具，功能非常强大。可以查看堆栈，dmesg日志，内核数据结构，反汇编等等。crash支持多种工具生成的转存文件格式，如kdump，LKCD，netdump和diskdump，而且还可以分析虚拟机Xen和Kvm上生成的内核转存文件。同时crash还可以调试运行时系统，直接运行crash即可，ubuntu下内核映象存放在/proc/kcore。

运行时系统调试

crash和linux内核是紧密耦合的，会随着内核的变化持续更新，它向前兼容的，新的crash工具可以分析老内核的转存文件。如果你的内核版本较新，crash无法解析，可以尝试安装最新的crash工具。

常用命令

下面介绍常用命令的使用，主要参考了crash_whitepaper和crash工具自带的帮助文档。crash_whitepaper介绍了开发的初衷，编译，命令的分类和使用以及如何添加自己的命令，是一个非常好的参考文献。我用的版本是crash-7.2.6和gdb-7.6，使用时可以使用“help command”来查看详细的帮助文档，详细的命令列表见附件。

帮助文档

crash在加载内核转存文件是会输出系统基本信息，如出问题的进程(bash - 2613)，系统内存大小(7.9GB)，系统架构(x86_64)等等，可以看到这个dump是sysrq触发的一个panic系统崩溃。

KERNEL: ../kernel-src/linux-4.19.53/vmlinux

DUMPFILE: crash/201907070732/dump.201907070732 [PARTIAL DUMP]

CPUS: 4

DATE: Sun Jul 7 07:31:34 2019

UPTIME: 00:10:27

LOAD AVERAGE: 0.14, 0.16, 0.12

TASKS: 584

NODENAME: glbian-OptiPlex-990

RELEASE: 4.19.53

VERSION: #1 SMP Sun Jun 23 11:01:25 CST 2019

MACHINE: x86_64 (3292 Mhz)

MEMORY: 7.9 GB

PANIC: "sysrq: SysRq : Trigger a crash"

PID: 2613

COMMAND: "bash"

TASK: ffff8b7df3cdae00 [THREAD_INFO: ffff8b7df3cdae00]

CPU: 2

STATE: TASK_RUNNING (SYSRQ)

查看堆栈

一般可以先查看堆栈(bt)，看看系统死在什么地方，进而确定调查方向。可以看到这个dump的异常发生在sysrq的处理函数里面。

crash> bt

PID: 2613 TASK: ffff8b7df3cdae00 CPU: 2 COMMAND: "bash"

'#0 [ffffa0f442cd7a08] machine_kexec at ffffffff99a69313

'#1 [ffffa0f442cd7a68] __crash_kexec at ffffffff99b3e6b9

'#2 [ffffa0f442cd7b30] crash_kexec at ffffffff99b3f441

'#3 [ffffa0f442cd7b50] oops_end at ffffffff99a32bed

'#4 [ffffa0f442cd7b78] no_context at ffffffff99a7997c

'#5 [ffffa0f442cd7bd8] __bad_area_nosemaphore at ffffffff99a79d15

'#6 [ffffa0f442cd7c20] bad_area at ffffffff99a79f86

'#7 [ffffa0f442cd7c48] __do_page_fault at ffffffff99a7a486

'#8 [ffffa0f442cd7cc0] do_page_fault at ffffffff99a7a60d

'#9 [ffffa0f442cd7cf0] page_fault at ffffffff9a6010ae

[exception RIP: sysrq_handle_crash+22]

RIP: ffffffff9a034066 RSP: ffffa0f442cd7da8 RFLAGS: 00010286

RAX: ffffffff9a034050 RBX: 0000000000000063 RCX: 0000000000000006

RDX: 0000000000000000 RSI: 0000000000000096 RDI: 0000000000000063

RBP: ffffa0f442cd7da8 R8: 00000000000002f2 R9: 0000000000000007

R10: 0000000000000000 R11: ffffffff9b39c3ed R12: 0000000000000004

R13: 0000000000000000 R14: ffffffff9afa7300 R15: ffff8b7de5af9100

ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018

'#10 [ffffa0f442cd7db0] __handle_sysrq at ffffffff9a0347e8

'#11 [ffffa0f442cd7de0] write_sysrq_trigger at ffffffff9a034cbf

... ...

另外可以加参数显示函数偏移，函数所在的文件和每一帧的具体内容，从而对照源码和汇编代码，查看函数入参和局部变量。

crash> bt -slf

PID: 2613 TASK: ffff8b7df3cdae00 CPU: 2 COMMAND: "bash"

'#0 [ffffa0f442cd7a08] machine_kexec+451 at ffffffff99a69313

/home/glbian/data/kernel-src/linux-4.19.53/arch/x86/kernel/machine_kexec_64.c: 346

ffffa0f442cd7a10: 0000a0f442cd7a50 ffff8b7c40000000

ffffa0f442cd7a20: 0000000024001000 ffff8b7c64001000

ffffa0f442cd7a30: 0000000024000000 a05cedc0dfb99200

ffffa0f442cd7a40: a05cedc0dfb99200 ffffa0f442cd7cf8

ffffa0f442cd7a50: 0000000000000009 ffffa0f442cd7cf8

ffffa0f442cd7a60: ffffa0f442cd7b28 ffffffff99b3e6b9

... ...

’#8 [ffffa0f442cd7cc0] do_page_fault+45 at ffffffff99a7a60d

/home/glbian/data/kernel-src/linux-4.19.53/arch/x86/mm/fault.c: 1470

ffffa0f442cd7cc8: ffff8b7e6500d140 0000000000000000

ffffa0f442cd7cd8: 0000000000000000 0000000000000000

ffffa0f442cd7ce8: ffffa0f442cd7cf9 ffffffff9a6010ae

'#9 [ffffa0f442cd7cf0] page_fault+30 at ffffffff9a6010ae

/home/glbian/data/kernel-src/linux-4.19.53/arch/x86/entry/entry_64.S: 1181

[exception RIP: sysrq_handle_crash+22]

RIP: ffffffff9a034066 RSP: ffffa0f442cd7da8 RFLAGS: 00010286

RAX: ffffffff9a034050 RBX: 0000000000000063 RCX: 0000000000000006

RDX: 0000000000000000 RSI: 0000000000000096 RDI: 0000000000000063

RBP: ffffa0f442cd7da8 R8: 00000000000002f2 R9: 0000000000000007

R10: 0000000000000000 R11: ffffffff9b39c3ed R12: 0000000000000004

R13: 0000000000000000 R14: ffffffff9afa7300 R15: ffff8b7de5af9100

ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018

/home/glbian/data/kernel-src/linux-4.19.53/drivers/tty/sysrq.c: 147

ffffa0f442cd7cf8: ffff8b7de5af9100 ffffffff9afa7300

ffffa0f442cd7d08: 0000000000000000 0000000000000004

ffffa0f442cd7d18: ffffa0f442cd7da8 0000000000000063

ffffa0f442cd7d28: ffffffff9b39c3ed 0000000000000000

ffffa0f442cd7d38: 0000000000000007 00000000000002f2

ffffa0f442cd7d48: ffffffff9a034050 0000000000000006

ffffa0f442cd7d58: 0000000000000000 0000000000000096

ffffa0f442cd7d68: 0000000000000063 ffffffffffffffff

ffffa0f442cd7d78: ffffffff9a034066 0000000000000010

ffffa0f442cd7d88: 0000000000010286 ffffa0f442cd7da8

ffffa0f442cd7d98: 0000000000000018 0000000000000000

ffffa0f442cd7da8: ffffa0f442cd7dd8 ffffffff9a0347e8

'#10 [ffffa0f442cd7db0] __handle_sysrq+136 at ffffffff9a0347e8

/home/glbian/data/kernel-src/linux-4.19.53/drivers/tty/sysrq.c: 583

ffffa0f442cd7db8: 0000000000000002 fffffffffffffffb

ffffa0f442cd7dc8: ffffa0f442cd7ee8 0000563d45717780

ffffa0f442cd7dd8: ffffa0f442cd7df0 ffffffff9a034cbf

... ...

可以用dis命令进行返汇编，查看对应地址的代码逻辑。

>crash> dis -r ffffffff9a6010ae

0xffffffff9a601090 : data32 xchg %ax,%ax

0xffffffff9a601093 : callq 0xffffffff9a601230

0xffffffff9a601098 : mov %rsp,%rdi

0xffffffff9a60109b : mov 0x78(%rsp),%rsi

0xffffffff9a6010a0 : movq $0xffffffffffffffff,0x78(%rsp)

0xffffffff9a6010a9 : callq 0xffffffff99a7a5e0

0xffffffff9a6010ae : jmpq 0xffffffff9a601330

>crash> dis -f ffffffff9a6010ae

0xffffffff9a6010ae : jmpq 0xffffffff9a601330

0xffffffff9a6010b3 : nopl (%rax)

0xffffffff9a6010b6 : nopw %cs:0x0(%rax,%rax,1)

有时会出现堆栈被破坏的情况，可以用-t/-T来把整个stack的信息dump出来，往往可以看到一些蛛丝马迹。

crash> bt -t

PID: 2613 TASK: ffff8b7df3cdae00 CPU: 2 COMMAND: "bash"

START: machine_kexec at ffffffff99a69313

[ffffa0f442cd7a08] machine_kexec at ffffffff99a69313

[ffffa0f442cd7a68] __crash_kexec at ffffffff99b3e6b9

[ffffa0f442cd7ac0] sysrq_handle_crash at ffffffff9a034050

[ffffa0f442cd7af0] sysrq_handle_crash at ffffffff9a034066

[ffffa0f442cd7b30] crash_kexec at ffffffff99b3f441

[ffffa0f442cd7b38] __die at ffffffff99a33375

[ffffa0f442cd7b50] oops_end at ffffffff99a32bed

[ffffa0f442cd7b78] no_context at ffffffff99a7997c

[ffffa0f442cd7bd8] __bad_area_nosemaphore at ffffffff99a79d15

[ffffa0f442cd7c20] bad_area at ffffffff99a79f86

[ffffa0f442cd7c48] __do_page_fault at ffffffff99a7a486

[ffffa0f442cd7cc0] do_page_fault at ffffffff99a7a60d

[ffffa0f442cd7cf0] page_fault at ffffffff9a6010ae

[ffffa0f442cd7d48] sysrq_handle_crash at ffffffff9a034050

[ffffa0f442cd7d78] sysrq_handle_crash at ffffffff9a034066

[ffffa0f442cd7db0] __handle_sysrq at ffffffff9a0347e8

[ffffa0f442cd7de0] write_sysrq_trigger at ffffffff9a034cbf

[ffffa0f442cd7df8] proc_reg_write at ffffffff99d2a0ee

[ffffa0f442cd7e18] __vfs_write at ffffffff99ca8a0a

[ffffa0f442cd7e40] apparmor_file_permission at ffffffff99e53a0a

[ffffa0f442cd7e50] security_file_permission at ffffffff99e06cf1

[ffffa0f442cd7e78] _cond_resched at ffffffff9a4153f9

[ffffa0f442cd7ea0] vfs_write at ffffffff99ca8d11

[ffffa0f442cd7ed8] ksys_write at ffffffff99ca8fcc

[ffffa0f442cd7f20] __x64_sys_write at ffffffff99ca906a

[ffffa0f442cd7f30] do_syscall_64 at ffffffff99a0428a

[ffffa0f442cd7f50] entry_SYSCALL_64_after_hwframe at ffffffff9a600088

RIP: 00007ff47e1ef154 RSP: 00007ffee9226298 RFLAGS: 00000246

RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 00007ff47e1ef154

RDX: 0000000000000002 RSI: 0000563d45717780 RDI: 0000000000000001

RBP: 0000563d45717780 R8: 000000000000000a R9: 0000000000000001

R10: 000000000000000a R11: 0000000000000246 R12: 00007ff47e4cb760

R13: 0000000000000002 R14: 00007ff47e4c72a0 R15: 00007ff47e4c6760

ORIG_RAX: 0000000000000001 CS: 0033 SS: 002b

默认bt会dump问题线程的场景，还可以用bt -a/-c查看所有当前CPU或指定cpu的堆栈。

crash> bt -c 1

PID: 0 TASK: ffff8b7e64165c00 CPU: 1 COMMAND: "swapper/1"

'#0 [fffffe0000034e38] crash_nmi_callback at ffffffff99a5d3d7

'#1 [fffffe0000034e48] nmi_handle at ffffffff99a33691

... ...

'#12 [ffffa0f440cd7f50] secondary_startup_64 at ffffffff99a000d4

crash> bt -a

PID: 0 TASK: ffffffff9ae13740 CPU: 0 COMMAND: "swapper/0"

... ...

PID: 0 TASK: ffff8b7e64165c00 CPU: 1 COMMAND: "swapper/1"

... ...

PID: 2613 TASK: ffff8b7df3cdae00 CPU: 2 COMMAND: "bash"

... ...

PID: 0 TASK: ffff8b7e642c4500 CPU: 3 COMMAND: "swapper/3"

... ...

也可以用set命令来改变线程环境，从而查看别的cpu上的堆栈情况。

crash> set 1

PID: 1

COMMAND: "systemd"

TASK: ffff8b7e6413c500 [THREAD_INFO: ffff8b7e6413c500]

CPU: 3

STATE: TASK_INTERRUPTIBLE

crash> bt

PID: 1 TASK: ffff8b7e6413c500 CPU: 3 COMMAND: "systemd"

'#0 [ffffa0f440c6fce0] __schedule at ffffffff9a414ba7

'#1 [ffffa0f440c6fd80] schedule at ffffffff9a41519c

'#2 [ffffa0f440c6fd90] schedule_hrtimeout_range_clock at ffffffff9a419691

'#3 [ffffa0f440c6fe20] schedule_hrtimeout_range at ffffffff9a4196b3

'#4 [ffffa0f440c6fe30] ep_poll at ffffffff99cf8941

'#5 [ffffa0f440c6fee0] do_epoll_wait at ffffffff99cf8ae0

'#6 [ffffa0f440c6ff20] __x64_sys_epoll_wait at ffffffff99cf8b0e

'#7 [ffffa0f440c6ff30] do_syscall_64 at ffffffff99a0428a

'#8 [ffffa0f440c6ff50] entry_SYSCALL_64_after_hwframe at ffffffff9a600088

RIP: 00007ffa791c6bb7 RSP: 00007ffc1c00b9d0 RFLAGS: 00000293

RAX: ffffffffffffffda RBX: 0000000000000004 RCX: 00007ffa791c6bb7

RDX: 00000000000000eb RSI: 00007ffc1c00ba10 RDI: 0000000000000004

RBP: 00007ffc1c00ba10 R8: 0000000000000000 R9: 7465677261742e79

R10: 00000000ffffffff R11: 0000000000000293 R12: 00000000000000eb

R13: 00000000ffffffff R14: 00007ffc1c00ba10 R15: 0000000000000001

ORIG_RAX: 00000000000000e8 CS: 0033 SS: 002b

系统日志

log命令可以用来查看系统的日志，“log -a”可以读取还没有从内核日志缓存到用户空间日志缓存的日志。

也可以重定向到文件(log > logfile)。

crash> log

... ...

[ 1610.759133] sysrq: SysRq : Trigger a crash

[ 1610.759147] BUG: unable to handle kernel NULL pointer dereference at 0000000000000000

[ 1610.759150] PGD 0 P4D 0

[ 1610.759154] Oops: 0002 [#1] SMP PTI

[ 1610.759159] CPU: 2 PID: 2613 Comm: bash Kdump: loaded Not tainted 4.19.53 #1

[ 1610.759161] Hardware name: Dell Inc. OptiPlex 990/0RVG2C, BIOS A13 04/02/2012

[ 1610.759167] RIP: 0010:sysrq_handle_crash+0x16/0x20

[ 1610.759170] Code: e8 9f fb ff ff e9 c0 fe ff ff 90 90 90 90 90 90 90 90 90 90 66 66 66 66 90 55 48 89 e5 c7 05 85 10 36 01 01 00 00 00 0f ae f8 04 25 00 00 00 00 01 5d c3 66 66 66 66 90 55 c7 05 40 fa e2 00

[ 1610.759173] RSP: 0018:ffffa0f442cd7da8 EFLAGS: 00010286

[ 1610.759176] RAX: ffffffff9a034050 RBX: 0000000000000063 RCX: 0000000000000006

[ 1610.759178] RDX: 0000000000000000 RSI: 0000000000000096 RDI: 0000000000000063

[ 1610.759180] RBP: ffffa0f442cd7da8 R08: 00000000000002f2 R09: 0000000000000007

[ 1610.759182] R10: 0000000000000000 R11: ffffffff9b39c3ed R12: 0000000000000004

[ 1610.759184] R13: 0000000000000000 R14: ffffffff9afa7300 R15: ffff8b7de5af9100

[ 1610.759186] FS: 00007ff47eb0a740(0000) GS:ffff8b7e65880000(0000) knlGS:0000000000000000

[ 1610.759189] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033

[ 1610.759191] CR2: 0000000000000000 CR3: 0000000205db0003 CR4: 00000000000606e0

[ 1610.759193] Call Trace:

[ 1610.759199] __handle_sysrq+0x88/0x140

[ 1610.759203] write_sysrq_trigger+0x2f/0x40

[ 1610.759208] proc_reg_write+0x3e/0x60

[ 1610.759212] __vfs_write+0x3a/0x190

[ 1610.759216] ? apparmor_file_permission+0x1a/0x20

[ 1610.759220] ? security_file_permission+0x31/0xc0

[ 1610.759224] ? _cond_resched+0x19/0x40

[ 1610.759226] vfs_write+0xb1/0x1a0

[ 1610.759229] ksys_write+0x5c/0xe0

[ 1610.759232] __x64_sys_write+0x1a/0x20

[ 1610.759237] do_syscall_64+0x5a/0x120

[ 1610.759241] entry_SYSCALL_64_after_hwframe+0x44/0xa9

[ 1610.759245] RIP: 0033:0x7ff47e1ef154

[ 1610.759247] Code: 89 02 48 c7 c0 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 66 90 48 8d 05 b1 07 2e 00 8b 00 85 c0 75 13 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 54 f3 c3 66 90 41 54 55 49 89 d4 53 48 89 f5

[ 1610.759249] RSP: 002b:00007ffee9226298 EFLAGS: 00000246 ORIG_RAX: 0000000000000001

[ 1610.759252] RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 00007ff47e1ef154

[ 1610.759254] RDX: 0000000000000002 RSI: 0000563d45717780 RDI: 0000000000000001

[ 1610.759256] RBP: 0000563d45717780 R08: 000000000000000a R09: 0000000000000001

[ 1610.759258] R10: 000000000000000a R11: 0000000000000246 R12: 00007ff47e4cb760

[ 1610.759260] R13: 0000000000000002 R14: 00007ff47e4c72a0 R15: 00007ff47e4c6760

[ 1610.759263] Modules linked in: nls_iso8859_1 intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel crct10dif_pclmul crc32_pclmul ghash_clmulni_intel snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic pcbc aesni_intel snd_hda_intel snd_hda_codec snd_hda_core snd_hwdep aes_x86_64 snd_pcm snd_seq_midi snd_seq_midi_event snd_rawmidi input_leds crypto_simd cryptd snd_seq snd_seq_device snd_timer dcdbas snd glue_helper intel_cstate intel_rapl_perf lpc_ich serio_raw soundcore sch_fq_codel mei_me mei mac_hid parport_pc ppdev lp parport ip_tables x_tables autofs4 hid_generic usbhid hid uas usb_storage i915 kvmgt vfio_mdev mdev vfio_iommu_type1 vfio kvm irqbypass i2c_algo_bit cec rc_core drm_kms_helper psmouse syscopyarea sysfillrect video sysimgblt fb_sys_fops ahci drm libahci e1000e

[ 1610.759320] CR2: 0000000000000000

查看数据结构

struct和union可以用来查看结构体和共用体，用法相同，下面看一些struct

打印的例子。把指定地址的内容以task_struct结构体解析打印，如果不带地址会显示结构体定义和大小。

1 打印task_struct结构体

crash> task_struct ffff8b7df3cdae00 -x

struct task_struct {

thread_info = {

flags = 0x80000000,

status = 0x0

},

state = 0x0,

stack = 0xffffa0f442cd4000,

usage = {

counter = 0x2

},

... ...

2 打印task_struct定义和大小。

struct task_struct {

[0x0] struct thread_info thread_info;

[0x10] volatile long state;

[0x18] void *stack;

... ...

[0x1288] void *security;

[0x12c0] struct thread_struct thread;

}

SIZE: 0x23c0

3 查看成员变量

crash> task_struct.stack_refcount ffff8b7df3cdae00 -xo

struct task_struct {

[ffff8b7df3cdc080] atomic_t stack_refcount;

}

4 查看指针成员变量

crash> task_struct.mm ffff8b7df3cdae00

mm = 0xffff8b7e5af06600

crash> task_struct.mm ffff8b7df3cdae00 -p

struct mm_struct *mm = 0xffff8b7e5af06600

-> {

{

mmap = 0xffff8b7dec0520c8,

mm_rb = {

rb_node = 0xffff8b7dec003b78

},

vmacache_seqnum = 17,

get_unmapped_area = 0xffffffff99a35760,

此外还可以查看数组内容，per-cpu变量，以及其他一些功能，详细可参考帮助文档。

查看和搜索内存

除了打印数据结构，有时需要查看和搜索内存内容，看有没有制定的数据模式。

1 查看系统版本信息

crash> rd -a linux_banner

ffffffff9aa00100: Linux version 4.19.53 (glbian@glbian-OptiPlex-990) (gcc vers

ffffffff9aa0013c: ion 7.4.0 (Ubuntu 7.4.0-1ubuntu1~18.04.1)) #1 SMP Sun Jun 23

ffffffff9aa00178: 11:01:25 CST 2019

查看内存内容

crash> rd ffffa0f442cd7a08 32

ffffa0f442cd7a08: ffffffff99a69313 0000a0f442cd7a50 ........Pz.B....

ffffa0f442cd7a18: ffff8b7c40000000 0000000024001000 ...@|......

....

ffffa0f442cd7a38: a05cedc0dfb99200 a05cedc0dfb99200 ..............

ffffa0f442cd7a48: ffffa0f442cd7cf8 0000000000000009 .|.B............

ffffa0f442cd7a58: ffffa0f442cd7cf8 ffffa0f442cd7b28 .|.B....({.B....

ffffa0f442cd7a68: ffffffff99b3e6b9 ffff8b7de5af9100 ............}...

ffffa0f442cd7a78: ffffffff9afa7300 0000000000000000 .s..............

ffffa0f442cd7a88: 0000000000000004 ffffa0f442cd7da8 .........}.B....

ffffa0f442cd7a98: 0000000000000063 ffffffff9b39c3ed c.........9.....

ffffa0f442cd7aa8: 0000000000000000 0000000000000007 ................

ffffa0f442cd7ab8: 00000000000002f2 ffffffff9a034050 ........P@......

ffffa0f442cd7ac8: 0000000000000006 0000000000000000 ................

ffffa0f442cd7ad8: 0000000000000096 0000000000000063 ........c.......

ffffa0f442cd7ae8: ffffffffffffffff ffffffff9a034066 ........f@......

ffffa0f442cd7af8: 0000000000000010 0000000000010286 ................

3 打印符号表

crash> rd ffffa0f442cd7a08 32 -s

ffffa0f442cd7a08: machine_kexec+451 0000a0f442cd7a50

ffffa0f442cd7a18: ffff8b7c40000000 0000000024001000

ffffa0f442cd7a28: ffff8b7c64001000 0000000024000000

ffffa0f442cd7a38: a05cedc0dfb99200 a05cedc0dfb99200

ffffa0f442cd7a48: ffffa0f442cd7cf8 0000000000000009

ffffa0f442cd7a58: ffffa0f442cd7cf8 ffffa0f442cd7b28

ffffa0f442cd7a68: __crash_kexec+105 ffff8b7de5af9100

ffffa0f442cd7a78: sysrq_crash_op 0000000000000000

ffffa0f442cd7a88: 0000000000000004 ffffa0f442cd7da8

ffffa0f442cd7a98: 0000000000000063 text.45672+13

ffffa0f442cd7aa8: 0000000000000000 0000000000000007

ffffa0f442cd7ab8: 00000000000002f2 sysrq_handle_crash

ffffa0f442cd7ac8: 0000000000000006 0000000000000000

ffffa0f442cd7ad8: 0000000000000096 0000000000000063

ffffa0f442cd7ae8: ffffffffffffffff sysrq_handle_crash+22

ffffa0f442cd7af8: 0000000000000010 0000000000010286

4 查看指定内存区域内容

crash> rd ffffa0f442cd7a08 -e ffffa0f442cd7a68

ffffa0f442cd7a08: ffffffff99a69313 0000a0f442cd7a50 ........Pz.B....

ffffa0f442cd7a18: ffff8b7c40000000 0000000024001000 ...@|......

....

ffffa0f442cd7a38: a05cedc0dfb99200 a05cedc0dfb99200 ..............

ffffa0f442cd7a48: ffffa0f442cd7cf8 0000000000000009 .|.B............

ffffa0f442cd7a58: ffffa0f442cd7cf8 ffffa0f442cd7b28 .|.B....({.B....

5 搜索指定内存

crash> search -s ffffa0f442cd7a08 -e ffffa0f442cd7db0 ffffffff9b39c3ed

ffffa0f442cd7aa0: ffffffff9b39c3ed

ffffa0f442cd7d28: ffffffff9b39c3ed

6 搜索匹配数据

crash> search -p babe0000 -m ffff

1c4cc6530: babec685

21f7d35b8: babe4550

crash>

查看线程状态

1 查看所有线程状态

crash> ps

PID PPID CPU TASK ST %MEM VSZ RSS COMM

0 0 0 ffffffff9ae13740 RU 0.0 0 0 [swapper/0]

0 0 1 ffff8b7e64165c00 RU 0.0 0 0 [swapper/1]

0 0 2 ffff8b7e64162e00 RU 0.0 0 0 [swapper/2]

0 0 3 ffff8b7e642c4500 RU 0.0 0 0 [swapper/3]

1 0 3 ffff8b7e6413c500 IN 0.1 225916 9716 systemd

2 0 2 ffff8b7e64138000 IN 0.0 0 0 [kthreadd]

2 查看父线程树

crash> ps -p 2613

PID: 0 TASK: ffffffff9ae13740 CPU: 0 COMMAND: "swapper/0"

PID: 1 TASK: ffff8b7e6413c500 CPU: 3 COMMAND: "systemd"

PID: 1081 TASK: ffff8b7e5dc81700 CPU: 1 COMMAND: "gdm3"

PID: 2114 TASK: ffff8b7e584f2e00 CPU: 0 COMMAND: "gdm-session-wor"

PID: 2136 TASK: ffff8b7e63cc4500 CPU: 1 COMMAND: "gdm-x-session"

PID: 2149 TASK: ffff8b7e5dfaae00 CPU: 0 COMMAND: "gnome-session-b"

PID: 2254 TASK: ffff8b7e5e04dc00 CPU: 0 COMMAND: "gnome-shell"

PID: 2582 TASK: ffff8b7dec3bae00 CPU: 0 COMMAND: "terminator"

PID: 2592 TASK: ffff8b7dec05ae00 CPU: 1 COMMAND: "bash"

PID: 2611 TASK: ffff8b7df3f8ae00 CPU: 0 COMMAND: "sudo"

PID: 2612 TASK: ffff8b7dec3b9700 CPU: 3 COMMAND: "su"

PID: 2613 TASK: ffff8b7df3cdae00 CPU: 2 COMMAND: "bash"

3 查看子线程

crash> ps -c 2582

PID: 2582 TASK: ffff8b7dec3bae00 CPU: 0 COMMAND: "terminator"

PID: 2592 TASK: ffff8b7dec05ae00 CPU: 1 COMMAND: "bash"

PID: 2600 TASK: ffff8b7df3f88000 CPU: 0 COMMAND: "bash"

PID: 2787 TASK: ffff8b7df9f80000 CPU: 3 COMMAND: "bash"

4 查看线程运行时间

crash> ps -t 2613

PID: 2613 TASK: ffff8b7df3cdae00 CPU: 2 COMMAND: "bash"

RUN TIME: 00:00:00

START TIME: 1296209749767

UTIME: 36000000

STIME: 16000000

5 查看活动线程

crash> ps -A

PID PPID CPU TASK ST %MEM VSZ RSS COMM

0 0 0 ffffffff9ae13740 RU 0.0 0 0 [swapper/0]

0 0 1 ffff8b7e64165c00 RU 0.0 0 0 [swapper/1]

0 0 3 ffff8b7e642c4500 RU 0.0 0 0 [swapper/3]

2613 2612 2 ffff8b7df3cdae00 RU 0.0 28708 4352 bash

6 查看内核线程

crash> ps -k

PID PPID CPU TASK ST %MEM VSZ RSS COMM

0 0 0 ffffffff9ae13740 RU 0.0 0 0 [swapper/0]

0 0 1 ffff8b7e64165c00 RU 0.0 0 0 [swapper/1]

0 0 2 ffff8b7e64162e00 RU 0.0 0 0 [swapper/2]

0 0 3 ffff8b7e642c4500 RU 0.0 0 0 [swapper/3]

2 0 2 ffff8b7e64138000 IN 0.0 0 0 [kthreadd]

7 查看用户态线程

crash> ps -u

PID PPID CPU TASK ST %MEM VSZ RSS COMM

1 0 3 ffff8b7e6413c500 IN 0.1 225916 9716 systemd

298 1 3 ffff8b7e5879c500 IN 0.4 126508 38028 systemd-journal

318 1 0 ffff8b7e584f5c00 IN 0.1 48004 6360 systemd-udevd

822 1 2 ffff8b7e59c71700 IN 0.1 70756 6176 systemd-resolve

824 1 2 ffff8b7e586e5c00 IN 0.1 146108 5540 systemd-timesyn

834 1 3 ffff8b7e63881700 IN 0.1 146108 5540 sd-resolve

863 1 3 ffff8b7e5d790000 IN 0.1 51612 6112 dbus-daemon

864 1 1 ffff8b7e5d794500 IN 0.1 427264 9404 ModemManager

8 查看最后运行时间戳

crash> ps -l

[1610759003323] [IN] PID: 2582 TASK: ffff8b7dec3bae00 CPU: 0 COMMAND: "terminator"

[1610758998404] [ID] PID: 211 TASK: ffff8b7e585aae00 CPU: 3 COMMAND: "kworker/u32:5"

[1610758938747] [RU] PID: 2613 TASK: ffff8b7df3cdae00 CPU: 2 COMMAND: "bash"

[1610758009873] [IN] PID: 2587 TASK: ffff8b7e06cd5c00 CPU: 2 COMMAND: "gdbus"

crash> ps -m

[0 00:00:00.000] [IN] PID: 2582 TASK: ffff8b7dec3bae00 CPU: 0 COMMAND: "terminator"

[0 00:00:00.000] [ID] PID: 211 TASK: ffff8b7e585aae00 CPU: 3 COMMAND: "kworker/u32:5"

[0 00:00:00.000] [RU] PID: 2613 TASK: ffff8b7df3cdae00 CPU: 2 COMMAND: "bash"

[0 00:00:00.000] [IN] PID: 2587 TASK: ffff8b7e06cd5c00 CPU: 2 COMMAND: "gdbus"

[0 00:00:00.001] [IN] PID: 2138 TASK: ffff8b7e26801700 CPU: 0 COMMAND: "Xorg"

9 查看线程资源限制

crash> ps -r 2613

PID: 2613 TASK: ffff8b7df3cdae00 CPU: 2 COMMAND: "bash"

RLIMIT CURRENT MAXIMUM

CPU (unlimited) (unlimited)

FSIZE (unlimited) (unlimited)

DATA (unlimited) (unlimited)

STACK 8388608 (unlimited)

CORE 0 (unlimited)

RSS (unlimited) (unlimited)

NPROC 30393 30393

NOFILE 1024 1048576

MEMLOCK 16777216 16777216

AS (unlimited) (unlimited)

LOCKS (unlimited) (unlimited)

SIGPENDING 30393 30393

MSGQUEUE 819200 819200

NICE 0 0

RTPRIO 0 0

RTTIME (unlimited) (unlimited)

Context切换

有些命令是线程上线文相关的，比如bt，可以用set命令来进行线程上下文切换。

1 切换到指定线程

crash> set ffff8b7e6413c500

PID: 1

COMMAND: "systemd"

TASK: ffff8b7e6413c500 [THREAD_INFO: ffff8b7e6413c500]

CPU: 3

STATE: TASK_INTERRUPTIBLE

crash> bt

PID: 1 TASK: ffff8b7e6413c500 CPU: 3 COMMAND: "systemd"

'#0 [ffffa0f440c6fce0] __schedule at ffffffff9a414ba7

'#1 [ffffa0f440c6fd80] schedule at ffffffff9a41519c

'#2 [ffffa0f440c6fd90] schedule_hrtimeout_range_clock at ffffffff9a419691

'#3 [ffffa0f440c6fe20] schedule_hrtimeout_range at ffffffff9a4196b3

'#4 [ffffa0f440c6fe30] ep_poll at ffffffff99cf8941

'#5 [ffffa0f440c6fee0] do_epoll_wait at ffffffff99cf8ae0

'#6 [ffffa0f440c6ff20] __x64_sys_epoll_wait at ffffffff99cf8b0e

'#7 [ffffa0f440c6ff30] do_syscall_64 at ffffffff99a0428a

'#8 [ffffa0f440c6ff50] entry_SYSCALL_64_after_hwframe at ffffffff9a600088

RIP: 00007ffa791c6bb7 RSP: 00007ffc1c00b9d0 RFLAGS: 00000293

RAX: ffffffffffffffda RBX: 0000000000000004 RCX: 00007ffa791c6bb7

RDX: 00000000000000eb RSI: 00007ffc1c00ba10 RDI: 0000000000000004

RBP: 00007ffc1c00ba10 R8: 0000000000000000 R9: 7465677261742e79

R10: 00000000ffffffff R11: 0000000000000293 R12: 00000000000000eb

R13: 00000000ffffffff R14: 00007ffc1c00ba10 R15: 0000000000000001

ORIG_RAX: 00000000000000e8 CS: 0033 SS: 002b

2 切会panic线程

crash> set -p

PID: 2613

COMMAND: "bash"

TASK: ffff8b7df3cdae00 [THREAD_INFO: ffff8b7df3cdae00]

CPU: 2

STATE: TASK_RUNNING (SYSRQ)

加载module符号表

1 查看当前加载的module

crash> mod

MODULE NAME SIZE OBJECT FILE

ffffffffc019d0c0 vfio_iommu_type1 24576 (not loaded) [CONFIG_KALLSYMS]

ffffffffc01a4440 uas 24576 (not loaded) [CONFIG_KALLSYMS]

ffffffffc01b0b40 rc_core 45056 (not loaded) [CONFIG_KALLSYMS]

ffffffffc01e76c0 e1000e 249856 (not loaded) [CONFIG_KALLSYMS]

ffffffffc01fcbc0 usbhid 49152 (not loaded) [CONFIG_KALLSYMS]

ffffffffc0207580 libahci 32768 (not loaded) [CONFIG_KALLSYMS]

2 加载所有module符号表

crash> mod -S

MODULE NAME SIZE OBJECT FILE

ffffffffc019d0c0 vfio_iommu_type1 24576 /lib/modules/4.19.53/kernel/drivers/vfio/vfio_iommu_type1.ko

ffffffffc01a4440 uas 24576 /lib/modules/4.19.53/kernel/drivers/usb/storage/uas.ko

ffffffffc01b0b40 rc_core 45056 /lib/modules/4.19.53/kernel/drivers/media/rc/rc-core.ko

ffffffffc01e76c0 e1000e 249856 /lib/modules/4.19.53/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko

ffffffffc01fcbc0 usbhid 49152 /lib/modules/4.19.53/kernel/drivers/hid/usbhid/usbhid.ko

3 加载指定module符号表

crash> mod -s rc_core /lib/modules/4.19.53/kernel/drivers/media/rc/rc-core.ko

MODULE NAME SIZE OBJECT FILE

ffffffffc01b0b40 rc_core 45056 /lib/modules/4.19.53/kernel/drivers/media/rc/rc-core.ko

crash> mod

MODULE NAME SIZE OBJECT FILE

ffffffffc019d0c0 vfio_iommu_type1 24576 (not loaded) [CONFIG_KALLSYMS]

ffffffffc01a4440 uas 24576 (not loaded) [CONFIG_KALLSYMS]

ffffffffc01b0b40 rc_core 45056 /lib/modules/4.19.53/kernel/drivers/media/rc/rc-core.ko

ffffffffc01e76c0 e1000e 249856 (not loaded) [CONFIG_KALLSYMS]

ffffffffc01fcbc0 usbhid 49152 (not loaded) [CONFIG_KALLSYMS]

其他命令

还有很多针对某些内核模块的命令，比如kmem，vm，tree，list，pte等等，参考附件命令列表，后面在使用过程中再学习和研究。

命令扩展

crash还支持用户添加在自己的调试命令。可以直接在Crash源码里添加新的命令，更多的是创建一个共享库，用extend动态加载。帮助文档里有一个简单的例子，在crash源码目录下新建一个test.c，把示例代码拷贝进去，就可以进行编译。

gcc -nostartfiles -shared -rdynamic -o echo.so echo.c -fPIC -D $(TARGET_CFLAGS)

crash> sys

KERNEL: ../../kernel-src/linux-4.19.53/vmlinux

DUMPFILE: 201907070732/dump.201907070732 [PARTIAL DUMP]

CPUS: 4

DATE: Sun Jul 7 07:31:34 2019

UPTIME: 00:10:27

LOAD AVERAGE: 0.14, 0.16, 0.12

TASKS: 584

NODENAME: glbian-OptiPlex-990

RELEASE: 4.19.53

VERSION: #1 SMP Sun Jun 23 11:01:25 CST 2019

MACHINE: x86_64 (3292 Mhz)

MEMORY: 7.9 GB

PANIC: "sysrq: SysRq : Trigger a crash"

可以用sys命令查看机器架构，我的及其machine-type选x86-64，编译命令如下：gcc -shared -rdynamic -o test.so test.c -fPIC -Dx86_64 _D_FILE_OFFSET_BITS=64

生成test.so。可以用extend直接加载，加载成功后可以看到帮助菜单多了一条echo命令，我们可以基于echo示例开发自己的命令。

crash> extend ../../src/crash-7.2.6/test.so

../../src/crash-7.2.6/test.so: shared object loaded

crash> extend

SHARED OBJECT COMMANDS

../../src/crash-7.2.6/test.so echo

crash> help

‘* extend mach runq union

alias files mod search vm

ascii foreach mount set vtop

bpf fuser net sig waitq

bt gdb p struct whatis

btop help ps swap wr

dev ipcs pte sym q

dis irq ptob sys

echo kmem ptov task

eval list rd timer

exit log repeat tree

结语

系统崩溃通常是非常棘手的问题，需要非常熟悉内核和相应的子模块，再结合crash工具进行分析，总之需要在实践中累积经验，实践出真知。

附件

Crash命令列表

命令

功能

*

指针快捷健

alias

命令快捷键

ascii

ASCII码转换和码表

bpf

eBPF - extended Berkeley Filter

bt

堆栈查看

btop

地址页表转换

dev

设备数据查询

dis

返汇编

eval

计算器

exit

退出

extend

命令扩展

files

打开的文件查看

foreach

循环查看

fuser

文件使用者查看

gdb

调用gdb执行命令

help

帮助

ipcs

查看system V IPC工具

irq

查看irq数据

kmem

查看Kernel内存

list

查看链表

log

查看系统消息缓存

mach

查看平台信息

mod

加载符号表

mount

Mount文件系统数据

net

网络命令

p

查看数据结构

ps

查看进程状态信息

pte

查看页表

ptob

页表地址转换

ptov

物理地址虚拟地址转换

rd

查看内存

repeat

重复执行

runq

查看run queue上的线程

search

搜索内存

set

设置线程环境和Crash内部变量

sig

查询线程消息

struct

查询结构体

swap

查看swap信息

sym

符号和虚拟地址转换

sys

查看系统信息

task

查看task_struct和thread_thread信息

timer

查看timer队列

tree

查看radix树和rb树

union

查看union结构体

vm

查看虚拟内存

vtop

虚拟地址物理地址转换

waitq

查看wait queue上的进程

whatis

符号表查询

wr

改写内存

q

退出

图片发自简书App