目前遇到一個崩潰問題記錄一下！
使用crash 分析結果如下：
crash> sys KERNEL: vmlinux DUMPFILE: kernel_dump_file_debug [PARTIAL DUMP] CPUS: 32 DATE: Thu Jul 8 16:06:13 2021 UPTIME: 12 days, 01:19:36 LOAD AVERAGE: 4.57, 5.64, 5.97 TASKS: 832 NODENAME: localhost RELEASE: 2.6.39-gentoo-r3-wafg2-47137 VERSION: #18 SMP Wed Dec 30 21:37:53 JST 2020 MACHINE: x86_64 (2599 Mhz) MEMORY: 128 GB PANIC: "[1039338.727675] Kernel panic - not syncing: softlockup: hung tasks"
crash> bt PID: 22501 TASK: ffff881ff4340690 CPU: 1 COMMAND: "xxxxproess" #0 [ffff88107fc238b0] machine_kexec at ffffffff810243b6 #1 [ffff88107fc23920] crash_kexec at ffffffff810773b9 #2 [ffff88107fc239f0] panic at ffffffff815f35e0 #3 [ffff88107fc23a70] watchdog_timer_fn at ffffffff81089a38 #4 [ffff88107fc23aa0] __run_hrtimer.clone.28 at ffffffff8106303a #5 [ffff88107fc23ad0] hrtimer_interrupt at ffffffff81063541 #6 [ffff88107fc23b30] smp_apic_timer_interrupt at ffffffff81020b92 #7 [ffff88107fc23b50] apic_timer_interrupt at ffffffff815f6553 #8 [ffff88107fc23bb8] igb_xmit_frame_ring at ffffffffa006a754 [igb] #9 [ffff88107fc23c70] igb_xmit_frame at ffffffffa006ada4 [igb] #10 [ffff88107fc23ca0] dev_hard_start_xmit at ffffffff814d588d #11 [ffff88107fc23d10] sch_direct_xmit at ffffffff814e87f7 #12 [ffff88107fc23d60] dev_queue_xmit at ffffffff814d5c2e #13 [ffff88107fc23db0] transmit_skb at ffffffffa0111032 [wafg2] #14 [ffff88107fc23dc0] forward_skb at ffffffffa01113b4 [wafg2] #15 [ffff88107fc23df0] dev_rx_skb at ffffffffa0111875 [wafg2] #16 [ffff88107fc23e40] igb_poll at ffffffffa006d6fc [igb] #17 [ffff88107fc23f10] net_rx_action at ffffffff814d437a #18 [ffff88107fc23f60] __do_softirq at ffffffff8104f3bf #19 [ffff88107fc23fb0] call_softirq at ffffffff815f6d9c --- <IRQ stack> --- #20 [ffff881f2ebcfae0] __skb_queue_purge at ffffffff8153af65 #21 [ffff881f2ebcfb00] do_softirq at ffffffff8100d1c4 #22 [ffff881f2ebcfb20] _local_bh_enable_ip.clone.8 at ffffffff8104f311 #23 [ffff881f2ebcfb30] local_bh_enable at ffffffff8104f336 #24 [ffff881f2ebcfb40] inet_csk_listen_stop at ffffffff8152a94b #25 [ffff881f2ebcfb80] tcp_close at ffffffff8152c8aa #26 [ffff881f2ebcfbb0] inet_release at ffffffff8154a44d #27 [ffff881f2ebcfbd0] sock_release at ffffffff814c409f #28 [ffff881f2ebcfbf0] sock_close at ffffffff814c4111 #29 [ffff881f2ebcfc00] fput at ffffffff810d4c85 #30 [ffff881f2ebcfc50] filp_close at ffffffff810d1ea0 #31 [ffff881f2ebcfc80] put_files_struct at ffffffff8104d4d9 #32 [ffff881f2ebcfcd0] exit_files at ffffffff8104d5b4 #33 [ffff881f2ebcfcf0] do_exit at ffffffff8104d821 #34 [ffff881f2ebcfd70] do_group_exit at ffffffff8104df5c #35 [ffff881f2ebcfda0] get_signal_to_deliver at ffffffff810570b2 #36 [ffff881f2ebcfe20] do_signal at ffffffff8100ae52 #37 [ffff881f2ebcff20] do_notify_resume at ffffffff8100b47e #38 [ffff881f2ebcff50] int_signal at ffffffff815f5e63 RIP: 00007fd9e52e1cdd RSP: 00007fd9a7cfa370 RFLAGS: 00000293 RAX: 000000000000001b RBX: 00000000000000fb RCX: ffffffffffffffff RDX: 000000000000001b RSI: 00007fd96a77e05e RDI: 00000000000000fb RBP: 00007fd9a8513e80 R8: 00000000007a7880 R9: 0000000000000000 R10: 0000000000000000 R11: 0000000000000293 R12: 000000000000001b R13: 00007fd96a77e05e R14: 000000000000001b R15: 0000000000735240 ORIG_RAX: 0000000000000001 CS: 0033 SS: 002b
首先弄明白 “Kernel panic - not syncing: softlockup: hung tasks” 這個結果是怎么出現，它代表着什么意思？也就是翻譯翻譯這個結論!!
lockup分為soft lockup和hard lockup
soft lockup是指內核中有BUG導致在內核模式下一直循環的時間超過n s（n為配置參數），而其他進程得不到運行的機會；實現方式：內核對於每一個cpu都有一個監控進程watchdog/x 每秒鍾會對比進程時間戳，對比時間戳就可以知道運行情況了，進程長時間沒有運行，時間戳沒有更新，超過一定時間就報錯。
hard lockup的發生是由於禁止了CPU的所有中斷超過一定時間(幾秒)這種情況下，外部設備發生的中斷無法處理，內核認為此時發生了所謂的hard lockup
A ‘softlockup’ is defined as a bug that causes the kernel to loop in kernel mode for more than 20 seconds (see “Implementation” below for details), without giving other tasks a chance to run. The current stack
trace is displayed upon detection and, by default, the system will stay locked up. Alternatively, the kernel can be configured to panic; a sysctl, “kernel.softlockup_panic”, a kernel parameter,
“softlockup_panic” (see “Documentation/kernel-parameters.txt” for details), and a compile option,“BOOTPARAM_SOFTLOCKUP_PANIC”, are provided for this. A ‘hardlockup’ is defined as a bug that causes the CPU to loop in kernel mode for more than 10 seconds (see “Implementation” below for details), without letting other interrupts have a chance to run.
Similarly to the softlockup case, the current stack trace is displayed upon detection and the system will stay locked up unless the default behavior is changed, which can be done through a sysctl,
‘hardlockup_panic’, a compile time knob,“BOOTPARAM_HARDLOCKUP_PANIC”, and a kernel parameter, “nmi_watchdog”
那就看為啥cpu 沒有被調度過來了?? 看了一下鬼知道!!! 干飯去----->下午繼續
目前crash 查看相關命令為：
- 使用bt -slf 函數所在的文件和每一幀的具體內容，從而對照源碼和匯編代碼，查看函數入參 等詳細情況
- 具體使用 help bt 就行和gdb 一樣
- dis [-r][-l][-u][-b [num]] [address | symbol | (expression)] [count]
- 該命令是disassemble的縮寫。把一個命令或者函數分解成匯編代碼。
-
Sym命令
sym [-l] | [-M] | [-m module] | [-p|-n] | [-q string] | [symbol | vaddr]
把一個標志符轉換到它所對應的虛擬地址，或者把虛擬地址轉換為它所對應的標志符
- struct 用於查看數據結構原形
View Codecrash> struct -o request_sock struct request_sock { [0] struct request_sock *dl_next; [8] u16 mss; [10] u8 retrans; [11] u8 cookie_ts; [12] u32 window_clamp; [16] u32 rcv_wnd; [20] u32 ts_recent; [24] unsigned long expires; [32] const struct request_sock_ops *rsk_ops; [40] struct sock *sk; [48] u32 secid; [52] u32 peer_secid; } SIZE: 56
- 目前看到mem使用正常
crash> kmem -i PAGES TOTAL PERCENTAGE TOTAL MEM 33001378 125.9 GB ---- FREE 31408525 119.8 GB 95% of TOTAL MEM USED 1592853 6.1 GB 4% of TOTAL MEM SHARED 107702 420.7 MB 0% of TOTAL MEM BUFFERS 3207 12.5 MB 0% of TOTAL MEM CACHED 721460 2.8 GB 2% of TOTAL MEM SLAB 472316 1.8 GB 1% of TOTAL MEM TOTAL SWAP 0 0 ---- SWAP USED 0 0 100% of TOTAL SWAP SWAP FREE 0 0 0% of TOTAL SWAP
- 命令列表
View Code
Crash命令列表 命令 功能 * 指針快捷健 alias 命令快捷鍵 ascii ASCII碼轉換和碼表 bpf eBPF - extended Berkeley Filter bt 堆棧查看 btop 地址頁表轉換 dev 設備數據查詢 dis 返匯編 eval 計算器 exit 退出 extend 命令擴展 files 打開的文件查看 foreach 循環查看 fuser 文件使用者查看 gdb 調用gdb執行命令 help 幫助 ipcs 查看system V IPC工具 irq 查看irq數據 kmem 查看Kernel內存 list 查看鏈表 log 查看系統消息緩存 mach 查看平台信息 mod 加載符號表 mount Mount文件系統數據 net 網絡命令 p 查看數據結構 ps 查看進程狀態信息 pte 查看頁表 ptob 頁表地址轉換 ptov 物理地址虛擬地址轉換 rd 查看內存 repeat 重復執行 runq 查看run queue上的線程 search 搜索內存 set 設置線程環境和Crash內部變量 sig 查詢線程消息 struct 查詢結構體 swap 查看swap信息 sym 符號和虛擬地址轉換 sys 查看系統信息 task 查看task_struct和thread_thread信息 timer 查看timer隊列 tree 查看radix樹和rb樹 union 查看union結構體 vm 查看虛擬內存 vtop 虛擬地址物理地址轉換 waitq 查看wait queue上的進程 whatis 符號表查詢 wr 改寫內存 q 退出
crash> bt -T PID: 22501 TASK: ffff881ff4340690 CPU: 1 COMMAND: "先休息" [ffff881f2ebcf3e0] put_dec at ffffffff8127ac94 [ffff881f2ebcf3f0] put_dec at ffffffff8127ac94 [ffff881f2ebcf410] number.clone.1 at ffffffff8127b9a1 [ffff881f2ebcf450] number.clone.1 at ffffffff8127b9a1 [ffff881f2ebcf460] put_dec at ffffffff8127ac94 [ffff881f2ebcf480] number.clone.1 at ffffffff8127b9a1 [ffff881f2ebcf490] __kmalloc_node_track_caller at ffffffff810ce77f [ffff881f2ebcf4c0] number.clone.1 at ffffffff8127b9a1 [ffff881f2ebcf4d0] get_partial_node at ffffffff810cc603 [ffff881f2ebcf500] number.clone.1 at ffffffff8127b9a1 [ffff881f2ebcf510] get_partial_node at ffffffff810cc603 [ffff881f2ebcf550] vsnprintf at ffffffff8127c36f [ffff881f2ebcf590] arch_local_irq_save at ffffffff810709ee [ffff881f2ebcf5b0] _raw_spin_unlock_irqrestore at ffffffff815f5156 [ffff881f2ebcf5d0] _raw_spin_unlock_irqrestore at ffffffff815f5156 [ffff881f2ebcf5e0] console_unlock at ffffffff8104b2bf [ffff881f2ebcf620] vprintk at ffffffff8104b706 [ffff881f2ebcf660] vprintk at ffffffff8104b706 [ffff881f2ebcf690] common_interrupt at ffffffff815f54ce [ffff881f2ebcf700] swiotlb_dma_mapping_error at ffffffff8128a0d3 [ffff881f2ebcf720] igb_xmit_frame_ring at ffffffffa006a3c1 [igb] [ffff881f2ebcf740] swiotlb_dma_mapping_error at ffffffff8128a0d3 [ffff881f2ebcf760] igb_xmit_frame_ring at ffffffffa006a3c1 [igb] [ffff881f2ebcf778] swiotlb_map_page at ffffffff8128a87e [ffff881f2ebcf790] local_bh_enable at ffffffff8104f336 [ffff881f2ebcf7f0] igb_xmit_frame at ffffffffa006ada4 [igb] [ffff881f2ebcf820] dev_hard_start_xmit at ffffffff814d588d [ffff881f2ebcf880] _raw_spin_lock at ffffffff815f50fc [ffff881f2ebcf890] sch_direct_xmit at ffffffff814e881f [ffff881f2ebcf8c0] _local_bh_enable_ip.clone.8 at ffffffff8104f2b9 [ffff881f2ebcf8d0] local_bh_enable at ffffffff8104f336 [ffff881f2ebcf8e0] dev_queue_xmit at ffffffff814d5dc0 [ffff881f2ebcf930] mac_build_and_send_pkt at ffffffffa010dca6 [wafg2] [ffff881f2ebcf950] ip_finish_output2 at ffffffff8152568a [ffff881f2ebcf980] ip_finish_output at ffffffff81525792 [ffff881f2ebcf9a0] ip_output at ffffffff815261a7
crash> log
[1039338.458914] Second detect insufficient ring room. Requested: 22. [1039338.459474] Second detect insufficient ring room. Requested: 22. [1039338.460095] Second detect insufficient ring room. Requested: 22. [1039338.460628] Second detect insufficient ring room. Requested: 22. [1039338.461218] Second detect insufficient ring room. Requested: 22. [1039338.461792] Second detect insufficient ring room. Requested: 22. [1039338.462317] Second detect insufficient ring room. Requested: 22. [1039338.462936] Second detect insufficient ring room. Requested: 22. [1039338.463755] Second detect insufficient ring room. Requested: 22. [1039338.646254] Second detect insufficient ring room. Requested: 22. [1039338.646769] Second detect insufficient ring room. Requested: 22. [1039338.647356] Second detect insufficient ring room. Requested: 22. [1039338.679837] Second detect insufficient ring room. Requested: 22. [1039338.680431] Second detect insufficient ring room. Requested: 22. [1039338.680961] Second detect insufficient ring room. Requested: 22. [1039338.681491] Second detect insufficient ring room. Requested: 22. [1039338.682080] Second detect insufficient ring room. Requested: 22. [1039338.682696] Second detect insufficient ring room. Requested: 22. [1039338.683330] Second detect insufficient ring room. Requested: 22. [1039338.683845] Second detect insufficient ring room. Requested: 22. [1039338.684741] Second detect insufficient ring room. Requested: 22. [1039338.685251] Second detect insufficient ring room. Requested: 22. [1039338.727267] BUG: soft lockup - CPU#1 stuck for 67s! [wafd:22501] [1039338.727277] CPU 1 [1039338.727285] [1039338.727292] RIP: 0010:[<ffffffffa006a754>] [<ffffffffa006a754>] igb_xmit_frame_ring+0x744/0xd10 [igb] [1039338.727301] RSP: 0018:ffff88107fc23be8 EFLAGS: 00000216 [1039338.727303] RAX: 0000000000000000 RBX: ffffffff8106230e RCX: 0000000000000100 [1039338.727305] RDX: 00000000000000f3 RSI: 0000000000000000 RDI: 0000000062300000 [1039338.727307] RBP: ffff88107fc23c68 R08: 0000000ef7858840 R09: 00000000008d8000 [1039338.727309] R10: 0000000000000000 R11: 00000000000010c0 R12: ffffffff815f6553 [1039338.727311] R13: ffff88107fc23b58 R14: 0000000000000032 R15: ffff88103f64de00 [1039338.727313] FS: 00007fd9a7d0b700(0000) GS:ffff88107fc20000(0000) knlGS:0000000000000000 [1039338.727315] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [1039338.727317] CR2: 00007fd96a7a0000 CR3: 000000000195f000 CR4: 00000000000406e0
objdump只反匯編想要的函數
#!/bin/bash vmlinux=$1 symbol=$2 if [ -z "$vmlinux" ]; then echo "usage : $0 vmlinux symbol" exit fi startaddress=$(nm -n $vmlinux | grep "\w\s$symbol" | awk '{print "0x"$1;exit}') endaddress=$(nm -n $vmlinux | grep -A1 "\w\s$symbol" | awk '{getline; print "0x"$1;exit}') if [ -z "$symbol" ]; then echo "dump all symbol" objdump -d $vmlinux else echo "start-address: $startaddress, end-address: $endaddress" objdump -d $vmlinux --start-address=$startaddress --stop-address=$endaddress fi
對比 代碼查看 應該就是 xmit pkt 的時候dma 出現錯誤
但是為什么 ？ 是什么原因導致？ 怎么分析 啃代碼？？
crash> dis -l ffffffffa006a3c1 0xffffffffa006a3c1 <igb_xmit_frame_ring+945>: test %eax,%eax crash> dis -l 0xffffffffa006a754 0xffffffffa006a754 <igb_xmit_frame_ring+1860>: add $0x58,%rsp swiotlb_dma_mapping_error at ffffffff8128a0d3 [ffff881f2ebcf720] igb_xmit_frame_ring at ffffffffa006a3c1 [igb] [ffff881f2ebcf740] swiotlb_dma_mapping_error at ffffffff8128a0d3 [ffff881f2ebcf760] igb_xmit_frame_ring at ffffffffa006a3c1 [igb] [ffff881f2ebcf778] swiotlb_map_page at ffffffff8128a87e
這是gdb 驅動的結果
ps:驅動出現的問題, 那么gdb直接調試驅動的 ko 文件, 如果是源內核出現的 OOPS, 那么只能用 gdb 對 vmlinux(內核根目錄下) 文件進行調試
或者 gdb 調試對應的.o文件 disass 對應的函數名稱 找到其 偏移地址對應的地址 ；然后使用addrline -a 地址 -e file 找出對應代碼
根據log以及panic堆棧 估計就是網卡一直在發包導致的吧！！！但是 根本原因不知道啊！！！！！
log分析可知： 是unused_desc不足導致的問題
/* We need to check again in a case another CPU has just * made room available. */ if (igb_desc_unused(tx_ring) < size) return -EBUSY;
err = request_irq(adapter->msix_entries[vector].vector, igb_msix_ring, 0, q_vector->name, q_vector); static irqreturn_t igb_msix_ring(int irq, void *data) { struct igb_q_vector *q_vector = data; /* Write the ITR value calculated from the previous interrupt. */ igb_write_itr(q_vector); napi_schedule(&q_vector->napi); return IRQ_HANDLED; } static inline void napi_schedule(struct napi_struct *n) { if (napi_schedule_prep(n)) __napi_schedule(n); } void __napi_schedule(struct napi_struct *n) { unsigned long flags; local_irq_save(flags); ____napi_schedule(this_cpu_ptr(&softnet_data), n); local_irq_restore(flags); } /* Called with irq disabled */ static inline void ____napi_schedule(struct softnet_data *sd, struct napi_struct *napi) { list_add_tail(&napi->poll_list, &sd->poll_list); __raise_softirq_irqoff(NET_RX_SOFTIRQ); }
發送數據，但是硬中斷最終觸發的軟中斷卻是 NET_RX_SOFTIRQ; 如果要是有源源不斷的數據 是不是會 收包 發包 一直循環？？
void __qdisc_run(struct Qdisc *q) { int quota = weight_p; int packets; while (qdisc_restart(q, &packets)) { /* * Ordered by possible occurrence: Postpone processing if * 1. we've exceeded packet quota * 2. another process needs the CPU; */ quota -= packets; if (quota <= 0 || need_resched()) { __netif_schedule(q); break; } } qdisc_run_end(q); }
while 循環調用 qdisc_restart()，后者取出一個 skb，然后嘗試通過 sch_direct_xmit() 來發送；sch_direct_xmit 調用 dev_hard_start_xmit 來向驅動 程序進行實際發送。任何無法發送的 skb 都重新入隊，將在 NET_TX softirq 中進行 發送。
目前是by pass 中斷收發包， 繼續分析
目前這邊對比分析：認為是網卡中斷都負載在第一個cpu 上導致， 雖然有16核cpu，但是只有一個cpu 在處理！！ 采用rss 即可
后續還是會繼續研究其vmcore
對於vmcore 可以使用一下方式 查看二季指針
如果要查看二階指針的值，可以通過rd命令需要先獲取一級指針的值，然后再用struct 結構體名 + addr獲取具體的值 crash> files 1076 PID: 1076 TASK: ffff882034f68000 CPU: 9 COMMAND: "python" ROOT: / CWD: / FD FILE DENTRY INODE TYPE PATH 0 ffff88203480a300 ffff88107f4e00c0 ffff88103f99b700 CHR /dev/null 1 ffff88203480a300 ffff88107f4e00c0 ffff88103f99b700 CHR /dev/null 2 ffff88203480a300 ffff88107f4e00c0 ffff88103f99b700 CHR /dev/null 3 ffff881036921e00 ffff8810375449c0 ffff88107f4526b0 SOCK 4 ffff881036921ec0 ffff881037544480 ffff882078c00980 UNKN [eventpoll] 5 ffff881037149bc0 ffff8810377c3800 ffff88107f452ef0 SOCK 6 ffff881037149ec0 ffff8810377c20c0 ffff88107f42fbf0 SOCK 7 ffff8810371480c0 ffff8810377c3740 ffff88107f42f930 SOCK 8 ffff881037148c00 ffff8810377c2cc0 ffff88107f7fe330 SOCK 9 ffff8810371495c0 ffff8810377c3a40 ffff88107f7fa970 SOCK 10 ffff8820364ae600 ffff88201d7223c0 ffff88203bcd73b0 SOCK 11 ffff8820364aea80 ffff88201d723680 ffff88203bcd7670 SOCK 12 ffff8820364af200 ffff88201d723740 ffff88203bcd7930 SOCK 13 ffff8820364af380 ffff88201d723800 ffff88203bcd7bf0 SOCK 14 ffff88203eba15c0 ffff88201ccc4240 ffff88203be53730 SOCK 15 ffff88203eba1140 ffff88201ccc4300 ffff88203be539f0 SOCK 16 ffff88203eba0300 ffff88201ccc43c0 ffff88203be53cb0 SOCK 17 ffff88203eba0a80 ffff88201ccc4480 ffff88203be53f70 SOCK 18 ffff88203eba1740 ffff88201ccc4540 ffff88203be54230 SOCK 19 ffff88203eba0000 ffff88201ccc4600 ffff88203be544f0 SOCK 20 ffff88203eba1500 ffff88201ccc46c0 ffff88203be547b0 SOCK 21 ffff88203eba0600 ffff88201ccc4780 ffff88203be54a70 SOCK 22 ffff88203eba1b00 ffff88201ccc4840 ffff88203be54d30 SOCK 23 ffff88203eba18c0 ffff88201ccc4900 ffff88203be54ff0 SOCK 24 ffff88203eba0fc0 ffff88201ccc49c0 ffff88203be552b0 SOCK 25 ffff88203aac9440 ffff88201ccc4a80 ffff88203be55570 SOCK 26 ffff88203aac8c00 ffff88201ccc4b40 ffff88203be55830 SOCK 27 ffff88203aac86c0 ffff88201ccc4c00 ffff88203be55af0 SOCK 28 ffff88203aac8780 ffff88201ccc4cc0 ffff88203be55db0 SOCK 29 ffff88203aac95c0 ffff88201ccc4d80 ffff88203be56070 SOCK 30 ffff88203aac83c0 ffff88201ccc4e40 ffff88203be56330 SOCK 31 ffff88203aac9680 ffff88201ccc4f00 ffff88203be565f0 SOCK 32 ffff88203aac9800 ffff88201ccc4fc0 ffff88203be568b0 SOCK 33 ffff88203aac8d80 ffff88201ccc5080 ffff88203be56b70 SOCK 34 ffff88203aac8900 ffff88201ccc5140 ffff88203be56e30 SOCK 35 ffff88203aac8f00 ffff88201ccc5200 ffff88203be570f0 SOCK 36 ffff88203aac8180 ffff88201ccc52c0 ffff88203be573b0 SOCK crash> files 1076 PID: 1076 TASK: ffff882034f68000 CPU: 9 COMMAND: "python" ROOT: / CWD: / FD FILE DENTRY INODE TYPE PATH 0 ffff88203480a300 ffff88107f4e00c0 ffff88103f99b700 CHR /dev/null 1 ffff88203480a300 ffff88107f4e00c0 ffff88103f99b700 CHR /dev/null 2 ffff88203480a300 ffff88107f4e00c0 ffff88103f99b700 CHR /dev/null 3 ffff881036921e00 ffff8810375449c0 ffff88107f4526b0 SOCK 4 ffff881036921ec0 ffff881037544480 ffff882078c00980 UNKN [eventpoll] 5 ffff881037149bc0 ffff8810377c3800 ffff88107f452ef0 SOCK 6 ffff881037149ec0 ffff8810377c20c0 ffff88107f42fbf0 SOCK 7 ffff8810371480c0 ffff8810377c3740 ffff88107f42f930 SOCK 8 ffff881037148c00 ffff8810377c2cc0 ffff88107f7fe330 SOCK 9 ffff8810371495c0 ffff8810377c3a40 ffff88107f7fa970 SOCK 10 ffff8820364ae600 ffff88201d7223c0 ffff88203bcd73b0 SOCK 11 ffff8820364aea80 ffff88201d723680 ffff88203bcd7670 SOCK 12 ffff8820364af200 ffff88201d723740 ffff88203bcd7930 SOCK 13 ffff8820364af380 ffff88201d723800 ffff88203bcd7bf0 SOCK 14 ffff88203eba15c0 ffff88201ccc4240 ffff88203be53730 SOCK 15 ffff88203eba1140 ffff88201ccc4300 ffff88203be539f0 SOCK 16 ffff88203eba0300 ffff88201ccc43c0 ffff88203be53cb0 SOCK 17 ffff88203eba0a80 ffff88201ccc4480 ffff88203be53f70 SOCK 18 ffff88203eba1740 ffff88201ccc4540 ffff88203be54230 SOCK 19 ffff88203eba0000 ffff88201ccc4600 ffff88203be544f0 SOCK 20 ffff88203eba1500 ffff88201ccc46c0 ffff88203be547b0 SOCK 21 ffff88203eba0600 ffff88201ccc4780 ffff88203be54a70 SOCK 22 ffff88203eba1b00 ffff88201ccc4840 ffff88203be54d30 SOCK 23 ffff88203eba18c0 ffff88201ccc4900 ffff88203be54ff0 SOCK 24 ffff88203eba0fc0 ffff88201ccc49c0 ffff88203be552b0 SOCK 25 ffff88203aac9440 ffff88201ccc4a80 ffff88203be55570 SOCK 26 ffff88203aac8c00 ffff88201ccc4b40 ffff88203be55830 SOCK 27 ffff88203aac86c0 ffff88201ccc4c00 ffff88203be55af0 SOCK 28 ffff88203aac8780 ffff88201ccc4cc0 ffff88203be55db0 SOCK 29 ffff88203aac95c0 ffff88201ccc4d80 ffff88203be56070 SOCK 30 ffff88203aac83c0 ffff88201ccc4e40 ffff88203be56330 SOCK 31 ffff88203aac9680 ffff88201ccc4f00 ffff88203be565f0 SOCK 32 ffff88203aac9800 ffff88201ccc4fc0 ffff88203be568b0 SOCK 33 ffff88203aac8d80 ffff88201ccc5080 ffff88203be56b70 SOCK 34 ffff88203aac8900 ffff88201ccc5140 ffff88203be56e30 SOCK 35 ffff88203aac8f00 ffff88201ccc5200 ffff88203be570f0 SOCK 36 ffff88203aac8180 ffff88201ccc52c0 ffff88203be573b0 SOCK crash> struct task_struct.files ffff882034f68000 files = 0xffff8820779739c0 crash> struct files_struct 0xffff8820779739c0 struct files_struct { count = { counter = 3 }, fdt = 0xffff8820779739d0, fdtab = { max_fds = 64, fd = 0xffff882077973a58, close_on_exec = 0xffff882077973a48, open_fds = 0xffff882077973a50, rcu = { next = 0x0, func = 0xffffffff810e801b <free_fdtable_rcu> }, next = 0x0 }, file_lock = { { rlock = { raw_lock = { slock = 3753500601 } } } }, next_fd = 37, close_on_exec_init = { fds_bits = {0} }, open_fds_init = { fds_bits = {137438953471} }, fd_array = {0xffff88203480a300, 0xffff88203480a300, 0xffff88203480a300, 0xffff881036921e00, 0xffff881036921ec0, 0xffff881037149bc0, 0xffff881037149ec0, 0xffff8810371480c0, 0xffff881037148c00, 0xffff8810371495c0, 0xffff8820364ae600, 0xffff8820364aea80, 0xffff8820364af200, 0xffff8820364af380, 0xffff88203eba15c0, 0xffff88203eba1140, 0xffff88203eba0300, 0xffff88203eba0a80, 0xffff88203eba1740, 0xffff88203eba0000, 0xffff88203eba1500, 0xffff88203eba0600, 0xffff88203eba1b00, 0xffff88203eba18c0, 0xffff88203eba0fc0, 0xffff88203aac9440, 0xffff88203aac8c00, 0xffff88203aac86c0, 0xffff88203aac8780, 0xffff88203aac95c0, 0xffff88203aac83c0, 0xffff88203aac9680, 0xffff88203aac9800, 0xffff88203aac8d80, 0xffff88203aac8900, 0xffff88203aac8f00, 0xffff88203aac8180, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0} } crash> struct files_struct struct files_struct { atomic_t count; struct fdtable *fdt; struct fdtable fdtab; spinlock_t file_lock; int next_fd; struct embedded_fd_set close_on_exec_init; struct embedded_fd_set open_fds_init; struct file *fd_array[64]; } SIZE: 704 crash> struct files_struct.fdt 0xffff8820779739c0 fdt = 0xffff8820779739d0 crash> struct fdtable 0xffff8820779739d0 struct fdtable { max_fds = 64, fd = 0xffff882077973a58, close_on_exec = 0xffff882077973a48, open_fds = 0xffff882077973a50, rcu = { next = 0x0, func = 0xffffffff810e801b <free_fdtable_rcu> }, next = 0x0 } crash> rd 0xffff882077973a58 ffff882077973a58: ffff88203480a300 ...4 ... fd[0] crash> rd 0xffff882077973a60 ffff882077973a60: ffff88203480a300 ...4 ... crash> rd 0xffff882077973a68 ffff882077973a68: ffff88203480a300 ...4 ... crash> rd 0xffff882077973a70 ffff882077973a70: ffff881036921e00 ...6....fd[3] crash>
Crash命令列表
| 命令 | 功能 |
|---|---|
| * | 指針快捷健 |
| alias | 命令快捷鍵 |
| ascii | ASCII碼轉換和碼表 |
| bpf | eBPF - extended Berkeley Filter |
| bt | 堆棧查看 |
| btop | 地址頁表轉換 |
| dev | 設備數據查詢 |
| dis | 返匯編 |
| eval | 計算器 |
| exit | 退出 |
| extend | 命令擴展 |
| files | 打開的文件查看 |
| foreach | 循環查看 |
| fuser | 文件使用者查看 |
| gdb | 調用gdb執行命令 |
| help | 幫助 |
| ipcs | 查看system V IPC工具 |
| irq | 查看irq數據 |
| kmem | 查看Kernel內存 |
| list | 查看鏈表 |
| log | 查看系統消息緩存 |
| mach | 查看平台信息 |
| mod | 加載符號表 |
| mount | Mount文件系統數據 |
| net | 網絡命令 |
| p | 查看數據結構 |
| ps | 查看進程狀態信息 |
| pte | 查看頁表 |
| ptob | 頁表地址轉換 |
| ptov | 物理地址虛擬地址轉換 |
| rd | 查看內存 |
| repeat | 重復執行 |
| runq | 查看run queue上的線程 |
| search | 搜索內存 |
| set | 設置線程環境和Crash內部變量 |
| sig | 查詢線程消息 |
| struct | 查詢結構體 |
| swap | 查看swap信息 |
| sym | 符號和虛擬地址轉換 |
| sys | 查看系統信息 |
| task | 查看task_struct和thread_thread信息 |
| timer | 查看timer隊列 |
| tree | 查看radix樹和rb樹 |
| union | 查看union結構體 |
| vm | 查看虛擬內存 |
| vtop | 虛擬地址物理地址轉換 |
| waitq | 查看wait queue上的進程 |
| whatis | 符號表查詢 |
| wr | 改寫內存 |
| q | 退出 |
表格轉載自鏈接：https://www.jianshu.com/p/ad03152a0a53