最近在整理Linux調試方面的文檔，正好碰到了一個內存泄漏踩棧的問題，借此機會記錄一下分析過程。
首先，發現問題之后，趕緊看一下產生coredump文件沒有，果不其然，產生了coredump，果斷上gdb調試。
$ arm-buildroot-linux-gnueabi-gdb ./linecard ~/core_tMscRcv_165
GNU gdb (GDB) 7.10.1 Copyright (C) 2015 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "--host=x86_64-unknown-linux-gnu --target=arm-buildroot-linux-gnueabi". Type "show configuration" for configuration details. For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from ./linecard...done.
warning: exec file is newer than core file.
[New LWP 276]
... ...
[New LWP 303]
warning: .dynamic section for "./libethernet_oam.so" is not at the expected address (wrong library or version mismatch?)
warning: .dynamic section for "./libigmp_adapter.so" is not at the expected address (wrong library or version mismatch?)
warning: .dynamic section for "./libmsc.so" is not at the expected address (wrong library or version mismatch?)
warning: .dynamic section for "./libolt_config.so" is not at the expected address (wrong library or version mismatch?)
warning: .dynamic section for "./libonu_config.so" is not at the expected address (wrong library or version mismatch?)
warning: .dynamic section for "./libonu_vlan.so" is not at the expected address (wrong library or version mismatch?)
warning: .dynamic section for "./libpolicy.so" is not at the expected address (wrong library or version mismatch?)
warning: .dynamic section for "./libppt.so" is not at the expected address (wrong library or version mismatch?)
warning: .dynamic section for "./librms.so" is not at the expected address (wrong library or version mismatch?)
warning: .dynamic section for "./libtime_sync.so" is not at the expected address (wrong library or version mismatch?)
warning: .dynamic section for "./libvoice.so" is not at the expected address (wrong library or version mismatch?)
warning: .dynamic section for "./libprivate_com.so" is not at the expected address (wrong library or version mismatch?)
warning: .dynamic section for "./libonu_ability.so" is not at the expected address (wrong library or version mismatch?)
warning: .dynamic section for "./libomci.so" is not at the expected address (wrong library or version mismatch?)
warning: .dynamic section for "./libconfig_data_gpon.so" is not at the expected address (wrong library or version mismatch?)
warning: .dynamic section for "./liblineid_adapter.so" is not at the expected address (wrong library or version mismatch?)
warning: .dynamic section for "./libbep.so" is not at the expected address (wrong library or version mismatch?)
warning: Could not load shared library symbols for 11 libraries, e.g. /usr/local/libfhdrv_kdrv_board_impl.so.
Use the "info sharedlibrary" command to see the complete listing.
Do you need "set solib-search-path" or "set sysroot"?
Core was generated by `./linecard'.
Program terminated with signal SIGABRT, Aborted. /* 最開始要確定是什么信號量導致程序異常退出，這樣很容易縮小范圍，猜測大概可能發生了什么問題 */
#0 0xe9d2f630 in ?? ()
[Current thread is 1 (LWP 276)]
貌似找不到符號表，庫文件匹配不上，怎么辦？用set solib-search-path設置一下庫文件路徑就好了！
(gdb) set solib-search-path /media/new/linyao/2_xPON/xPON/src/config_data_gpon/squashfs-root/mnt/work/linecard_app:/media/new/linyao/2_xPON/xPON/src/config_data_gpon/squashfs-root/usr/local:/media/new/linyao/2_xPON/xPON/src/config_data_gpon/squashfs-root/lib:/media/new/linyao/2_xPON/xPON/src/config_data_gpon/squashfs-root/usr/lib
Reading symbols from /media/new/linyao/2_xPON/xPON/src/config_data_gpon/squashfs-root/usr/local/libfhdrv_kdrv_board_impl.so...done.
Reading symbols from /media/new/linyao/2_xPON/xPON/src/config_data_gpon/squashfs-root/usr/local/libvirtual_netdev_drv.so...done.
Reading symbols from /media/new/linyao/2_xPON/xPON/src/config_data_gpon/squashfs-root/lib/libpthread.so.0...(no debugging symbols found)...done.
Reading symbols from /media/new/linyao/2_xPON/xPON/src/config_data_gpon/squashfs-root/lib/librt.so.1...(no debugging symbols found)...done.
Reading symbols from /media/new/linyao/2_xPON/xPON/src/config_data_gpon/squashfs-root/lib/libdl.so.2...(no debugging symbols found)...done.
Reading symbols from /media/new/linyao/2_xPON/xPON/src/config_data_gpon/squashfs-root/lib/libc.so.6...(no debugging symbols found)...done.
Reading symbols from /media/new/linyao/2_xPON/xPON/src/config_data_gpon/squashfs-root/lib/ld-linux.so.3...(no debugging symbols found)...done.
Reading symbols from /media/new/linyao/2_xPON/xPON/src/config_data_gpon/squashfs-root/usr/lib/libstdc++.so.6...(no debugging symbols found)...done.
Reading symbols from /media/new/linyao/2_xPON/xPON/src/config_data_gpon/squashfs-root/lib/libm.so.6...(no debugging symbols found)...done.
Reading symbols from /media/new/linyao/2_xPON/xPON/src/config_data_gpon/squashfs-root/lib/libgcc_s.so.1...(no debugging symbols found)...done.
Reading symbols from /media/new/linyao/2_xPON/xPON/src/config_data_gpon/squashfs-root/lib/libnss_files.so.2...(no debugging symbols found)...done.
/media/new/linyao/2_xPON/xPON/src/config_data_gpon/squashfs-root/mnt/work/linecard_app
用bt命令看一下調用棧：
(gdb) bt
#0 0xe9d2f630 in raise () from /media/new/linyao/2_xPON/xPON/src/config_data_gpon/squashfs-root/lib/libc.so.6
#1 0xe9d309c8 in abort () from /media/new/linyao/2_xPON/xPON/src/config_data_gpon/squashfs-root/lib/libc.so.6
#2 0xf5a52158 in diag_displayTermPack (ucTermType=<optimized out>, usTermCmd=<optimized out>, pucData=0xc6e61440 "\001", usDataLen=<optimized out>)
at ../../code/diag/diag/diag_display.c:147
#3 0xf5a62288 in wos_taskDiagMsgProc (pulPara=0xf5b20da8 <g_astWosPerfMsgDispatchStat+44>) at ../../code/diag/wos/wos_task.c:1973
#4 <signal handler called> /* 觸發信號量異常處理函數 */
#5 0xe9c71ca0 in __dynamic_cast () from /media/new/linyao/2_xPON/xPON/src/config_data_gpon/squashfs-root/usr/lib/libstdc++.so.6
#6 0xf6416f70 in cfg_mod_set_onu_local_mng_interface_config_flag (pon_no=0, onu_no=0, cfg_flag=0 '\000')
at /media/new/linyao/2_xPON/xPON/src/config_data_gpon/config_module_cinterface.cpp:35769
#7 0xf6cfa60c in str_ipaddr_to_general_ip (pgeneral_ip=0x1 <error: Cannot access memory at address 0x1>,
str_ipaddr=0x80075 <error: Cannot access memory at address 0x80075>) at /media/new/linyao/2_xPON/xPON/src/voice/voice_service_global.cpp:945
#8 0xf5d382bc in CMscCommandObject::Parse (this=0x25a7a38, buf=0xde83a22c "\016\002\237\001\001\004\025", buf_len=676, session_id=0, volt_id=0 '\000', is_head=1 '\001')
at /media/new/jenkins/workspace/workspace/201716-OLT-CBB-PONSYSTEM_LINUX-fsl61293-coverity/src/service_module/msc_command_object.cpp:201
#9 0xf5d39d48 in CServiceObject::DispatchMscMessage (this=0x2598818, cmd_id=1045, pbuf=0xde83a22c "\016\002\237\001\001\004\025", buf_len=676, session_id=0,
volt_id=0 '\000', is_head=1 '\001') at /media/new/jenkins/workspace/workspace/201716-OLT-CBB-PONSYSTEM_LINUX-fsl61293-coverity/src/service_module/service_object.cpp:46
#10 0xf5d29f28 in CPonSystem::DispatchMscMessage (this=0x849a80, cmd_id=1045, pbuf=0xde83a22c "\016\002\237\001\001\004\025", buf_len=676, session_id=0, is_head=1 '\001',
volt_id=0 '\000') at /media/new/jenkins/workspace/workspace/201716-OLT-CBB-PONSYSTEM_LINUX-fsl61293-coverity/src/service_module/pon_system.hpp:566
#11 0xf5d25494 in pon_system_dispatch_msc (cmd_id=1045, pbuf=0xde83a22c "\016\002\237\001\001\004\025", buf_len=676, session_id=0, is_head=1 '\001')
at /media/new/jenkins/workspace/workspace/201716-OLT-CBB-PONSYSTEM_LINUX-fsl61293-coverity/src/service_module/pon_system_cinterface.cpp:385
#12 0xf728c46c in MSC_RegProcGswCmdFunToManager (cmdId=1045, cmdType=0 '\000', pFun=0xde83a22c, pPrmt=0x2a4 <error: Cannot access memory at address 0x2a4>)
at /media/new/linyao/2_xPON/xPON/src/msc/msc_main.c:1477
#13 0xf72897dc in __gnu_cxx::__normal_iterator<CServiceObject**, std::vector<CServiceObject*, std::allocator<CServiceObject*> > >::__normal_iterator (this=0x0,
__i=@0x1: <error reading variable>)
at /opt/toolchains/crosstools-arm-gcc-5.3-linux-4.1-glibc-2.24-binutils-2.25/usr/arm-buildroot-linux-gnueabi/include/c++/5.3.0/bits/stl_iterator.h:740
#14 0xf72897dc in __gnu_cxx::__normal_iterator<CServiceObject**, std::vector<CServiceObject*, std::allocator<CServiceObject*> > >::__normal_iterator (this=0xe3400002,
__i=@0xe300c406: <error reading variable>)
at /opt/toolchains/crosstools-arm-gcc-5.3-linux-4.1-glibc-2.24-binutils-2.25/usr/arm-buildroot-linux-gnueabi/include/c++/5.3.0/bits/stl_iterator.h:740
#15 0xf72897dc in __gnu_cxx::__normal_iterator<CServiceObject**, std::vector<CServiceObject*, std::allocator<CServiceObject*> > >::__normal_iterator (this=0x0,
__i=@0x0: <error reading variable>)
at /opt/toolchains/crosstools-arm-gcc-5.3-linux-4.1-glibc-2.24-binutils-2.25/usr/arm-buildroot-linux-gnueabi/include/c++/5.3.0/bits/stl_iterator.h:740
Backtrace stopped: previous frame inner to this frame (corrupt stack?)
frame 4觸發信號量異常處理函數，那么說明問題肯定出在frame 5。
(gdb) frame 5
#5 0xe9c71ca0 in __dynamic_cast () from /media/new/justin/2_xPON/xPON/src/config_data_gpon/squashfs-root/usr/lib/libstdc++.so.6
(gdb) info registers
r0 0xb8d70860 3101100128
r1 0xf5d6c6c0 4124493504
r2 0xf5d6c9e0 4124494304
r3 0x0 0
r4 0xb8d70860 3101100128
r5 0xf65f1808 4133427208
r6 0x79e0 31200
r7 0xf5af8ed4 4121923284
r8 0xf5af903c 4121923644
r9 0x64 100
r10 0xf5b1a118 4122059032
r11 0xc6e581ac 3336929708
r12 0x2564 9572
sp 0xc6e58120 0xc6e58120
lr 0xf6416f70 -163483792
pc 0xe9c71ca0 0xe9c71ca0 <__dynamic_cast+16>
cpsr 0xa0070010 -1610153968
(gdb) disassemble
Dump of assembler code for function __dynamic_cast:
0xe9c71c90 <+0>: ldr r12, [r0]
0xe9c71c94 <+4>: push {r4, r5, r6, r7, r8, lr}
0xe9c71c98 <+8>: mov r4, r0
0xe9c71c9c <+12>: sub sp, sp, #40 ; 0x28
=> 0xe9c71ca0 <+16>: ldr lr, [r12, #-8] /* 出錯指令，將r12-8指向內存地址的數據加載到lr，通過前面打印的寄存器值發現r12此時為0x2564，在0號page，明顯的非法地址訪問 */
(gdb) x/40wx $sp
0xc6e58120: 0xf5d60101 0xb8d6d50c 0xf5d6d1c4 0xf65f1808
0xc6e58130: 0xc6e5815c 0xf5d09c3c 0xc6e5815c 0x00000002
0xc6e58140: 0x0013849e 0x00080075 0xf6e05bb4 0xf65f1808
0xc6e58150: 0x000079e0 0xf5af8ed4 0xf5af903c 0xf6416f70
0xc6e58160: 0x00000000 0xf5ce3f5c 0x000079e0 0x00000415
0xc6e58170: 0x001381ac 0x00080075 0x023f3460 0x00000101
0xc6e58180: 0x023f3460 0x00080101 0x00083460 0x00000000
0xc6e58190: 0xb8d703b8 0xb8d70338 0xb8d4d4b0 0x8000007a
0xc6e581a0: 0xf6e05bb4 0x00000008 0xc6e60944 0xf6cfa60c
0xc6e581b0: 0xc6e60870 0x00000001 0x00000001 0x00000001
出錯指令為ldr lr, [r12, #-8] 此時r12的值為0x2564，指向內存0頁，為非法地址，因此該錯誤為非法地址訪問。
上一行指令sub sp, sp, #40說明該函數的棧大小為40個字節，打印所有棧數據。
此時pc地址在libstdc++.so.6庫的代碼段__dynamic_cast函數 lr地址（函數返回地址）在libconfig_xxx.so的代碼段，即調用ONU_CONFIGOBJECT()宏處。
(gdb) list *0xf6416f70
0xf6416f70 is in cfg_mod_set_onu_local_mng_interface_config_flag(PON_NO, ONU_NO, unsigned char) (/media/new/linyao/2_xPON/xPON/src/config_data_gpon/config_module_cinterface.cpp:35769).
35764 MAPLE_LOG(ULLOG_PRI_ERROR, "onuno %d\r\n", onu_no);
35765 return ERR_INVALID_ONU_NO;
35766 }
35767
35768 /* »嗃¶Տ
35769 COnuConfigObject * onu_config_object = ONU_CONFIGOBJECT(pon_no, onu_no);
35770 if (NULL != onu_config_object)
35771 {
35772 /* ¼ԋ
35773 synchronized(onu_config_object)
再回溯一條指令，是push {r4, r5, r6, r7, r8, lr}壓棧指令，lr壓棧，在棧數據中找到lr值，在0xc6e5815c處，由此反推出r4~r8的值：
r8 0xf5af903c
r7 0xf5af8ed4
r6 0x000079e0
r5 0xf65f1808
r4 0xf6e05bb4
再往上一條指令時ldr r12, [r0]，將r0指向內存的值加載到r12寄存器中。
這幾條指令中，r0的值沒有發生改變，打印r0指向內存區域的值。
(gdb) x/64wx $r0-64 （打印以r0指向地址為中心的128字節數據）
0xb8d70820: 0x00000000 0x00000000 0x00000000 0x00000000
0xb8d70830: 0xfe0f0000 0xffffffff 0x0a00ffff 0x0e888888
0xb8d70840: 0xf90f0081 0x01000608 0x04060008 0x0a000100
0xb8d70850: 0x0e888888 0x0e64190a 0x00000000 0x190a0000
0xb8d70860: 0x00002564 0x00000000 0x00000000 0x00000000
0xb8d70870: 0x00000000 0x00080001 0x00750000 0x00000013
0xb8d70880: 0x00000000 0x00000002 0x00000000 0x31312d38
0xb8d70890: 0x39312d37 0x0000322f 0x00000000 0x00000000
0xb8d708a0: 0x00000000 0x00000000 0x00000000 0xb8d708e0
0xb8d708b0: 0xb8d4d4b0 0x00000000 0x00000000 0x00000000
0xb8d708c0: 0xb8d708b8 0xb8d708b8 0x00000000 0x00000000
0xb8d708d0: 0x00000000 0x00000013 0x00000002 0x000004ad
0xb8d708e0: 0xf65f1448 0x00000000 0x00000001 0xb8d70860
0xb8d708f0: 0x00000000 0x00000000 0x00000000 0x00000000
0xb8d70900: 0x00000000 0x00000000 0x00000000 0x00000000
0xb8d70910: 0x00000000 0x00000000 0x00000000 0x00000000
r0的值為0x2564，即前面出問題的地址。那么，r0的值從哪里來的呢？帶着疑問進入上一層函數
(gdb) frame 6
#6 0xf6416f70 in cfg_mod_set_onu_local_mng_interface_config_flag (pon_no=0, onu_no=0, cfg_flag=0 '\000')
at /media/new/justin/2_xPON/xPON/src/config_data_gpon/config_module_cinterface.cpp:35769
35769 COnuConfigObject * onu_config_object = ONU_CONFIGOBJECT(pon_no, onu_no);
(gdb) info registers
r0 0xb8d70860 3101100128
r1 0xf5d6c6c0 4124493504
r2 0xf5d6c9e0 4124494304
r3 0x0 0
r4 0xf6e05bb4 4141898676
r5 0xf65f1808 4133427208
r6 0x79e0 31200
r7 0xf5af8ed4 4121923284
r8 0xf5af903c 4121923644
r9 0x64 100
r10 0xf5b1a118 4122059032
r11 0xc6e581ac 3336929708
r12 0x2564 9572
sp 0xc6e58160 0xc6e58160
lr 0xf6416f70 -163483792
pc 0xf6416f70 0xf6416f70 <cfg_mod_set_onu_local_mng_interface_config_flag(PON_NO, ONU_NO, unsigned char)+252>
cpsr 0xa0070010 -1610153968
(gdb) info frame
Stack level 6, frame at 0xc6e581b0:
pc = 0xf6416f70 in cfg_mod_set_onu_local_mng_interface_config_flag (/media/new/justin/2_xPON/xPON/src/config_data_gpon/config_module_cinterface.cpp:35769); saved pc = 0xf6cfa60c
called by frame at 0xc6e60948, caller of frame at 0xc6e58160
source language c++.
Arglist at 0xc6e581ac, args: pon_no=0, onu_no=0, cfg_flag=0 '\000'
Locals at 0xc6e581ac, Previous frame's sp is 0xc6e581b0
Saved registers:
r4 at 0xc6e581a0, r5 at 0xc6e581a4, r11 at 0xc6e581a8, lr at 0xc6e581ac
反匯編確定frame 6返回地址行
(gdb) disassemble （根據寄存器值和棧數據，返回代碼執行流程，灰色為不執行代碼，藍色為跳轉地址）
Dump of assembler code for function cfg_mod_set_onu_local_mng_interface_config_flag(PON_NO, ONU_NO, unsigned char):
0xf6416e74 <+0>: push {r4, r5, r11, lr}
0xf6416e78 <+4>: add r11, sp, #12
0xf6416e7c <+8>: sub sp, sp, #32
0xf6416e80 <+12>: mov r3, r1
0xf6416e84 <+16>: strh r0, [r11, #-30] ; 0xffffffe2
0xf6416e88 <+20>: strh r3, [r11, #-32] ; 0xffffffe0
0xf6416e8c <+24>: mov r3, r2
0xf6416e90 <+28>: strb r3, [r11, #-33] ; 0xffffffdf
0xf6416e94 <+32>: ldr r5, [pc, #556] ; 0xf64170c8 <cfg_mod_set_onu_local_mng_interface_config_flag(PON_NO, ONU_NO, unsigned char)+596>
0xf6416e98 <+36>: add r5, pc, r5
0xf6416e9c <+40>: ldrb r3, [r11, #-33] ; 0xffffffdf
0xf6416ea0 <+44>: cmp r3, #1
0xf6416ea4 <+48>: bls 0xf6416ed4 <cfg_mod_set_onu_local_mng_interface_config_flag(PON_NO, ONU_NO, unsigned char)+96>
0xf6416ea8 <+52>: ldrb r3, [r11, #-33] ; 0xffffffdf
0xf6416eac <+56>: str r3, [sp]
0xf6416eb0 <+60>: ldr r3, [pc, #532] ; 0xf64170cc <cfg_mod_set_onu_local_mng_interface_config_flag(PON_NO, ONU_NO, unsigned char)+600>
0xf6416eb4 <+64>: add r3, pc, r3
0xf6416eb8 <+68>: ldr r2, [pc, #528] ; 0xf64170d0 <cfg_mod_set_onu_local_mng_interface_config_flag(PON_NO, ONU_NO, unsigned char)+604>
0xf6416ebc <+72>: add r2, pc, r2
0xf6416ec0 <+76>: mov r1, #13
0xf6416ec4 <+80>: mov r0, #10
0xf6416ec8 <+84>: bl 0xf63afc70 <ulogPrintf@plt>
0xf6416ecc <+88>: mov r4, #-2147483645 ; 0x80000003
0xf6416ed0 <+92>: b 0xf64170a4 <cfg_mod_set_onu_local_mng_interface_config_flag(PON_NO, ONU_NO, unsigned char)+560>
0xf6416ed4 <+96>: ldrh r3, [r11, #-30] ; 0xffffffe2
0xf6416ed8 <+100>: mov r0, r3
0xf6416edc <+104>: bl 0xf63aa6cc <valid_port_no@plt>
(gdb) disassemble 0xf63aa6cc
Dump of assembler code for function valid_port_no@plt:
0xf63aa6cc <+0>: add r12, pc, #2097152 ; 0x200000
0xf63aa6d0 <+4>: add r12, r12, #290816 ; 0x47000
0xf63aa6d4 <+8>: ldr pc, [r12, #3688]! ; 0xe68
End of assembler dump.
0xf6416ee0 <+108>: mov r3, r0
0xf6416ee4 <+112>: cmp r3, #0
0xf6416ee8 <+116>: movne r3, #1
0xf6416eec <+120>: moveq r3, #0
0xf6416ef0 <+124>: uxtb r3, r3
0xf6416ef4 <+128>: cmp r3, #0
0xf6416ef8 <+132>: beq 0xf6416f20 <cfg_mod_set_onu_local_mng_interface_config_flag(PON_NO, ONU_NO, unsigned char)+172>
0xf6416efc <+136>: ldrh r3, [r11, #-30] ; 0xffffffe2
0xf6416f00 <+140>: ldr r2, [pc, #460] ; 0xf64170d4 <cfg_mod_set_onu_local_mng_interface_config_flag(PON_NO, ONU_NO, unsigned char)+608>
0xf6416f04 <+144>: add r2, pc, r2
0xf6416f08 <+148>: mov r1, #13
0xf6416f0c <+152>: mov r0, #10
0xf6416f10 <+156>: bl 0xf63afc70 <ulogPrintf@plt>
0xf6416f14 <+160>: mov r4, #134 ; 0x86
0xf6416f18 <+164>: movt r4, #32768 ; 0x8000
0xf6416f1c <+168>: b 0xf64170a4 <cfg_mod_set_onu_local_mng_interface_config_flag(PON_NO, ONU_NO, unsigned char)+560>
0xf6416f20 <+172>: ldrh r3, [r11, #-32] ; 0xffffffe0
0xf6416f24 <+176>: mov r0, r3 >>>>>>>>>Look Here>>>>>>>>> r0 = r3 = [r11, #-32],
0xf6416f28 <+180>: bl 0xf63ae9bc <valid_onu_no@plt>
(gdb) disassemble 0xf63ae9bc
Dump of assembler code for function valid_onu_no@plt:
0xf63ae9bc <+0>: add r12, pc, #2097152 ; 0x200000
0xf63ae9c0 <+4>: add r12, r12, #282624 ; 0x45000
0xf63ae9c4 <+8>: ldr pc, [r12, #456]! ; 0x1c8
End of assembler dump.
0xf6416f2c <+184>: mov r3, r0
0xf6416f30 <+188>: cmp r3, #0
0xf6416f34 <+192>: movne r3, #1
0xf6416f38 <+196>: moveq r3, #0
0xf6416f3c <+200>: uxtb r3, r3
0xf6416f40 <+204>: cmp r3, #0
0xf6416f44 <+208>: beq 0xf6416f6c <cfg_mod_set_onu_local_mng_interface_config_flag(PON_NO, ONU_NO, unsigned char)+248>
0xf6416f48 <+212>: ldrh r3, [r11, #-32] ; 0xffffffe0
0xf6416f4c <+216>: ldr r2, [pc, #388] ; 0xf64170d8 <cfg_mod_set_onu_local_mng_interface_config_flag(PON_NO, ONU_NO, unsigned char)+612>
0xf6416f50 <+220>: add r2, pc, r2
0xf6416f54 <+224>: mov r1, #13
0xf6416f58 <+228>: mov r0, #10
0xf6416f5c <+232>: bl 0xf63afc70 <ulogPrintf@plt>
0xf6416f60 <+236>: mov r4, #135 ; 0x87
0xf6416f64 <+240>: movt r4, #32768 ; 0x8000
0xf6416f68 <+244>: b 0xf64170a4 <cfg_mod_set_onu_local_mng_interface_config_flag(PON_NO, ONU_NO, unsigned char)+560>
0xf6416f6c <+248>: ldrh r1, [r11, #-32] ; 0xffffffe0 >>>>>>>>>>[r11 - 32]物理內存地址(0xc6e5818c)的值為0x00000000，但是此時r1寄存器值為0xf5d6c6c0，說明棧的數據被踩了
=> 0xf6416f70 <+252>: ldrh r3, [r11, #-30] ; 0xffffffe2
0xf6416f74 <+256>: mov r2, #0
0xf6416f78 <+260>: mov r0, r3
0xf6416f7c <+264>: bl 0xf63a8aa0 <_ZN17CCardConfigObject15GetConfigObjectEtth@plt>
0xf6416f80 <+268>: cmp r0, #0
0xf6416f84 <+272>: bne 0xf6416f90 <cfg_mod_set_onu_local_mng_interface_config_flag(PON_NO, ONU_NO, unsigned char)+284>
分析r1 = [r11 - 32]，此時r1寄存器值為0xf5d6c6c0，但是棧數據中保存的值為0x0000 0000，說明棧數據被踩了。
(gdb) p/x $r11-32
$1 = 0xc6e5818c
(gdb) x/wx 0xc6e5818c
0xc6e5818c: 0x00000000
分析r0 = r3 = [r11 - 30], r11是棧基址，sp是棧頂指針。此時r0寄存器值為0xb8d70860，但是棧數據中保存的值為0x03b8 0000，證明棧數據果然被踩了。
(gdb) p/x $r11-30
$2 = 0xc6e5818e
(gdb) x/wx 0xc6e5818e
0xc6e5818e: 0x03b80000
那么棧數據被誰踩了呢？茫茫進程中，萬千線程，怎么破？
(gdb) info registers
r0 0xb8d70860 3101100128
r1 0xf5d6c6c0 4124493504
r2 0xf5d6c9e0 4124494304
r3 0x0 0
r4 0xf6e05bb4 4141898676
r5 0xf65f1808 4133427208
r6 0x79e0 31200
r7 0xf5af8ed4 4121923284
r8 0xf5af903c 4121923644
r9 0x64 100
r10 0xf5b1a118 4122059032
r11 0xc6e581ac 3336929708
r12 0x2564 9572
sp 0xc6e58160 0xc6e58160
lr 0xf6416f70 -163483792
pc 0xf6416f70 0xf6416f70 <cfg_mod_set_onu_local_mng_interface_config_flag(PON_NO, ONU_NO, unsigned char)+252>
cpsr 0xa0070010 -1610153968
查看一下棧數據，貌似沒有發現什么異常。
(gdb) x /128wx $sp-256
0xc6e58060: 0x008499d0 0xf5b1a118 0x000000a9 0xf5d482d8
0xc6e58070: 0x000000a9 0xf5ce41f4 0xf5d6ca30 0xb8d4d4b0
0xc6e58080: 0x00000000 0xf5b1a118 0x000000a9 0xf5d482d8
0xc6e58090: 0xc6e5809c 0xf5ce41f4 0xf65f1808 0x000079e0
0xc6e580a0: 0xf5af8ed4 0xf5af903c 0x00000000 0xb8d70860
0xc6e580b0: 0xc6e580bc 0xf5a6da18 0xc6e580ec 0xf5ce4310
0xc6e580c0: 0x0000006b 0xf5d4c410 0x000003e8 0x023f3460
0xc6e580d0: 0xf5d6ca30 0x00000000 0x00000000 0x023f3460
0xc6e580e0: 0x0000006b 0x00000000 0xc6e580fc 0xf5ce3f5c
0xc6e580f0: 0xb8d6d50c 0xc6e5811c 0xc6e58134 0xf5d0ee0c
0xc6e58100: 0x000003e8 0xc6e5810c 0xf5d6d1c4 0x00000013
0xc6e58110: 0x00000002 0xb8d4d4b0 0xb8d6d4f4 0x023f3460
0xc6e58120: 0xf5d60101 0xb8d6d50c 0xf5d6d1c4 0xf65f1808
0xc6e58130: 0xc6e5815c 0xf5d09c3c 0xc6e5815c 0x00000002
0xc6e58140: 0x0013849e 0x00080075 0xf6e05bb4 0xf65f1808
0xc6e58150: 0x000079e0 0xf5af8ed4 0xf5af903c 0xf6416f70
0xc6e58160: 0x00000000 0xf5ce3f5c 0x000079e0 0x00000415
0xc6e58170: 0x001381ac 0x00080075 0x023f3460 0x00000101
0xc6e58180: 0x023f3460 0x00080101 0x00083460 0x00000000
0xc6e58190: 0xb8d703b8 0xb8d70338 0xb8d4d4b0 0x8000007a
0xc6e581a0: 0xf6e05bb4 0x00000008 0xc6e60944 0xf6cfa60c
0xc6e581b0: 0xc6e60870 0x00000001 0x00000001 0x00000001
0xc6e581c0: 0x00000000 0xeeeeeeee 0xeeeeeeee 0x0000024f
0xc6e581d0: 0xde83a281 0x025a7a38 0x00000000 0x00000000
0xc6e581e0: 0x00000000 0x00000000 0x00000000 0x00000000
0xc6e581f0: 0x00000000 0x00000000 0x00000000 0x00000000
0xc6e58200: 0x00000000 0x00000000 0x00000000 0x00000000
0xc6e58210: 0x00000000 0x00000000 0x00000000 0x00000000
0xc6e58220: 0x00000000 0x00000000 0x00000000 0x00000000
0xc6e58230: 0x00000000 0x00000000 0x00000000 0x00000000
0xc6e58240: 0x00000000 0x00000000 0x00000000 0x00000000
0xc6e58250: 0x00000000 0x00000000 0x00000000 0x00000000
既然出問題的r0指向的內存地址，而且0x2564地址出現的很詭異，明顯也是被踩了，那就看一下r0指向的地址附近的數據
查看以r0指向地址為中心的128字節數據
(gdb) x /64wx $r0-64
0xb8d70820: 0x00000000 0x00000000 0x00000000 0x00000000
0xb8d70830: 0xfe0f0000 0xffffffff 0x0a00ffff 0x0e888888
0xb8d70840: 0xf90f0081 0x01000608 0x04060008 0x0a000100
0xb8d70850: 0x0e888888 0x0e64190a 0x00000000 0x190a0000
0xb8d70860: 0x00002564 0x00000000 0x00000000 0x00000000 ------->0x2564赫然在列！
0xb8d70870: 0x00000000 0x00080001 0x00750000 0x00000013
0xb8d70880: 0x00000000 0x00000002 0x00000000 0x31312d38
0xb8d70890: 0x39312d37 0x0000322f 0x00000000 0x00000000
0xb8d708a0: 0x00000000 0x00000000 0x00000000 0xb8d708e0
0xb8d708b0: 0xb8d4d4b0 0x00000000 0x00000000 0x00000000
0xb8d708c0: 0xb8d708b8 0xb8d708b8 0x00000000 0x00000000
0xb8d708d0: 0x00000000 0x00000013 0x00000002 0x000004ad
0xb8d708e0: 0xf65f1448 0x00000000 0x00000001 0xb8d70860
0xb8d708f0: 0x00000000 0x00000000 0x00000000 0x00000000
0xb8d70900: 0x00000000 0x00000000 0x00000000 0x00000000
0xb8d70910: 0x00000000 0x00000000 0x00000000 0x00000000
以字節形式顯示:
(gdb) x/128bx $r0-64
0xb8d70820: 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
0xb8d70828: 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
0xb8d70830: 0x00 0x00 0x0f 0xfe 0xff 0xff 0xff 0xff ---->ff:ff:ff:ff:ff:ff 目的MAC地址，廣播地址
0xb8d70838: 0xff 0xff 0x00 0x0a 0x88 0x88 0x88 0x0e ----->源MAC地址
0xb8d70840: 0x81 0x00 0x0f 0xf9 0x08 0x06 0x00 0x01 -----> 0x8100 幀類型 ARP數據包
0xb8d70848: 0x08 0x00 0x06 0x04 0x00 0x01 0x00 0x0a
0xb8d70850: 0x88 0x88 0x88 0x0e 0x0a 0x19 0x64 0x0e
0xb8d70858: 0x00 0x00 0x00 0x00 0x00 0x00 0x0a 0x19
0xb8d70860: 0x64 0x25 0x00 0x00 0x00 0x00 0x00 0x00
0xb8d70868: 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
0xb8d70870: 0x00 0x00 0x00 0x00 0x01 0x00 0x08 0x00
0xb8d70878: 0x00 0x00 0x75 0x00 0x13 0x00 0x00 0x00
0xb8d70880: 0x00 0x00 0x00 0x00 0x02 0x00 0x00 0x00
0xb8d70888: 0x00 0x00 0x00 0x00 0x38 0x2d 0x31 0x31
0xb8d70890: 0x37 0x2d 0x31 0x39 0x2f 0x32 0x00 0x00
0xb8d70898: 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
這個時候，大神出現，一眼就看出了踩內存的這一段數據是一個ARP網絡數據包。趕緊ifconfig看一下:
源MAC地址和eth0的MAC地址完全一致，終於抓到了罪魁禍首！！！
最終定位是網卡驅動使用了一段保留內存段，但是Linux內核初始化DDR時，保留內存段設置沒有對齊，導致該內存段參與了動態內存分配，用戶態進程和內核網卡驅動同時往這段內存寫數據，ARP報文踩了進程的子線程的棧，導致該異常發生。