多線程中如何使用gdb精確定位死鎖問題

https://my.oschina.net/u/4614991/blog/4494424

傳統gdb調試多線程死鎖方法

（1）attach id關聯到發生死鎖的進程id

(gdb) attach 109Attaching to process 109[New LWP 110][New LWP 111][New LWP 112][Thread debugging using libthread_db enabled]Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".0x00007fa33f9e8d2d in __GI___pthread_timedjoin_ex (threadid=140339109693184, thread_return=0x0, abstime=0x0,block=<optimized out>) at pthread_join_common.c:8989 pthread_join_common.c: No such file or directory.

（2）info threads查看當前進程中所有線程的信息，也可以查看到部分堆棧信息

(gdb) info threadsId Target Id Frame* 1 Thread 0x7fa33ff10740 (LWP 109) "out" 0x00007fa33f9e8d2d in __GI___pthread_timedjoin_ex (threadid=140339109693184, thread_return=0x0, abstime=0x0, block=<optimized out>) at pthread_join_common.c:892 Thread 0x7fa33ec80700 (LWP 110) "out" __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:1353 Thread 0x7fa33e470700 (LWP 111) "out" __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:1354 Thread 0x7fa33dc60700 (LWP 112) "out" __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135

這里可以看到2、3、4線程都在lock_wait狀態，基本上可以看出或許是否問題，但是不一定，這里需要多次info threads看看這些線程有沒有什么變化，多次如果都沒有變化那基本上就是發生了死鎖。

（3）thread id進入具體線程

(gdb) thread 2[Switching to thread 2 (Thread 0x7fa33ec80700 (LWP 110))]#0 __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135135 ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S: No such file or directory.

（4）bt查看當前線程堆棧信息

(gdb) bt#0 __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135#1 0x00007fa33f9ea023 in __GI___pthread_mutex_lock (mutex=0x7fa340204180 <mutex2>) at ../nptl/pthread_mutex_lock.c:78#2 0x00007fa340000fff in __gthread_mutex_lock(pthread_mutex_t*) ()#3 0x00007fa3400015b2 in std::mutex::lock() ()#4 0x00007fa3400016d8 in std::lock_guard<std::mutex>::lock_guard(std::mutex&) ()#5 0x00007fa34000109b in FuncA() ()#6 0x00007fa340001c07 in void std::__invoke_impl<void, void (*)()>(std::__invoke_other, void (*&&)()) ()

調試到這里基本已經差不多了，針對pthread_mutex_t卻可以打印出被哪個線程持有，之后再重復步驟3和4，就可以確定哪幾個線程以及哪幾把鎖發生的死鎖，而針對於std::mutex，gdb沒法打印具體的mutex的信息，不能看出來mutex是被哪個線程持有，只能依次進入線程查看堆棧信息。

然而針對於c++11的std::mutex有沒有什么好辦法定位死鎖呢？

有。

可以算作第五步，繼續：

（5）source加載deadlock.py腳本

(gdb) source -v deadlock.pyType "deadlock" to detect deadlocks.

（6）輸入deadlock檢測死鎖

(gdb) deadlock[Switching to thread 3 (Thread 0x7f5585670700 (LWP 123))]#0 __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135135 in ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S[Switching to thread 4 (Thread 0x7f5584e60700 (LWP 124))]#0 __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135135 in ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S[Switching to thread 2 (Thread 0x7f5585e80700 (LWP 122))]#0 __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135135 in ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S#1 0x00007f5586bea023 in __GI___pthread_mutex_lock (mutex=0x7f5587404180 <mutex2>) at ../nptl/pthread_mutex_lock.c:78[Switching to thread 3 (Thread 0x7f5585670700 (LWP 123))]#0 __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135#1 0x00007f5586bea023 in __GI___pthread_mutex_lock (mutex=0x7f55874041c0 <mutex3>) at ../nptl/pthread_mutex_lock.c:78[Switching to thread 4 (Thread 0x7f5584e60700 (LWP 124))]#0 __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135#1 0x00007f5586bea023 in __GI___pthread_mutex_lock (mutex=0x7f5587404140 <mutex1>) at ../nptl/pthread_mutex_lock.c:78Found deadlock!Thread 2 (LWP 122) is waiting on pthread_mutex_t (0x00007f5587404180) held by Thread 3 (LWP 123)Thread 3 (LWP 123) is waiting on pthread_mutex_t (0x00007f55874041c0) held by Thread 4 (LWP 124)Thread 4 (LWP 124) is waiting on pthread_mutex_t (0x00007f5587404140) held by Thread 2 (LWP 122)

直接看結果，腳本檢測出了死鎖，並指明了具體的哪幾個線程造成的死鎖，根據輸出信息可以明顯看出來線程鎖形成的環造成了死鎖，找到了具體是哪幾個線程構成的死鎖環，就可以查看相應線程的堆棧信息查看到哪幾把鎖正在等待。

死鎖檢測腳本的原理：

還是拿上面圖舉例：

• 線程A已經持有mutex1，想要申請mutex2，拿到mutex2后才可以釋放mutex1和mutex2，而此時mutex2被線程B占用。

• 線程B已經持有mutex2，想要申請mutex3，拿到mutex3后才可以釋放mutex2和mutex3，而此時mutex3被線程C占用。

• 線程C已經持有mutex3，想要申請mutex1，拿到mutex1后才可以釋放mutex3和mutex1，而此時mutex1被線程A占用。

如圖，三個線程形成了一個環，死鎖檢測就是檢查線程之間是否有環的存在。單獨檢查死鎖的環比較容易，這里延申下還涉及到簡單環的概念，因為正常檢測出來的環可能是個大環，不是權值頂點數最少的環，如果檢測的環的頂點數較多，加大定位的代價，腳本就是檢測的簡單環，這里涉及到強連通分量算法和簡單環算法，比較繁瑣就不過多介紹了，腳本來源於facebook的folly庫（這里推薦看下google的abseil和facebook的folly，都是好東西），代碼較長在文中不好列出，

如何在代碼中檢測死鎖

和上面介紹的原理相同，在線程加鎖過程中始終維護一張圖，記錄線程之間的關系

A->B, B->C, C->A