多线程中如何使用gdb精确定位死锁问题

https://my.oschina.net/u/4614991/blog/4494424

传统gdb调试多线程死锁方法

（1）attach id关联到发生死锁的进程id

(gdb) attach 109Attaching to process 109[New LWP 110][New LWP 111][New LWP 112][Thread debugging using libthread_db enabled]Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".0x00007fa33f9e8d2d in __GI___pthread_timedjoin_ex (threadid=140339109693184, thread_return=0x0, abstime=0x0,block=<optimized out>) at pthread_join_common.c:8989 pthread_join_common.c: No such file or directory.

（2）info threads查看当前进程中所有线程的信息，也可以查看到部分堆栈信息

(gdb) info threadsId Target Id Frame* 1 Thread 0x7fa33ff10740 (LWP 109) "out" 0x00007fa33f9e8d2d in __GI___pthread_timedjoin_ex (threadid=140339109693184, thread_return=0x0, abstime=0x0, block=<optimized out>) at pthread_join_common.c:892 Thread 0x7fa33ec80700 (LWP 110) "out" __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:1353 Thread 0x7fa33e470700 (LWP 111) "out" __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:1354 Thread 0x7fa33dc60700 (LWP 112) "out" __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135

这里可以看到2、3、4线程都在lock_wait状态，基本上可以看出或许是否问题，但是不一定，这里需要多次info threads看看这些线程有没有什么变化，多次如果都没有变化那基本上就是发生了死锁。

（3）thread id进入具体线程

(gdb) thread 2[Switching to thread 2 (Thread 0x7fa33ec80700 (LWP 110))]#0 __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135135 ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S: No such file or directory.

（4）bt查看当前线程堆栈信息

(gdb) bt#0 __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135#1 0x00007fa33f9ea023 in __GI___pthread_mutex_lock (mutex=0x7fa340204180 <mutex2>) at ../nptl/pthread_mutex_lock.c:78#2 0x00007fa340000fff in __gthread_mutex_lock(pthread_mutex_t*) ()#3 0x00007fa3400015b2 in std::mutex::lock() ()#4 0x00007fa3400016d8 in std::lock_guard<std::mutex>::lock_guard(std::mutex&) ()#5 0x00007fa34000109b in FuncA() ()#6 0x00007fa340001c07 in void std::__invoke_impl<void, void (*)()>(std::__invoke_other, void (*&&)()) ()

调试到这里基本已经差不多了，针对pthread_mutex_t却可以打印出被哪个线程持有，之后再重复步骤3和4，就可以确定哪几个线程以及哪几把锁发生的死锁，而针对于std::mutex，gdb没法打印具体的mutex的信息，不能看出来mutex是被哪个线程持有，只能依次进入线程查看堆栈信息。

然而针对于c++11的std::mutex有没有什么好办法定位死锁呢？

有。

可以算作第五步，继续：

（5）source加载deadlock.py脚本

(gdb) source -v deadlock.pyType "deadlock" to detect deadlocks.

（6）输入deadlock检测死锁

(gdb) deadlock[Switching to thread 3 (Thread 0x7f5585670700 (LWP 123))]#0 __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135135 in ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S[Switching to thread 4 (Thread 0x7f5584e60700 (LWP 124))]#0 __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135135 in ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S[Switching to thread 2 (Thread 0x7f5585e80700 (LWP 122))]#0 __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135135 in ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S#1 0x00007f5586bea023 in __GI___pthread_mutex_lock (mutex=0x7f5587404180 <mutex2>) at ../nptl/pthread_mutex_lock.c:78[Switching to thread 3 (Thread 0x7f5585670700 (LWP 123))]#0 __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135#1 0x00007f5586bea023 in __GI___pthread_mutex_lock (mutex=0x7f55874041c0 <mutex3>) at ../nptl/pthread_mutex_lock.c:78[Switching to thread 4 (Thread 0x7f5584e60700 (LWP 124))]#0 __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135#1 0x00007f5586bea023 in __GI___pthread_mutex_lock (mutex=0x7f5587404140 <mutex1>) at ../nptl/pthread_mutex_lock.c:78Found deadlock!Thread 2 (LWP 122) is waiting on pthread_mutex_t (0x00007f5587404180) held by Thread 3 (LWP 123)Thread 3 (LWP 123) is waiting on pthread_mutex_t (0x00007f55874041c0) held by Thread 4 (LWP 124)Thread 4 (LWP 124) is waiting on pthread_mutex_t (0x00007f5587404140) held by Thread 2 (LWP 122)

直接看结果，脚本检测出了死锁，并指明了具体的哪几个线程造成的死锁，根据输出信息可以明显看出来线程锁形成的环造成了死锁，找到了具体是哪几个线程构成的死锁环，就可以查看相应线程的堆栈信息查看到哪几把锁正在等待。

死锁检测脚本的原理：

还是拿上面图举例：

• 线程A已经持有mutex1，想要申请mutex2，拿到mutex2后才可以释放mutex1和mutex2，而此时mutex2被线程B占用。

• 线程B已经持有mutex2，想要申请mutex3，拿到mutex3后才可以释放mutex2和mutex3，而此时mutex3被线程C占用。

• 线程C已经持有mutex3，想要申请mutex1，拿到mutex1后才可以释放mutex3和mutex1，而此时mutex1被线程A占用。

如图，三个线程形成了一个环，死锁检测就是检查线程之间是否有环的存在。单独检查死锁的环比较容易，这里延申下还涉及到简单环的概念，因为正常检测出来的环可能是个大环，不是权值顶点数最少的环，如果检测的环的顶点数较多，加大定位的代价，脚本就是检测的简单环，这里涉及到强连通分量算法和简单环算法，比较繁琐就不过多介绍了，脚本来源于facebook的folly库（这里推荐看下google的abseil和facebook的folly，都是好东西），代码较长在文中不好列出，

如何在代码中检测死锁

和上面介绍的原理相同，在线程加锁过程中始终维护一张图，记录线程之间的关系

A->B, B->C, C->A