valgrind包含多個工具，通過--tool=xxx指定，最被大家熟知是memcheck，主要解決內存泄露，越界訪問，未初始化卻去引用等問題，它是默認選項，如果未指定--tool，默認就是memcheck了。而在多線程編程中，最常見的bug有：數據競爭（data race），死鎖，錯誤的使用POSIX接口等問題。這些問題可以通過valgrind工具集里面的helgrind來探測。即：
valgrind --tool=helgrind yourprogram
-
利用診斷輸出文件靜態分析
一般探測出來的診斷信息會直接輸出在終端上，但是有時候被診斷的程序可能也會輸出信息到終端上，兩者混淆在一起不方便閱讀。這時可以通過參數--log-file=指定診斷日志文件來解決這個問題，診斷信息會輸出到你指定的文件里面。
valgrind --tool=helgrind --log-file=./dump.log yourprogram
甚至還可以是fd, socket。分別由參數--log-fd=，--log-socket=指定。
診斷信息通常如下：
==11525== Possible data race during read of size 1 at 0x6532D40 by thread #3
==11525== Locks held: 1, at address 0x1FFEFFFCA0
==11525== at 0x5068856: ??? (in /usr/lib/x86_64-linux-gnu/libmosquitto.so.1)
==11525== by 0x1140C6: mqttclient_subscribe (mqttwrapper.c:1138)
==11525== by 0x1162CB: bar (test_mqtt.c:99)
==11525== by 0x4C36C26: ??? (in /usr/lib/valgrind/vgpreload_helgrind-amd64-linux.so)
==11525== by 0x4E496DA: start_thread (pthread_create.c:463)
==11525== by 0x585DA3E: clone (clone.S:95)
==11525==
==11525== This conflicts with a previous write of size 1 by thread #6
==11525== Locks held: 1, at address 0x6532BA0
==11525== at 0x506948B: ??? (in /usr/lib/x86_64-linux-gnu/libmosquitto.so.1)
==11525== by 0x5068A65: ??? (in /usr/lib/x86_64-linux-gnu/libmosquitto.so.1)
==11525== by 0x5065D58: mosquitto_loop_read (in /usr/lib/x86_64-linux-gnu/libmosquitto.so.1)
==11525== by 0x50660D0: ??? (in /usr/lib/x86_64-linux-gnu/libmosquitto.so.1)
==11525== by 0x5066465: mosquitto_loop_forever (in /usr/lib/x86_64-linux-gnu/libmosquitto.so.1)
==11525== by 0x506B25A: ??? (in /usr/lib/x86_64-linux-gnu/libmosquitto.so.1)
==11525== by 0x4C36C26: ??? (in /usr/lib/valgrind/vgpreload_helgrind-amd64-linux.so)
==11525== by 0x4E496DA: start_thread (pthread_create.c:463)
==11525== Address 0x6532d40 is 656 bytes inside a block of size 832 alloc'd
==11525== at 0x4C32F45: calloc (in /usr/lib/valgrind/vgpreload_helgrind-amd64-linux.so)
==11525== by 0x50650CE: mosquitto_new (in /usr/lib/x86_64-linux-gnu/libmosquitto.so.1)
==11525== by 0x11271C: client_create (mqttwrapper.c:239)
==11525== by 0x1139B6: mqttclient_create (mqttwrapper.c:837)
==11525== by 0x1160F1: mqtt_init (test_mqtt.c:32)
==11525== by 0x1162B3: bar (test_mqtt.c:97)
==11525== by 0x4C36C26: ??? (in /usr/lib/valgrind/vgpreload_helgrind-amd64-linux.so)
==11525== by 0x4E496DA: start_thread (pthread_create.c:463)
==11525== by 0x585DA3E: clone (clone.S:95)
==11525== Block was alloc'd by thread #3
如上面的信息所示：
==11525== Possible data race during read of size 1 at 0x6532D40 by thread #3
==11525== Locks held: 1, at address 0x1FFEFFFCA0
==11525== at 0x5068856: ??? (in /usr/lib/x86_64-linux-gnu/libmosquitto.so.1)
==11525== by 0x1140C6: mqttclient_subscribe (mqttwrapper.c:1138)
==11525== by 0x1162CB: bar (test_mqtt.c:99)
==11525== by 0x4C36C26: ??? (in /usr/lib/valgrind/vgpreload_helgrind-amd64-linux.so)
==11525== by 0x4E496DA: start_thread (pthread_create.c:463)
==11525== by 0x585DA3E: clone (clone.S:95)
這里輸出了調用棧，也寫明了dada race所操作的地址0x6532D40。但是沒法判斷具體是哪個變量，並且錯誤產生時也無法暫停程序，用gdb attach來分析。但是動態分析程序，像用gdb那樣分析程序對分析問題是有極大幫助的。
-
使用gdb + valgrind動態分析
好在valgrind有一個叫vgdb的東西，可以運行一個gdb server，然后我們用gdb的 target remote來連接它，於是可以愉快的使用gdb動態分析問題，並且可以在問題產生時，立馬停止程序，分析現場。
--vgdb=yes|no使用或關閉vgdb--vgdb-error=0探測到多少個錯誤就暫停程序，如果是0，則表示發現錯誤立即暫停。
具體命令如下：
valgrind --tool=helgrind --vgdb=yes --vgdb-error=0 ./test_mqtt
這時gdb server就暫停程序，等待gdb來連接才開始繼續運行下去。另起一個終端，直接運行gdb，然后輸入 target remote | vgdb , continue
thomas@thomas-virtual-machine:~$ gdb
GNU gdb (Ubuntu 8.1-0ubuntu3.2) 8.1.0.20180409-git
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word".
(gdb) target remote | vgdb
(gdb) c
用gdb來連接gdbserver總不太方便，借助IDE可以更舒服的分析問題。推薦使用Clion這款IDE，詳細設置見官方文檔。稍微不同的是'target remote' args不是文檔里面gdbserver ip : port的形式，而是| vgdb這種管道形式。然后就是設置你的符號文件，即你應用程序。