python下的jstack - pystack

背景

• python 多進程任務，卡在某個地方沒有繼續執行也沒有報出異常，進程被hang住
• 日志沒有捕獲到相關信息，需要知道進程阻塞在哪里，為什么阻塞
• jvm提供了jstack、jmap類工具進行性能分析，python有沒有類似於pystack的工具
• python jstack pystack 調試工具
• 依賴gdb
• 結論：http接口訪問忘記設置超時，導致接口響應緩慢情況下任務死等。

安裝

yum install gdb
pip install pystack-debugger

內網安裝

pip download pystack-debugger
wget -c -t 10 10.222.77.121:12306/pystack_debugger-0.9.0-py2-none-any.whl
wget -c -t 10 10.222.77.121:12306/click-5.1-py2.py3-none-any.whl
pip install ./click-5.1-py2.py3-none-any.whl
pip install ./pystack_debugger-0.9.0-py2-none-any.whl

pystack 使用（其他作業，背景中debug作業信息已丟失）

• threads

pystack 22374

pystack 22374
Dumping Threads....


  File "/usr/lib64/python2.7/threading.py", line 784, in __bootstrap
    self.__bootstrap_inner()
  File "/usr/lib64/python2.7/threading.py", line 811, in __bootstrap_inner
    self.run()
  File "/usr/lib64/python2.7/threading.py", line 764, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/mnt/disk1/control_center/weibox/yarn-component/tools/utils/_waic_base_job_definition.py", line 157, in wait
    time.sleep(self.watch_dog_duration)

---------------

  File "/data0/control_center/weibox/yarn-weibox/tools/waic_weibox_submit.py", line 17, in <module>
    kimy.submit()
  File "/mnt/disk1/control_center/weibox/yarn-component/tools/utils/_waic_base_job_definition.py", line 175, in submit
    self.job_execute()
  File "/mnt/disk1/control_center/weibox/yarn-component/tools/utils/_waic_base_job_definition.py", line 521, in job_execute
    (response_code, response_out, response_err) = helpers.run_shell("bash " + self.weibox_submit_name)
  File "/mnt/disk1/control_center/weibox/yarn-component/tools/utils/helpers.py", line 218, in run_shell
    stdout, stderr = p.communicate()
  File "/usr/lib64/python2.7/subprocess.py", line 800, in communicate
    return self._communicate(input)
  File "/usr/lib64/python2.7/subprocess.py", line 1401, in _communicate
    stdout, stderr = self._communicate_with_poll(input)
  File "/usr/lib64/python2.7/subprocess.py", line 1455, in _communicate_with_poll
    ready = poller.poll()
  File "<string>", line 1, in <module>
  File "<string>", line 1, in <module>

---

• greenlet

pystack 22374 --include-greenlet

pystack 22374 --include-greenlet
Dumping Threads....


  File "/usr/lib64/python2.7/threading.py", line 784, in __bootstrap
    self.__bootstrap_inner()
  File "/usr/lib64/python2.7/threading.py", line 811, in __bootstrap_inner
    self.run()
  File "/usr/lib64/python2.7/threading.py", line 764, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/mnt/disk1/control_center/weibox/yarn-component/tools/utils/_waic_base_job_definition.py", line 157, in wait
    time.sleep(self.watch_dog_duration)

---------------

  File "/data0/control_center/weibox/yarn-weibox/tools/waic_weibox_submit.py", line 17, in <module>
    kimy.submit()
  File "/mnt/disk1/control_center/weibox/yarn-component/tools/utils/_waic_base_job_definition.py", line 175, in submit
    self.job_execute()
  File "/mnt/disk1/control_center/weibox/yarn-component/tools/utils/_waic_base_job_definition.py", line 521, in job_execute
    (response_code, response_out, response_err) = helpers.run_shell("bash " + self.weibox_submit_name)
  File "/mnt/disk1/control_center/weibox/yarn-component/tools/utils/helpers.py", line 218, in run_shell
    stdout, stderr = p.communicate()
  File "/usr/lib64/python2.7/subprocess.py", line 800, in communicate
    return self._communicate(input)
  File "/usr/lib64/python2.7/subprocess.py", line 1401, in _communicate
    stdout, stderr = self._communicate_with_poll(input)
  File "/usr/lib64/python2.7/subprocess.py", line 1455, in _communicate_with_poll
    ready = poller.poll()
  File "<string>", line 1, in <module>
  File "<string>", line 1, in <module>

Dumping Greenlets....


  File "/data0/control_center/weibox/yarn-weibox/tools/waic_weibox_submit.py", line 17, in <module>
    kimy.submit()
  File "/mnt/disk1/control_center/weibox/yarn-component/tools/utils/_waic_base_job_definition.py", line 175, in submit
    self.job_execute()
  File "/mnt/disk1/control_center/weibox/yarn-component/tools/utils/_waic_base_job_definition.py", line 521, in job_execute
    (response_code, response_out, response_err) = helpers.run_shell("bash " + self.weibox_submit_name)
  File "/mnt/disk1/control_center/weibox/yarn-component/tools/utils/helpers.py", line 218, in run_shell
    stdout, stderr = p.communicate()
  File "/usr/lib64/python2.7/subprocess.py", line 800, in communicate
    return self._communicate(input)
  File "/usr/lib64/python2.7/subprocess.py", line 1401, in _communicate
    stdout, stderr = self._communicate_with_poll(input)
  File "/usr/lib64/python2.7/subprocess.py", line 1455, in _communicate_with_poll
    ready = poller.poll()
  File "<string>", line 1, in <module>
  File "<string>", line 1, in <module>
  File "<string>", line 1, in <genexpr>