Simple module that allows you to explore deadlocks in multi threaded programs.
Download
Copy to clipboard1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 | """Stack tracer for multi-threaded applications.
Usage:
import stacktracer
stacktracer.start_trace("trace.html",interval=5,auto=True) # Set auto flag to always update file!
....
stacktracer.stop_trace()
"""
import sys
import traceback
from pygments import highlight
from pygments.lexers import PythonLexer
from pygments.formatters import HtmlFormatter
# Taken from http://bzimmer.ziclix.com/2008/12/17/python-thread-dumps/
def stacktraces():
code = []
for threadId, stack in sys._current_frames().items():
code.append("\n# ThreadID: %s" % threadId)
for filename, lineno, name, line in traceback.extract_stack(stack):
code.append('File: "%s", line %d, in %s' % (filename, lineno, name))
if line:
code.append(" %s" % (line.strip()))
return highlight("\n".join(code), PythonLexer(), HtmlFormatter(
full=False,
# style="native",
noclasses=True,
))
# This part was made by nagylzs
import os
import time
import threading
class TraceDumper(threading.Thread):
"""Dump stack traces into a given file periodically."""
def __init__(self,fpath,interval,auto):
"""
@param fpath: File path to output HTML (stack trace file)
@param auto: Set flag (True) to update trace continuously.
Clear flag (False) to update only if file not exists.
(Then delete the file to force update.)
@param interval: In seconds: how often to update the trace file.
"""
assert(interval>0.1)
self.auto = auto
self.interval = interval
self.fpath = os.path.abspath(fpath)
self.stop_requested = threading.Event()
threading.Thread.__init__(self)
def run(self):
while not self.stop_requested.isSet():
time.sleep(self.interval)
if self.auto or not os.path.isfile(self.fpath):
self.stacktraces()
def stop(self):
self.stop_requested.set()
self.join()
try:
if os.path.isfile(self.fpath):
os.unlink(self.fpath)
except:
pass
def stacktraces(self):
fout = file(self.fpath,"wb+")
try:
fout.write(stacktraces())
finally:
fout.close()
_tracer = None
def trace_start(fpath,interval=5,auto=True):
"""Start tracing into the given file."""
global _tracer
if _tracer is None:
_tracer = TraceDumper(fpath,interval,auto)
_tracer.setDaemon(True)
_tracer.start()
else:
raise Exception("Already tracing to %s"%_tracer.fpath)
def trace_stop():
"""Stop tracing."""
global _tracer
if _tracer is None:
raise Exception("Not tracing, cannot stop.")
else:
_trace.stop()
_trace = None
|
I often write multi threaded programs (mostly networking) and sometimes I run into a deadlock problem. This code fragment was very useful for me to detect such problems.
Usage:
import stacktracer stacktracer.trace_start("trace.html")
Then you will have a "trace.html" file showing the stack traces of all threads in your program. By default, the file is updated automatically in every 5 seconds. Whenever the program runs into a deadlock, cannot be terminated by Ctrl+C (or not responding to external network events, signals etc), I just wait 5 seconds, look at the HTML file and I can tell what it is waiting for.
It is especially useful to detect deadlocks because you can see which thread is waiting for what. If you detect a circle, you have a deadlock.
Thank you soooo much for this gem! I have spent almost a week debugging a program and with this recipe, I was able to figure out where the problem was in 5 minutes.
Ohh I'm glad finally actucally used it. :-)
This is indeed very useful. I'm running bits of this code from an embdded console as I type this :)
btw I submitted related bug to pycharm: http://youtrack.jetbrains.com/issue/PY-6237 if jetbrains implement threaded debugging properly perhaps this deadlock trick becomes unnecessary
Very nice. Minor issue: docstring says to use start_trace/stop_trace, while the code has trace_start/trace_stop.