Welcome, guest | Sign In | My Account | Store | Cart

Simple module that allows you to explore deadlocks in multi threaded programs.

Python, 101 lines
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
"""Stack tracer for multi-threaded applications.


Usage:

import stacktracer
stacktracer.start_trace("trace.html",interval=5,auto=True) # Set auto flag to always update file!
....
stacktracer.stop_trace()
"""



import sys
import traceback
from pygments import highlight
from pygments.lexers import PythonLexer
from pygments.formatters import HtmlFormatter
 
 # Taken from http://bzimmer.ziclix.com/2008/12/17/python-thread-dumps/
 
def stacktraces():
    code = []
    for threadId, stack in sys._current_frames().items():
        code.append("\n# ThreadID: %s" % threadId)
        for filename, lineno, name, line in traceback.extract_stack(stack):
            code.append('File: "%s", line %d, in %s' % (filename, lineno, name))
            if line:
                code.append("  %s" % (line.strip()))
 
    return highlight("\n".join(code), PythonLexer(), HtmlFormatter(
      full=False,
      # style="native",
      noclasses=True,
    ))


# This part was made by nagylzs
import os
import time
import threading

class TraceDumper(threading.Thread):
    """Dump stack traces into a given file periodically."""
    def __init__(self,fpath,interval,auto):
        """
        @param fpath: File path to output HTML (stack trace file)
        @param auto: Set flag (True) to update trace continuously.
            Clear flag (False) to update only if file not exists.
            (Then delete the file to force update.)
        @param interval: In seconds: how often to update the trace file.
        """
        assert(interval>0.1)
        self.auto = auto
        self.interval = interval
        self.fpath = os.path.abspath(fpath)
        self.stop_requested = threading.Event()
        threading.Thread.__init__(self)
    
    def run(self):
        while not self.stop_requested.isSet():
            time.sleep(self.interval)
            if self.auto or not os.path.isfile(self.fpath):
                self.stacktraces()
    
    def stop(self):
        self.stop_requested.set()
        self.join()
        try:
            if os.path.isfile(self.fpath):
                os.unlink(self.fpath)
        except:
            pass
    
    def stacktraces(self):
        fout = file(self.fpath,"wb+")
        try:
            fout.write(stacktraces())
        finally:
            fout.close()


_tracer = None
def trace_start(fpath,interval=5,auto=True):
    """Start tracing into the given file."""
    global _tracer
    if _tracer is None:
        _tracer = TraceDumper(fpath,interval,auto)
        _tracer.setDaemon(True)
        _tracer.start()
    else:
        raise Exception("Already tracing to %s"%_tracer.fpath)

def trace_stop():
    """Stop tracing."""
    global _tracer
    if _tracer is None:
        raise Exception("Not tracing, cannot stop.")
    else:
        _trace.stop()
        _trace = None

I often write multi threaded programs (mostly networking) and sometimes I run into a deadlock problem. This code fragment was very useful for me to detect such problems.

Usage:

import stacktracer stacktracer.trace_start("trace.html")

Then you will have a "trace.html" file showing the stack traces of all threads in your program. By default, the file is updated automatically in every 5 seconds. Whenever the program runs into a deadlock, cannot be terminated by Ctrl+C (or not responding to external network events, signals etc), I just wait 5 seconds, look at the HTML file and I can tell what it is waiting for.

It is especially useful to detect deadlocks because you can see which thread is waiting for what. If you detect a circle, you have a deadlock.

5 comments

Tim Vazquez 12 years, 5 months ago  # | flag

Thank you soooo much for this gem! I have spent almost a week debugging a program and with this recipe, I was able to figure out where the problem was in 5 minutes.

Laszlo Nagy (author) 12 years, 5 months ago  # | flag

Ohh I'm glad finally actucally used it. :-)

Dima Tisnek 12 years ago  # | flag

This is indeed very useful. I'm running bits of this code from an embdded console as I type this :)

Dima Tisnek 12 years ago  # | flag

btw I submitted related bug to pycharm: http://youtrack.jetbrains.com/issue/PY-6237 if jetbrains implement threaded debugging properly perhaps this deadlock trick becomes unnecessary

Ken Shirriff 6 years, 11 months ago  # | flag

Very nice. Minor issue: docstring says to use start_trace/stop_trace, while the code has trace_start/trace_stop.