Welcome, guest | Sign In | My Account | Store | Cart

This provides code to allow any python program which uses it to be interrupted at the current point, and communicated with via a normal python interactive console. This allows the locals, globals and associated program state to be investigated, as well as calling arbitrary functions and classes.

To use, a process should import the module, and call listen() at any point during startup. To interrupt this process, the script can be run directly, giving the process Id of the process to debug as the parameter.

Python, 136 lines
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
try: import readline  # For readline input support
except: pass

import sys, os, traceback, signal, codeop, cStringIO, cPickle, tempfile

def pipename(pid):
    """Return name of pipe to use"""
    return os.path.join(tempfile.gettempdir(), 'debug-%d' % pid)

class NamedPipe(object):
    def __init__(self, name, end=0, mode=0666):
        """Open a pair of pipes, name.in and name.out for communication
        with another process.  One process should pass 1 for end, and the
        other 0.  Data is marshalled with pickle."""
        self.in_name, self.out_name = name +'.in',  name +'.out',
        try: os.mkfifo(self.in_name,mode)
        except OSError: pass
        try: os.mkfifo(self.out_name,mode)
        except OSError: pass
        
        # NOTE: The order the ends are opened in is important - both ends
        # of pipe 1 must be opened before the second pipe can be opened.
        if end:
            self.inp = open(self.out_name,'r')
            self.out = open(self.in_name,'w')
        else:
            self.out = open(self.out_name,'w')
            self.inp = open(self.in_name,'r')
        self._open = True

    def is_open(self):
        return not (self.inp.closed or self.out.closed)
        
    def put(self,msg):
        if self.is_open():
            data = cPickle.dumps(msg,1)
            self.out.write("%d\n" % len(data))
            self.out.write(data)
            self.out.flush()
        else:
            raise Exception("Pipe closed")
        
    def get(self):
        txt=self.inp.readline()
        if not txt: 
            self.inp.close()
        else:
            l = int(txt)
            data=self.inp.read(l)
            if len(data) < l: self.inp.close()
            return cPickle.loads(data)  # Convert back to python object.
            
    def close(self):
        self.inp.close()
        self.out.close()
        try: os.remove(self.in_name)
        except OSError: pass
        try: os.remove(self.out_name)
        except OSError: pass

    def __del__(self):
        self.close()
        
def remote_debug(sig,frame):
    """Handler to allow process to be remotely debugged."""
    def _raiseEx(ex):
        """Raise specified exception in the remote process"""
        _raiseEx.ex = ex
    _raiseEx.ex = None
    
    try:
        # Provide some useful functions.
        locs = {'_raiseEx' : _raiseEx}
        locs.update(frame.f_locals)  # Unless shadowed.
        globs = frame.f_globals
        
        pid = os.getpid()  # Use pipe name based on pid
        pipe = NamedPipe(pipename(pid))
    
        old_stdout, old_stderr = sys.stdout, sys.stderr
        txt = ''
        pipe.put("Interrupting process at following point:\n" + 
               ''.join(traceback.format_stack(frame)) + ">>> ")
        
        try:
            while pipe.is_open() and _raiseEx.ex is None:
                line = pipe.get()
                if line is None: continue # EOF
                txt += line
                try:
                    code = codeop.compile_command(txt)
                    if code:
                        sys.stdout = cStringIO.StringIO()
                        sys.stderr = sys.stdout
                        exec code in globs,locs
                        txt = ''
                        pipe.put(sys.stdout.getvalue() + '>>> ')
                    else:
                        pipe.put('... ')
                except:
                    txt='' # May be syntax err.
                    sys.stdout = cStringIO.StringIO()
                    sys.stderr = sys.stdout
                    traceback.print_exc()
                    pipe.put(sys.stdout.getvalue() + '>>> ')
        finally:
            sys.stdout = old_stdout # Restore redirected output.
            sys.stderr = old_stderr
            pipe.close()

    except Exception:  # Don't allow debug exceptions to propogate to real program.
        traceback.print_exc()
        
    if _raiseEx.ex is not None: raise _raiseEx.ex
    
def debug_process(pid):
    """Interrupt a running process and debug it."""
    os.kill(pid, signal.SIGUSR1)  # Signal process.
    pipe = NamedPipe(pipename(pid), 1)
    try:
        while pipe.is_open():
            txt=raw_input(pipe.get()) + '\n'
            pipe.put(txt)
    except EOFError:
        pass # Exit.
    pipe.close()

def listen():
    signal.signal(signal.SIGUSR1, remote_debug) # Register for remote debugging.

if __name__=='__main__':
    if len(sys.argv) != 2:
        print "Error: Must provide process id to debug"
    else:
        pid = int(sys.argv[1])
        debug_process(pid)

This was written to deal with a problem I had with a long-running background process that would occassionally get stuck after a long time. This was difficult to reproduce when testing, so I wrote the above code so that when it did happen, I'd be able to break in and see what was going on.

Its implemented by first sending the process to debug a signal, and then opening a pair of pipes with the name /tmp/debug-pid.in and /tmp/debug-pid.out. The remote process, on receiving the signal, opens the other end of this pipe and these are used to pass code to be executed from the debugging process, and read responses from the debugee.

There are a few warnings to make:

  • There is absolutely no security here - pretty much anyone who can write to the pipe can gain full control of any process using this. Use only for developer environments, not live systems!

  • Sending a signal can interrupt whatever I/O or activity the process is currently doing, so you won't always just be able to detach again and let it run unchanged.

  • It uses signals to wake the process, so currently only works on unix-like systems that support this.

  • Untested with threads

5 comments

Brian McErlean (author) 13 years, 2 months ago  # | flag

f_locals is already giving a copy anyway - you can't change the local variables of any frame other than the one you're in (and only then because we're using exec). I'm using a copy of that just to give a place to inject functions into the namespace without shadowing actual locals we may want to inspect.

Daniel Lepage 13 years, 2 months ago  # | flag

Why do you make a copy of frame.f_locals instead of using it directly? Direct access would let you modify the state of the program you're debugging.

haridsv 11 years, 5 months ago  # | flag

Nice recipe. It would be even better if we could do a pdb.Pdb().set_trace(frame) at this point, but I am not sure if it is possible to interface pdb with the pipe.

Stefan Behnel 8 years, 5 months ago  # | flag

Might be a good idea to integrate this with IPython:

http://ipython.org/ipython-doc/stable/interactive/reference.html#embedding-ipython

zhubo198511 6 years, 2 months ago  # | flag

why need locs = {'_raiseEx' : _raiseEx} ?