This recipe shows how to execute a unix shell command and capture the output and error streams in python. By contrast, os.system() sends both streams directly to the shell.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45
import os, popen2, fcntl, FCNTL, select def makeNonBlocking(fd): fl = fcntl.fcntl(fd, FCNTL.F_GETFL) try: fcntl.fcntl(fd, FCNTL.F_SETFL, fl | FCNTL.O_NDELAY) except AttributeError: fcntl.fcntl(fd, FCNTL.F_SETFL, fl | FCNTL.FNDELAY) def getCommandOutput(command): child = popen2.Popen3(command, 1) # capture stdout and stderr from command child.tochild.close() # don't need to talk to child outfile = child.fromchild outfd = outfile.fileno() errfile = child.childerr errfd = errfile.fileno() makeNonBlocking(outfd) # don't deadlock! makeNonBlocking(errfd) outdata = errdata = '' outeof = erreof = 0 while 1: ready = select.select([outfd,errfd],,) # wait for input if outfd in ready: outchunk = outfile.read() if outchunk == '': outeof = 1 outdata = outdata + outchunk if errfd in ready: errchunk = errfile.read() if errchunk == '': erreof = 1 errdata = errdata + errchunk if outeof and erreof: break select.select(,,,.1) # give a little time for buffers to fill err = child.wait() if err != 0: raise RuntimeError, '%s failed w/ exit code %d\n%s' % (command, err, errdata) return outdata def getCommandOutput2(command): child = os.popen(command) data = child.read() err = child.close() if err: raise RuntimeError, '%s failed w/ exit code %d' % (command, err) return data
The presented getCommandOutput(command) function will execute a command and will return the command's output. If the command fails, an exception will be raised with the text captured from the command's stderr.
Most of complexity of this code is due to the difficulty of capturing both the output and error streams of the child process at the same time. Normal (blocking) read calls may deadlock if the child is trying to write to one stream and the parent is waiting for data on the other stream, so the streams must be set to non-blocking and select() must be used to wait for data on the streams.
Note: the second select call adds a 0.1 second sleep after each read. This (counterintuitively) allows the code to run much faster since it gives the child time to put more data in the buffer. Without this, the parent may try to read only a few chars at a time which can be very expensive.
If you only want to capture the output and don't mind the error stream going to the terminal, you can use the much simpler code presented in getCommandOutput2. If you want to suppress the error stream altogether, you can append '2>/dev/null' to the command (e.g. 'ls -1 2>/dev/null').
Note: Python (as of version 2.0) now includes the os.popen4 function which combines the output and error streams of the child process. However, the streams are combined in a potentially very messy way depending on how they are buffered in the child process.