Welcome, guest | Sign In | My Account | Store | Cart

This context manager provides a convenient, Pythonic way to temporarily replace the file descriptors of stdout and stderr, redirecting to either os.devnull or files of your choosing. Swapping the C-level file descriptors is required when suppressing output from compiled extension modules, such as those built using F2PY. It functions equally well for pure-Python code. UPDATE: (see below).

Python, 108 lines
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
import os

class Silence:
    """Context manager which uses low-level file descriptors to suppress
    output to stdout/stderr, optionally redirecting to the named file(s).
    
    >>> import sys, numpy.f2py
    >>> # build a test fortran extension module with F2PY
    ...
    >>> with open('hellofortran.f', 'w') as f:
    ...     f.write('''\
    ...       integer function foo (n)
    ...           integer n
    ...           print *, "Hello from Fortran!"
    ...           print *, "n = ", n
    ...           foo = n
    ...       end
    ...       ''')
    ...
    >>> sys.argv = ['f2py', '-c', '-m', 'hellofortran', 'hellofortran.f']
    >>> with Silence():
    ...     # assuming this succeeds, since output is suppressed
    ...     numpy.f2py.main()
    ...
    >>> import hellofortran
    >>> foo = hellofortran.foo(1)
     Hello from Fortran!
     n =  1
    >>> print "Before silence"
    Before silence
    >>> with Silence(stdout='output.txt', mode='w'):
    ...     print "Hello from Python!"
    ...     bar = hellofortran.foo(2)
    ...     with Silence():
    ...         print "This will fall on deaf ears"
    ...         baz = hellofortran.foo(3)
    ...     print "Goodbye from Python!"
    ...
    ...
    >>> print "After silence"
    After silence
    >>> # ... do some other stuff ...
    ...
    >>> with Silence(stderr='output.txt', mode='a'):
    ...     # appending to existing file
    ...     print >> sys.stderr, "Hello from stderr"
    ...     print "Stdout redirected to os.devnull"
    ...
    ...
    >>> # check the redirected output
    ...
    >>> with open('output.txt', 'r') as f:
    ...     print "=== contents of 'output.txt' ==="
    ...     print f.read()
    ...     print "================================"
    ...
    === contents of 'output.txt' ===
    Hello from Python!
     Hello from Fortran!
     n =  2
    Goodbye from Python!
    Hello from stderr
    
    ================================
    >>> foo, bar, baz
    (1, 2, 3)
    >>>

    """
    def __init__(self, stdout=os.devnull, stderr=os.devnull, mode='w'):
        self.outfiles = stdout, stderr
        self.combine = (stdout == stderr)
        self.mode = mode
        
    def __enter__(self):
        import sys
        self.sys = sys
        # save previous stdout/stderr
        self.saved_streams = saved_streams = sys.__stdout__, sys.__stderr__
        self.fds = fds = [s.fileno() for s in saved_streams]
        self.saved_fds = map(os.dup, fds)
        # flush any pending output
        for s in saved_streams: s.flush()

        # open surrogate files
        if self.combine: 
            null_streams = [open(self.outfiles[0], self.mode, 0)] * 2
            if self.outfiles[0] != os.devnull:
                # disable buffering so output is merged immediately
                sys.stdout, sys.stderr = map(os.fdopen, fds, ['w']*2, [0]*2)
        else: null_streams = [open(f, self.mode, 0) for f in self.outfiles]
        self.null_fds = null_fds = [s.fileno() for s in null_streams]
        self.null_streams = null_streams
        
        # overwrite file objects and low-level file descriptors
        map(os.dup2, null_fds, fds)

    def __exit__(self, *args):
        sys = self.sys
        # flush any pending output
        for s in self.saved_streams: s.flush()
        # restore original streams and file descriptors
        map(os.dup2, self.saved_fds, self.fds)
        sys.stdout, sys.stderr = self.saved_streams
        # clean up
        for s in self.null_streams: s.close()
        for fd in self.saved_fds: os.close(fd)
        return False

Tools like F2PY are a wonderful way to wrap legacy code in beautiful Python bindings. However, it can be cumbersome to deal with suppressing/capturing an extension module's standard output, since the usual tricks like replacing sys.stdout are not effective. This context manager handles the process of swapping in and out file descriptors so the interfacing code can remain simple and readable. It sure beats digging into digging through the Fortran to rewrite every PRINT statement!

The same effect can be accomplished with a try...finally block, but a context manager is much more reusable. I've avoided hard-coding the file numbers (1 and 2 for stdout and stderr) to allow for nested Silence contexts (for instance, to capture all output except in a specific case, where output should be redirected differently).

While Silence will work on pure-Python code just as well, it's important to note that it is not capable of sending output to StringIO objects, since they do not implement a fileno() method. In fact, in this implementation it cannot use open file handles at all--a potential point of improvement.

I found the trick of using os.dup2 in this answer on Stack Overflow. A writeup on my blog includes a walkthrough with few additional examples.

UPDATE: I've revised the code slightly:

  • The class now accounts for some cases where the standard streams weren't being properly flushed on __enter__ and __exit__.
  • For more robust nesting capability, Silence now saves and restores the magic sys.__stdout__ and sys.__stderr__ streams.
  • When the stdout and stderr are to be combined, new unbuffered instances of sys.stdout and sys.stderr are opened so output is written in the proper sequence, instead of writing all stdout first (since it is flushed first). This will probably only work when capturing pure-Python output but it's nice nonetheless.
  • Some redundant lines have been removed from the code, and more detailed examples and documentation have been added to the docstring.

UPDATE 2: Added for fd in self.saved_fds: os.close(fd) to __exit__ to properly release file descriptors. (Thanks, Andrew!)

2 comments

Andrew 9 years, 10 months ago  # | flag

The current implementation leaks file descriptors. I discovered this when I ran some code that used this context manager in a long loop. It seems that because the saved file descriptors are never properly released the system limit can quite easily be reached. I patched my version of the code to include an extra line in the __exit__ method before the return line:

for fd in self.saved_fds: os.close(fd)

This seems to fix the problem.

Diego 5 years, 3 months ago  # | flag

While this code is over 4 years old, I feel it is incredibly useful. However, it does not seem to work with Python 3. It throws the error:

null_streams = [open(self.outfiles[0], self.mode, 0)] * 2
ValueError: can't have unbuffered text I/O

This seems to be an issue with Python (see for example 'this link').

I don't know that much about python to solve it myself so, is there any suggestion?

Best regards