Welcome, guest | Sign In | My Account | Store | Cart

A generator function which takes a file object (assumed to be some sort of pipe or socket, open for reading), and yields lines from it without blocking. If there is no input available, it will yield an endless stream of empty strings until input becomes available again; caller is responsible for not going into a busy loop. (Newlines are normalized but not stripped, so if there is actually a blank line in the input, the value yielded will be '\n'.) The intended use case is a thread which must respond promptly to input from a pipe, and also something else which cannot be fed to select (e.g. a queue.Queue). Note that the file object is ignored except for its fileno.

Only tested on Unix. Only tested on 3.4; ought to work with any python that has bytearray, locale.getpreferredencoding, and fcntl.

Python, 43 lines
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
def nonblocking_readlines(f):
    """Generator which yields lines from F (a file object, used only for
       its fileno()) without blocking.  If there is no data, you get an
       endless stream of empty strings until there is data again (caller
       is expected to sleep for a while).
       Newlines are normalized to the Unix standard.
    """

    fd = f.fileno()
    fl = fcntl.fcntl(fd, fcntl.F_GETFL)
    fcntl.fcntl(fd, fcntl.F_SETFL, fl | os.O_NONBLOCK)
    enc = locale.getpreferredencoding(False)

    buf = bytearray()
    while True:
        try:
            block = os.read(fd, 8192)
        except BlockingIOError:
            yield ""
            continue

        if not block:
            if buf:
                yield buf.decode(enc)
                buf.clear()
            break

        buf.extend(block)

        while True:
            r = buf.find(b'\r')
            n = buf.find(b'\n')
            if r == -1 and n == -1: break

            if r == -1 or r > n:
                yield buf[:(n+1)].decode(enc)
                buf = buf[(n+1):]
            elif n == -1 or n > r:
                yield buf[:r].decode(enc) + '\n'
                if n == r+1:
                    buf = buf[(r+2):]
                else:
                    buf = buf[(r+1):]