Welcome, guest | Sign In | My Account | Store | Cart

Decorator-based implementation of PEP 380 (yield from). This is the simple version (no special handling of nested "yield _from"s).

Python, 151 lines
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
from compat.functools import wraps as _wraps
from sys import exc_info as _exc_info

class _from(object):
    def __init__(self, EXPR):
        self.iterator = iter(EXPR)

def supergenerator(genfunct):
    """Implements PEP 380. Use as:

        @supergenerator
        def genfunct(*args):
            try:
                sent1 = (yield val1)
                ,,,
                retval = yield _from(iterator)
                ...
            except Exception, e:
                # caller did generator.throw
                pass
            finally:
                pass             # closing
    """

    @_wraps(genfunct)
    def wrapper(*args, **kwargs):
        gen = genfunct(*args, **kwargs)

        try:

            # if first poll of gen raises StopIteration
            # or any other Exception, we propagate
            item = gen.next()

            # OUTER loop
            while True:

                # yield _from(EXPR)
                # semantics based on PEP 380, Revised**12, 19 April
                if isinstance(item, _from):
                    _i = item.iterator
                    try:
                        # first poll of the subiterator
                        _y = _i.next()
                    except StopIteration, _e:
                        # subiterator exhausted on first poll
                        # extract return value
                        _r = _e.args if _e.args else (None,)
                    else:

                        # INNER loop
                        while True:
                            try:
                                # yield what the subiterator did
                                _s = (yield _y)

                            except GeneratorExit, _e:
                                # close the subiterator if possible
                                try:
                                    _close = _i.close
                                except AttributeError:
                                    pass
                                else:
                                    _close()
                                # finally clause will gen.close()
                                raise _e

                            except BaseException:
                                # caller did wrapper.throw
                                _x = _exc_info()
                                # throw to the subiterator if possible
                                try:
                                    _throw = _i.throw
                                except AttributeError:
                                    # doesn't attempt to close _i?
                                    # if gen raises StopIteration
                                    # or any other Exception, we propagate
                                    item = gen.throw(*_x)
                                    _r = None
                                    # fall through to INTERSECTION A
                                    # then to OUTER loop
                                    pass
                                else:
                                    try:
                                        _y = _throw(*_x)
                                    except StopIteration, _e:
                                        _r = _e.args if _e.args else (None,)
                                        # fall through to INTERSECTION A
                                        # then to INTERSECTION B
                                        pass
                                    else:
                                        # restart INNER loop
                                        continue

                                # INTERSECTION A
                                # restart OUTER loop or proceed to B?
                                if _r is None: break

                            else:
                                try:
                                    # re-poll the subiterator
                                    if _s is None:
                                        _y = _i.next()
                                    else:
                                        _y = _i.send(_s)
                                except StopIteration, _e:
                                    # subiterator is exhausted
                                    # extract return value
                                    _r = _e.args if _e.args else (None,)
                                    # fall through to INTERSECTION B
                                    pass
                                else:
                                    # restart INNER loop
                                    continue

                            # INTERSECTION B
                            # done yielding from subiterator
                            # send retvalue to gen

                            # if gen raises StopIteration
                            # or any other Exception, we propagate
                            item = gen.send(_r[0])

                            # restart OUTER loop
                            break

                # traditional yield from gen
                else:
                    try:
                        sent = (yield item)
                    except Exception:
                        # caller did wrapper.throw
                        _x = _exc_info()
                        # if gen raises StopIteration
                        # or any other Exception, we propagate
                        item = gen.throw(*_x)
                    else:
                        # if gen raises StopIteration
                        # or any other Exception, we propagate
                        item = gen.send(sent)

                # end of OUTER loop, restart it
                pass

        finally:
            # gen raised Exception
            # or caller did wrapper.close()
            # or wrapper was garbage collected
            gen.close()

    return wrapper

http://www.python.org/dev/peps/pep-0380/ proposes new syntax ("yield from") for generators to delegate control to a "subgenerator" (really to any iterator). Any send/next/throw/close calls to the delegating generator are forwarded to the delegee, until the delegee is exhausted.

This is being considered for inclusion in Python 2.7, but I wanted a way to play around with the design pattern now (and in case the PEP isn't soon accepted, and on older Python installations regardless of what happens with future versions of Python).

So I came up with this decorator-based solution. The "supergenerator" decorator wraps the delegating generator with a control handler that takes care of directing send/next/throw/close calls to the delegator or delegee, as appropriate.

Sample usage is described in the decorator's docstring.

Delegees can pass return values to the "yield _from" call by "raise StopIteration(retval)". Example:

@supergenerator
def gen1funct():
    for i in xrange(3):
        sent = yield i
        print "sent1: %r" % (sent,)
    delegee = gen2funct()
    retval = yield _from(delegee)
    print "return value: %r"  % (retval,)

def gen2funct():
    for i in xrange(3,6):
        sent = yield i
        print "sent2: %r" % (sent,)
    raise StopIteration(100)

gen=gen1funct()
try:
    i = gen.next()
    while True:
        print "yielded: %r" % (i,)
        i = gen.send(i*10)
except StopIteration:
    print "yielded: %r" % (i,)
    print "stopped"

Result:

yielded: 0
sent1: 0
yielded: 1
sent1: 10
yielded: 2
sent1: 20
yielded: 3
sent2: 30
yielded: 4
sent2: 40
yielded: 5
sent2: 50
return value: 100
yielded: 5
stopped

This is the "simple" version of my implementation. It doesn't do any special handling of nested "yield _from"s. As a result, there's an extra level of generator-switching overhead for each delegating generator. If there are many levels of nesting, this will add up.

However, I also have an "optimized" implementation which automatically keeps track of nested "yield _from"s, and delegates directly to the most deeply-nested delegees. See http://code.activestate.com/recipes/576728/.

The flow control in this simple implementation is less straightforward than it might be. I did that in order to maximize the similarity between the simple and optimized versions.

1 comment

hannanaha 7 years, 7 months ago  # | flag

Thank you for this recipe. I tried doing this with context managers, but the way of the decorator is superior.

I think there is a small bug in this implementation. If the original generator passed as EXPR is an empty generator (immediately raises StopIteration on first next() call), then the Outer loop will go on forever. This can be easily fixed by adding:

item = gen.send(_r[0])

after line 51:

_r = _e.args if _e.args else (None,)

Though there probably is a more elegant fix.