Decorator-based implementation of PEP 380 (yield from). This is the simple version (no special handling of nested "yield _from"s).
Download
Copy to clipboard1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 | from compat.functools import wraps as _wraps
from sys import exc_info as _exc_info
class _from(object):
def __init__(self, EXPR):
self.iterator = iter(EXPR)
def supergenerator(genfunct):
"""Implements PEP 380. Use as:
@supergenerator
def genfunct(*args):
try:
sent1 = (yield val1)
,,,
retval = yield _from(iterator)
...
except Exception, e:
# caller did generator.throw
pass
finally:
pass # closing
"""
@_wraps(genfunct)
def wrapper(*args, **kwargs):
gen = genfunct(*args, **kwargs)
try:
# if first poll of gen raises StopIteration
# or any other Exception, we propagate
item = gen.next()
# OUTER loop
while True:
# yield _from(EXPR)
# semantics based on PEP 380, Revised**12, 19 April
if isinstance(item, _from):
_i = item.iterator
try:
# first poll of the subiterator
_y = _i.next()
except StopIteration, _e:
# subiterator exhausted on first poll
# extract return value
_r = _e.args if _e.args else (None,)
else:
# INNER loop
while True:
try:
# yield what the subiterator did
_s = (yield _y)
except GeneratorExit, _e:
# close the subiterator if possible
try:
_close = _i.close
except AttributeError:
pass
else:
_close()
# finally clause will gen.close()
raise _e
except BaseException:
# caller did wrapper.throw
_x = _exc_info()
# throw to the subiterator if possible
try:
_throw = _i.throw
except AttributeError:
# doesn't attempt to close _i?
# if gen raises StopIteration
# or any other Exception, we propagate
item = gen.throw(*_x)
_r = None
# fall through to INTERSECTION A
# then to OUTER loop
pass
else:
try:
_y = _throw(*_x)
except StopIteration, _e:
_r = _e.args if _e.args else (None,)
# fall through to INTERSECTION A
# then to INTERSECTION B
pass
else:
# restart INNER loop
continue
# INTERSECTION A
# restart OUTER loop or proceed to B?
if _r is None: break
else:
try:
# re-poll the subiterator
if _s is None:
_y = _i.next()
else:
_y = _i.send(_s)
except StopIteration, _e:
# subiterator is exhausted
# extract return value
_r = _e.args if _e.args else (None,)
# fall through to INTERSECTION B
pass
else:
# restart INNER loop
continue
# INTERSECTION B
# done yielding from subiterator
# send retvalue to gen
# if gen raises StopIteration
# or any other Exception, we propagate
item = gen.send(_r[0])
# restart OUTER loop
break
# traditional yield from gen
else:
try:
sent = (yield item)
except Exception:
# caller did wrapper.throw
_x = _exc_info()
# if gen raises StopIteration
# or any other Exception, we propagate
item = gen.throw(*_x)
else:
# if gen raises StopIteration
# or any other Exception, we propagate
item = gen.send(sent)
# end of OUTER loop, restart it
pass
finally:
# gen raised Exception
# or caller did wrapper.close()
# or wrapper was garbage collected
gen.close()
return wrapper
|
http://www.python.org/dev/peps/pep-0380/ proposes new syntax ("yield from") for generators to delegate control to a "subgenerator" (really to any iterator). Any send/next/throw/close calls to the delegating generator are forwarded to the delegee, until the delegee is exhausted.
This is being considered for inclusion in Python 2.7, but I wanted a way to play around with the design pattern now (and in case the PEP isn't soon accepted, and on older Python installations regardless of what happens with future versions of Python).
So I came up with this decorator-based solution. The "supergenerator" decorator wraps the delegating generator with a control handler that takes care of directing send/next/throw/close calls to the delegator or delegee, as appropriate.
Sample usage is described in the decorator's docstring.
Delegees can pass return values to the "yield _from" call by "raise StopIteration(retval)". Example:
@supergenerator
def gen1funct():
for i in xrange(3):
sent = yield i
print "sent1: %r" % (sent,)
delegee = gen2funct()
retval = yield _from(delegee)
print "return value: %r" % (retval,)
def gen2funct():
for i in xrange(3,6):
sent = yield i
print "sent2: %r" % (sent,)
raise StopIteration(100)
gen=gen1funct()
try:
i = gen.next()
while True:
print "yielded: %r" % (i,)
i = gen.send(i*10)
except StopIteration:
print "yielded: %r" % (i,)
print "stopped"
Result:
yielded: 0
sent1: 0
yielded: 1
sent1: 10
yielded: 2
sent1: 20
yielded: 3
sent2: 30
yielded: 4
sent2: 40
yielded: 5
sent2: 50
return value: 100
yielded: 5
stopped
This is the "simple" version of my implementation. It doesn't do any special handling of nested "yield _from"s. As a result, there's an extra level of generator-switching overhead for each delegating generator. If there are many levels of nesting, this will add up.
However, I also have an "optimized" implementation which automatically keeps track of nested "yield _from"s, and delegates directly to the most deeply-nested delegees. See http://code.activestate.com/recipes/576728/.
The flow control in this simple implementation is less straightforward than it might be. I did that in order to maximize the similarity between the simple and optimized versions.
Thank you for this recipe. I tried doing this with context managers, but the way of the decorator is superior.
I think there is a small bug in this implementation. If the original generator passed as EXPR is an empty generator (immediately raises StopIteration on first next() call), then the Outer loop will go on forever. This can be easily fixed by adding:
after line 51:
Though there probably is a more elegant fix.