This is an example of how functional ideas of infinite lists can be used in python to optimize memory usage of certain problems.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 | from itertools import *
from collections import deque
# First a naive approach. At each generation we pop the first element and append
# it to the back. This is highly memmory deficient.
def rotations(it):
""" rotations([0,1,2]) --> [[0, 1, 2], [1, 2, 0], [2, 0, 1]] """
l = list(it)
for i in range(len(l)):
yield iter(l)
l = l[1:]+[l[0]]
# A much better approach would seam to be using a deque, which rotates in O(1),
# However this does have the negative effect, that generating the next rotation
# before the current has been iterated through, will result in a RuntimeError
# because the deque has mutated during iteration.
def rotations(it):
""" rotations([0,1,2]) --> [[0, 1, 2], [1, 2, 0], [2, 0, 1]] """
l = deque(it)
for i in range(len(l)):
yield iter(l)
l.rotate()
# The trick is to use subsets of infinite lists. First we define the function tails,
# which is standard in many functal languages.
# Because of the way tee is implemented in itertools, the below implementation will
# use memory only propertional to the offset difference between the generated
# iterators.
def tails(it):
""" tails([1,2,3,4,5]) --> [[1,2,3,4,5], [2,3,4,5], [3,4,5], [4,5], [5], []] """
while True:
tail, it = tee(it)
yield tail
next(it)
# We can now define two new rotations functions.
# The first one is very similar to the above, but since we never keep all list
# elements, we need an extra length parameter.
def rotations(it, N):
""" rotations([0,1,2]) --> [[0, 1, 2], [1, 2, 0], [2, 0, 1]] """
return (islice(rot,N) for rot in islice(tails(cycle(it)),N))
# The above works fine for things like:
# >>> for rot in rotations(range(4), 4):
# ... print (list(rot))
# ...
# [0, 1, 2, 3]
# [1, 2, 3, 0]
# [2, 3, 0, 1]
# [3, 0, 1, 2]
#
# But that is not really where it shines, since the lists are iterated one after
# another, and so the tails memory usage becomes linear.
#
# In many cases tails and infinite lists lets us get away with an even simpler
# rotaions function:
def rotations(it, N):
""" rotations([0,1,2]) --> [[0, 1, 2], [1, 2, 0], [2, 0, 1]] """
return islice(tails(cycle(it)),N)
# This one works great for instances where we are topcut anyway:
# >>> for rot in rotations(range(4), 4):
# ... print (list(zip(range(4), rot)))
# ...
# [(0, 0), (1, 1), (2, 2), (3, 3)]
# [(0, 1), (1, 2), (2, 3), (3, 0)]
# [(0, 2), (1, 3), (2, 0), (3, 1)]
# [(0, 3), (1, 0), (2, 1), (3, 2)]
|
In particular the above suggests an optimized tails function to be placed in the itertools library.