The 3 Python array packages are not able to create multidimensional arrays from iterators. This recipe allows you to do that.
Download
Copy to clipboard1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 | # Public domain.
import numpy
def array_from_iter(it, typecode):
try:
ans = numpy.array([it.next()], typecode)
except StopIteration:
raise ValueError('iterator contains 0 items')
shape0 = ans.shape[1:]
for (i, x) in enumerate(it):
ans.resize((i+2,)+shape0)
ans[i+1] = x
return ans
def f():
yield [1,2,3]
yield [4,5,6]
print array_from_iter(f(), 'b') # Prints [[1,2,3],[4,5,6]]
# Alternatively, the same code above works with numarray and Numeric.
# Just replace 'numpy' with 'numarray' or 'Numeric'.
|
In some cases, you may have an iterator that returns a large number of lists, so that calling list() on your iterator will cause the Python interpreter to exceed the available physical memory. If your iterator of lists is "rectangular" (in the sense of a rectangular multidimensional array) and uses homogeneous types, then the array might fit in memory if constructed by Numeric.array, numarray.array, or numpy.array.
The three array packages do not support initialization from an iterator in general: for example, numarray.array(iter([[1,2,3],[4,5,6]])) fails. The above recipe will produce the correct result: array_from_iter(iter([[1,2,3],[4,5,6]]),'b') will produce the 2x3 array [[1,2,3],[4,5,6]].
Note that the iterator passed to array_from_iter() must have at least 1 element: if it has 0 elements, then ValueError is raised.
