blocked-table 1.08 (beta)

Efficient on-disk sorted table format.

blocked-table

This package provides an efficient, 2-level indexed table in pure Python. The index structure allows for billions of keys to be added while retaining low memory usage.

Tables are expected to contain string keys; values are encoded using the 'json' package and optionally compressed using gzip or snappy compression.

usage

Building a table:

import blocked_table
tb = blocked_table.TableBuilder('/tmp/mytable')
for i in range(1000000):
  tb.add(str(i), 'Value %d' % i)
del tb

Reading a table:

t = blocked_table.open('/tmp/mytable')
for k, v in t.iteritems(): print k
v = t.lookup('this is a key')

Utility functions:

d = dict([('key %d' % i, 'value %d' % i) for i in range(1000000)])
dictionary_to_table(d, '/tmp/mytable')