keas.pbstate 0.1.1

An object database foundation based on Google's Protocol Buffers

Mixins

A class can mix in properties that access sub-messages. This is useful when subclassing (although subclassing should be avoided in general).

Here is a class that mixes the ContactPB properties and the AddressPB properties in a single class.

>>> class MixedContact(object):
...     __metaclass__ = ProtobufState
...     protobuf_type = ContactPB
...     protobuf_mixins = ('address',)

>>> mc = MixedContact()
>>> mc.line1 = u'180 Market St.'
>>> mc.line1
u'180 Market St.'
>>> mc.address.line1
u'180 Market St.'

Serialization

The ProtobufState also provides __getstate__ and __setstate__ methods, which Python uses for serialization purposes.

Try to serialize the object without providing all of the required fields.

>>> c.__getstate__()
Traceback (most recent call last):
...
EncodeError: Required field AddressPB.city is not set.

Finish filling out the required fields, then serialize.

>>> c.address.city = u'Toronto'
>>> c.create_time = 1001
>>> c.__getstate__()
('\x08\xe9\x07\x12\x08John Doe\x1a#\n\x10100 First Avenue\x1a\x07Toronto2\x06Canada', {})

Create a contact and copy its state from c.

>>> c_dup = Contact.__new__(Contact)
>>> c_dup.__setstate__(c.__getstate__())
>>> c_dup.name
u'John Doe'
>>> c_dup.address.country
u'Canada'

Create another contact, but this time provide no address information.

>>> c2 = Contact()
>>> c2.create_time = 1002
>>> c2.name = u'Mary Anne'
>>> c2.__getstate__()
('\x08\xea\x07\x12\tMary Anne', {})

Object References

Classes using the ProtobufState metaclass support references to arbitrary objects through the use of the 'protobuf_refs' attribute.

Our Contact class has a 'guardians' attribute that contains a list of references. The ProtobufState metaclass treats any message or sub-message with a _p_refid field as a reference.

Add a guardian to c2, but don't say who the guardian is yet.

>>> guardian_ref = c2.guardians.add()

Call protobuf_refs.set() to make guardian_ref refer to c.

>>> c2.protobuf_refs.set(guardian_ref, c)

Let's go over what happened. The set method generated a reference ID, then that ID was assigned to guardian_ref._p_refid, and the refid and target object were added to the internal state of the protobuf_refs instance. Any message with a _p_refid field is a reference. Every _p_refid field should be of type uint32.

Read the reference.

>>> c2.protobuf_refs.get(guardian_ref) is c
True

Verify the reference gets serialized correctly.

>>> data, targets = c2.__getstate__()
>>> targets[c2.guardians[0]._p_refid] is c
True

Delete the reference.

>>> c2.protobuf_refs.delete(guardian_ref)
>>> c2.protobuf_refs.get(guardian_ref, 'gone')
'gone'

Verify the reference is no longer contained in the serialized state.

>>> data, targets = c2.__getstate__()
>>> len(targets)
0

Features Designed for ZODB

This package provides enough features for storing ProtobufState objects in ZODB, although without the keas.pbpersist package, the stored objects will still be wrapped inside a Python pickle, making them hard for languages other than Python to access. See the keas.pbpersist package for a straightforward method of storing ProtobufState objects in ZODB without using Python pickles.

In ZODB, objects have a _p_changed attribute to indicate when they are dirty. The ProtobufState metaclass causes instances to modify the _p_changed attribute if it exists; it is set to True whenever the message changes.

Here is a PersistentContact class, which has a _p_changed attribute. (We also define a FakePersistent base class in order to avoid depending on ZODB.)

>>> class FakePersistent(object):
...     __slots__ = ('_changed',)
...     def _get_changed(self):
...         return getattr(self, '_changed', False)
...     def _set_changed(self, value):
...         self._changed = value
...         if not value:
...             # reset the _cache_byte_size_dirty flags
...             self.protobuf.ByteSize()
...     _p_changed = property(_get_changed, _set_changed)
...
>>> class PersistentContact(FakePersistent):
...     __metaclass__ = ProtobufState
...     protobuf_type = ContactPB
...

>>> c3 = PersistentContact()
>>> c3._p_changed
False
>>> c3.create_time = 1003
>>> c3.name = u'Snoopy'
>>> c3._p_changed = False

Reading an attribute does not set _p_changed.

>>> c3.name
u'Snoopy'
>>> c3._p_changed
False

Writing an attribute sets _p_changed.

>>> c3.name = u'Woodstock'
>>> c3._p_changed
True

Adding to a repeated element sets _p_changed.

>>> c3._p_changed = False
>>> c3._p_changed
False
>>> c3.guardians.add()
<keas.pbstate.testclasses_pb2.Ref object at ...>
>>> c3._p_changed
True
>>> del c3.guardians[0]

A copy of c3 should initially have _p_changed = False; setting an attribute should set _p_changed to true.

>>> c4 = PersistentContact.__new__(PersistentContact)
>>> c4.__setstate__(c3.__getstate__())
>>> c4._p_changed
False
>>> c4.name = u'Linus'
>>> c4._p_changed
True

The tuple returned by __getstate__ is actually a subclass of tuple. The StateTuple suggests to the ZODB serializer that it can save the state in a different format than the default pickle format.

>>> type(c.__getstate__())
<class 'keas.pbstate.state.StateTuple'>
>>> c.__getstate__().serial_format
'protobuf'

Edge Cases

Synthesize a refid hash collision. This is a rare occurrence, but this package should handle it transparently as long as no single object holds more than about one billion (2**30) references to other objects.

First make a reference:

>>> guardian_ref = c2.guardians.add()
>>> c2.protobuf_refs.set(guardian_ref, c)

Covertly change the target of that reference:

>>> c2.protobuf_refs._targets[guardian_ref._p_refid] = mc

Add a new reference to the original target. The first generated refid will collide, but he protobuf_refs should should choose a different refid automatically.

>>> guardian2_ref = c2.guardians.add()
>>> c2.protobuf_refs.set(guardian2_ref, c)
>>> guardian_ref._p_refid == guardian2_ref._p_refid
False

Exception Conditions

Deleting message attributes is not allowed.

>>> del c.name
Traceback (most recent call last):
...
AttributeError: can't delete attribute
>>> del mc.line1
Traceback (most recent call last):
...
AttributeError: can't delete attribute

Mixin names are checked.

>>> class MixedUpContact(object):
...     __metaclass__ = ProtobufState
...     protobuf_type = ContactPB
...     protobuf_mixins = ('bogus',)
Traceback (most recent call last):
...
AttributeError: Field 'bogus' not defined for protobuf type <...>

Create a broken reference by setting a reference using the wrong protobuf_refs. To prevent this condition, the protobuf_refs attribute and the first argument to protobuf_refs.set() must descend from the same containing object.

>>> c.guardians.add()
<keas.pbstate.testclasses_pb2.Ref object at ...>
>>> c2.protobuf_refs.set(c.guardians[0], c)
>>> c.__getstate__()
Traceback (most recent call last):
...
KeyError: 'Object contains broken references: <Contact object at ...>'
>>> del c.guardians[0]

Don't omit the protobuf_type attribute.

>>> class FailedContact(object):
...     __metaclass__ = ProtobufState
Traceback (most recent call last):
...
TypeError: Class ...FailedContact needs a protobuf_type attribute