Welcome, guest | Sign In | My Account | Store | Cart

A small function that walks over pretty much any Python object and yields the objects contained within (if any) along with the path to reach them. I wrote it and am using it to validate a deserialized data-structure, but you can probably use it for many things.

Python, 59 lines
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
from collections import Mapping, Set, Sequence 

# dual python 2/3 compatability, inspired by the "six" library
string_types = (str, unicode) if str is bytes else (str, bytes)
iteritems = lambda mapping: getattr(mapping, 'iteritems', mapping.items)()

def objwalk(obj, path=(), memo=None):
    if memo is None:
        memo = set()
    iterator = None
    if isinstance(obj, Mapping):
        iterator = iteritems
    elif isinstance(obj, (Sequence, Set)) and not isinstance(obj, string_types):
        iterator = enumerate
    if iterator:
        if id(obj) not in memo:
            memo.add(id(obj))
            for path_component, value in iterator(obj):
                for result in objwalk(value, path + (path_component,), memo):
                    yield result
            memo.remove(id(obj))
    else:
        yield path, obj

# optional test code from here on
import unittest

class TestObjwalk(unittest.TestCase):
    def assertObjwalk(self, object_to_walk, *expected_results):
        return self.assertEqual(tuple(sorted(expected_results)), tuple(sorted(objwalk(object_to_walk))))
    def test_empty_containers(self):
        self.assertObjwalk({})
        self.assertObjwalk([])
    def test_single_objects(self):
        for obj in (None, 42, True, "spam"):
            self.assertObjwalk(obj, ((), obj))
    def test_plain_containers(self):
        self.assertObjwalk([1, True, "spam"], ((0,), 1), ((1,), True), ((2,), "spam"))
        self.assertObjwalk({None: 'eggs', 'bacon': 'ham', 'spam': 1},
                           ((None,), 'eggs'), (('spam',), 1), (('bacon',), 'ham'))
        # sets are unordered, so we dont test the path, only that no object is missing
        self.assertEqual(set(obj for path, obj in objwalk(set((1,2,3)))), set((1,2,3)))
    def test_nested_containers(self):
        self.assertObjwalk([1, [2, [3, 4]]],
                           ((0,), 1), ((1,0), 2), ((1, 1, 0), 3), ((1, 1, 1), 4))
        self.assertObjwalk({1: {2: {3: 'spam'}}},
                           ((1,2,3), 'spam'))
    def test_repeating_containers(self):
        repeated = (1,2)
        self.assertObjwalk([repeated, repeated],
                           ((0, 0), 1), ((0, 1), 2), ((1, 0), 1), ((1, 1), 2))
    def test_recursive_containers(self):
        recursive = [1, 2]
        recursive.append(recursive)
        recursive.append(3)
        self.assertObjwalk(recursive, ((0,), 1), ((1,), 2), ((3,), 3))

if __name__ == '__main__':
    unittest.main()

Example use: In one configuration mechanism I implemented, there exists an UNCONFIGURED sentinel that marks configuration items that are required but aren't set. If a loaded configuration has unconfigured items in it, it's considered invalid. I used objwalk to recursively traverse the configuration object and possibly raise an exception telling where in the configuration were unconfigured elements found.

4 comments

zetah 12 years, 4 months ago  # | flag

This is great I got lucky and found it with Google and it does what I needed - iterate over nasty object

Thanks for making it

julien tayon 12 years, 2 months ago  # | flag

I pretty much inspired some of my code from your code.

You are duely quoted as the source of the code (the code is python public licence or same).

It helps doing fun stuff, like watching for a dict in a dict (of dict ( of dict ... ) ) ) matching criterion. So thanks.

https://github.com/jul/ADictAdd_iction/blob/master/vector_dict/VectorDict.py#L156 (I know it is the old version, I will use the better one if I am allowed to).

Please contact me if I violated any IP stuff, and I would gladly conform to any of your demand even getting rid of it, if you consider this solution.

Yaniv Aknin (author) 12 years, 2 months ago  # | flag

Nah, use the IP as you wish, that's the idea.

julien tayon 12 years, 1 month ago  # | flag

cool. Btw, I was told on #python to wait an endless time for yield from PEP implementation, and that this kind of code is bad usage and should not be promoted.

(I use your function to flatten nested generators).

I did not reply, but I had the idea now and could not wait, and it fitted my needs. :)

Practicality beats purity :)

Thanks.

Created by Yaniv Aknin on Tue, 13 Dec 2011 (PSF)
Python recipes (4591)
Yaniv Aknin's recipes (1)

Required Modules

  • (none specified)

Other Information and Tasks