ActiveState Code

Recipe 252498: A Generator That Helps Simplify Queue Consumers

My Queue usage typically involves a producer thread and a consumer thread. The producer calls queue.put(value) until it's done at which point it calls queue.put(sentinel). My consumers almost always look like this:

while True:     value = queue.get()     if value != sentinel:         # do something with value     else:         break

That logic can be abstracted away into a generator function that allows consumer code to look like this instead:

for value in iterQueue(queue, sentinel):     # do something with value

Python 
1
2
3
4
5
6
7
8
def iterQueue(queue, sentinel):
    """Iterate over the values in queue until sentinel is reached."""
    while True:
        value = queue.get()
        if value != sentinel:
            yield value
        else:
            return

Discussion

Generators functions provide a powerful tool for abstracting looping logic into separate functions. This can make code much easier to read.

Comments

  1. 1. At 2:27 a.m. on 21 may 2004, Paul Moore said:

    This is built in (but fairly obscure!). You don't need this - the built in iter() function allows you to do this:

    for val in iter(queue.get, sentinel):
    # process val

    Until I saw this recipe, I'd forgotten about it, though!

  2. 2. At 4:09 p.m. on 12 nov 2005, Ori Peleg said:

    Cool! I want to pass arguments to Queue.get as well, so I added:

    def iterQueue(queue, sentinel, **kwargs):
...
   value = queue.get(**kwargs)
...

    Now I can

    iterQueue(queue, sentinel, timeout=10)

    to my heart's desire :-)

Sign in to comment