Latest recipes by George Sakkis

Simultaneous topdown and bottomup variant of os.walk() (alt. title: "Delete .pyc files and empty directories recursively") (Python)

2010-04-21T20:16:09-07:00

Python recipe 577206 by George Sakkis (os_walk).

The standard lib os.walk() function provides a topdown parameter that determines whether entries are yielded in a top-down or a bottom-up order. Sometimes though you may want each directory yielded twice; once before any of its children directories (and recursively their descendants) are yielded and once after they are all yielded. The walk2() function below does this by yielding 4-tuples; the first 3 elements are the same yielded by os.walk() and the 4th is True the first time (topdown) and False the second (bottomup).

An example is deleting all .pyc files and empty directories under some root dir, but excluding specific directories (e.g. VCS specific dirs). The exclusion check should be done topdown (we don't want to descend into any directory that must be excluded) but the check for empty directories has to be done bottom up, since a directory containing only .pyc files will be non-empty initially but empty after removing the files.

Browser history data structure (Python)

2009-12-25T17:26:57-08:00

Python recipe 576991 by George Sakkis (datastructures, datastuctures). Revision 6.

The BrowserHistory class encapsulates the history of moving from location to location, as in Web browsing context; the recipe is not restricted to Web browsing though. See docstrings for more details and usage.

The current implementation requires Python 2.6.

Context manager for restoring a value (Python)

2010-01-09T07:14:06-08:00

Python recipe 576977 by George Sakkis (contextlib, context_manager). Revision 8.

Often one wants to rebind a name or modify a mutable object, perform a bunch of actions and finally restore the name/object to its original state. An example is redirecting stdout/stderr temporarily (http://www.diveintopython.org/scripts_and_streams/stdin_stdout_stderr.html). The restoring context manager shown below simplifies this pattern::

import sys

# prints in console
print "hello world!"

with restoring('sys.stdout'):
    with open('hello.txt', 'w') as sys.stdout:
        # prints in file
        print "hello world!"

# prints in console again
print "hello world!"

Handling ties for top largest/smallest elements (Python)

2009-04-07T18:57:35-07:00

Python recipe 576712 by George Sakkis (heapq, largest, smallest, top). Revision 8.

The heapq module provides efficient functions for getting the top-N smallest and largest elements of an iterable. A caveat of these functions is that if there are ties (i.e. equal elements with respect to the comparison key), some elements may end up in the returned top-N list while some equal others may not:

>>> nsmallest(3, [4,3,-2,-3,2], key=abs)
[-2, 2, 3]

Although 3 and -3 are equal with respect to the key function, only one of them is chosen to be returned. For several applications, an all-or-nothing approach with respect to ties is preferable or even required.

A new optional boolean parameter 'ties' is proposed to accomodate these cases. If ties=True and the iterable contains more than N elements, the length of the returned sorted list can be lower than N if not all ties at the last position can fit in the list:

>>> nsmallest(3, [4,3,-2,-3,2], key=abs, ties=True)
[-2, 2]

Yet another Null object (Python)

2008-11-11T15:24:20-08:00

Python recipe 576562 by George Sakkis (design_pattern, dummy, null).

This recipe builds on two previously posted recipes for a null or dummy object by modifying a few methods (e.g. as in SQL, Null == Null is Null, not True), supporting most (all?) special methods (e.g. int(Null)) and providing correct pickling/unpickling.

Records (Python)

2008-11-04T06:52:42-08:00

Python recipe 576555 by George Sakkis (datastructures, namedtuple, record).

This is a recipe similar in functionality and exec-style optimized implementation to the very well received namedtuple (http://code.activestate.com/recipes/500261/) that was included in Python 2.6. The main difference is that records, unlike named tuples, are mutable. In addition, fields can have a default value. Instead of subclassing tuple or list, the implementation create a regular class with __slots__.

Introspecting call arguments (Python)

2009-05-10T11:32:18-07:00

Python recipe 551779 by George Sakkis (algorithms). Revision 2.

This recipe implements in pure Python the algorithm used by the interpreter for binding the values passed as parameters in a function call to the formal arguments of the function.

This is useful for decorators that want to take into account this binding when wrapping a function (e.g. for type checking).

Edit (2008/5/10): Added a second returned value, missing_args: a tuple of the formal parameters whose value was not provided (i.e. those using the respective default value). This is useful in cases where one want to distinguish f() from f(None) given "def f(x=None):".

Iterating over fixed size blocks (Python)

2008-01-12T17:58:28-08:00

Python recipe 542194 by George Sakkis (algorithms).

This recipe shows a generator that breaks an iterable into chunks of fixed size. It addresses the general use case of having to (or wanting to) constrain the number of items to be processed at a time, for example because of resource limitations. It can very easily wrap blocks of code that work on iterables: just replace <pre>process(all_items)</pre>with <pre>for some_items in iterblock(all_items, 100): process(some_items)</pre>

Roundrobin generator (Python)

2007-09-07T19:09:36-07:00

Python recipe 528936 by George Sakkis (algorithms).

This recipe implements a round-robin generator, a generator that cycles through N iterables until all of them are exhausted:

>>> list(roundrobin('abc', [], range(4),  (True,False)))
['a', 0, True, 'b', 1, False, 'c', 2, 3]

Generic block iterator (Python)

2008-01-17T03:55:45-08:00

Python recipe 521877 by George Sakkis (algorithms). Revision 4.

A common task, especially in text processing, is to break some input sequence into chunks (lines, paragraphs, records, etc.) and process them one by one. The iterators of builtin strings and files can be considered such chunkers, breaking the object into characters and lines respectively. This recipe is a generic way to break any iterable into consecutive blocks, specified by a delimiter or a pair of (start,end) delimiters.

Yet another threadpool module (Python)

2007-03-25T05:37:06-07:00

Python recipe 502291 by George Sakkis (oop). Revision 4.

A small yet flexible threadpool implementation. I wasn't quite satisfied with other existing solutions, so I rolled one on my own, building on top of the threadpool module by Christopher Arndt.

See the example in the end and the docstrings (in Epydoc format) for the details.

Re-evaluatable default argument expressions (Python)

2007-02-04T05:04:41-08:00

Python recipe 502206 by George Sakkis . Revision 2.

This recipe extends the standard python semantics with respect to default function arguments by allowing "deferred" expressions, expressions that are evaluated on every call instead of just once at function definition time.

Generating combinations in blocks (Python)

2007-01-26T02:30:08-08:00

Python recipe 501151 by George Sakkis (algorithms). Revision 2.

This recipe is yet another combination generator, with an extra twist: the generated combinations can be yielded in blocks of a given size. This comes useful (or necessary) when processing arbitrarily long or infinite iterables in bounded-size blocks.

Fixed keys mapping type (Python)

2007-01-12T00:29:57-08:00

Python recipe 499373 by George Sakkis (oop). Revision 3.

A memory efficient implementation of a mapping type for mappings with fixed keys.

Rich iterator wrapper (Python)

2006-11-19T00:03:21-08:00

Python recipe 498272 by George Sakkis (oop).

This recipe may be of interest to those who make heavy use of the itertools module. It provides a wrapper class that exposes most itertools functions as methods, plus a few more. Moreover, two frequently used itertools functions, chain and islice, are conveniently exposed as addition and slicing operators, respectively.

Reusing default function arguments (Python)

2005-10-12T20:37:53-07:00

Python recipe 440702 by George Sakkis (oop). Revision 2.

This recipe applies the "once and only once" principle to function default values. Often two or more callables specify the same default values for one or more arguments. This is especially typical when overriding a method. Using the defaultsfrom(func) decorator, a method may 'inherit' the default values from the super method. More generally, any function may inherit the default values from another one.

Safe heap queue class (Python)

2005-10-09T14:57:08-07:00

Python recipe 440673 by George Sakkis (algorithms). Revision 3.

The Heap class is an improvement over a previous recipe (http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/437116) that also wraps the heapq module. There are two main differences from the other recipe: - All methods on this heap preserve the heap property invariant; therefore there is no need for is_heap(). - When creating a new heap, an optional 'key' argument can be specified to determine the comparison key for the items to be pushed into the heap.

Instantiating pickled instances transparently (Python)

2005-09-19T22:26:08-07:00

Python recipe 440579 by George Sakkis (oop).

This recipe addresses the following two needs: - Object construction for some class is expensive. - Objects of this class need to be instantiated across multiple runs of the program. For example, object instantiaton may involve reading one or more big files, connecting to a database or a network socket with considerable expected delay, etc.

The autopickle decorator deals with this problem by wrapping the __init__ of the class. The first time a specific instance is created, it is also pickled to a file. In all subsequent attempts to create the same instance, the pickled instance is loaded and returned instead. If unpickling the file is faster than creating the instance normally, all but the first instantiations are faster than the normal one.

The instance determines the path of the file to be pickled to by calling its getPickleFilename() method. This takes the same arguments given in __init__ and it has the responsibility to specify a valid and distinct path for the instance.

Property decorator for python 2.4 (Python)

2005-04-25T23:32:11-07:00

Python recipe 410698 by George Sakkis (oop). Revision 3.

This recipe refines an older recipe on creating class properties ( http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/205183). The refinement consists of: - Using a decorator (introduced in python 2.4) to "declare" a function in class scope as property. - Using a trace function to capture the locals() of the decorated function, instead of requiring the latter to return locals(), as in the older recipe.

Implementing Java inner classes using descriptors (Python)

2005-07-08T22:08:04-07:00

Python recipe 409366 by George Sakkis (oop). Revision 2.

Java programs use extensively "inner" (or nested) classes. There are actually four kinds of such classes (http://www.unix.org.ua/orelly/java-ent/jnut/ch03_08.htm), namely static and non-static member classes, local classes and anonymous inner classes. Python supports directly the last two through classes defined in nested scopes and lambdas. This recipe implements the first two kinds, static and non-static member classes.

A non-static member class is associated with an instance of the outer class and has implicit access to its __dict__. Thus, if the normal attribute lookup fails for an instance of the inner class, it continues to the outer instance.

Similarly, a static member class is associated with the outer class, instead of a specific instance of the outer class.

In both cases, the outer object (instance or class) can be explicitly accessed though the '__outer__' attribute.