A Simple Namespace Class (Python)

2011-10-03T21:12:41-07:00

Python recipe 577887 by Eric Snow (dict, namespaces). Revision 3.

"Namespaces are one honking great idea -- let's do more of those!" -- The Zen of Python

For when you want a simple, easy namespace, but you don't want it cluttered up with Python's object machinery.

Inspect a PYC File (Python)

2011-09-29T20:07:10-07:00

Python recipe 577880 by Eric Snow (bytecode, inspection, pyc). Revision 3.

This is a revamp of a recipe that Ned Batchelder posted on his blog a few years ago (thanks Ned!). That page is pretty insightful, recipe aside.

This recipe works for all versions of Python back to 2.4 (at least). Warning: using one version of Python to inspect a pyc file from another Python version may not work too well.

Apply decorators to all functions in a module (Python)

2011-06-09T22:51:28-07:00

Python recipe 577742 by Eric Snow (decorators, import, modules). Revision 2.

Use my modulehacker recipe (recipe 577740) to apply decorators to any number of modules, or even (nearly) all of them.

See What the Builtins Are (Python)

2011-11-04T22:44:09-07:00

Python recipe 577888 by Eric Snow (builtins). Revision 3.

The built-ins are all those functions, constants, and types that are there when you need them most, like list, any, and None. This recipe is a simple way of looking at the built-ins. Check below for a thorough explanation.

Mixin and Overlay (Python)

2011-06-01T05:16:47-07:00

Python recipe 577730 by Eric Snow (classes, inheritance, mixins).

The Two Sides of Classes

Class inheritance either adds new attributes to a class, or changes existing attributes. However, this distinction is not made in Python's class system. This recipe is a stab at providing that separation. It takes advantage of metaclasses and class decorators to do it.

For this recipe the two sides of class inheritance are called mixins and overlays. Mixins add new attributes to a class. Overlays change existing attributes, which in the case of methods means changing the behavior of the methods. However, overlays do not add any attributes.

Separation of Concerns

Another issue with class inheritance is that abstraction rarely breaks down into perfect trees with a clear separation of concerns. Python allows multiple inheritance, which can help, but requires cooperation between classes in the diamond hierarchy. Mixins and overlays help with this problem. Hopefully, that will be evidenced by the recipe and subsequent examples.

With this approach, the main single inheritance line can focus on the core abstraction and mixins/overlays can be used to extend the classes in other directions. This does not solve all the problems regarding separation of concerns, but it solves some. I hope to address the rest in another recipe that centers around delegation through a component architecture on instances.

Interfaces

Since Python 2.6 we have had Abstract Base Classes providing a mechanism for promising what interfaces an object provides. I'll show in one of the examples how an ABC can be split into a interface portion and a mixin portion. The recipe takes advantage of the ABC functionality in the Python type system. I expect that mixins and overlays would be a good fit with one of my other recipes (Recipe 577711).

The Recipe Classes

This recipe provides a standard approach to applying mixin classes without using inheritance. This is done through a metaclass. The metaclass builds a __mixins__ attribute on the class and provides a mixes_in class decorator. Applying that decorator to the class will add the attributes in __mixins__ to the decorated class. However, if any of those names are already bound on the name then it will fail. The decorator returns the modified class.

Note: Traditional mixins are typically done through multiple inheritance, as opposed to class decorators.

This recipe also provides a companion to mixin classes, called overlays. This is done through a metaclass in exactly the same way as the mixins, but provides an __overlays__ attribute and a class decorator called overlays. In contrast to mixins, if an overlay attribute is missing on the decorated class, it will fail. This is because the decorator will return a new class that inherits from the decorated class, and overrides the attributes in __overlays__.

Note: Metaclasses are used here because we need to pull from the class namespace there. A class decorator does not afford us the same functionality without more complexity.

Note: This recipe should work fine in 2.7 with a switch to the __metaclass__ syntax.

Mixins by Inheritance vs. by Decorator...Let's Try Decorators (Python)

2011-08-12T15:29:21-07:00

Python recipe 577824 by Eric Snow (decorators, mixins).

Using mixin classes via inheritance has its pros and cons. Here is an easy alternative via a decorator. As a bonus, you can mix in attributes from any object, not just classes.

A class decorator for creating named tuples (Python)

2013-02-14T19:58:13-08:00

Python recipe 578456 by Eric Snow (metaprogramming, namedtuple). Revision 3.

This class decorator factory is useful for replacing the following:

class MyTuple(namedtuple('MyTuple', "a b c")):
    """Something special."""
    @classmethod
    def from_defaults(cls, a, b=None, c=5):
        return cls(a, b, c)

or even:

class MyTuple(namedtuple('MyTuple', "a b c")):
    """Something special."""
    def __new__(cls, a, b=None, c=5):
        return super().__new__(cls, a, b, c)

with this:

@as_namedtuple("a b c", None, c=5)
class MyTuple:
    """Something special."""

I found that I often subclass named tuples to add on some functionality or even just a nice docstring. Plus with the class syntax there's no missing that a class is bound to the name (and it's a little easier to search for the definition). When you subclass a named tuple the boilerplate involved really jumps out.

One of the main reasons Adding support for defaults to namedtuple would mitigate the need for that functionality here, but I'm not going to hold my breath on that.

One nice (though minor) thing is that you don't have to repeat the name when defining the namedtuple.

Simple Lock Classes Based on threading.Lock() (Python)

2011-10-19T21:26:39-07:00

Python recipe 577914 by Eric Snow (lock, locking).

Nothing fancy here. Locks can take many forms. Here are a few.

Simple Abstract "Constants" to Use When @abstractproperty is Overkill or Misleading (Python)

2011-08-12T23:35:45-07:00

Python recipe 577761 by Eric Snow (attributes, classes).

Use these instead of abstract properties when you don't plan on the abstract attribute being implemented with a property. And you can still give your attribute a docstring!

Smarter Default Arguments (Python)

2011-08-12T23:06:57-07:00

Python recipe 577786 by Eric Snow (default_arguments, deferred, function, functions). Revision 3.

Improved handling of mutable and deferred default arguments. After the recipe you'll find an explanation of what that means.

Works for 2.7 with minor tweaks (getfullargspec --> getargspec).

Using Metaclasses and Class Decorators to Inherit Function Docstrings (Python)

2011-06-11T00:59:35-07:00

Python recipe 577743 by Eric Snow (decorator, decorators, docstring, inheritance, metaclass, metaclasses). Revision 4.

You'll find three different approaches to copying the method's docstring to the overriding method on a child class.

The function decorator approach is limited by the fact that you have to know the class when you call the decorator, so it can't be used inside a class body. However, recipe #577746 provides a function decorator that does not have this limitation.

Emulate Keyword-Only Arguments in Python 2 (Python)

2011-11-03T18:31:52-07:00

Python recipe 577940 by Eric Snow (argument, keyword_only, kwonly, parameter).

Python 3 introduced a useful feature: keyword-only arguments. In order to get the same effect in Python 2, you must use **kwargs in your parameter list. Then, at the beginning of your function body you must manually extract what would be the keyword-only arguments once you upgrade to 3.

This recipe helps reduce the boilerplate to a single function call. You still don't get those parameters in your "def" clause (where they are more obvious), but at least it reduces the clutter.

Class-only Methods (Python)

2013-03-08T17:00:53-08:00

Python recipe 578486 by Eric Snow (metaprogramming).

We use the classmethod builtin to tie methods to the class rather than the instance. This is really useful for a variety of things, like alternate contructors. However, sometimes you don't want to expose the method on the instances, just on the class.

For instance, it usually doesn't make a lot of sense to have direct access to alternate constructors from instances; the main constructor, via the __call__ method, is only available on the class. Why? So that instances call implement their own call semantics. This is similar to why the methods (and other class attributes) on namedtuples all have names starting with an underscore.

To restrict a method just to the class, currently you must use a metaclass. Most people consider this black magic so they just use @classmethod. Furthermore, implementing a metaclass is simply more verbose than decorating a method with classmethod. Plus you have to deal with things like metaclass conflicts and the fact that metaclasses are inherited.

So...here is a sibling to classmethod that makes a method class-only without introducing metaclass complexity. You use it the same way as you would classmethod.

Validating classes and objects against an Abstract Base Class (Python)

2011-05-21T19:14:19-07:00

Python recipe 577711 by Eric Snow (abc, validation). Revision 4.

Abstract Bases Classes in Python provide great features for describing interfaces programmatically. By default a subclass is validated against all its ABC parents at instantiation time (in object.__new__). This recipe aims to provide for validation against an ABC of:

any class at definition time (including subclasses and registered classes),
any object at any time.

I have included an example of the reason I did all this. It allows you to implement an ABC in the instance rather than the class.

If the classes argument to validate is None then it tries to build the list of classes from the object's MRO. If the ABCMeta.register method facilitated an __implements__ list on classes, we could also use that to validate against the registered "base" classes.

The code I have provided is for Python 3, but it should work in 2.7 with a little modification.

This code borrows from Lib/abc.py and objects/typeobject.c

A Protocol for Making Objects Immutable (Python)

2011-10-07T03:59:45-07:00

Python recipe 577895 by Eric Snow (freeze, immutable, mutable, unfreeze).

Python already provides immutable versions of many of the mutable built-in types. Dict is the notable exception. Regardless, here is a protocol that objects may implement that facilitates turning immutable object mutable and vice-versa.

A MutableMapping that Can Use Unhashable Objects as Keys (Python)

2012-04-04T20:36:59-07:00

Python recipe 578096 by Eric Snow .

The catch is that the unhashable objects aren't actually stored in the mapping. Only their IDs are. Thus you must also store the actual objects somewhere else.

Simple but Powerful Grouped Constants -- A Step Toward NamedConstants (Python)

2011-08-12T23:21:32-07:00

Python recipe 577755 by Eric Snow (constants). Revision 6.

The recipe provides an easy-to-use class for a group of "constants". It provides helper functions for getting the constant values just right. It also exposes the mechanism it uses to bind the contents of an iterable into the namespace of another object.

For this binding and for the grouped constants, this recipe makes it easy to dynamically generate the mapped values you want to expose.

The next step is to make the values aware of the context in which they are bound and to strengthen their association with the group they are in. And that is one of the recipes I'm working on next!

Make a Class's Name Available in its Definition Body (Python)

2011-08-04T21:38:40-07:00

Python recipe 577813 by Eric Snow (classes).

Since a class object is created after the body is executed, it can't be available to the class body. Even the name is unavailable, at least by default. However, you can use the __prepare__() method in a metaclass to stick it in there. This recipe is a simple demonstration of how.

A Script that Adds a Directory to sys.path Permanently (Python)

2012-02-16T23:15:11-08:00

Python recipe 578043 by Eric Snow (script, sys_path).

This script takes advantage of PEP 370, "Per user site-packages directory". It manages .pth files, which are are non-volatile (unlike manually adding to sys.path). See http://docs.python.org/library/site.html.

Turn a Function Into a Class (Python)

2011-10-05T18:38:43-07:00

Python recipe 577822 by Eric Snow (classes, functions).

The only catch is that the function has to return locals() at the end. And it doesn't do the __prepare__ part of 3.x metaclasses.

Most viewed recipes by Eric Snow