A powerful functional programming technique is the use of pipelines of functions. If you have used shell scripting languages like
bash, you will have used this technique. For instance, to count the number of files or directories, you might say:
ls | wc -l. The output of the first command is piped into the input of the second, and the result returned.
We can set up a similar pipeline using Lisp-like Map, Filter and Reduce special functions. Unlike the standard Python
reduce, these are designed to operate in a pipeline, using the same
| syntax used by bash and other shell scripting languages:
>>> data = range(1000) >>> data | Filter(lambda n: 20 < n < 30) | Map(float) | List [21.0, 22.0, 23.0, 24.0, 25.0, 26.0, 27.0, 28.0, 29.0]
The standard Python functional tools is much less attractive, as you have to write the functions in the opposite order to how they are applied to the data. This makes it harder to follow the logic of the expression.
>>> list(map(float, filter(lambda n: 20 < n < 30, data))) [21.0, 22.0, 23.0, 24.0, 25.0, 26.0, 27.0, 28.0, 29.0]
We can also end the pipeline with a call to
Reduce to collate the sequence into a single value. Here we take a string, extract all the digits, convert to ints, and multiply:
>>> from operator import mul >>> "abcd12345xyz" | Filter(str.isdigit) | Map(int) | Reduce(mul) 120
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40
class Map: def __init__(self, func): self.func = func def __ror__(self, iterable): for obj in iterable: yield self.func(obj) class Filter: def __init__(self, func): self.func = func def __ror__(self, iterable): for obj in iterable: if self.func(obj): yield obj _SENTINEL = object() class Reduce: def __init__(self, func, start=_SENTINEL): self.func = func self.start = start def __ror__(self, iterable): it = iter(iterable) func = self.func result = self.start if result is _SENTINEL: result = next(it, _SENTINEL) if result is _SENTINEL: raise ValueError('empty iterable') for obj in it: result = func(result, obj) return result class Apply: def __init__(self, func): self.func = func def __ror__(self, iterable): return self.func(iterable) List = Apply(list)
Martin Fowler discusses this technique, its history in Smalltalk, and its use in functional programming languages here.
This code shown here uses a minimal set of standard functional tools,
List to convert the result back to a list. We can create a much richer set of tools using the
Apply class, for instance, to reverse the sequence:
>>> Reverse = Apply(reversed) >>> "abcd" | Reverse | List ['d', 'c', 'b', 'a']
Similarly to extract the minimum value, we might use
The pipeline tools shown here return lists or iterators, so you can use them with other Python functions, such as the
itertools module, for-loops, built-ins like
list, or anything else you want.