A powerful functional programming technique is the use of pipelines of functions. If you have used shell scripting languages like bash, you will have used this technique. For instance, to count the number of files or directories, you might say: ls | wc -l. The output of the first command is piped into the input of the second, and the result returned.
We can set up a similar pipeline using Lisp-like Map, Filter and Reduce special functions. Unlike the standard Python map, filter and reduce, these are designed to operate in a pipeline, using the same | syntax used by bash and other shell scripting languages:
>>> data = range(1000)
>>> data | Filter(lambda n: 20 < n < 30) | Map(float) | List
[21.0, 22.0, 23.0, 24.0, 25.0, 26.0, 27.0, 28.0, 29.0]
The standard Python functional tools is much less attractive, as you have to write the functions in the opposite order to how they are applied to the data. This makes it harder to follow the logic of the expression.
>>> list(map(float, filter(lambda n: 20 < n < 30, data)))
[21.0, 22.0, 23.0, 24.0, 25.0, 26.0, 27.0, 28.0, 29.0]
We can also end the pipeline with a call to Reduce to collate the sequence into a single value. Here we take a string, extract all the digits, convert to ints, and multiply:
>>> from operator import mul
>>> "abcd12345xyz" | Filter(str.isdigit) | Map(int) | Reduce(mul)
120
Download
Copy to clipboard1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 | class Map:
def __init__(self, func):
self.func = func
def __ror__(self, iterable):
for obj in iterable:
yield self.func(obj)
class Filter:
def __init__(self, func):
self.func = func
def __ror__(self, iterable):
for obj in iterable:
if self.func(obj):
yield obj
_SENTINEL = object()
class Reduce:
def __init__(self, func, start=_SENTINEL):
self.func = func
self.start = start
def __ror__(self, iterable):
it = iter(iterable)
func = self.func
result = self.start
if result is _SENTINEL:
result = next(it, _SENTINEL)
if result is _SENTINEL:
raise ValueError('empty iterable')
for obj in it:
result = func(result, obj)
return result
class Apply:
def __init__(self, func):
self.func = func
def __ror__(self, iterable):
return self.func(iterable)
List = Apply(list)
|
Martin Fowler discusses this technique, its history in Smalltalk, and its use in functional programming languages here.
This code shown here uses a minimal set of standard functional tools, Map, Filter and Reduce, plus List to convert the result back to a list. We can create a much richer set of tools using the Apply class, for instance, to reverse the sequence:
>>> Reverse = Apply(reversed)
>>> "abcd" | Reverse | List
['d', 'c', 'b', 'a']
Similarly to extract the minimum value, we might use Apply(min).
The pipeline tools shown here return lists or iterators, so you can use them with other Python functions, such as the itertools module, for-loops, built-ins like set and list, or anything else you want.
Very nice Steven :) Is there a PyPi library/package to this effect that you know of?