Latest recipes by Steven D'Aprano http://code.activestate.com/recipes/users/4172944/new/2017-06-25T17:17:43-07:00ActiveState Code RecipesGuard against an exception in the wrong place (Python)
2017-06-25T17:17:43-07:00Steven D'Apranohttp://code.activestate.com/recipes/users/4172944/http://code.activestate.com/recipes/580808-guard-against-an-exception-in-the-wrong-place/
<p style="color: grey">
Python
recipe 580808
by <a href="/recipes/users/4172944/">Steven D'Aprano</a>
(<a href="/recipes/tags/context/">context</a>, <a href="/recipes/tags/exception/">exception</a>, <a href="/recipes/tags/guard/">guard</a>, <a href="/recipes/tags/manager/">manager</a>).
Revision 2.
</p>
<p>Sometimes exception handling can obscure bugs unless you guard against a particular exception occurring in a certain place. One example is that <a href="https://www.python.org/dev/peps/pep-0479/">accidentally raising <code>StopIteration</code> inside a generator</a> will halt the generator instead of displaying a traceback. That was solved by PEP 479, which automatically has such <code>StopIteration</code> exceptions change to <code>RuntimeError</code>. See the discussion below for further examples.</p>
<p>Here is a class which can be used as either a decorator or context manager for guarding against the given exceptions. It takes an exception (or a tuple of exceptions) as argument, and if the wrapped code raises that exception, it is re-raised as another exception type (by default <code>RuntimeError</code>).</p>
<p>For example:</p>
<pre class="prettyprint"><code>try:
with exception_guard(ZeroDivisionError):
1/0 # raises ZeroDivisionError
except RuntimeError:
print ('ZeroDivisionError replaced by RuntimeError')
@exception_guard(KeyError)
def demo():
return {}['key'] # raises KeyError
try:
demo()
except RuntimeError:
print ('KeyError replaced by RuntimeError')
</code></pre>
Collection Pipeline in Python (Python)
2016-03-16T14:45:02-07:00Steven D'Apranohttp://code.activestate.com/recipes/users/4172944/http://code.activestate.com/recipes/580625-collection-pipeline-in-python/
<p style="color: grey">
Python
recipe 580625
by <a href="/recipes/users/4172944/">Steven D'Aprano</a>
(<a href="/recipes/tags/bash/">bash</a>, <a href="/recipes/tags/filter/">filter</a>, <a href="/recipes/tags/map/">map</a>, <a href="/recipes/tags/pipe/">pipe</a>, <a href="/recipes/tags/pipline/">pipline</a>).
</p>
<p>A powerful functional programming technique is the use of pipelines of functions. If you have used shell scripting languages like <code>bash</code>, you will have used this technique. For instance, to count the number of files or directories, you might say: <code>ls | wc -l</code>. The output of the first command is piped into the input of the second, and the result returned.</p>
<p>We can set up a similar pipeline using Lisp-like Map, Filter and Reduce special functions. Unlike the standard Python <code>map</code>, <code>filter</code> and <code>reduce</code>, these are designed to operate in a pipeline, using the same <code>|</code> syntax used by bash and other shell scripting languages:</p>
<pre class="prettyprint"><code>>>> data = range(1000)
>>> data | Filter(lambda n: 20 < n < 30) | Map(float) | List
[21.0, 22.0, 23.0, 24.0, 25.0, 26.0, 27.0, 28.0, 29.0]
</code></pre>
<p>The standard Python functional tools is much less attractive, as you have to write the functions in the opposite order to how they are applied to the data. This makes it harder to follow the logic of the expression.</p>
<pre class="prettyprint"><code>>>> list(map(float, filter(lambda n: 20 < n < 30, data)))
[21.0, 22.0, 23.0, 24.0, 25.0, 26.0, 27.0, 28.0, 29.0]
</code></pre>
<p>We can also end the pipeline with a call to <code>Reduce</code> to collate the sequence into a single value. Here we take a string, extract all the digits, convert to ints, and multiply:</p>
<pre class="prettyprint"><code>>>> from operator import mul
>>> "abcd12345xyz" | Filter(str.isdigit) | Map(int) | Reduce(mul)
120
</code></pre>
Safely and atomically write to a file (Python)
2016-03-23T14:14:26-07:00Steven D'Apranohttp://code.activestate.com/recipes/users/4172944/http://code.activestate.com/recipes/579097-safely-and-atomically-write-to-a-file/
<p style="color: grey">
Python
recipe 579097
by <a href="/recipes/users/4172944/">Steven D'Aprano</a>
(<a href="/recipes/tags/atomic/">atomic</a>, <a href="/recipes/tags/file/">file</a>, <a href="/recipes/tags/save/">save</a>).
Revision 3.
</p>
<p>Saving the user's data is risky. If you write to a file directly, and an error occurs during the write, you may corrupt the file and lose the user's data. One approach to prevent this is to write to a temporary file, then only when you know the file has been written successfully, over-write the original. This function returns a context manager which can make this more convenient.</p>
<p>Update: this now uses <code>os.replace</code> when available, and hopefully will work better on Windows.</p>
Call out to an external editor (Python)
2014-09-01T18:26:51-07:00Steven D'Apranohttp://code.activestate.com/recipes/users/4172944/http://code.activestate.com/recipes/578926-call-out-to-an-external-editor/
<p style="color: grey">
Python
recipe 578926
by <a href="/recipes/users/4172944/">Steven D'Aprano</a>
(<a href="/recipes/tags/editing/">editing</a>, <a href="/recipes/tags/editor/">editor</a>, <a href="/recipes/tags/external/">external</a>, <a href="/recipes/tags/text/">text</a>).
Revision 2.
</p>
<p>Here's a function that lets you use Python to wrap calls to an external editor. The editor can be an command line editor, like venerable old "ed", or something more powerful like nano, vim or emacs, and even GUI editors. After the editor quits, the text you typed in the editor is returned by the function.</p>
<p>A simple example, using the (rather cryptic) 'ed' editor on Linux. For the benefit of those unfamiliar with 'ed', I have annotated the editor session with comments.</p>
<pre class="prettyprint"><code>>>> status, text = edit('ed')
0 ## ed prints the initial number of lines
a ## start "append" mode
Hello World!
Goodbye now
. ## stop appending
w ## write the file to disk
25 ## ed prints the number of bytes written
q ## quit ed and return to Python
>>> status
0
>>> print text
Hello World!
Goodbye now
</code></pre>
Yet Another NamedTuple (Python)
2014-08-02T02:56:12-07:00Steven D'Apranohttp://code.activestate.com/recipes/users/4172944/http://code.activestate.com/recipes/578918-yet-another-namedtuple/
<p style="color: grey">
Python
recipe 578918
by <a href="/recipes/users/4172944/">Steven D'Aprano</a>
(<a href="/recipes/tags/shortcuts/">shortcuts</a>).
</p>
<p>Yet another look at Raymond Hettinger's excellent "namedtuple" factory. Unlike Raymond's version, this one minimizes the use of exec. In the original, the entire inner class is dynamically generated then exec'ed, leading to the bulk of the code being inside a giant string template. This version uses a regular inner class for everything except the __new__ method.</p>
Method chaining or cascading (Python)
2016-09-01T12:34:17-07:00Steven D'Apranohttp://code.activestate.com/recipes/users/4172944/http://code.activestate.com/recipes/578770-method-chaining-or-cascading/
<p style="color: grey">
Python
recipe 578770
by <a href="/recipes/users/4172944/">Steven D'Aprano</a>
(<a href="/recipes/tags/cascade/">cascade</a>, <a href="/recipes/tags/cascading/">cascading</a>, <a href="/recipes/tags/chaining/">chaining</a>, <a href="/recipes/tags/method/">method</a>).
</p>
<p>A frequently missed feature of built-ins like lists and dicts is the ability to chain method calls like this:</p>
<pre class="prettyprint"><code>x = []
x.append(1).append(2).append(3).reverse().append(4)
# x now equals [3, 2, 1, 4]
</code></pre>
<p>Unfortunately this doesn't work, as mutator methods return <code>None</code> rather than <code>self</code>. One possibility is to design your class from the beginning with method chaining in mind, but what do you do with those like the built-ins which aren't?</p>
<p>This is sometimes called <a href="https://en.wikipedia.org/wiki/Method_cascading">method cascading</a>. Here's a proof-of-concept for an adapter class which turns any object into one with methods that can be chained.</p>
Using ChainMap for embedded namespaces (Python)
2012-10-03T18:12:50-07:00Steven D'Apranohttp://code.activestate.com/recipes/users/4172944/http://code.activestate.com/recipes/578279-using-chainmap-for-embedded-namespaces/
<p style="color: grey">
Python
recipe 578279
by <a href="/recipes/users/4172944/">Steven D'Aprano</a>
(<a href="/recipes/tags/chainmap/">chainmap</a>, <a href="/recipes/tags/metaclass/">metaclass</a>, <a href="/recipes/tags/namespace/">namespace</a>).
</p>
<p>The Zen of Python tells us:</p>
<p><em>Namespaces are one honking great idea -- let's do more of those!</em></p>
<p>Python already has an excellent namespace type, the module, but the problem with modules is that they have to live in a separate file, and sometimes you want the convenience of a single file while still encapsulating your code into namespaces. That's where classes are the usual solution, but classes need to be instantiated and methods need to be defined with a <code>self</code> parameter.</p>
<p>C++ has "namespaces" for encapsulating related objects and dividing the global scope into sub-scopes. Can we do the same in Python?</p>
<p>With a bit of metaclass trickery and the new ChainMap type from Python 3.3, we can!</p>
Get single keypress (Python)
2011-12-06T08:46:33-08:00Steven D'Apranohttp://code.activestate.com/recipes/users/4172944/http://code.activestate.com/recipes/577977-get-single-keypress/
<p style="color: grey">
Python
recipe 577977
by <a href="/recipes/users/4172944/">Steven D'Aprano</a>
(<a href="/recipes/tags/getch/">getch</a>, <a href="/recipes/tags/key/">key</a>).
</p>
<p>Here's a platform-independent module that exposes a single function, getch, which reads stdin for a single character. It uses msvcrt.getch on Windows, and should work on any platform that supports the tty and termios modules (e.g. Linux).</p>
<p>This has been tested on Python 2.4, 2.5 and 2.6 on Linux.</p>
Benchmark code with the with statement (Python)
2011-10-08T09:53:06-07:00Steven D'Apranohttp://code.activestate.com/recipes/users/4172944/http://code.activestate.com/recipes/577896-benchmark-code-with-the-with-statement/
<p style="color: grey">
Python
recipe 577896
by <a href="/recipes/users/4172944/">Steven D'Aprano</a>
(<a href="/recipes/tags/benchmark/">benchmark</a>, <a href="/recipes/tags/speed/">speed</a>, <a href="/recipes/tags/time/">time</a>, <a href="/recipes/tags/timer/">timer</a>).
</p>
<p>Inspired by <a href="http://preshing.com/20110924/timing-your-code-using-pythons-with-statement">this post</a> I wrote this context manager to benchmark code blocks or function calls.</p>
<p>Usage is incredibly simple:</p>
<pre class="prettyprint"><code>with Timer():
... # code to benchmark goes here
</code></pre>
<p>The time taken (in seconds) will be printed when the code block completes. To capture the time taken programmatically is almost as easy:</p>
<pre class="prettyprint"><code>t = Timer()
with t:
... # code to benchmark goes here
time_taken = t.interval
</code></pre>
<p>Due to the difficulties of timing small snippets of code <em>accurately</em>, you should only use this for timing code blocks or function calls which take a significant amount of time to process. For micro-benchmarks, you should use the <code>timeit</code> module.</p>
Equally-spaced floats part 2 (Python)
2011-09-27T15:48:00-07:00Steven D'Apranohttp://code.activestate.com/recipes/users/4172944/http://code.activestate.com/recipes/577881-equally-spaced-floats-part-2/
<p style="color: grey">
Python
recipe 577881
by <a href="/recipes/users/4172944/">Steven D'Aprano</a>
(<a href="/recipes/tags/float/">float</a>, <a href="/recipes/tags/range/">range</a>, <a href="/recipes/tags/spread/">spread</a>).
</p>
<p>See the recipe <a href="http://code.activestate.com/recipes/577878">here</a> for a description of the problem.</p>
<p>This variant allows the user to choose between two APIs, either by supplying the count, start value and end value:</p>
<pre class="prettyprint"><code>>>> list(spread(5, 1.0, end=2.0))
[1.0, 1.2, 1.4, 1.6, 1.8]
</code></pre>
<p>or the count, start value and step size:</p>
<pre class="prettyprint"><code>>>> list(spread(5, 1.0, step=-0.25))
[1.0, 0.75, 0.5, 0.25, 0.0]
</code></pre>
<p>As before, the count argument specifies how many subdivisions are made. The optional argument <code>mode</code> selects whether the start and end values are included. By default, start is included and end is not, and exactly count values are returned.</p>
Generate equally-spaced floats (Python)
2011-09-24T18:49:39-07:00Steven D'Apranohttp://code.activestate.com/recipes/users/4172944/http://code.activestate.com/recipes/577878-generate-equally-spaced-floats/
<p style="color: grey">
Python
recipe 577878
by <a href="/recipes/users/4172944/">Steven D'Aprano</a>
(<a href="/recipes/tags/float/">float</a>, <a href="/recipes/tags/range/">range</a>, <a href="/recipes/tags/spread/">spread</a>).
</p>
<p>Generating a list of equally-spaced floats can surprising due to floating point rounding. See, for example, the recipe for a <a href="http://code.activestate.com/recipes/577068">floating point range</a>. One way of avoiding some surprises is by changing the API: instead of specifying a start, stop and step values, instead use a start, stop and count:</p>
<pre class="prettyprint"><code>>>> list(spread(0.0, 2.1, 7))
[0.0, 0.3, 0.6, 0.9, 1.2, 1.5, 1.8]
</code></pre>
<p>Like <a href="http://code.activestate.com/recipes/577068">frange</a> <code>spread</code> takes an optional mode argument to select whether the start and end values are included. By default, start is included and end is not, and exactly count values are returned.</p>
Search sequences for sub-sequence (Python)
2011-08-19T05:17:00-07:00Steven D'Apranohttp://code.activestate.com/recipes/users/4172944/http://code.activestate.com/recipes/577850-search-sequences-for-sub-sequence/
<p style="color: grey">
Python
recipe 577850
by <a href="/recipes/users/4172944/">Steven D'Aprano</a>
(<a href="/recipes/tags/find/">find</a>, <a href="/recipes/tags/searching/">searching</a>, <a href="/recipes/tags/sequence/">sequence</a>, <a href="/recipes/tags/string/">string</a>, <a href="/recipes/tags/sublist/">sublist</a>, <a href="/recipes/tags/substring/">substring</a>).
</p>
<p>The list and tuple index() method and <code>in</code> operator test for element containment, unlike similar tests for strings, which checks for sub-strings:</p>
<pre class="prettyprint"><code>>>> "12" in "0123"
True
>>> [1, 2] in [0, 1, 2, 3]
False
</code></pre>
<p>These two functions, search and rsearch, act like str.find() except they operate on any arbitrary sequence such as lists:</p>
<pre class="prettyprint"><code>>>> search([1, "a", "b", 2, 3], ["b", 2])
2
</code></pre>
Integer square root function (Python)
2011-08-04T05:29:16-07:00Steven D'Apranohttp://code.activestate.com/recipes/users/4172944/http://code.activestate.com/recipes/577821-integer-square-root-function/
<p style="color: grey">
Python
recipe 577821
by <a href="/recipes/users/4172944/">Steven D'Aprano</a>
(<a href="/recipes/tags/integer/">integer</a>, <a href="/recipes/tags/isqrt/">isqrt</a>, <a href="/recipes/tags/math/">math</a>, <a href="/recipes/tags/mathematics/">mathematics</a>, <a href="/recipes/tags/maths/">maths</a>, <a href="/recipes/tags/root/">root</a>, <a href="/recipes/tags/square/">square</a>).
</p>
<p>The <em>integer square root</em> function, or isqrt, is equivalent to floor(sqrt(x)) for non-negative x. For small x, the most convenient way to calculate isqrt is by calling int(x**0.5) or int(math.sqrt(x)), but if x is a large enough integer, the sqrt calculation overflows.</p>
<p>You can calculate the isqrt without any floating point maths, using just pure integer maths, allowing the function to operate with numbers far larger than possible with floats:</p>
<pre class="prettyprint"><code>>>> n = 1234567*(10**1000)
>>> n2 = n*n
>>> math.sqrt(n2)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
OverflowError: long int too large to convert to float
>>> isqrt(n2) == n
True
</code></pre>
Enhancing dir() with globs (Python)
2011-07-01T04:03:46-07:00Steven D'Apranohttp://code.activestate.com/recipes/users/4172944/http://code.activestate.com/recipes/577774-enhancing-dir-with-globs/
<p style="color: grey">
Python
recipe 577774
by <a href="/recipes/users/4172944/">Steven D'Aprano</a>
(<a href="/recipes/tags/dir/">dir</a>, <a href="/recipes/tags/filter/">filter</a>, <a href="/recipes/tags/glob/">glob</a>, <a href="/recipes/tags/globbing/">globbing</a>).
</p>
<p>dir() is useful for interactively exploring the attributes and methods of objects at the command line. But sometimes dir() returns a lot of information:</p>
<pre class="prettyprint"><code>>>> len(dir(decimal.Decimal)) # too much information!
137
</code></pre>
<p>It can be helpful to filter the list of names returned. This enhanced version of dir does exactly that, using simple string globbing:</p>
<pre class="prettyprint"><code>>>> dir(decimal.Decimal, '*log*') # just attributes with "log" in the name
['_fill_logical', '_islogical', '_log10_exp_bound', 'log10', 'logb',
'logical_and', 'logical_invert', 'logical_or', 'logical_xor']
</code></pre>
Calculate the MRO of a class (Python)
2011-06-11T08:31:09-07:00Steven D'Apranohttp://code.activestate.com/recipes/users/4172944/http://code.activestate.com/recipes/577748-calculate-the-mro-of-a-class/
<p style="color: grey">
Python
recipe 577748
by <a href="/recipes/users/4172944/">Steven D'Aprano</a>
(<a href="/recipes/tags/c3/">c3</a>, <a href="/recipes/tags/classes/">classes</a>, <a href="/recipes/tags/method/">method</a>, <a href="/recipes/tags/mro/">mro</a>, <a href="/recipes/tags/order/">order</a>, <a href="/recipes/tags/resolution/">resolution</a>).
</p>
<p>This function allows you to calculate the Method Resolution Order (MRO, or sometimes linearization) of a class or base classes. This is the so-called "C3" algorithm, as used by Python (new-style classes, from version 2.3 and higher). The MRO is the order of base classes that Python uses to search for methods and attributes. For single inheritance, the MRO is obvious and straight-forward and not very exciting, but for multiple inheritance it's not always obvious what the MRO should be.</p>
Validate ABNs (Australian Business Numbers) (Python)
2011-05-13T03:22:12-07:00Steven D'Apranohttp://code.activestate.com/recipes/users/4172944/http://code.activestate.com/recipes/577692-validate-abns-australian-business-numbers/
<p style="color: grey">
Python
recipe 577692
by <a href="/recipes/users/4172944/">Steven D'Aprano</a>
(<a href="/recipes/tags/abn/">abn</a>, <a href="/recipes/tags/validation/">validation</a>).
</p>
<p>This is a validation function for Australian Business Numbers (ABNs). It tests whether an integer or string is a valid ABN (but not whether it is a <em>legal</em> ABN, i.e. if it belongs to the person or business entity that is quoting it).</p>
Validate ACNs (Australian Company Numbers) (Python)
2011-05-13T03:15:35-07:00Steven D'Apranohttp://code.activestate.com/recipes/users/4172944/http://code.activestate.com/recipes/577691-validate-acns-australian-company-numbers/
<p style="color: grey">
Python
recipe 577691
by <a href="/recipes/users/4172944/">Steven D'Aprano</a>
(<a href="/recipes/tags/acn/">acn</a>, <a href="/recipes/tags/validation/">validation</a>).
</p>
<p>This is a validation function for Australian Company Numbers (ACNs). It tests whether the integer or string is a valid ACN (but not whether it is a <em>legal</em> ACN, i.e. if it belongs to the company that is quoting it).</p>
map_longest and map_strict (Python)
2011-05-06T17:35:17-07:00Steven D'Apranohttp://code.activestate.com/recipes/users/4172944/http://code.activestate.com/recipes/577687-map_longest-and-map_strict/
<p style="color: grey">
Python
recipe 577687
by <a href="/recipes/users/4172944/">Steven D'Aprano</a>
(<a href="/recipes/tags/iteration/">iteration</a>, <a href="/recipes/tags/map/">map</a>).
Revision 2.
</p>
<p>In Python 3, the map builtin silently drops any excess items in its input:</p>
<pre class="prettyprint"><code>>>> a = [1, 2, 3]
>>> b = [2, 3, 4]
>>> c = [3, 4, 5, 6, 7]
>>> list(map(lambda x,y,z: x*y+z, a, b, c))
[5, 10, 17]
</code></pre>
<p>In Python 2, map pads the shorter items with None, while itertools.imap drops the excess items. Inspired by this, and by itertools.zip_longest, I have map_longest that takes a value to fill missing items with, and map_strict that raises an exception if a value is missing.</p>
<pre class="prettyprint"><code>>>> list(map_longest(lambda x,y,z: x*y+z, a, b, c, fillvalue=0))
[5, 10, 17, 6, 7]
>>> list(map_strict(lambda x,y,z: x*y+z, a, b, c))
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 9, in map_strict
ValueError: too few items in iterable
</code></pre>
Locate and import Python's standard regression tests (Python)
2010-09-17T11:31:31-07:00Steven D'Apranohttp://code.activestate.com/recipes/users/4172944/http://code.activestate.com/recipes/577394-locate-and-import-pythons-standard-regression-test/
<p style="color: grey">
Python
recipe 577394
by <a href="/recipes/users/4172944/">Steven D'Aprano</a>
(<a href="/recipes/tags/import/">import</a>, <a href="/recipes/tags/path/">path</a>, <a href="/recipes/tags/regression/">regression</a>, <a href="/recipes/tags/testing/">testing</a>).
</p>
<p>The Python standard library comes with an extensive set of regression tests. I had need to import and run some of these tests from my own code, but the test directory is not in Python's path. This simple helper function solves the problem.</p>
Approximately Equal (Python)
2010-03-17T17:05:51-07:00Steven D'Apranohttp://code.activestate.com/recipes/users/4172944/http://code.activestate.com/recipes/577124-approximately-equal/
<p style="color: grey">
Python
recipe 577124
by <a href="/recipes/users/4172944/">Steven D'Aprano</a>
(<a href="/recipes/tags/comparisons/">comparisons</a>, <a href="/recipes/tags/distance/">distance</a>, <a href="/recipes/tags/math/">math</a>, <a href="/recipes/tags/maths/">maths</a>, <a href="/recipes/tags/numeric/">numeric</a>, <a href="/recipes/tags/string/">string</a>).
</p>
<p>Generic "approximately equal" function for any object type, with customisable error tolerance.</p>
<p>When called with float arguments, approx_equal(x, y[, tol[, rel]) compares x and y numerically, and returns True if y is within either absolute error tol or relative error rel of x, otherwise return False. The function defaults to sensible default values for tol and rel.</p>
<p>For any other pair of objects, approx_equal() looks for a method __approx_equal__ and, if found, calls it with arbitrary optional arguments. This allows types to define their own concept of "close enough".</p>