Latest recipes by Steven D'Aprano Code RecipesGuard against an exception in the wrong place (Python) 2017-06-25T17:17:43-07:00Steven D'Aprano <p style="color: grey"> Python recipe 580808 by <a href="/recipes/users/4172944/">Steven D'Aprano</a> (<a href="/recipes/tags/context/">context</a>, <a href="/recipes/tags/exception/">exception</a>, <a href="/recipes/tags/guard/">guard</a>, <a href="/recipes/tags/manager/">manager</a>). Revision 2. </p> <p>Sometimes exception handling can obscure bugs unless you guard against a particular exception occurring in a certain place. One example is that <a href="">accidentally raising <code>StopIteration</code> inside a generator</a> will halt the generator instead of displaying a traceback. That was solved by PEP 479, which automatically has such <code>StopIteration</code> exceptions change to <code>RuntimeError</code>. See the discussion below for further examples.</p> <p>Here is a class which can be used as either a decorator or context manager for guarding against the given exceptions. It takes an exception (or a tuple of exceptions) as argument, and if the wrapped code raises that exception, it is re-raised as another exception type (by default <code>RuntimeError</code>).</p> <p>For example:</p> <pre class="prettyprint"><code>try: with exception_guard(ZeroDivisionError): 1/0 # raises ZeroDivisionError except RuntimeError: print ('ZeroDivisionError replaced by RuntimeError') @exception_guard(KeyError) def demo(): return {}['key'] # raises KeyError try: demo() except RuntimeError: print ('KeyError replaced by RuntimeError') </code></pre> Collection Pipeline in Python (Python) 2016-03-16T14:45:02-07:00Steven D'Aprano <p style="color: grey"> Python recipe 580625 by <a href="/recipes/users/4172944/">Steven D'Aprano</a> (<a href="/recipes/tags/bash/">bash</a>, <a href="/recipes/tags/filter/">filter</a>, <a href="/recipes/tags/map/">map</a>, <a href="/recipes/tags/pipe/">pipe</a>, <a href="/recipes/tags/pipline/">pipline</a>). </p> <p>A powerful functional programming technique is the use of pipelines of functions. If you have used shell scripting languages like <code>bash</code>, you will have used this technique. For instance, to count the number of files or directories, you might say: <code>ls | wc -l</code>. The output of the first command is piped into the input of the second, and the result returned.</p> <p>We can set up a similar pipeline using Lisp-like Map, Filter and Reduce special functions. Unlike the standard Python <code>map</code>, <code>filter</code> and <code>reduce</code>, these are designed to operate in a pipeline, using the same <code>|</code> syntax used by bash and other shell scripting languages:</p> <pre class="prettyprint"><code>&gt;&gt;&gt; data = range(1000) &gt;&gt;&gt; data | Filter(lambda n: 20 &lt; n &lt; 30) | Map(float) | List [21.0, 22.0, 23.0, 24.0, 25.0, 26.0, 27.0, 28.0, 29.0] </code></pre> <p>The standard Python functional tools is much less attractive, as you have to write the functions in the opposite order to how they are applied to the data. This makes it harder to follow the logic of the expression.</p> <pre class="prettyprint"><code>&gt;&gt;&gt; list(map(float, filter(lambda n: 20 &lt; n &lt; 30, data))) [21.0, 22.0, 23.0, 24.0, 25.0, 26.0, 27.0, 28.0, 29.0] </code></pre> <p>We can also end the pipeline with a call to <code>Reduce</code> to collate the sequence into a single value. Here we take a string, extract all the digits, convert to ints, and multiply:</p> <pre class="prettyprint"><code>&gt;&gt;&gt; from operator import mul &gt;&gt;&gt; "abcd12345xyz" | Filter(str.isdigit) | Map(int) | Reduce(mul) 120 </code></pre> Safely and atomically write to a file (Python) 2016-03-23T14:14:26-07:00Steven D'Aprano <p style="color: grey"> Python recipe 579097 by <a href="/recipes/users/4172944/">Steven D'Aprano</a> (<a href="/recipes/tags/atomic/">atomic</a>, <a href="/recipes/tags/file/">file</a>, <a href="/recipes/tags/save/">save</a>). Revision 3. </p> <p>Saving the user's data is risky. If you write to a file directly, and an error occurs during the write, you may corrupt the file and lose the user's data. One approach to prevent this is to write to a temporary file, then only when you know the file has been written successfully, over-write the original. This function returns a context manager which can make this more convenient.</p> <p>Update: this now uses <code>os.replace</code> when available, and hopefully will work better on Windows.</p> Call out to an external editor (Python) 2014-09-01T18:26:51-07:00Steven D'Aprano <p style="color: grey"> Python recipe 578926 by <a href="/recipes/users/4172944/">Steven D'Aprano</a> (<a href="/recipes/tags/editing/">editing</a>, <a href="/recipes/tags/editor/">editor</a>, <a href="/recipes/tags/external/">external</a>, <a href="/recipes/tags/text/">text</a>). Revision 2. </p> <p>Here's a function that lets you use Python to wrap calls to an external editor. The editor can be an command line editor, like venerable old "ed", or something more powerful like nano, vim or emacs, and even GUI editors. After the editor quits, the text you typed in the editor is returned by the function.</p> <p>A simple example, using the (rather cryptic) 'ed' editor on Linux. For the benefit of those unfamiliar with 'ed', I have annotated the editor session with comments.</p> <pre class="prettyprint"><code>&gt;&gt;&gt; status, text = edit('ed') 0 ## ed prints the initial number of lines a ## start "append" mode Hello World! Goodbye now . ## stop appending w ## write the file to disk 25 ## ed prints the number of bytes written q ## quit ed and return to Python &gt;&gt;&gt; status 0 &gt;&gt;&gt; print text Hello World! Goodbye now </code></pre> Yet Another NamedTuple (Python) 2014-08-02T02:56:12-07:00Steven D'Aprano <p style="color: grey"> Python recipe 578918 by <a href="/recipes/users/4172944/">Steven D'Aprano</a> (<a href="/recipes/tags/shortcuts/">shortcuts</a>). </p> <p>Yet another look at Raymond Hettinger's excellent "namedtuple" factory. Unlike Raymond's version, this one minimizes the use of exec. In the original, the entire inner class is dynamically generated then exec'ed, leading to the bulk of the code being inside a giant string template. This version uses a regular inner class for everything except the __new__ method.</p> Method chaining or cascading (Python) 2016-09-01T12:34:17-07:00Steven D'Aprano <p style="color: grey"> Python recipe 578770 by <a href="/recipes/users/4172944/">Steven D'Aprano</a> (<a href="/recipes/tags/cascade/">cascade</a>, <a href="/recipes/tags/cascading/">cascading</a>, <a href="/recipes/tags/chaining/">chaining</a>, <a href="/recipes/tags/method/">method</a>). </p> <p>A frequently missed feature of built-ins like lists and dicts is the ability to chain method calls like this:</p> <pre class="prettyprint"><code>x = [] x.append(1).append(2).append(3).reverse().append(4) # x now equals [3, 2, 1, 4] </code></pre> <p>Unfortunately this doesn't work, as mutator methods return <code>None</code> rather than <code>self</code>. One possibility is to design your class from the beginning with method chaining in mind, but what do you do with those like the built-ins which aren't?</p> <p>This is sometimes called <a href="">method cascading</a>. Here's a proof-of-concept for an adapter class which turns any object into one with methods that can be chained.</p> Using ChainMap for embedded namespaces (Python) 2012-10-03T18:12:50-07:00Steven D'Aprano <p style="color: grey"> Python recipe 578279 by <a href="/recipes/users/4172944/">Steven D'Aprano</a> (<a href="/recipes/tags/chainmap/">chainmap</a>, <a href="/recipes/tags/metaclass/">metaclass</a>, <a href="/recipes/tags/namespace/">namespace</a>). </p> <p>The Zen of Python tells us:</p> <p><em>Namespaces are one honking great idea -- let's do more of those!</em></p> <p>Python already has an excellent namespace type, the module, but the problem with modules is that they have to live in a separate file, and sometimes you want the convenience of a single file while still encapsulating your code into namespaces. That's where classes are the usual solution, but classes need to be instantiated and methods need to be defined with a <code>self</code> parameter.</p> <p>C++ has "namespaces" for encapsulating related objects and dividing the global scope into sub-scopes. Can we do the same in Python?</p> <p>With a bit of metaclass trickery and the new ChainMap type from Python 3.3, we can!</p> Get single keypress (Python) 2011-12-06T08:46:33-08:00Steven D'Aprano <p style="color: grey"> Python recipe 577977 by <a href="/recipes/users/4172944/">Steven D'Aprano</a> (<a href="/recipes/tags/getch/">getch</a>, <a href="/recipes/tags/key/">key</a>). </p> <p>Here's a platform-independent module that exposes a single function, getch, which reads stdin for a single character. It uses msvcrt.getch on Windows, and should work on any platform that supports the tty and termios modules (e.g. Linux).</p> <p>This has been tested on Python 2.4, 2.5 and 2.6 on Linux.</p> Benchmark code with the with statement (Python) 2011-10-08T09:53:06-07:00Steven D'Aprano <p style="color: grey"> Python recipe 577896 by <a href="/recipes/users/4172944/">Steven D'Aprano</a> (<a href="/recipes/tags/benchmark/">benchmark</a>, <a href="/recipes/tags/speed/">speed</a>, <a href="/recipes/tags/time/">time</a>, <a href="/recipes/tags/timer/">timer</a>). </p> <p>Inspired by <a href="">this post</a> I wrote this context manager to benchmark code blocks or function calls.</p> <p>Usage is incredibly simple:</p> <pre class="prettyprint"><code>with Timer(): ... # code to benchmark goes here </code></pre> <p>The time taken (in seconds) will be printed when the code block completes. To capture the time taken programmatically is almost as easy:</p> <pre class="prettyprint"><code>t = Timer() with t: ... # code to benchmark goes here time_taken = t.interval </code></pre> <p>Due to the difficulties of timing small snippets of code <em>accurately</em>, you should only use this for timing code blocks or function calls which take a significant amount of time to process. For micro-benchmarks, you should use the <code>timeit</code> module.</p> Equally-spaced floats part 2 (Python) 2011-09-27T15:48:00-07:00Steven D'Aprano <p style="color: grey"> Python recipe 577881 by <a href="/recipes/users/4172944/">Steven D'Aprano</a> (<a href="/recipes/tags/float/">float</a>, <a href="/recipes/tags/range/">range</a>, <a href="/recipes/tags/spread/">spread</a>). </p> <p>See the recipe <a href="">here</a> for a description of the problem.</p> <p>This variant allows the user to choose between two APIs, either by supplying the count, start value and end value:</p> <pre class="prettyprint"><code>&gt;&gt;&gt; list(spread(5, 1.0, end=2.0)) [1.0, 1.2, 1.4, 1.6, 1.8] </code></pre> <p>or the count, start value and step size:</p> <pre class="prettyprint"><code>&gt;&gt;&gt; list(spread(5, 1.0, step=-0.25)) [1.0, 0.75, 0.5, 0.25, 0.0] </code></pre> <p>As before, the count argument specifies how many subdivisions are made. The optional argument <code>mode</code> selects whether the start and end values are included. By default, start is included and end is not, and exactly count values are returned.</p> Generate equally-spaced floats (Python) 2011-09-24T18:49:39-07:00Steven D'Aprano <p style="color: grey"> Python recipe 577878 by <a href="/recipes/users/4172944/">Steven D'Aprano</a> (<a href="/recipes/tags/float/">float</a>, <a href="/recipes/tags/range/">range</a>, <a href="/recipes/tags/spread/">spread</a>). </p> <p>Generating a list of equally-spaced floats can surprising due to floating point rounding. See, for example, the recipe for a <a href="">floating point range</a>. One way of avoiding some surprises is by changing the API: instead of specifying a start, stop and step values, instead use a start, stop and count:</p> <pre class="prettyprint"><code>&gt;&gt;&gt; list(spread(0.0, 2.1, 7)) [0.0, 0.3, 0.6, 0.9, 1.2, 1.5, 1.8] </code></pre> <p>Like <a href="">frange</a> <code>spread</code> takes an optional mode argument to select whether the start and end values are included. By default, start is included and end is not, and exactly count values are returned.</p> Search sequences for sub-sequence (Python) 2011-08-19T05:17:00-07:00Steven D'Aprano <p style="color: grey"> Python recipe 577850 by <a href="/recipes/users/4172944/">Steven D'Aprano</a> (<a href="/recipes/tags/find/">find</a>, <a href="/recipes/tags/searching/">searching</a>, <a href="/recipes/tags/sequence/">sequence</a>, <a href="/recipes/tags/string/">string</a>, <a href="/recipes/tags/sublist/">sublist</a>, <a href="/recipes/tags/substring/">substring</a>). </p> <p>The list and tuple index() method and <code>in</code> operator test for element containment, unlike similar tests for strings, which checks for sub-strings:</p> <pre class="prettyprint"><code>&gt;&gt;&gt; "12" in "0123" True &gt;&gt;&gt; [1, 2] in [0, 1, 2, 3] False </code></pre> <p>These two functions, search and rsearch, act like str.find() except they operate on any arbitrary sequence such as lists:</p> <pre class="prettyprint"><code>&gt;&gt;&gt; search([1, "a", "b", 2, 3], ["b", 2]) 2 </code></pre> Integer square root function (Python) 2011-08-04T05:29:16-07:00Steven D'Aprano <p style="color: grey"> Python recipe 577821 by <a href="/recipes/users/4172944/">Steven D'Aprano</a> (<a href="/recipes/tags/integer/">integer</a>, <a href="/recipes/tags/isqrt/">isqrt</a>, <a href="/recipes/tags/math/">math</a>, <a href="/recipes/tags/mathematics/">mathematics</a>, <a href="/recipes/tags/maths/">maths</a>, <a href="/recipes/tags/root/">root</a>, <a href="/recipes/tags/square/">square</a>). </p> <p>The <em>integer square root</em> function, or isqrt, is equivalent to floor(sqrt(x)) for non-negative x. For small x, the most convenient way to calculate isqrt is by calling int(x**0.5) or int(math.sqrt(x)), but if x is a large enough integer, the sqrt calculation overflows.</p> <p>You can calculate the isqrt without any floating point maths, using just pure integer maths, allowing the function to operate with numbers far larger than possible with floats:</p> <pre class="prettyprint"><code>&gt;&gt;&gt; n = 1234567*(10**1000) &gt;&gt;&gt; n2 = n*n &gt;&gt;&gt; math.sqrt(n2) Traceback (most recent call last): File "&lt;stdin&gt;", line 1, in &lt;module&gt; OverflowError: long int too large to convert to float &gt;&gt;&gt; isqrt(n2) == n True </code></pre> Enhancing dir() with globs (Python) 2011-07-01T04:03:46-07:00Steven D'Aprano <p style="color: grey"> Python recipe 577774 by <a href="/recipes/users/4172944/">Steven D'Aprano</a> (<a href="/recipes/tags/dir/">dir</a>, <a href="/recipes/tags/filter/">filter</a>, <a href="/recipes/tags/glob/">glob</a>, <a href="/recipes/tags/globbing/">globbing</a>). </p> <p>dir() is useful for interactively exploring the attributes and methods of objects at the command line. But sometimes dir() returns a lot of information:</p> <pre class="prettyprint"><code>&gt;&gt;&gt; len(dir(decimal.Decimal)) # too much information! 137 </code></pre> <p>It can be helpful to filter the list of names returned. This enhanced version of dir does exactly that, using simple string globbing:</p> <pre class="prettyprint"><code>&gt;&gt;&gt; dir(decimal.Decimal, '*log*') # just attributes with "log" in the name ['_fill_logical', '_islogical', '_log10_exp_bound', 'log10', 'logb', 'logical_and', 'logical_invert', 'logical_or', 'logical_xor'] </code></pre> Calculate the MRO of a class (Python) 2011-06-11T08:31:09-07:00Steven D'Aprano <p style="color: grey"> Python recipe 577748 by <a href="/recipes/users/4172944/">Steven D'Aprano</a> (<a href="/recipes/tags/c3/">c3</a>, <a href="/recipes/tags/classes/">classes</a>, <a href="/recipes/tags/method/">method</a>, <a href="/recipes/tags/mro/">mro</a>, <a href="/recipes/tags/order/">order</a>, <a href="/recipes/tags/resolution/">resolution</a>). </p> <p>This function allows you to calculate the Method Resolution Order (MRO, or sometimes linearization) of a class or base classes. This is the so-called "C3" algorithm, as used by Python (new-style classes, from version 2.3 and higher). The MRO is the order of base classes that Python uses to search for methods and attributes. For single inheritance, the MRO is obvious and straight-forward and not very exciting, but for multiple inheritance it's not always obvious what the MRO should be.</p> Validate ABNs (Australian Business Numbers) (Python) 2011-05-13T03:22:12-07:00Steven D'Aprano <p style="color: grey"> Python recipe 577692 by <a href="/recipes/users/4172944/">Steven D'Aprano</a> (<a href="/recipes/tags/abn/">abn</a>, <a href="/recipes/tags/validation/">validation</a>). </p> <p>This is a validation function for Australian Business Numbers (ABNs). It tests whether an integer or string is a valid ABN (but not whether it is a <em>legal</em> ABN, i.e. if it belongs to the person or business entity that is quoting it).</p> Validate ACNs (Australian Company Numbers) (Python) 2011-05-13T03:15:35-07:00Steven D'Aprano <p style="color: grey"> Python recipe 577691 by <a href="/recipes/users/4172944/">Steven D'Aprano</a> (<a href="/recipes/tags/acn/">acn</a>, <a href="/recipes/tags/validation/">validation</a>). </p> <p>This is a validation function for Australian Company Numbers (ACNs). It tests whether the integer or string is a valid ACN (but not whether it is a <em>legal</em> ACN, i.e. if it belongs to the company that is quoting it).</p> map_longest and map_strict (Python) 2011-05-06T17:35:17-07:00Steven D'Aprano <p style="color: grey"> Python recipe 577687 by <a href="/recipes/users/4172944/">Steven D'Aprano</a> (<a href="/recipes/tags/iteration/">iteration</a>, <a href="/recipes/tags/map/">map</a>). Revision 2. </p> <p>In Python 3, the map builtin silently drops any excess items in its input:</p> <pre class="prettyprint"><code>&gt;&gt;&gt; a = [1, 2, 3] &gt;&gt;&gt; b = [2, 3, 4] &gt;&gt;&gt; c = [3, 4, 5, 6, 7] &gt;&gt;&gt; list(map(lambda x,y,z: x*y+z, a, b, c)) [5, 10, 17] </code></pre> <p>In Python 2, map pads the shorter items with None, while itertools.imap drops the excess items. Inspired by this, and by itertools.zip_longest, I have map_longest that takes a value to fill missing items with, and map_strict that raises an exception if a value is missing.</p> <pre class="prettyprint"><code>&gt;&gt;&gt; list(map_longest(lambda x,y,z: x*y+z, a, b, c, fillvalue=0)) [5, 10, 17, 6, 7] &gt;&gt;&gt; list(map_strict(lambda x,y,z: x*y+z, a, b, c)) Traceback (most recent call last): File "&lt;stdin&gt;", line 1, in &lt;module&gt; File "&lt;stdin&gt;", line 9, in map_strict ValueError: too few items in iterable </code></pre> Locate and import Python's standard regression tests (Python) 2010-09-17T11:31:31-07:00Steven D'Aprano <p style="color: grey"> Python recipe 577394 by <a href="/recipes/users/4172944/">Steven D'Aprano</a> (<a href="/recipes/tags/import/">import</a>, <a href="/recipes/tags/path/">path</a>, <a href="/recipes/tags/regression/">regression</a>, <a href="/recipes/tags/testing/">testing</a>). </p> <p>The Python standard library comes with an extensive set of regression tests. I had need to import and run some of these tests from my own code, but the test directory is not in Python's path. This simple helper function solves the problem.</p> Approximately Equal (Python) 2010-03-17T17:05:51-07:00Steven D'Aprano <p style="color: grey"> Python recipe 577124 by <a href="/recipes/users/4172944/">Steven D'Aprano</a> (<a href="/recipes/tags/comparisons/">comparisons</a>, <a href="/recipes/tags/distance/">distance</a>, <a href="/recipes/tags/math/">math</a>, <a href="/recipes/tags/maths/">maths</a>, <a href="/recipes/tags/numeric/">numeric</a>, <a href="/recipes/tags/string/">string</a>). </p> <p>Generic "approximately equal" function for any object type, with customisable error tolerance.</p> <p>When called with float arguments, approx_equal(x, y[, tol[, rel]) compares x and y numerically, and returns True if y is within either absolute error tol or relative error rel of x, otherwise return False. The function defaults to sensible default values for tol and rel.</p> <p>For any other pair of objects, approx_equal() looks for a method __approx_equal__ and, if found, calls it with arbitrary optional arguments. This allows types to define their own concept of "close enough".</p>