Top-rated recipes by Alfe

groupby() For Unsorted Input (Python)

2017-05-12T10:40:58-07:00

Python recipe 580800 by Alfe (algorithm, datastructures, generators, grouping, lazy).

We all know the groupby() which is available in the itertools standard module. This one yields groups of consecutive elements in the input which are meant to be together in one group. For non-consecutive elements this will yield more than one group for the same key.

So effectively, groupby() only reformats a flat list into bunches of elements from that list without reordering anything. In practice this means that for input sorted by key this works perfect, but for unsorted input it might yield several groups for the same key (with groups for other keys in between). Typically needed, though, is a grouping with reordering if necessary.

I implemented a likewise lazy function (yielding generators) which also accepts ungrouped input.

Context Manager for an Arbitrary Number of Files in Python (Python)

2017-03-13T13:27:50-07:00

Python recipe 580763 by Alfe (contextmanager, file, open). Revision 2.

The pattern using with together with open() to automatically close a file after leaving the context is well known. To open a fixed number you can simply nest these statements or use the comma notation. For having a context which represents an arbitrary number of open files you can use the ExitStack class, but only for Python 3.3+.

For other Python versions I'm using the following class which I named Files. The presented implementation is only for reading files (for keeping it clear). Extending it for having various file modes should not pose a problem.

Retry Decorator in Python (Python)

2017-01-11T10:10:30-08:00

Python recipe 580745 by Alfe (aspect, except, exception, retry, try).

This is a Python decorator which helps implementing an aspect oriented implementation of a retrying of certain steps which might fail sometimes. A typical example for this would be communication processes with the outside world, e. g. HTTP requests, allocation of some resource, etc. To use it, refactor the step in question into a (local) function and decorate this with the retry decorator. See examples in the discussion sector below.

Python Exception Chains (or Trees) (Python)

2013-02-04T15:15:22-08:00

Python recipe 578252 by Alfe (cause, chain, exception, python, reason, reraise, tree). Revision 2.

I have here an approach for chaining exceptions in case a lower layer (library) raises an exception which is caught in an upper layer (application) and later given as cause when a different exception is raised. Passing this cause exception is meant to offer access to the stack trace of the inner exception for debugging.

This approach is implemented in Python 3 and in Java, so it definitely makes sense; you also quickly find questions on Stackoverflow concerning it.

I even extended this feature by not only using chains of exceptions but also trees. Trees, why trees? Because I had situations in which my application layer tried various approaches using the library layer. If all failed (raised an exception), my application layer also raised an exception; this is the case in which I wanted to pass more than one cause exception into my own exception (hence the tree of causes).

My approach uses a special Exception class from which all my exceptions will inherit; standard exception must be wrapped (directly after catching, to preserve the exception stack trace). Please see the examples contained in the code below. The exception itself is rather small.

I'd be happy to hear any comments regarding memory leaks (I didn't find any but one never knows), usability, enhancements or similar.

Variable Abbreviations (Python)

2017-06-22T11:57:20-07:00

Python recipe 580807 by Alfe (abbreviations, contextmanager, variables, with).

One sometimes has nice long speaking names vor variables, maybe things like buildingList[foundIndex].height, but would like to address these in a shorter fashion to be used within a formula or similar where lots of longs names tend to confuse any reader trying to understand the formula. Physicists use one-letter names for a reason.

For this I wrote a small context provider which allows using short names instead of long ones:

with Abbr(h=buildingList[foundIndex].height, g=gravitationalConstant):
    fallTime = sqrt(2 * h / g)
    endSpeed = sqrt(2 * h * g)
print("Fall time:", fallTime)
print("End speed:", endSpeed)

For longer formulas this can reduce ugly multi-line expressions to clearly readable one-liners.

One could use this:

h = buildingList[foundIndex].height
g = gravitationalConstant
fallTime = sqrt(2 * h / g)
endSpeed = sqrt(2 * h * g)
del g, h
print("Fall time:", fallTime)
print("End speed:", endSpeed)

to achieve the same result, but

it would not look as clean and
the context provider solves the typical issues like cleanup on exception etc.

Just using local variables without cleanup (like above without the del statement) also is an option of course, but that would clutter the variable name space unnecessarily.

CAVEATS: The implementation of Abbr() is a hack. If used as intended and described here, it should work just fine, though. But the hackish nature forces me to mention some things: Since at compile time the compiler decides that the h and g in the example must be global variables (because they aren't assigned in the function), it produces byte code accessing global variables. The context provider changes the global variable structure to fill the needs. (Overridden already existing global variables of the same name get restored properly at context exit.) This means some things:

One cannot have a local variable of the same name in the frame surrounding the context manager.
Existing global variables are changed during the time of the context manager; so using names like sys or os for abbreviations might be a bad idea due to side-effects.

Unit Testing Nested Functions (Python)

2016-11-10T10:23:11-08:00

Python recipe 580716 by Alfe (introspection, nested, unittests). Revision 3.

Python allows the declaration of nested functions. These are typically hard to unit test because using just the normal ways of calling they cannot be called from outside their surrounding function. So they cannot be considered a clearly separated unit and thus cannot be unit tested.

This is a drawback of using them, so many developers (especially the ones deep into test driven development who strive to have a high unit test coverage) tend to avoid them in favor for standalone functions which can be called from the unit tests without any hassle.

But not all solutions with nested functions can be written as elegant with standalone functions. Nested functions are powerful insofar that they can access the local variables of the surrounding function without any need to pass them into the nested function, thus the code can in many cases stay neat and tidy while using a standalone function instead might raise the need to pass the complete context in form of a bunch of parameters. Also, using nested functions makes their local usage clear to any reader and keeps the name space tight.

But at least in the standard CPython (i. e. not necessarily in Jython, etc.) the implementation of functions (and methods) allows to find the nested function's code, wrap it properly to give it its needed context and then call it from the outside. I wrote a small module which helps doing exactly this.

Tetris (Python)

2016-06-09T12:49:41-07:00

Python recipe 580680 by Alfe (tetris).

Just the game of Tetris, implemented in Python some years ago. Contains some fancy add-ons.

Data (A Class For Arbitrary Data) (Python)

2015-08-03T15:22:40-07:00

Python recipe 579092 by Alfe (class, data, lightweight).

A class which is designed to be easy to use when one needs a piece of data with a minimum of source required.

Usage:

shop = Data(owner="Homer", address="down the street", ice=Data(flavor="vanilla", amount=3))
print shop
Data:
  owner = 'Homer'
  ice = Data:
          amount = 3
          flavor = 'vanilla'
  address = 'down the street'

Monitor Progress of File Descriptors of Another Process (Python)

2014-05-30T01:10:53-07:00

Python recipe 578882 by Alfe (file_descriptor, monitor, prediction, proc, process, progress, watch).

This tool (inspired by azat@stackoverflow, see http://stackoverflow.com/a/16082562/1281485) allows to watch the progress of the file descriptors of another process. This can be used, for example, if you transfer a file to another host and the transferring program does not show any progress indication itself. Instead of waiting blindly until the routine is done, with this tool you can use Linux's proc file system to monitor the progress of the other process while it walks through the file.

The tool continuously monitors the position in and the size of the files the given process's file descriptors point to. For growing (or shrinking, but that's very unusual) files, a time when it was (or will be) empty is computed ("emptyTime"), for moving file descriptors (the typical case), the time when it started at position 0 ("startTime"), the time when it will reach the current size of the file ("reachTime") and when it will meet with the end of a growing (or shrinking) file is computed ("meetTime").

For fixed-size files the meetTime will be the same as the reachTime of course. The meetTime only makes sense in case a file is growing and at the same time read (e. g. when a movie is downloaded to a file by one process and converted by a different process; using this tool can tell you when the converter process might run dry on the input because the download wasn't fast enough, and in this case you maybe can pause the converter to prevent this situation).

The tool is designed as a library; the display of the information is independent from the gathering of the data. Please feel free to create more fancy displays, add percentage output etc.

Minimalistic Word Wrap using Regex (Python)

2012-06-07T14:31:44-07:00

Python recipe 578162 by Alfe (minimalistic, python, regex, word, wrap).

I know there is the module textwrap and other recipes like http://code.activestate.com/recipes/148061-one-liner-word-wrap-function/

But in some constellations my recipe for a very simple word wrap might come in handy nevertheless.

Stack Environment (Python)

2017-06-22T10:24:05-07:00

Python recipe 578147 by Alfe (environment, global, stack, stackenv, variable). Revision 3.

The environment variables of processes get inherited by their children who can modify them and pass them on to their children. The tree of processes is similar to the call tree of a running Python process, but a similar mechanism like the environment variables is missing. I'm offering a solution for this; each stack frame takes the place of a process, hence I call it StackEnv. It comes in handy if you want to pass information from one frame to a frame far below without patching all the calls in between to pass the data (which might belong to a framework you don't want to change).

Usecases:

You call a framework which calls back your code before returning (e. g. in passed objects you provide). You want to pass some information to your code without relying on global variables or similar constructs which weren't thread-safe nor re-entrant.
You want to pass pseudo-global but in fact situation-related information (e. g. the verbosity based on the situation the call comes from) without handing the information down in each and every call which can happen below yours.
You want to give called methods the option to override the decision of the caller method regarding this information. (E. g. the caller decides that verbosity should be True, but the called method then calls another method and decides that the verbosity in this case should be False.)

Alike processes, called frames cannot influence the values for calling frames (but of course mutable values can be used to pass information upwards; this normal kind of "abuse" isn't prevented).

Importing:

from stackEnv import stackEnv

Setting a stackEnv variable:

stackEnv.verbose = True

Using the value of a stackEnv variable:

if stackEnv.verbose: print "being verbose now"

Testing a stackEnv variable:

if 'verbose' in stackEnv: print "having information on verbosity"

Overriding a stackEnv variable's value for the rest of this frame and its calls:

stackEnv.verbose = False

Removing a stackEnv variable for the rest of this frame and its calls:

del stackEnv.verbose

Some more useful API of this class can be found in the unit test included.

Gimp Paperwhite Scriptfu (Python)

2014-03-25T10:17:07-07:00

Python recipe 578857 by Alfe (gimp, image_processing, paperwhite, plugin, script, scriptfu).

This Gimp-plugin (written in Python) changes a photograph of a paper document so that the paper background appears white again without overly lighting the text.