Popular recipes by Denis Barmenkov

"tail -f" with inode monitor (Python)

2010-09-21T07:37:38-07:00

Python recipe 577398 by Denis Barmenkov (inode, monitor, tail). Revision 3.

Sometimes tail -f launched for log file miss the point when program recreates log file. Script in this recipe monitors inode changes for specified file and restarts tail if needed.

Relative path from one directory to another without explicit string functions (unix only) (Python)

2011-04-11T13:02:32-07:00

Python recipe 577356 by Denis Barmenkov (path, relative, unix). Revision 4.

I saw a recipe 208993 messed up with os.sep and '../' and decide to write near-pure-Python version. os.sep used in string expressions only for testing for root directory.

Function deal with Unix paths (root: "/"), Windows systems are not supported (root: "C:\").

xnview backup files remove utility (Python)

2010-07-06T18:09:43-07:00

Python recipe 577281 by Denis Barmenkov (backup, cleanup, remove, xnview). Revision 3.

Currently XnView image viewer (versions up to 1.97.6) correctly rotate images only when option "[x] make backup" set.

Backup files have added name part '.xnbak' before extension:

original file: IMG0001.jpg
backup file: IMG0001.xnbak.jpg

Attached script take a root directory in command line and remove backup files created by XnView only when rotated file present in same directory.

Render tables for text interface (Python)

2010-04-20T18:02:51-07:00

Python recipe 577202 by Denis Barmenkov (format, table, text). Revision 2.

Sometime pprint module is not enough for formatting data for console or log file output. This module provide function which fill the gap.

Sample function call:

nums = [ '1', '2', '3', '4' ]
speeds = [ '100', '10000', '1500', '12' ]
desc = [ '', 'label 1', 'none', 'very long description' ]
lines = format_table( [(nums, ALIGN_RIGHT|PADDING_ALL, 'NUM'), 
                       (speeds, ALIGN_RIGHT|PADDING_ALL, 'SPEED'), 
                       (desc, ALIGN_LEFT|PADDING_ALL, 'DESC')] )

Output:

=======================================
| NUM | SPEED | DESC                  |
=======================================
|   1 |   100 |                       |
|   2 | 10000 | label 1               |
|   3 |  1500 | none                  |
|   4 |    12 | very long description |
=======================================

Make unique file name (Python)

2010-04-18T21:07:52-07:00

Python recipe 577200 by Denis Barmenkov (file, name, rename). Revision 2.

Sometimes it is important to save data in the file but the file with the specified name already exists. This function creates a file name that is similar to the original by adding a unique numeric suffix. This avoids the renaming of existing files.

File read/write routines (Python)

2010-04-18T17:18:50-07:00

Python recipe 577199 by Denis Barmenkov (binary, file, read, text, write). Revision 2.

For small dirty hacks in Perl has a module File::Slurp. I wrote two simple functions when I moving from Perl to Python - one for reading files and second for writing files. Valuable data is list of lines or blob (additionally specified argument binmode=1).

Fix mbox files after importing EML into TB using ImportExportTools (Python)

2010-05-02T13:21:00-07:00

Python recipe 577214 by Denis Barmenkov (eml, from, import, importexporttools, mbox, tb, thunderbird). Revision 2.

I've found a bug in import EML file into Thunderbird using ImportExportTools addon: when I import eml file into TB there are a 'From' line added to mbox followed with EML file contents. TB maintains right 'From' line for messages fetched from mailservers:

From - Tue Apr 27 19:42:22 2010

ImportExportTools formats this line wrong I suppose that used some system function with default specifier so I saw in mbox file:

From - Sat May 01 2010 15:07:31 GMT+0400 (Russian Daylight Time)

So there are two errors: 1) sequence 'time year' broken into 'year time' 2) extra trash with GMT info along with time zone name

This prevents the mbox file parsing using Python standard library (for sample) because there are a hardcoded regexp for matching From line (file lib/mailbox.py, class UnixMailbox):

_fromlinepattern = r"From \s*[^\s]+\s+\w\w\w\s+\w\w\w\s+\d?\d\s+" \
                   r"\d?\d:\d\d(:\d\d)?(\s+[^\s]+)?\s+\d\d\d\d\s*$"

Attached script fixes incorrect From lines so parsing those mboxes using Python standard library will become ok.

Modified os.walk which return current directory depth (Python)

2009-11-06T16:16:42-08:00

Python recipe 576933 by Denis Barmenkov (directory_depth, os_walk, relative_path). Revision 2.

On some task I need to collect file names under specified directory with distance from it. Standard os.walk function do not return depth value.

One solution -- find function which will calculate relative distance from top directory to file.

Another [presented] solution -- modify os.walk so it returns depth level as fourth tuple's value.

Align text filter (Python)

2010-03-09T07:34:43-08:00

Python recipe 577093 by Denis Barmenkov (align, pipe, text, wrap). Revision 4.

Python 2.3 module textwrap can justify text in three modes: left/right/center. Sometimes 'align' mode become more useful.

Precise console progress meter with ETA calculation (Python)

2010-04-18T21:11:40-07:00

Python recipe 577002 by Denis Barmenkov (console, eta, meter, progress). Revision 13.

After several attempts to use third-party modules I wrote my own console progress meter.

Bonus list:

calculation of ETA based on last update points. More accuracy when comparing with calculation ETA based on process start time (process can survive after Hibernate, but ETA has lost his accuracy)
ability to write progress meter to sys.stderr
update_left() method for multithreaded programs :)

One-word logging.basicConfig wrapper (Python)

2010-01-06T06:54:09-08:00

Python recipe 576994 by Denis Barmenkov (basicconfig, logging, wrapper). Revision 2.

Every Python logging manual have this code:

logging.basicConfig(level=logging.DEBUG, filename='debug.log',
                    format='%(asctime)s %(levelname)s: %(message)s',
                    datefmt='%Y-%m-%d %H:%M:%S')

This is a simple function which call it for you. All you need is remember one function name; useful on little scripts.

Patch extension binaries compiled for previous (different) Python version (Python)

2010-03-12T15:50:04-08:00

Python recipe 576873 by Denis Barmenkov (binaries, dll, patch, precompiled, pyd, windows). Revision 4.

It is helpful to download and install precompiled binaries of Python modules especially for Windows. Second way is longer: get module sources, build the C compiler environment, compile binaries, beware of compiler warnings, install module.

Often precompiled binaries can be patched to meet used Python version.

See 'discussions' below for details.

simple samefile() for windows (local drive case only) (Python)

2009-09-09T02:26:35-07:00

Python recipe 576886 by Denis Barmenkov (os_path_samefile, samefile, win32, windows). Revision 4.

A simple replacement of the os.path.samefile() function not existing on the Windows platform. MAC/Unix supported in standard way :).

EXIF-date-based JPEG files rename using PIL (Python)

2009-02-10T07:28:48-08:00

Python recipe 576646 by Denis Barmenkov (date, exif, jfif, jpeg, jpg, photo, pil, rename). Revision 4.

Rename JPEG files according to EXIF-date using PIL [library].

If global variable CREATE_HARDLINK is set, script creates Windows (XP) batch file for creating hardlink version of source files.

PIL available here: http://www.pythonware.com/products/pil/

binascii.crc32 result to hex (Python)

2007-06-14T06:25:57-07:00

Python recipe 521907 by Denis Barmenkov (algorithms).

Function becomes useful when we need hex value of crc32() result, even if it is negative.

print statement: battle with UnicodeEncodeError (Python)

2009-02-10T07:22:46-08:00

Python recipe 523011 by Denis Barmenkov (sysadmin). Revision 5.

I received UnicodeEncodeError when playing with various codepages in source code/files/standard streams. Sometime I receive UnicodeEncodeError when script launched via scheduler or in long running batch when parsing unpredictable [alien ;)] HTML.

Function console() helps avoid this exceptions by converting erroneous charatcters to standard python representation.

to do in future: make a codec-wrapper for safe using in statements like this:

sys.stdout=codecs.getwriter('cp866')(sys.stdout)

Saving [logging.] disk log fragment into separated file (Python)

2007-05-08T07:26:44-07:00

Python recipe 519624 by Denis Barmenkov (sysadmin).

Sometime I want extract part of disk-based log file, created by standard logging module) into separated file on disk. This recipe shows simple technique to acquire this.

Sample: processing tasks in loop, so on exit I have few logs: on log per task.

How to disablе debug logging in release version (Python)

2006-07-12T08:34:36-07:00

Python recipe 496870 by Denis Barmenkov (debugging). Revision 3.

When you release your program to client, its a good idea to disable all the debug messages. It is possible via custom configuring debug levels at all modules, but may be implemented using simple wrapper around logging.getLogger() function.

Getting SYSTEM environment variable under Windows (Python)

2009-02-10T07:24:02-08:00

Python recipe 473800 by Denis Barmenkov (sysadmin). Revision 2.

As you can see in Windows Control Panel 'System' applet there are two groups of environment variables: USER and SYSTEM. Here presents function for retrieve SYSTEM variable value.

Align text string using spaces between words to fit specified width (Python)

2010-03-09T07:04:19-08:00

Python recipe 414870 by Denis Barmenkov (text). Revision 5.

Python 2.3 module textwrap can justify text in three modes: left/right/center. Sometimes 'align' mode become more useful.