Get columns of data from text files (Python)

2010-10-28T16:18:19-07:00

Python recipe 577444 by alinium (columns, file, parsing).

Read in a tab-delimited (or any separator-delimited like CSV) file and store each column in a list that can be referenced from a dictionary. The keys for the dictionary are the headings for the columns (if any). All data is read in as strings.

Decoding Binary Files (Python)

2011-03-15T00:11:10-07:00

Python recipe 577610 by Yony Kochinski (binary, decode, file, python). Revision 2.

One way to read files that contain binary fields is to use the struct module. However, to do this properly one must learn struct's format characters, which may look especially cryptic when sprinkled around the code. So instead, I use a wrapper object that presents a simple interface as well as type names that are more inline with many IDLs.

Composing a POSTable HTTP request with multipart/form-data Content-Type to simulate a form/file upload. (Python)

2014-03-08T17:34:38-08:00

Python recipe 578846 by István Pásztor (field, file, form, html, httpclient, mime, multipart, post, upload, web). Revision 5.

This code is useful if you are using a http client and you want to simulate a request similar to that of a browser that submits a form containing several input fields (including file upload fields). I've used this with python 2.x.

Find file in subdirectory (Python)

2010-02-02T11:35:53-08:00

Python recipe 577027 by Daniel Cohn (file, find, subdirectory). Revision 2.

Walks subdirectories to find a file and returns . Default start location is the current working directory. Optionally, a different directory can be set as the search's starting location.

Sorting big files the Python 2.6 way (Python)

2009-05-30T21:51:09-07:00

Python recipe 576755 by Gabriel Genellina (binary, file, sort, sorting, text). Revision 3.

This is just a rewrite of Recipe 466302 "Sorting big files the Python 2.4 way", taking advantage of heapq.merge, context managers, and other niceties of newer Python versions. It can be used to sort very large files (millions of records) in Python. No record termination character is required, hence a record may contain embedded binary data, newlines, etc. You can specify how many temporary files to use and where they are located.

Convert HTML text to PDF with Beautiful Soup and xtopdf (Python)

2015-01-28T22:20:53-08:00

Python recipe 579014 by Vasudev Ram (file, format, pdf, pdfwriter, python, reportgeneration, reporting, reportlab, xtopdf).

This recipe shows how to convert the text in an HTML document to PDF. It uses the Beautiful Soup and xtopdf Python libraries. Beautiful Soup is a library for HTML parsing and content extraction. xtopdf is a library for PDF creation from other formats, including text and many others.

Multiprocess-safe logging file-handler + interprocess RLock (Python)

2010-09-22T17:30:10-07:00

Python recipe 577395 by Jan Kaliszewski (concurrency, file, file_lock, flock, handler, logging, process, rlock, threading, threadsafe). Revision 11.

A Python 2.x/3.x-compatibile multiprocess-safe logging file-handler (logging.FileHandler replacement, designed for logging to a single file from multiple independent processes) together with a simple interprocess recursive lock -- universal abstract classes + Unix/Linux implementation.

Update: It's is a deeply revised version. Especially, now it --

is Python 2.4, 2.5, 2.6, 3.1 -compatibile (previously Py>=2.6 was needed); probably works also with 2.7, 3.0 and 3.2 (but not tested if it does);
is multiprocess-safe as well as thread-safe (proviously thread safety within a process was missed);
is based on public interfaces only (previously FileHandler._open() was called and overriden);
implement full RLock instance interface, as documented for threading.RLock (previously non-blocking mode and context-manager interface were missing).

The module contains:

Unix/Linux-only example implementation (with flock-based locking): FLockRLock and FLockFileHandler classes.
universal abstract classes -- which may be useful at developing implementation for non-Unix platforms: MultiprocessRLock, MultiprocessFileHandler, LockedFileHandler,

Also a quick-and-dirty test was added.

It is still an alpha version -- I'll be very grateful for any feedback.

Further updates:

2010-09-20: Some corrections, especially: non-blocking mode bug in MultiprocessRLock.acquire() fixed; _test() function improved; plus fixes in the description below.
2010-09-22: _test() improved and moved to description section. Mistaken copyright-notice removed.

Safely and atomically write to a file (Python)

2016-03-23T14:14:26-07:00

Python recipe 579097 by Steven D'Aprano (atomic, file, save). Revision 3.

Saving the user's data is risky. If you write to a file directly, and an error occurs during the write, you may corrupt the file and lose the user's data. One approach to prevent this is to write to a temporary file, then only when you know the file has been written successfully, over-write the original. This function returns a context manager which can make this more convenient.

Update: this now uses os.replace when available, and hopefully will work better on Windows.

Searching .dll and .exe files in PATH (Python)

2014-10-06T09:22:53-07:00

Python recipe 576522 by Michal Niklas (dll, exe, file, version, windows). Revision 7.

Returns the pathnames of the file (.exe or .dll) which would be loaded/executed in the current environment. It uses some dirs from configuration (SystemDir, WindowsDir) and dirs from PATH.

To obtain version info it uses code from: http://pywin32.hg.sourceforge.net/hgweb/pywin32/pywin32/file/tip/win32/Demos/getfilever.py

Example of usage:

    c:\tools\pyscripts\scripts>which_dll.py libpq.dll
    2008-06-09 02:58:26   167936 [b]    c:\postgresql\8.3\bin\libpq.dll ver:8.3.3.8160
    2008-03-17 01:47:50   167936 [b]    c:\tools\libpq.dll  ver:8.3.1.8075
    2008-03-17 01:47:50   167936 [b]    g:\public\libpq.dll ver:8.3.1.8075
        trying to load "libpq.dll" ...
        c:\postgresql\8.3\bin\libpq.dll loaded

Tail a continuously growing file (like tail -f filename) (Python)

2011-05-20T16:32:59-07:00

Python recipe 577710 by Jason Morriss (file, rotate, tail).

This is my version of a "File Tail" class for Python3 (will not work on Python2 w/o a couple of modifications). My original inspiration came from the perl File::Tail module.

Transparently handles files that get rotated or truncated.

Does not take 100% CPU.
Does not take up much memory.
Is capable of handling any size log file.
Not tested on Windows

Example:

from filetail import FileTail
tail = FileTail("/var/log/syslog")
for line in tail:
    print(line, end="")

fcntl.flock() (Unix file lock) behaviour sampling script (Python)

2010-09-22T00:11:12-07:00

Python recipe 577404 by Jan Kaliszewski (file, file_descriptor, file_lock, flock, process, thread). Revision 3.

A quick fcntl.flock(fcntl.LOCK_EX | fcntl.LOCK_NB) call sampling script: with one file object (and descriptor) or separate file objects (and different descriptors) pointing to the same filesystem path -- with/without threading or forking.

It's rather exemplum-and-educational piece of code than utility-script, unless somebody has to few slots in their memory to remember that flock is file-descriptor-tacked (then quick run of the script can save web-searching) :)

File browser for tkinter using idle GUI library (Python)

2017-04-03T13:37:34-07:00

Python recipe 580772 by Miguel Martínez López (browser, file, idlelib, tkinter).

Idle is installed by default on windows.

For Ubuntu, Linux Mint and Debian run:

  sudo apt-get install idle-python2.7

A tree structure is drawn on a Tkinter Canvas object. A tree item is an object with an icon and a text. The item maybe be expandable and/or editable. A tree item has two kind of icons: A normal icon and an icon when the item is selected. To create the tree structure, it's necessary to create a link between tree items, using a parent-child relationship.

The canvas is built using a idlelib.TreeWidget.ScrolledCanvas class. The frame attribute of this object contains a Frame Tkinter widget prepared for scrolling. This frame allocates a Tkinter Canvas and Tkinter Scrollbars. This is the signature:

   ScrolledCanvas(master, **options_for_the_tkinter_canvas)

It accepts exactly the same arguments than a Canvas widget.

A tree item should be a subclass of idlelib.TreeWidget.TreeItem.

The parent-child relationship between tree items is established using the idlelib.TreeWidget.TreeNode class.

This is the signature for TreeNode(canvas, parent, item):

canvas should be a ScrolledCanvas instance.
parent should be the parent item. Leave that to None to create a root node.
item should be the child item

FileTreeItem is a type of TreeItem. The only argument of a file tree item is a path.

Here there is an example of a custom tree item:

https://code.activestate.com/recipes/579077-bookmarks-browser-for-firefox/

To create your own tree items, it's required to subclass TreeItem. These are the methods that should be overrided:

GetText: It should return the text string to display.
GetLabelText: It should return label text string to display in front of text (Optional).
IsExpandable: It should return a boolean indicating whether the istem is expandable
IsEditable: It should return a boolean indicating whether the item's text may be edited.
SetText: Get the text to change if the item is is editable
GetIconName: Return name of icon to be displayed normally (Icons should be included in ICONDIR directory)
GetSelectedIconName: Return name of icon to be displayed when selected (Icons should be included in ICONDIR directory).
GetSubList: It should return a list of child items (Optional. If not defined, the element is not expandable)
OnDoubleClick: Called on a double-click on the item. (Optional)

Icons should be included in "Icons" subdirectory of path to idlelib library. If you want to use other path, just change ICONDIR variable to path to your icons:

 import idlelib
 idlelib.ICONDIR = "Your path to your icons"

Run this to find the path to idlelib module:

 python -c "import idlelib; print(idlelib.__file__)"

Python Awesome DD (Python)

2014-07-12T07:29:59-07:00

Python recipe 578907 by Mike 'Fuzzy' Partin (curses, dd, file, input, output, progress_bar). Revision 3.

Imagine if the *nix utilities 'dd', 'wget', and 'pv' had a baby. That baby would be named 'padd'. A 'dd' replacement (most commonly used options translate with no problem), that gives progress output like 'pv' and supports (currently) http(s) and ftp input sources, with more planned (sftp for one). Output support for ftp, sftp, and http(s) is also planned.

File comparison utility in Python (Python)

2016-03-26T18:31:11-07:00

Python recipe 580631 by Vasudev Ram (commandline, comparison, file, files, linux, python, unix, utilities).

This is a recipe to compare any two files via a Python command-line program. It is like a basic version of the cmp command of Unix or the fc.exe (file compare) command of Windows.

Storing a Single Username and Encrypted Password to a File (Python)

2013-01-31T02:07:25-08:00

Python recipe 578443 by Captain DeadBones (encryption, file, python).

This script was written for an article I wrote to check user passwords. It stores data using SHA one-way function. For more information Authentication Of Users And Passwords In Python

Make unique file name (Python)

2010-04-18T21:07:52-07:00

Python recipe 577200 by Denis Barmenkov (file, name, rename). Revision 2.

Sometimes it is important to save data in the file but the file with the specified name already exists. This function creates a file name that is similar to the original by adding a unique numeric suffix. This avoids the renaming of existing files.

Python one-liner to compare two files (Python)

2016-03-28T21:36:31-07:00

Python recipe 580632 by Vasudev Ram (comparison, file, files, programming, python).

As the title says ...

It prints True if the files compared are the same, and False if they differ (either in size or in content).

Find Duplicate Files (Python)

2014-10-12T21:14:05-07:00

Python recipe 578950 by FB36 (dictionary, directories, directory, disk, file, files, utility).

Finds duplicate files which have same size and same content in the same directory or two different directories.

Sort sections and keys in .ini file (Python)

2008-12-19T05:46:05-08:00

Python recipe 576587 by Michal Niklas (config, file, ini, sort). Revision 3.

I use this program when I want to make .ini file more readable or compare two .ini files

File read/write routines (Python)

2010-04-18T17:18:50-07:00

Python recipe 577199 by Denis Barmenkov (binary, file, read, text, write). Revision 2.

For small dirty hacks in Perl has a module File::Slurp. I wrote two simple functions when I moving from Perl to Python - one for reading files and second for writing files. Valuable data is list of lines or blob (additionally specified argument binmode=1).

Most viewed recipes tagged "file"