Popular Python recipes tagged "meta:requires=struct"

DBF reader and writer -- selective fields and nullreplace (Python)

2016-09-18T20:39:20-07:00

Python recipe 580696 by Tomas Nordin (database).

This fork assumes a desire for limited selection of field names. With huge files this might be necessary on some machines.

Also, assuming that the meaning of null in a dbf file means zero might be a mistake, so the fork adds an argument nullreplace as way to choose what to replace null with. Null is sometimes used to mean missing value. This change is decoupled from the selective names feature.

Examples for random float between 0 and 1 (Python)

2015-06-25T17:55:33-07:00

Python recipe 579072 by Stephen Chappell (example, random).

I was researching how floats are stored in a computer and was trying to think up a way to generate random values between 0 and 1. Python already provides an implementation allowing this already, and two of the functions below are directly inspired by that code, but the third is a slightly different way of doing the same thing. A similar version of this code has been used to implement similar functionality in C# at one time. Others might also find it useful if they want to create equivalent code in a separate language while having access to random bytes but not random floats. It should be noted that the various implementations get slower as you go down the list.

Drop a minimal, valid Windows executable file to disk, for testing (Python)

2015-03-03T10:47:32-08:00

Python recipe 579029 by Simon Harrison (executable, pe, windows). Revision 3.

Sometimes I need to create a valid windows executable file from a Python script for the sake of running a test.

WebSocket interface (Python)

2012-11-25T16:52:21-08:00

Python recipe 578348 by Nick Faro (javascript, network, socket, websocket). Revision 2.

This tries its best to be a replacement for the regular socket module.

It supports only sending and receiving but should be useful enough.

The only real difference should be that you can't specify the number of bytes is received, instead do

for message in socket.recv():
    print(message)

Revision 2: Added proper message receiving. Previously it just requested a ton of data. Now it reads 2 bytes, determines the length, then requests that much.

Python reader + writer for spss sav files (Linux, Mac & Windows) (Python)

2013-02-20T22:07:27-08:00

Python recipe 577811 by Albert-Jan Roskam (reader, sav, spss, writer). Revision 12.

Python Program to READ & WRITE Spss system files (.sav) (Linux,Mac or Windows)

Check https://pypi.python.org/pypi/savReaderWriter/ for the latest version (including the libraries!)
Requires libspssdio.so.1 (LINUX) or libspssdio.dylib (MAC) or spssio32.dll (WINDOWS) plus associated libaries, which can be freely downloaded from: https://www.ibm.com/developerworks/mydeveloperworks/wikis/home/wiki/We70df3195ec8_4f95_9773_42e448fa9029/page/Downloads%20for%20IBM%C2%AE%20SPSS%C2%AE%20Statistics?lang=en
It is recommended to download the v21 I/O files (required for writing zlib (.zsav) compressed files)
December 2012 (complete rewrite):
Added support for slicing, indexing, array slicing + other special methods
Added support for writing spss date fields
Added support for almost all meta data (missing values, sets, roles, etc.)
Added support for 64 bit Windows (tested with Win7) and other OSs (z/Linux, Solaris, HP Linux, IBM AIX (untested though)
Added support for reading and writing zlib compressed (.zsav) files
Removed pesky segfault error when freeing memory
Removed errors related to encoding
Changed some Reader defaults (verbose=False, returnHeader=False)
Renamed SavDataDictionaryReader into SavHeaderReader

LINUX:

Installation (tested on Linux Ubuntu 10):

additional packages/files needed are: intel-icc8-libs_8.0-1_i386.deb,libicu32_3.2-3_i386.deb, libstdc++5_3.3.6-20_i386.deb, libirc.so.
Run the following commands in your terminal: sudo apt-get install intel-icc8-libs; sudo apt-get install libicu32; sudo apt-get install libstdc++5.
Then convert libirc.a (static) to libirc.so (dynamic), save in same location as libspssdio.so.1: ar vx intel-icc8-libs_8.0-1_i386.deb; tar -xzvf data.tar.gz ./usr/lib/libirc.a; ar -x libirc.a.

Calling the program:

In the TERMINAL type: export LD_LIBRARY_PATH=/path/of/additional/sofiles; python /some/path/wrapperToProgram.py. You may also add ld_library_path to .bashrc
The wrapper starts with "from SavReaderWriter import *", followed by e.g. stuff from the if __name__ == '__main__' section

MAC OS:

you must put all the dylib files that come with the IBM SPSS_Statistics_InputOutput_Modules_* package in the macos directory somewhere that OS X can find them
one simple way to accomplish this is to copy them to /usr/lib

WINDOWS:

You can also find this dll in the installation directory of SPSS (although SPSS is _not_ needed!)
The .dll should be saved in the same location as this program.

USAGE: See docstrings + __main__ section

STL Writer (Python)

2012-08-22T18:46:34-07:00

Python recipe 578246 by Manfred Moitzi (3d).

Export 3D objects, build of faces with 3 or 4 vertices, as ASCII or Binary STL (Standard Triangulation Language) file.

For AMIGA-Heads Only. Peeking A Memory Address On A Classic AMIGA, Inside Text Mode Python. (Python)

2012-07-03T17:00:20-07:00

Python recipe 578189 by Barry Walker (amiga, demo, e_uae, peek, peeking, winuae).

For AMIGA-Heads Only...

This code is purely for a minimum of a stock Classic AMIGA A1200(HD), E-UAE, and WinUAE running AMIGA OS3.0x to 3.1x. It will NOT work on any other platform!

(It MIGHT work on AMIGA OS3.5x and 3.9x with Python 2.4.6, but totally untested and I hold no responsibility for the outcome when trying it.)

It is a DEMO to peek a byte inside _ANY_ memory address inside the standard CPU 68EC020 16MB boundary.

I experimented with this several years ago and have decided to finally release it. I do await the flak!

It works from Python Versions 1.4.0 to 2.0.1 for the classic AMIGA.

Ensure that the T: Volume exists on the running machine.

See inside the code and text below it also as to how it works...

Bazza, G0LCU...

How to Mutate a Float (Python)

2012-05-14T16:49:13-07:00

Python recipe 578132 by Stephen Chappell (ctypes, experiment, float, structure).

This is just an experiment to test my mental understanding of Python. It should not be used in any code as is violates the design principle that floats are to be immutable. The code also abuses ctypes and an understanding of how floats are currently arranged in memory. set_float is not guaranteed to work properly on any system, and may fail to work in the future if the data's arrangement changes.

Self-contained TWL06 Dictionary Module (515 KB) (Python)

2011-08-10T20:32:03-07:00

Python recipe 577835 by Michael Fogleman (dawg, dictionary, scrabble, trie, twl06, word). Revision 4.

A convenient, self-contained, 515 KB Scrabble dictionary module, ideal for use in word games.

Functionality:

Check if a word is in the dictionary.
Enumerate all words in the dictionary.
Determine what letters may appear after a given prefix.
Determine what words can be formed by anagramming a set of letters.

Sample usage:

>>> import twl
>>> twl.check('dog')
True
>>> twl.check('dgo')
False
>>> words = set(twl.iterator())
>>> len(words)
178691
>>> twl.children('dude')
['$', 'd', 'e', 's']
>>> list(twl.anagram('top'))
['op', 'opt', 'pot', 'to', 'top']

DistCC 'top' (Python)

2013-12-01T11:19:56-08:00

Python recipe 577933 by Mike 'Fuzzy' Partin (color, curses, distcc, distributed, monitor, pack, struct, top, unpack). Revision 3.

A small recipe for a curses based, 'top'-like monitor for DistCC. I know there is already distccmon-text, but I don't like it, and much prefer this sytle of monitoring. Note that I don't keep hosts around in the list like distccmon-gui/gnome. The screen is drawn for exactly what is currently in state. The terminal size is respected at initialization time, however resize events aren't handled. There is color designation of job types.

Decoding Binary Files (Python)

2011-03-15T00:11:10-07:00

Python recipe 577610 by Yony Kochinski (binary, decode, file, python). Revision 2.

One way to read files that contain binary fields is to use the struct module. However, to do this properly one must learn struct's format characters, which may look especially cryptic when sprinkled around the code. So instead, I use a wrapper object that presents a simple interface as well as type names that are more inline with many IDLs.

Python reader for spss sav files (Python)

2011-07-25T18:59:49-07:00

Python recipe 577650 by Albert-Jan Roskam (files, pasw, sav, spss, statistics). Revision 3.

Python Program to read Spss system files (.sav)

* version 2 (FASTER!) *
Requires spssio32.dll, which can be freely downloaded from: https://www.ibm.com/developerworks/mydeveloperworks/wikis/home/wiki/We70df3195ec8_4f95_9773_42e448fa9029/page/Downloads%20for%20IBM%C2%AE%20SPSS%C2%AE%20Statistics?lang=en
You can also find this dll in the installation directory of SPSS (although SPSS is _not_ needed!)
The .dll should be saved in the same location as this program.
Parameters: savFileName: the file name of the spss data file; returnHeader: Boolean that indicates whether the first record should be a list of variable names (default: True); recodeSysmisTo: indicates to which value missing values should be recoded (default: ""); selectVars: indicates which variables in the file should be selected.The variables should be specified as a list or a tuple of valid variable names. If None is specified, all variables in the file are used (default: None); verbose: Boolean that indicates whether information about the spss data file (e.g., number of cases, variable names, file size) should be printed on the screen (default: True).
rawMode: Boolean that indicates whether values should get SPSS-style formatting,and whether date variables (if present) should be converted to ISO-dates. If True, the program does not format any values, which increases processing speed. (default: = False) interfaceEncoding Indicates the mode in which text communicated to or from the I/O Module will be. Valid values are 'UTF-8' or 'CODEPAGE' (default = 'CODEPAGE')
Typical use: savFileName = "d:/someFile.sav" with SavReader(savFileName) as sav: header = sav.next() for line in sav: process(line)
Note: --New version: If you downloaded this previously, use the current version as it is MUCH faster!! --this code currently only works on Windows (32 bits). I might make it work on Linux Ubuntu 10 at some point. --date fields in spss are represented as the number of seconds since the Gregorian calendar. The program converts these, wherever possible, to ISO-dates (yyyy-mm-dd).

Any feedback is welcome! I'm still learning!

NBD server in python (Python)

2011-02-08T20:59:23-08:00

Python recipe 577569 by Dima Tisnek (analyze, block, device, disk, io, linux, nbd, network, python).

Linux Network Block Device server in Python

This is a simplified version based on Kragen Sitaker's http://lists.canonical.org/pipermail/kragen-hacks/2004-May/000397.html

Close is never actually called, at least not on the same connection -- linux C nbd-client -d seems to stall, perhaps it tries to open another socket?

This code doesn't check for error conditions, failed reads/writes, past end of disk, etc.

It prints io requests, you can analyze filesystem and user program io patterns.

LoggingWebMonitor - a central logging server and monitor. (Python)

2010-02-02T01:56:42-08:00

Python recipe 577025 by Gabriel Genellina (client_server, debugging, distributed, logging, remote, sysadmin, web). Revision 3.

LoggingWebMonitor listens for log records sent from other processes running in the same box or network. Collects and saves them concurrently in a log file. Shows a summary web page with the latest N records received.

Size of Python objects (revised). (Python)

2017-07-22T15:52:01-07:00

Python recipe 546530 by Jean Brouwers (getsizeof, memory, sizeof). Revision 21.

This recipe determines the size of Python objects in bytes and has been further enhanced to handle ints, namedtuples, arrays and NumPy types better. Functions alen and itemsize have been updated. Support for Python 2.5 and earlier and the tests/examples have been removed. See project Pympler for unit tests.

See also other, simpler recipes like this Compute memory footprint of an object and its contents.

IP and MAC addresses (Python)

2016-07-07T17:52:15-07:00

Python recipe 577191 by Jean Brouwers (ios, ip, ipv4, mac, macos, network, python2_4, python3_5, uuid, windows). Revision 3.

This module collects all IP and MAC addresses from several available sources on the underlying system. See the module documentation for more details, supported Python releases and platforms.

Pyscanlogger - Python Port scan detector (Python)

2010-03-17T07:27:15-07:00

Python recipe 576690 by Anand (network, programs, security). Revision 5.

A pure Python program to detect network port scanning attacks. Currently logs different TCP port scans. Can run in the background like a daemon and log attacks to a log file.

socket.recv -- three ways to turn it into recvall (Python)

2005-04-05T13:47:57-07:00

Python recipe 408859 by John Nielsen (network). Revision 5.

An issue with socket.recv is how to know when you are done receiving data. A TCP stream guarantees the bytes will not arrive out of order or be sent more than once. But you do not know the size of the data that will be sent to you. 100 bytes could be sent as group of 10 bytes or maybe in one shot. Ultimately, this means you have to use a loop in some fashion until you know it is done.

The basic recv returns an empty string when the socket is disconnected. From that you can build a simple loop that will work as long as the sender manages to disconnect the socket at the appropriate time. However, there could be situations where a local error will mask as a clean shutdown or maybe a close() is never called.

Three very basic methods are shown below that try to fix that problem. They use either a time-based, end marker, or size of payload method. Since you cannot be sure just what you are going to receive, you have to be careful that you get enough of a message to determine the size of payload or end marker.

I updated the recv_size method to allocate data in larger chunks if it gets a large stream of data, which can increase performance.

Rename MP3 files from ID3 tags (does not require external ID3 modules) (Python)

2011-11-05T15:27:32-07:00

Python recipe 576811 by ccpizza (id3, mp3, rename). Revision 5.

Rename MP3 files in the current folder according to ID3 tags. This is based on Ned Batchelder's id3reader class. I only added the code in the __main__ method.

When run without arguments, the script renames the files in the current folder. The directory can be specified explicitly as the first argument.
The files do not need to necessarily have the MP3 extension.
To move the files to directories based on album name use the -d switch.
Does not work with other file types, such as OGG, M4A, MP4, etc.
This is a quick and dirty script that works for me most of the times. If you need more power, try mutagen, picard, mp3tag or something similar.

Interrogating linux /dev/usb/hiddev0 in python (Python)

2009-07-07T01:32:17-07:00

Python recipe 576834 by Dima Tisnek (hid, hiddev, linux, usb).

What this recipe does:

Maps linux usb hid ioctls and related C structs to python; Call ioctls, make some sense of output. Prints all reports for the device with some info.

Works with python 2.4 (tested python 2.4.6 on linux amd64). Would need changes (e.g. print) for python 3.0. Might need changes (ioctl signed/unsigned "FIX") for newer python than tested.