Popular Python recipes tagged "compression"

Text Compressor 3.1 (Python)

2010-10-20T00:50:15-07:00

Python recipe 577433 by Stephen Chappell (compression, encode, encryption).

Compression, encryption, and data codecs are all related fields that most programmers will use ready-made solutions for. This recipe is a shallow adventure into the writing of original code and algorithms that explores a combination of those topics. Based on the work of recipe 502202, the code here is compliant with Python 3.1 and will run a test of itself when executed. From the program's report, one can gather that the novel procedures compress the source and accurately decompress it again. For those who wish to experiment further with the concept, note that fewer unique characters will yield higher compression ratios.

Creating a tar archive (without using the tarfile module) (Python)

2010-10-11T06:18:42-07:00

Python recipe 577422 by Benjamin Sergeant (compression, tar, unix).

Creating a tar file is easy if you read the spec (you can look it up on wikipedia). Not every kind of files are supported (it support regular files, folders ans symlinks) and it's generating archives for the original tar file format (path length are limited to 100 chars, no extended attributes, ...). It wasn't tested very much but it was a fun hack :) ... I cheated just a little by looking at the python tarfile code from the stdlib for the checksum computation.

A tar file is very simple, it's a list of header/payload for each entry (file|folder|symlink) you want to archive. There's only a payload for file contents. The header is 512 bytes long and can be written in ascii. Numbers (attributes) needs to be written in octal. The files themselves needs to be written in chunks of 512 bytes, which mean you have to fill the last chunk with zeros when the file size is not a multiple of 512 bytes.

Use it like that:

python batar.py /tmp/foo.tar `find .` &&  tar tf /tmp/foo.tar # or xf if you want to extract it

Python code minifier (Python)

2014-05-25T16:23:55-07:00

Python recipe 576704 by Dan McDougall (bz2, bzip2, comments, compression, docstring, embedded, gzip, minify, pack, regex, zlib). Revision 16.

Update 05/25/2014: Pyminifier 2.0 has been released and now lives on Github: https://github.com/liftoff/pyminifier (docs are here: http://liftoff.github.io/pyminifier/). The code below is very out-of-date but will be left alone for historical purposes.

Python Minifier: Reduces the size of Python code for use on embedded platforms. Performs the following:

Removes docstrings.
Removes comments.
Removes blank lines.
Minimizes code indentation.
Joins multiline pairs of parentheses, braces, and brackets (and removes extraneous whitespace within).
Preserves shebangs and encoding info (e.g. "# -- coding: utf-8 --")
NEW: Optionally, produces a bzip2 or gzip-compressed self-extracting python script containing the minified source for ultimate minification.

Update 09/23/2010: Version 1.4.1: Fixed an indentation bug when operators such as @ and open parens started a line.

Update 09/18/2010: Version 1.4:

Added some command line options to save the result to an output file.
Added the ability to save the result as a bzip2 or gzip-compressed self-extracting python script (which is kinda neat--try it!).
Updated some of the docstrings to provide more examples of what each function does.

Update 06/02/2010: Version 1.3: Rewrote several functions to use Python's built-in tokenizer module (which I just discovered despite being in Python since version 2.2). This negated the requirement for pyparsing and improved performance by an order of magnitude. It also fixed some pretty serious bugs with dedent() and reduce_operators().

PLEASE POST A COMMENT IF YOU ENCOUNTER A BUG!

Huffman coding, Encoder/Deconder (Python)

2009-01-04T04:11:37-08:00

Python recipe 576603 by Shao-chuan Wang (algorithm, compression, decompression, huffman_code). Revision 2.

Please refer to wikipedia: http://en.wikipedia.org/wiki/Huffman_coding

Huffman coding is an entropy encoding algorithm used for lossless data compression. The term refers to the use of a variable-length code table for encoding a source symbol (such as a character in a file) where the variable-length code table has been derived in a particular way based on the estimated probability of occurrence for each possible value of the source symbol. It was developed by David A. Huffman while he was a Ph.D. student at MIT, and published in the 1952 paper "A Method for the Construction of Minimum-Redundancy Codes".

Serve static web content from within a gzipped tarball to save space using CherryPy (Python)

2009-03-31T18:24:06-07:00

Python recipe 576706 by Dan McDougall (cherrypy, compression, embedded, gzip, html, http, network, routes, web, web_server).

This code lets you store all of your static website content inside a gzipped tarball while transparently serving it to HTTP clients on-demand. Perfect for embedded systems where space is limited.