Popular Python recipes tagged "meta:requires=tokenize"

Colorize Python -- Sourcecode Syntax Highlighting (Python)

2012-07-21T02:46:37-07:00

Python recipe 578178 by Raymond Hettinger (highlighting, syntax). Revision 20.

Generates colorized HTML, ANSI escaped text, or a LaTeX document from Python source code. Useful for publishing or viewing your code in a more readable way.

Python code minifier (Python)

2014-05-25T16:23:55-07:00

Python recipe 576704 by Dan McDougall (bz2, bzip2, comments, compression, docstring, embedded, gzip, minify, pack, regex, zlib). Revision 16.

Update 05/25/2014: Pyminifier 2.0 has been released and now lives on Github: https://github.com/liftoff/pyminifier (docs are here: http://liftoff.github.io/pyminifier/). The code below is very out-of-date but will be left alone for historical purposes.

Python Minifier: Reduces the size of Python code for use on embedded platforms. Performs the following:

Removes docstrings.
Removes comments.
Removes blank lines.
Minimizes code indentation.
Joins multiline pairs of parentheses, braces, and brackets (and removes extraneous whitespace within).
Preserves shebangs and encoding info (e.g. "# -- coding: utf-8 --")
NEW: Optionally, produces a bzip2 or gzip-compressed self-extracting python script containing the minified source for ultimate minification.

Update 09/23/2010: Version 1.4.1: Fixed an indentation bug when operators such as @ and open parens started a line.

Update 09/18/2010: Version 1.4:

Added some command line options to save the result to an output file.
Added the ability to save the result as a bzip2 or gzip-compressed self-extracting python script (which is kinda neat--try it!).
Updated some of the docstrings to provide more examples of what each function does.

Update 06/02/2010: Version 1.3: Rewrote several functions to use Python's built-in tokenizer module (which I just discovered despite being in Python since version 2.2). This negated the requirement for pyparsing and improved performance by an order of magnitude. It also fixed some pretty serious bugs with dedent() and reduce_operators().

PLEASE POST A COMMENT IF YOU ENCOUNTER A BUG!

typeparser (Python)

2007-04-15T04:24:53-07:00

Python recipe 511473 by Florian Leitner (algorithms). Revision 2.

Python type-string parser. The code evolved from a post in python-list on 11/22/05 by Fredrik Lundh on a dictionary parser. It parses a type-string to their type objects for all basic types. Raises SyntaxError and SemanticError on failures.

Supported types: * containers: defaultdict, deque, dict, list, tuple, set * basic types: Decimal, bool, float, int, long, str * None type

REQUIRES PYTHON >= 2.5

Convert a formula string with implied multiplication to proper form (Python)

2006-09-16T06:50:04-07:00

Python recipe 498106 by Ernesto Adorio .

This recipe will insert multiplication symbols and convert all grouping symbols to equivalent parentheses. Uses the tokenize module for compact coding.

Tabify (Python)

2006-07-19T04:25:01-07:00

Python recipe 496893 by Yuce Tekol (programs). Revision 3.

A little script for those of us who prefer tabs over spaces.

Generator expressions for database requests (Python)

2005-10-18T12:21:33-07:00

Python recipe 442447 by Pierre Quentel (database).

This recipe is a follow-up to #440653, which was easy to implement but very slow because the iteration required to read all the rows of a table

As suggested by Matteo Dell'Amico in a comment, it would be much better if we could write something like

query(r.name for r in plane_tbl if r.country == "France")

where the generator expression is first translated into an SQL select, so that the iteration on the instance of query only reads the rows selected by the SQL statement

The present recipe is an attempt to achieve this. The first problem is to get the source code of the generator expression. I use information from the stack frame to get the file name and the line number, then the tokenize module to read the elements of the generator expression in the source code

Then, to build the SQL statement, the source code must be parsed : this is done using the compiler package and "visitors" that walk the AST tree returned by compiler.parse and do operations on the nodes, depending on their type

Finally, once the SQL statement is built, the iteration on the query instance can start : for the first one, the SQL statement is executed ; then the iteration yields the selected rows one by one.

The items can be : - objects, with attribute names matching those in the generator expression, except that qualified names (table.name) are converted to table_name - dictionaries : the keys are the same as the attribute names above - lists

For instance : - iterating on query(name for r in plane_tbl) returns objects with an attribute name - iterating on query(r.name for r in plane_tbl) returns objects with an attribute r_name

This is because of iteration on tables which have the same field names

query((r.name,c.name) for r in plane_tbl for c in country_tbl if r.speed > 500 )

The type of the items is set by query.return_type = object, dict or list

Python source to XHTML colorizer (Python)

2005-10-26T00:36:06-07:00

Python recipe 442482 by Peter Krantz .

Converts Python source to a portable XHTML 1.0 strict document that includes a basic set of Dublin Core metadata. Based on the MoinMoin source colorizer.

Colorize Python source using the built-in tokenizer (Python)

2001-04-06T23:05:53-07:00

Python recipe 52298 by Jürgen Hermann (programs). Revision 3.

This code is part of MoinMoin (http://moin.sourceforge.net/) and converts Python source code to HTML markup, rendering comments, keywords, operators, numeric and string literals in different colors.

It shows how to use the built-in keyword, token and tokenize modules to scan Python source code and re-emit it with no changes to its original formatting (which is the hard part).

The test code at the bottom of the module formats itself and launches a browser with the result.