Popular recipes tagged "meta:requires=token"

Python config file parser (Python)

2008-08-02T12:55:21-07:00

Python recipe 576404 by Frithiof andreas Jensen .

Config - reads a configuration file.

This module parses a configuration file using a restricted Python syntax. The Python tokenizer/parser is used to read the file, unwanted expressions are removed from the parser output before the result is compiled and executed. The initialised configuration settings are returned in a dict.

Generator expressions for database requests (Python)

2005-10-18T12:21:33-07:00

Python recipe 442447 by Pierre Quentel (database).

This recipe is a follow-up to #440653, which was easy to implement but very slow because the iteration required to read all the rows of a table

As suggested by Matteo Dell'Amico in a comment, it would be much better if we could write something like

query(r.name for r in plane_tbl if r.country == "France")

where the generator expression is first translated into an SQL select, so that the iteration on the instance of query only reads the rows selected by the SQL statement

The present recipe is an attempt to achieve this. The first problem is to get the source code of the generator expression. I use information from the stack frame to get the file name and the line number, then the tokenize module to read the elements of the generator expression in the source code

Then, to build the SQL statement, the source code must be parsed : this is done using the compiler package and "visitors" that walk the AST tree returned by compiler.parse and do operations on the nodes, depending on their type

Finally, once the SQL statement is built, the iteration on the query instance can start : for the first one, the SQL statement is executed ; then the iteration yields the selected rows one by one.

The items can be : - objects, with attribute names matching those in the generator expression, except that qualified names (table.name) are converted to table_name - dictionaries : the keys are the same as the attribute names above - lists

For instance : - iterating on query(name for r in plane_tbl) returns objects with an attribute name - iterating on query(r.name for r in plane_tbl) returns objects with an attribute r_name

This is because of iteration on tables which have the same field names

query((r.name,c.name) for r in plane_tbl for c in country_tbl if r.speed > 500 )

The type of the items is set by query.return_type = object, dict or list

Python source to XHTML colorizer (Python)

2005-10-26T00:36:06-07:00

Python recipe 442482 by Peter Krantz .

Converts Python source to a portable XHTML 1.0 strict document that includes a basic set of Dublin Core metadata. Based on the MoinMoin source colorizer.

Colorize Python source using the built-in tokenizer (Python)

2001-04-06T23:05:53-07:00

Python recipe 52298 by Jürgen Hermann (programs). Revision 3.

This code is part of MoinMoin (http://moin.sourceforge.net/) and converts Python source code to HTML markup, rendering comments, keywords, operators, numeric and string literals in different colors.

It shows how to use the built-in keyword, token and tokenize modules to scan Python source code and re-emit it with no changes to its original formatting (which is the hard part).

The test code at the bottom of the module formats itself and launches a browser with the result.