Simple Web Crawler (Python)

2011-01-31T21:57:58-08:00

Python recipe 576551 by James Mills (crawler, network, parsing, web). Revision 2.

NOTE: This recipe has been updated with suggested improvements since the last revision.

This is a simple web crawler I wrote to test websites and links. It will traverse all links found to any given depth.

See --help for usage.

I'm posting this recipe as this kind of problem has been asked on the Python Mailing List a number of times... I thought I'd share my simple little implementation based on the standard library and BeautifulSoup.

--JamesMills

Get columns of data from text files (Python)

2010-10-28T16:18:19-07:00

Python recipe 577444 by alinium (columns, file, parsing).

Read in a tab-delimited (or any separator-delimited like CSV) file and store each column in a list that can be referenced from a dictionary. The keys for the dictionary are the headings for the columns (if any). All data is read in as strings.

Copy directory tree recursively while ignoring cvs, git and svn directories (Python)

2008-12-18T21:36:12-08:00

Python recipe 576588 by Senthil Kumaran (parsing).

I wanted to do a conditional copy of a directory tree. Noticed a ignore parameter introduced in Python 2.6. Thats very handy. This snippet gives the example of its usage.

How to parse a table in a PDF document (Python)

2016-04-10T22:43:57-07:00

Python recipe 580635 by Jorj X. McKie (cbz, epub, fitz, mupdf, openxps, parsing, pdf, pymupdf, table, xps). Revision 4.

A Python function that converts a table contained in a page of a PDF (or OpenXPS, EPUB, CBZ, XPS) document to a matrix-like Python object (list of lists of strings).

Flexible datetime parsing (Python)

2012-08-21T07:35:34-07:00

Python recipe 578245 by Glenn Hutchings (datetime, parsing).

The strptime() method of datetime accepts a format string that you have to specify in advance. What if you want to be more flexible in the kinds of date your program accepts? Here's a recipe for a function that tries many different formats until it finds one that works.

Expression Evaluator (Python)

2009-06-02T23:57:44-07:00

Python recipe 576790 by Stephen Chappell (evaluation, expressions, parsing). Revision 4.

After reading a portion of the book "The C# Programming Language: Third Edition," I found a section in the introduction that introduced abstract classes and methods that involved an example that included the concept of expression trees. The code was easy to implement since it just had to be copied out of the book. After playing around with the program a little and extending it, I thought that it would be fun to write a program in C# that could (interactively) evaluate expressions and display the results. Not knowing C# quite as well as Python led to the following program written and tested in Python 3.0 (not sure about previous languages).

The first section of the code includes port of the program from the aforementioned book along with extra code that allows for further features not originally included in the C# version. Those sections are clearly marked as being new code written by yours truly. The second area of the program has six functions that are profusely documented so as to explain how they go about parsing and processing expressions entered for evaluation. For those wishing to use the code, the "run" function should be all that you need. The final part of the module contains a test program that can be used to check the validity of the how well the program works.

The parser is not very complicated and will except expressions that are both normal to Python and completely illegal in Python. The main features are its ability to (1) identify simple assignment and mathematical operations, (2) identify constant floating point numbers, and (3) identify variables that would otherwise have no other meaning to the program. A limited number of error messages are given when appropriate but may leave one guessing what the problem really is. Mathematical operations are evaluated from left to right without regards to precedence, and assignment statements are evaluated from right to left.

Parse HTTP date-time string (Python)

2010-01-20T13:47:50-08:00

Python recipe 577015 by Sridhar Ratnakumar (datetime, http, parsing).

This recipe will help you parse datetime strings returned by HTTP servers following the RFC 2616 standard (which supports three datetime formats). Credit for this recipe goes to ΤΖΩΤΖΙΟΥ.

Remove the .pyc files from current directory tree and from svn (Python)

2009-02-03T23:38:43-08:00

Python recipe 576641 by Senthil Kumaran (parsing).

I had mistakenly checked in .pyc files into svn, So I took this approach of deleting all the .pyc files in the current working copy directory tree and then using svn remove to the remove from the repository. The following is the snippet I wrote then to for the purpose.

python xml parsing (Python)

2011-05-28T19:36:00-07:00

Python recipe 577727 by abhijeet vaidya (parsing, python, xml).

xml parsing to how to extract data

Parse call function for Py2.6 and Py2.7 (Python)

2009-02-28T20:13:15-08:00

Python recipe 576671 by Jervis Whitley (ast, call, function, namedtuple, nodevisitor, parsing). Revision 14.

In some cases it may be desirable to parse the string expression "f1(*args)" and return some of the key features of the represented function-like call.

This recipe returns the key features in the form of a namedtuple.

e.g. (for the above)

>>> explain("f1(*args)")
[ Call(func='f1', starargs='args') ]

The recipe will return a list of such namedtuples for "f1(*args)\nf2(*args)" Note that while the passed string expression must evaluate to valid python syntax, names needn't be declared in current scope.

Dragon Lexical Analyzer (Python)

2010-09-01T14:49:37-07:00

Python recipe 577380 by Jack Trainor (educational, lexical_analyzer, parsing). Revision 2.

The lexical analyzer from "Compliers: Principles, Techniques and Tools," Chapter 2, by Aho, Sethi, Ullman (1986) implemented in Python.

Simple regex engine, elementary Python (Python)

2010-07-10T10:43:30-07:00

Python recipe 577251 by Joost Behrends (cached, parse, parsing, recursion, regular_expressions). Revision 40.

A short engine for testing against a regex, understanding the 3 common quantifiers ?,+,* (non-greedy) working on characters, ., [...], [^...], \s, \S, bracketed patterns and group designators \N. Accepts unicode objects and fixed-width encoded strings (but problems with eventual comparisons of trailing bytes in multi-byte utf-letters). Captures up to 10 groups ( (?:...) implemented), which can be used for back referencing and in xreplace(). Captured groups are accessible after the search in the global list xGroups. | is supported, but only in groups and needing nested=True. nested=False is making '(' and ')' common letters.

This is not about Python or for Python, there it has little use beside re. But regarding that re needs about 6,000 lines you might agree with the author, that these 176 lines are powerful. This was the reason to publish it as a recipe - as a kind of (fairly complete) minimal example of a regex tester and as an example for corresponding recursive structures in data (TokenListCache) and code.

Working on this improved the author's understanding of regular expressions - especially of their eventual "greed". "Greedy" quantifiers are a concept, which has to be explained seperately and is coming unexpected: Whoever is scanning a text for '<.*>', s/he will search SGML tags, not the whole text. Even with the star's "greediness" the code has to take care, that '.*' doesn't eat the whole text finding no match for '<.*>' at all. Thus the standard syntax with greedy quantifiers cannot be simpler to implement than this with its mere 3 lines 101, 111 and 121 preventing any greed. Perhaps it is faster - otherwise it is difficult to understand, why the concept "greed" is existing at all.

This engine might be useful here and then under circumstances with nothing else available. Its brevity eases translation to other languages and it can work with arbitrary characters for STAR or PERHAPS (for example).

Extracting structured text or code (Python)

2011-05-18T13:04:01-07:00

Python recipe 577700 by Mike Sweeney (parsing, structured, text_processing, token). Revision 2.

This function uses the power of regular expressions to extract parts of a structured text string. It can build a token list from many types of code and data formats. It finds string types (with quotes) and nested structures that use parentheses, brackets, and braces. If you need to extract a different syntax, you can provide a custom token pattern in the function arguments.

MicroXml: Stand-alone library for basic XML features (Python)

2015-12-04T22:36:56-08:00

Python recipe 579133 by Jack Trainor (parsing, xml).

MicroXml provides stand-alone support for the basic, most-used features of XML -- tags, attributes, and element values. It produces a DOM tree of XML nodes. It's compatible with Python 2.7 and Python 3. MicroXml does not support DTDs, CDATAs and other advanced XML features.

MicroXml is easy to use and easy to view/navigate its nodes in a debugger. It also includes a minimal XPath-like implementation.

Nicer struct syntax thanks to Py3 metaclasses (Python)

2009-02-25T07:39:52-08:00

Python recipe 576666 by Daniel Brodie (binary, parsing, py3, struct).

This is a quick-hack module I wrote up in a couple of hours that allows for a nicer syntax to build up struct-like binary packing and unpacking. The point was to get it to be concise and as C-like as possible. This script requires python3 for it's improved metaclass support.

A Basic USe flag EDitor for Gentoo Linux supporting on-the-fly editing (Python)

2015-02-28T07:04:31-08:00

Python recipe 579028 by Mike 'Fuzzy' Partin (parsing, popen, subprocess, user_input).

This allows for on-the-fly editing. Simply drop abused.py into your path, and ensure that -a is not set in EMERGE_DEFAULT_OPTS in /etc/portage/make.conf. Then whenver you are installing new packages, use abused in place of emerge (eg: abused multitail) you will be presented with a list of use flags that are used in this action, and a prompt for editing any of them, simply hit enter with no changes to fire off the build.

Simple tabulator (Python)

2010-11-09T12:50:06-08:00

Python recipe 577458 by Noufal Ibrahim (parsing, text_processing).

This is a simple script to covert a top to bottom list of items into a left to right list.

 a 
 b 
 c
 d
 e
 f
 g
 h
 i
 j
 k
 l
 m

into

 a  b  c  d  e  f  g
 h  i  j  k  l  m

A few command line options allow some amount of customisation.

MicroXml: Stand-alone library for basic XML features (C++)

2016-02-18T19:33:07-08:00

C++ recipe 580612 by Jack Trainor (parsing, xml).

MicroXml provides stand-alone support for the basic, most-used features of XML -- tags, attributes, and element values. It produces a DOM tree of XML nodes. MicroXml does not support DTDs, CDATAs and other advanced XML features.

MicroXml is easy to use and provides easy access to view/navigate its nodes in a debugger.

Most viewed recipes tagged "parsing"