A Simple Webcrawler (Python)

2012-03-03T02:37:30-08:00

Python recipe 578060 by John (crawler, html, page, parser, scraping, urllib, urlopen, web).

This is my simple web crawler. It takes as input a list of seed pages (web urls) and 'scrapes' each page of all its absolute path links (i.e. links in the format http://) and adds those to a dictionary. The web crawler can take all the links found in the seed pages and then scrape those as well. You can continue scraping as deep as you like. You can control how "deep you go" by specifying the depth variable passed into the WebCrawler class function start_crawling(seed_pages,depth). Think of the depth as the recursion depth (or the number of web pages deep you go before returning back up the tree).

To make this web crawler a little more interesting I added some bells and whistles. I added the ability to pass into the WebCrawler class constructor a regular expression object. The regular expression object is used to "filter" the links found during scraping. For example, in the code below you will see:

cnn_url_regex = re.compile('(?<=[.]cnn)[.]com') # cnn_url_regex is a regular expression object

w = WebCrawler(cnn_url_regex)

This particular regular expression says:

1) Find the first occurence of the string '.com'

2) Then looking backwards from where '.com' was found it attempts to find '.cnn'

Why do this?

You can control where the crawler crawls. In this case I am constraining the crawler to operate on webpages within cnn.com.

Another feature I added was the ability to parse a given page looking for specific html tags. I chose as an example the <h1> tag. Once a <h1> tag is found I store all the words I find in the tag in a dictionary that gets associated with the page url.

Why do this?

My thought was that if I scraped the page for text I could eventually use this data for a search engine request. Say I searched for 'Lebron James'. And suppose that one of the pages my crawler scraped found an article that mentions Lebron James many times. In response to a search request I could return the link with the Lebron James article in it.

The web crawler is described in the WebCrawler class. It has 2 functions the user should call:

1) start_crawling(seed_pages,depth)

2) print_all_page_text() # this is only used for debug purposes

The rest of WebCrawler's functions are internal functions that should not be called by the user (think private in C++).

Upon construction of a WebCrawler object, it creates a MyHTMLParser object. The MyHTMLParser class inherits from the built-in Python class HTMLParser. I use the MyHTMLParser object when searching for the <h1> tag. The MyHTMLParser class creates instances of a helper class named Tag. The tag class is used in creating a "linked list" of tags.

So to get started with WebCrawler make sure to use Python 2.7.2. Enter the code a piece at a time into IDLE in the order displayed below. This ensures that you import libs before you start using them.

Once you have entered all the code into IDLE, you can start crawling the 'interwebs' by entering the following:

import re

cnn_url_regex = re.compile('(?<=[.]cnn)[.]com')

w = WebCrawler(cnn_url_regex)

w.start_crawling(['http://www.cnn.com/2012/02/24/world/americas/haiti-pm-resigns/index.html?hpt=hp_t3'],1)

Of course you can enter any page you want. But the regular expression object is already setup to filter on cnn.com. Remember the second parameter passed into the start_crawling function is the recursion depth.

Happy Crawling!

Composing a POSTable HTTP request with multipart/form-data Content-Type to simulate a form/file upload. (Python)

2014-03-08T17:34:38-08:00

Python recipe 578846 by István Pásztor (field, file, form, html, httpclient, mime, multipart, post, upload, web). Revision 5.

This code is useful if you are using a http client and you want to simulate a request similar to that of a browser that submits a form containing several input fields (including file upload fields). I've used this with python 2.x.

Image Downloader (Python)

2014-02-24T03:49:51-08:00

Python recipe 577385 by FB36 (html, http, url, urllib2, web). Revision 4.

Finds and downloads all images from any given URL.

Important note:

If your download location path has spaces then put quotes around it!

Website Text Search (Python)

2010-09-11T17:32:01-07:00

Python recipe 577388 by FB36 (html, http, url, urllib2, web). Revision 2.

Searches a website recursively for the given text string and prints all URLs containing it.

IFrame Example (HTML)

2011-04-21T02:11:57-07:00

HTML recipe 577671 by FB36 (html, iframe).

IFrame demo. It opens 4 sub-browser windows inside the webpage.

Generating Code Coverage HTML Output (Python)

2010-10-17T15:17:44-07:00

Python recipe 491274 by Sebastien Martini (coverage, debugging, html). Revision 11.

Code coverage testing is very useful especially for dynamic languages, it can easily give a view of the unused parts of your code. Not that this method is able to prove that statements are dead code, but instead help to prevent from syntax errors and to force yourself to think about your code, to find the right test cases reaching the unused statements.

In that sense, the module coverage.py made by Ned Batchelder is very useful and efficient.

The analysis returned by this module is very accurate, but as unreached lines numbers are not very readable by itself, this recipe simply generate an html output highlighting unreached lines. This recipe is directely based upon a source code colorizer derived from this recipe.

Convert text/enriched MIME to text/html (Python)

2009-06-09T15:08:40-07:00

Python recipe 576800 by Jack Trainor (email, html, mime, text_enriched).

Converts text stream in text/enriched MIME format from file or stdin to text/html output to file or stdout.

Python HTML Stripper (Python)

2013-04-08T13:58:00-07:00

Python recipe 578511 by Granning Stoline (html, python, stripper).

Python HTML Stripper

text-to-html (Python)

2009-03-05T23:10:50-08:00

Python recipe 576682 by nillgump nillgump (html).

将文本排版转化为html中相同的排版。为google blog服务

Easy to use, easy to read, python based HTML generation (Python)

2013-06-21T14:47:21-07:00

Python recipe 578436 by Pavlos (html, template, text). Revision 4.

I was looking for a simple way to generate HTML directly in python that does not require learning a new template 'language' nor requires the installation of a big complex package. Closest thing I found was James Casbon's attempt(https://gist.github.com/1461441). This is my version of the same idea.

(2013-04-21) added some simplifications and support for switching off string interpolation. Added to github:

https://github.com/pavlos-christoforou/web

Convert HTML to PDF with the PDFcrowd API (Python)

2015-03-07T20:22:54-08:00

Python recipe 579032 by Vasudev Ram (api, html, pdf, pdfcrowd).

This recipe shows how to use Python and the PDFcrowd API to convert HTML content to PDF. The HTML input can come from a remote URL, a local HTML file, or a string containing HTML.

Random URL (Python)

2010-09-12T22:23:09-07:00

Python recipe 577389 by FB36 (html, http, url, urllib2, web).

Finds and displays a random webpage from the Internet. (Warning: It may take a while!)

Serve static web content from within a gzipped tarball to save space using CherryPy (Python)

2009-03-31T18:24:06-07:00

Python recipe 576706 by Dan McDougall (cherrypy, compression, embedded, gzip, html, http, network, routes, web, web_server).

This code lets you store all of your static website content inside a gzipped tarball while transparently serving it to HTTP clients on-demand. Perfect for embedded systems where space is limited.

CommentEditor: HTML editor for online comments (Python)

2009-06-17T12:10:55-07:00

Python recipe 576814 by Jack Trainor (editor, html, wxwidgets). Revision 3.

Edit online comments with easy addition of HTML tags for bold, italics, underlining, blockquote and anchored links. Then check your work with the preview feature.

Requires wxWidgets.

Userfriendly Webpage Template (Python)

2010-05-04T07:38:03-07:00

Python recipe 577203 by david.gaarenstroom (cgi, html, httpserver, mvc, template, web, webdesign, webpagetemplate). Revision 5.

User friendly template class targeted towards Web-page usage and optimized for speed and efficiency.

Tags can be inserted in a template HTML file in a non-intrusive way, by using specially formatted comment strings. Therefore, the template-file can be viewed in a browser, even with prototype data embedded in it, which will later be replaced by dynamic content. Also, webdesigners can continue to work on the template and upload it without further modification.

Komodo Macro - Copy to clipboard code selection in HTML format (JavaScript)

2010-04-14T07:17:59-07:00

JavaScript recipe 577193 by Davide Ficano (clipboard, html, komodo, macro).

This macro makes easy to paste on documents (MS Office, Open Office, iWorks Pages) code snippets with syntax highlight and/or line numbers.

This macro copies the selected code text or whole document if no selection is present on clipboard in HTML format.

On OSX you need at least Komodo 6 because previous versions don't support data flavors

Website Mapper (Python)

2010-09-23T01:23:04-07:00

Python recipe 577392 by FB36 (html, http, url, web). Revision 3.

Prints the tree graph of the given URL.

Safe HTML string and unicode (Python)

2012-01-10T08:14:14-08:00

Python recipe 578008 by Garel Alex (html, security, web). Revision 2.

As you display message on a web page, you have to sanitize input data coming from users to avoid XSS. Here is a small recipe where we can use a special class for our string to be sure we get safe all the way long.

Pretty and Stated HTMLParsers (Python)

2013-12-14T00:28:36-08:00

Python recipe 578787 by Ádám Szieberth (html, htmlparser, state). Revision 2.

Extensions of html.parser.HTMLParser().

PrettyHTMLParser() does not splits data into chuncks by HTML entities. StatedHTMLParser() can have many state-dependent handlers which helps parsing HTML pages alot.

Komodo JS Macro -- Escape HTML Special Characters in current file (JavaScript)

2012-05-25T20:04:49-07:00

JavaScript recipe 578146 by Keegan Brown (characters, html, macro, special, xhtml).

Escapes all Special Characters to their HTML Special Character equivalent.

Most viewed recipes tagged "html"