Popular Python recipes tagged "meta:requires=urllib"

Composing a POSTable HTTP request with multipart/form-data Content-Type to simulate a form/file upload. (Python)

2014-03-08T17:34:38-08:00

Python recipe 578846 by István Pásztor (field, file, form, html, httpclient, mime, multipart, post, upload, web). Revision 5.

This code is useful if you are using a http client and you want to simulate a request similar to that of a browser that submits a form containing several input fields (including file upload fields). I've used this with python 2.x.

Recursive Multimedia (audio, video) M3U Playlist Generator (Python)

2014-02-08T01:03:36-08:00

Python recipe 578771 by Mano Bastardo (audio, ffmpeg, generate, m3u, mulitmedia, os, playlist, python, system, video). Revision 9.

Generate an m3u playlist searching recursively for multimedia files (video or audio) in the given directory. Information from ID3 tags will be extracted for audio files with FFmpeg available.

Music Downloader (Python)

2013-05-25T06:52:51-07:00

Python recipe 578530 by Christian Careaga (download, downloader, music, program, python, python_scripts, selenium, urllib, urllib2).

A Python Program i wrote that downloads music from the web

download the Activestate cook book recipe (Python)

2013-01-29T16:24:22-08:00

Python recipe 578439 by lwz7512 .

Small effort to store the python recipes to our local

similar effort by other people: 543267-i-will-download-all-of-the-recipes-from-the-python 535162-i-download-all-the-python-cookbook-recipes

download the Activestate cook book recipe (Python)

2012-07-04T12:14:22-07:00

Python recipe 578193 by Sudeep AM . Revision 2.

Small effort to store the python recipes to our local

similar effort by other people: 543267-i-will-download-all-of-the-recipes-from-the-python 535162-i-download-all-the-python-cookbook-recipes

KYSU (Keep Your Stuff Updated) (Python)

2012-07-29T01:12:01-07:00

Python recipe 578174 by Gamoholic (automatic, exe, installer, updated, updater). Revision 5.

I wrote this program to keep my Samba share up to date. The Samba share contains installers for the programs that I use to fix and update computers (ccleaner, mbam, java, .etc). It can also be used for ISO's such as clonezilla.

For the latest version of the script and the accompanying files please go to GitHub.

https://github.com/Gamoholic/KYSU

!!! Important! The only OS I have tested this on is Ubuntu. I will test Windows soon. If it doesn't work on any other OS please let me know! Also, I have no idea how well this works with Python 3. I may test that eventually.

Geocoding Lists via Google Maps (Python)

2012-05-11T05:06:27-07:00

Python recipe 578126 by Mano Bastardo (batch, coordinates, geocode, geocoding, google, google_maps, lat, latitude, list, list_comprehension, lng, longitude, map, web). Revision 2.

A simple script written as an experiment in geocoding addresses in a database. A list of addresses in the form of "100 Any Street, Anytown, CA, 10010" is passed to a Google Maps URL, and the latitude/longitude coordinates are extracted from the returned XML.

XML methods are not used in this script, but simple string searches instead.

A Simple Webcrawler (Python)

2012-03-03T02:37:30-08:00

Python recipe 578060 by John (crawler, html, page, parser, scraping, urllib, urlopen, web).

This is my simple web crawler. It takes as input a list of seed pages (web urls) and 'scrapes' each page of all its absolute path links (i.e. links in the format http://) and adds those to a dictionary. The web crawler can take all the links found in the seed pages and then scrape those as well. You can continue scraping as deep as you like. You can control how "deep you go" by specifying the depth variable passed into the WebCrawler class function start_crawling(seed_pages,depth). Think of the depth as the recursion depth (or the number of web pages deep you go before returning back up the tree).

To make this web crawler a little more interesting I added some bells and whistles. I added the ability to pass into the WebCrawler class constructor a regular expression object. The regular expression object is used to "filter" the links found during scraping. For example, in the code below you will see:

cnn_url_regex = re.compile('(?<=[.]cnn)[.]com') # cnn_url_regex is a regular expression object

w = WebCrawler(cnn_url_regex)

This particular regular expression says:

1) Find the first occurence of the string '.com'

2) Then looking backwards from where '.com' was found it attempts to find '.cnn'

Why do this?

You can control where the crawler crawls. In this case I am constraining the crawler to operate on webpages within cnn.com.

Another feature I added was the ability to parse a given page looking for specific html tags. I chose as an example the <h1> tag. Once a <h1> tag is found I store all the words I find in the tag in a dictionary that gets associated with the page url.

Why do this?

My thought was that if I scraped the page for text I could eventually use this data for a search engine request. Say I searched for 'Lebron James'. And suppose that one of the pages my crawler scraped found an article that mentions Lebron James many times. In response to a search request I could return the link with the Lebron James article in it.

The web crawler is described in the WebCrawler class. It has 2 functions the user should call:

1) start_crawling(seed_pages,depth)

2) print_all_page_text() # this is only used for debug purposes

The rest of WebCrawler's functions are internal functions that should not be called by the user (think private in C++).

Upon construction of a WebCrawler object, it creates a MyHTMLParser object. The MyHTMLParser class inherits from the built-in Python class HTMLParser. I use the MyHTMLParser object when searching for the <h1> tag. The MyHTMLParser class creates instances of a helper class named Tag. The tag class is used in creating a "linked list" of tags.

So to get started with WebCrawler make sure to use Python 2.7.2. Enter the code a piece at a time into IDLE in the order displayed below. This ensures that you import libs before you start using them.

Once you have entered all the code into IDLE, you can start crawling the 'interwebs' by entering the following:

import re

cnn_url_regex = re.compile('(?<=[.]cnn)[.]com')

w = WebCrawler(cnn_url_regex)

w.start_crawling(['http://www.cnn.com/2012/02/24/world/americas/haiti-pm-resigns/index.html?hpt=hp_t3'],1)

Of course you can enter any page you want. But the regular expression object is already setup to filter on cnn.com. Remember the second parameter passed into the start_crawling function is the recursion depth.

Happy Crawling!

Extract air quality data of Santigo , Chile on csv file (Python)

2011-06-30T13:39:29-07:00

Python recipe 577773 by jrovegno (aire, calidad, chile, data_mining, santiago).

Extract air quality data of Santigo , Chile on csv file

Extract data from feedjit (Python)

2011-05-03T19:50:27-07:00

Python recipe 577683 by jrovegno (data, feedjit). Revision 2.

Script to extract data from my live traffic feed from feedjit

Download all lolcat images from iCanHasCheezburger.com (Python)

2011-03-10T08:49:14-08:00

Python recipe 577603 by Rahul Anand (download, images, lolcat, python, web).

Running this python script will download all lolcat images from icanhascheezburger.com to the current folder. Download will start from the oldest image. Images are collected into subfolders lolcat0, lolcat1 etc, each containing 300 images. The script can be stopped and resumed at anytime. Make sure to create files lolconfig.txt and log.txt in the same folder before running the script. lolconfig.txt must have a string as follows in the beginning: 1496/1496/0. log.txt is an empty file in the beginning

Robot Pager (Search engines and others) (Python)

2010-10-10T19:33:20-07:00

Python recipe 577420 by Carlos del Ojo (automate, engine, paging, robot, search, websites). Revision 3.

This is a class to make easy the development of robots, to parse results over a website with a paging. For example Google, Yahoo, Bing, or any other page with paging system.

PagerEngine is the main class. I've developed three more clases implementing GoogleSearch, YahooSearch and BingSearch as examples.

Inheriting from PagerEngine (and having RexExp knowledge) you can easily develop other robots for other websites.

Script para descargar videos desde http://www.chilevision.cl/ (Python)

2010-08-23T20:20:15-07:00

Python recipe 577367 by jrovegno (chv, download, tv, video).

Requiere: - aria2c - wget Extras: Ofrece Descargar el resto de los videos Uso: # Comillas requeridas, problema parser cvh_video.py "http://www.chilevision.cl/home/index.php?option=com_content&task=view&id=YYYYY&Itemid=XXX"

Script para saber calida del aire Santiago de Chile (Python)

2011-06-02T00:55:02-07:00

Python recipe 577369 by jrovegno (aire, calidad, santiago). Revision 4.

Script para saber calida del aire Santiago de Chile

ur1.ca command-line client (Python)

2011-03-23T05:27:27-07:00

Python recipe 577236 by Cong (scraping, shortening, url, web). Revision 2.

(ur1.ca)[http://ur1.ca/] is the URL shortening services provided by status.net. This script makes it possible to access the service from the command line. This is done by scraping the returned page and look for the shortened URL.

Routines for programmatically authenticating with the Google Accounts system at Google App-Engine. (Python)

2010-05-20T20:39:50-07:00

Python recipe 577217 by Berend (auth, authentication, gae, google, http, sessions). Revision 2.

This takes two calls, one to the ClientLogin service of Google Accounts, and then a second to the login frontend of App Engine.

User credentials are provided to the first, which responds with a token. Passing that token to the _ah/login GAE endpoint then gives the cookie that can be used to make further authenticated requests.

Download chromium browser nightly builds for any OS (with proxy support) (Python)

2014-07-05T18:47:47-07:00

Python recipe 577162 by ccpizza (chrome, chromium, download, unzip). Revision 19.

Downloads the latest Chromium browser build from http://commondatastorage.googleapis.com/chromium-browser-continuous/ using urllib2 or wget (with Python versions below 2.5) and unzips the downloaded zip file to a predefined folder.

To use a custom proxy define the HTTP_PROXY system variable.

The script will figure out the OS but you can also pass the platform as the first parameter (one of win32, linux, linux64, mac).

Prerequisites (!! only for Python versions below 2.5):

wget - usually already installed on most linux systems. Windows users can get it here.
unzip - used for unpacking the archive; usually already installed on most linux systems. Windows users can get it here.

Both wget and unzip should be available in PATH (for Python 2.5+ native Python modules are used).

For most Linux distros this script does not make much sense since the built-in package managers do a better job of managing chromium builds and dependencies, but it still can be useful if you are using a stable Chromium build but would like to be able to test the nightly builds too.

For OSX an additional installation step will be performed using the install.sh that is included in the OSX build. The OSX installer will copy the package to ~/Applications/Chromium, and set some permissions that are required for Chromium to run. Running the unpacked zip without doing the installation will most likely will not work because of missing executable permissions on some files.

Retrieve Dell Warranty Information for all machines in AD Domain (Python)

2010-02-18T15:51:28-08:00

Python recipe 577056 by Kenneth Keiter (active_directory_scripts, ad, dell, domain, network, warranty, windows).

This snippet retrieves warranty information for all Dell machines in a domain and outputs a CSV of the results.

Should be run on a machine joined to an active directory or NT4 domain. May need to be updated to parse Dell's website if they update it (since they have no service tag API).

Improvements of the urllib.URLopen.retrieve() method (Python)

2010-01-16T04:50:07-08:00

Python recipe 577009 by Kévin Gomez (retrieve, urllib).

I improved the urllib.URLopen.retrieve() method so that it can restart a download if it failed. And like wget does (with wget -c), it restarts where it stopped. The number of maximum tries can be changed.

Wordze.com API bindings (Python)

2010-01-21T21:24:39-08:00

Python recipe 577018 by Sergei Lebedev (api, bindings, wordze, wordze_com). Revision 3.

Just in case anyone would need the bindings, here's a very basic implementation. For some reasons, wordze.com has very strict API usage limitations, so to use this efficiently, you'll probably need to cache every request-response.

Any comments and/or corrections are welcome. :)