Latest recipes tagged "urllib"

Urllib handler for Amazon S3 buckets (Python)

2014-11-06T18:31:44-08:00

Python recipe 578957 by Andrea Corbellini (aws, s3, urllib).

This is a handler for the standard urllib.request module capable of opening buckets stored on Amazon S3.

Here is an usage example:

>>> from urllib.request import build_opener
>>> opener = build_opener(S3Handler)
>>> response = opener.open('s3://bucket-name/key-name')
>>> response.read()
b'contents'

Music Downloader with Wx GUI! (Python)

2013-11-05T02:52:29-08:00

Python recipe 578681 by Christian Careaga (beautifulsoup, downloader, gui, music, python, urllib, urllib2, wxpyton).

Just type in a song and the artist and the program will get the YouTube video, convert it to an mp3 then download it! It has a high quality function and a medium quality function and also the user can choose the directory and name they want the file to be saved to!

It is the first time I've used threads and my second time with WxPython! I used BeautifulSoup for the scraping and I'm pretty familiar with that.just thought id share it with you guys and see if you have any feedback or suggestions!

also you may get an error saying self.convhtml doesn't exist just wait then retry

Here is a link to an screenshot:

http://adf.ly/XJaoU

if you want you can checkout the Github page:

http://adf.ly/XGL6P

also you will need to make a folder called Files and put a file called dir.txt and in the file write /Files. this is where the music will be downloaded to!

I just made the .exe so you can just use that and its easier! Here: http://adf.ly/XRjRH

Music Downloader (Python)

2013-05-25T06:52:51-07:00

Python recipe 578530 by Christian Careaga (download, downloader, music, program, python, python_scripts, selenium, urllib, urllib2).

A Python Program i wrote that downloads music from the web

Multithreading Downloader Class (Python)

2012-07-22T07:44:20-07:00

Python recipe 578220 by Itay Brandes (downloader, multithread, multithreading, urllib, urllib2, urlopen).

Garbs files from the web using multithreading in an attempt to enhance transfer rate.

Cosign Handler (Python)

2012-07-18T13:30:10-07:00

Python recipe 578217 by Colin Higgs (authentication, cosign, handler, urllib).

Handler (python 3.x urllib.request style) for web pages where cosign authentication is required.

See http://weblogin.org/ for details of the cosign authentication system.

A Simple Webcrawler (Python)

2012-03-03T02:37:30-08:00

Python recipe 578060 by John (crawler, html, page, parser, scraping, urllib, urlopen, web).

This is my simple web crawler. It takes as input a list of seed pages (web urls) and 'scrapes' each page of all its absolute path links (i.e. links in the format http://) and adds those to a dictionary. The web crawler can take all the links found in the seed pages and then scrape those as well. You can continue scraping as deep as you like. You can control how "deep you go" by specifying the depth variable passed into the WebCrawler class function start_crawling(seed_pages,depth). Think of the depth as the recursion depth (or the number of web pages deep you go before returning back up the tree).

To make this web crawler a little more interesting I added some bells and whistles. I added the ability to pass into the WebCrawler class constructor a regular expression object. The regular expression object is used to "filter" the links found during scraping. For example, in the code below you will see:

cnn_url_regex = re.compile('(?<=[.]cnn)[.]com') # cnn_url_regex is a regular expression object

w = WebCrawler(cnn_url_regex)

This particular regular expression says:

1) Find the first occurence of the string '.com'

2) Then looking backwards from where '.com' was found it attempts to find '.cnn'

Why do this?

You can control where the crawler crawls. In this case I am constraining the crawler to operate on webpages within cnn.com.

Another feature I added was the ability to parse a given page looking for specific html tags. I chose as an example the <h1> tag. Once a <h1> tag is found I store all the words I find in the tag in a dictionary that gets associated with the page url.

Why do this?

My thought was that if I scraped the page for text I could eventually use this data for a search engine request. Say I searched for 'Lebron James'. And suppose that one of the pages my crawler scraped found an article that mentions Lebron James many times. In response to a search request I could return the link with the Lebron James article in it.

The web crawler is described in the WebCrawler class. It has 2 functions the user should call:

1) start_crawling(seed_pages,depth)

2) print_all_page_text() # this is only used for debug purposes

The rest of WebCrawler's functions are internal functions that should not be called by the user (think private in C++).

Upon construction of a WebCrawler object, it creates a MyHTMLParser object. The MyHTMLParser class inherits from the built-in Python class HTMLParser. I use the MyHTMLParser object when searching for the <h1> tag. The MyHTMLParser class creates instances of a helper class named Tag. The tag class is used in creating a "linked list" of tags.

So to get started with WebCrawler make sure to use Python 2.7.2. Enter the code a piece at a time into IDLE in the order displayed below. This ensures that you import libs before you start using them.

Once you have entered all the code into IDLE, you can start crawling the 'interwebs' by entering the following:

import re

cnn_url_regex = re.compile('(?<=[.]cnn)[.]com')

w = WebCrawler(cnn_url_regex)

w.start_crawling(['http://www.cnn.com/2012/02/24/world/americas/haiti-pm-resigns/index.html?hpt=hp_t3'],1)

Of course you can enter any page you want. But the regular expression object is already setup to filter on cnn.com. Remember the second parameter passed into the start_crawling function is the recursion depth.

Happy Crawling!

Improvements of the urllib.URLopen.retrieve() method (Python)

2010-01-16T04:50:07-08:00

Python recipe 577009 by Kévin Gomez (retrieve, urllib).

I improved the urllib.URLopen.retrieve() method so that it can restart a download if it failed. And like wget does (with wget -c), it restarts where it stopped. The number of maximum tries can be changed.

Fetch all (new) xkcd strips (Python)

2009-08-15T12:15:17-07:00

Python recipe 576881 by xipe totec (file_download, urllib).

Downloads and saves all xkcd strips (with the exception of #404, as it's intentionally left 404...)