Welcome, guest | Sign In | My Account | Store | Cart

This recipe shows how you can grab a document from the web using urllib.py.

Python, 4 lines
1
2
3
4
from urllib import urlopen

doc = urlopen("http://www.python.org").read()
print doc

6 comments

William Trenker 20 years, 6 months ago  # | flag

Grab a document from the web. This is an amazing example of the power of python. These one-liners are great for beginners like me who want to tap into this power right up front!

How about a "Python Power" category for these simple but "not so obvious to the newbie" power tips.

Baptiste Lepilleur 18 years, 4 months ago  # | flag

Adding support for proxy. Nice, but any real-life usage require a proxy.

none none 18 years, 4 months ago  # | flag

Proxy in python. Set an environment variable HTTP_PROXY to your proxyserver:port So it'll look something like this:

set HTTP_PROXY=http://proxy.domain.com:8080

You need to have the http:// in front... or else it won't work!

Cheers,

Kraulin

Laszlo Kohegyi 17 years, 8 months ago  # | flag

Support for proxy authentication? Is there any way to work with a proxy server that requires authentication?

Magnus Bodin 17 years, 8 months ago  # | flag

Proxy auth urllib. There is an example here:

http://pydoc.org/2.3/urllib2.html



import urllib2

# set up authentication info
authinfo = urllib2.HTTPBasicAuthHandler()
authinfo.add_password('realm', 'host', 'username', 'password')

proxy_support = urllib2.ProxyHandler({"http" : "http://ahad-haam:3128"})

# build a new opener that adds authentication and caching FTP handlers
opener = urllib2.build_opener(proxy_support, authinfo, urllib2.CacheFTPHandler)

# install it
urllib2.install_opener(opener)

f = urllib2.urlopen('http://www.python.org/')
Abhineshwar Tomar 12 years, 10 months ago  # | flag

thats the thing about python, simple yet so powerful.

Created by Gisle Aas on Fri, 23 Feb 2001 (PSF)
Python recipes (4591)
Gisle Aas's recipes (3)

Required Modules

Other Information and Tasks