ActiveState Code

Recipe 52199: Grab a document from the web

This recipe shows how you can grab a document from the web using urllib.py.

Python 
1
2
3
4
from urllib import urlopen

doc = urlopen("http://www.python.org").read()
print doc

Comments

  1. 1. At 3:13 p.m. on 10 jun 2001, William Trenker said:

    Grab a document from the web. This is an amazing example of the power of python. These one-liners are great for beginners like me who want to tap into this power right up front!

    How about a "Python Power" category for these simple but "not so obvious to the newbie" power tips.

  2. 2. At 4:51 a.m. on 23 jul 2003, Baptiste Lepilleur said:

    Adding support for proxy. Nice, but any real-life usage require a proxy.

  3. 3. At 6:12 a.m. on 1 aug 2003, none none said:

    Proxy in python. Set an environment variable HTTP_PROXY to your proxyserver:port So it'll look something like this:

    set HTTP_PROXY=http://proxy.domain.com:8080

    You need to have the http:// in front... or else it won't work!

    Cheers,

    Kraulin

  4. 4. At 7:50 a.m. on 1 apr 2004, Laszlo Kohegyi said:

    Support for proxy authentication? Is there any way to work with a proxy server that requires authentication?

  5. 5. At 1:07 p.m. on 8 apr 2004, Magnus Bodin said:

    Proxy auth urllib. There is an example here:

    http://pydoc.org/2.3/urllib2.html



import urllib2

# set up authentication info
authinfo = urllib2.HTTPBasicAuthHandler()
authinfo.add_password('realm', 'host', 'username', 'password')

proxy_support = urllib2.ProxyHandler({"http" : "http://ahad-haam:3128"})

# build a new opener that adds authentication and caching FTP handlers
opener = urllib2.build_opener(proxy_support, authinfo, urllib2.CacheFTPHandler)

# install it
urllib2.install_opener(opener)

f = urllib2.urlopen('http://www.python.org/')
  6. 6. At 10:05 a.m. on 13 jan 2009, Abhineshwar Tomar said:

    thats the thing about python, simple yet so powerful.

Sign in to comment