Welcome, guest | Sign In | My Account | Store | Cart

This recipe shows how you can grab a document from the web using urllib.py.

Python, 4 lines
from urllib import urlopen

doc = urlopen("http://www.python.org").read()
print doc


William Trenker 15 years, 1 month ago  # | flag

Grab a document from the web. This is an amazing example of the power of python. These one-liners are great for beginners like me who want to tap into this power right up front!

How about a "Python Power" category for these simple but "not so obvious to the newbie" power tips.

Baptiste Lepilleur 13 years ago  # | flag

Adding support for proxy. Nice, but any real-life usage require a proxy.

none none 13 years ago  # | flag

Proxy in python. Set an environment variable HTTP_PROXY to your proxyserver:port So it'll look something like this:

set HTTP_PROXY=http://proxy.domain.com:8080

You need to have the http:// in front... or else it won't work!



Laszlo Kohegyi 12 years, 4 months ago  # | flag

Support for proxy authentication? Is there any way to work with a proxy server that requires authentication?

Magnus Bodin 12 years, 3 months ago  # | flag

Proxy auth urllib. There is an example here:


import urllib2

# set up authentication info
authinfo = urllib2.HTTPBasicAuthHandler()
authinfo.add_password('realm', 'host', 'username', 'password')

proxy_support = urllib2.ProxyHandler({"http" : "http://ahad-haam:3128"})

# build a new opener that adds authentication and caching FTP handlers
opener = urllib2.build_opener(proxy_support, authinfo, urllib2.CacheFTPHandler)

# install it

f = urllib2.urlopen('http://www.python.org/')
Abhineshwar Tomar 7 years, 6 months ago  # | flag

thats the thing about python, simple yet so powerful.

Add a comment

Sign in to comment