Welcome, guest | Sign In | My Account | Store | Cart
1

This recipe shows how you can grab a document from the web using urllib.py.

Python, 4 lines
1
2
3
4
from urllib import urlopen

doc = urlopen("http://www.python.org").read()
print doc

6 comments

William Trenker 12 years, 10 months ago  # | flag

Grab a document from the web. This is an amazing example of the power of python. These one-liners are great for beginners like me who want to tap into this power right up front!

How about a "Python Power" category for these simple but "not so obvious to the newbie" power tips.

Baptiste Lepilleur 10 years, 8 months ago  # | flag

Adding support for proxy. Nice, but any real-life usage require a proxy.

none none 10 years, 8 months ago  # | flag

Proxy in python. Set an environment variable HTTP_PROXY to your proxyserver:port So it'll look something like this:

set HTTP_PROXY=http://proxy.domain.com:8080

You need to have the http:// in front... or else it won't work!

Cheers,

Kraulin

Laszlo Kohegyi 10 years ago  # | flag

Support for proxy authentication? Is there any way to work with a proxy server that requires authentication?

Magnus Bodin 10 years ago  # | flag

Proxy auth urllib. There is an example here:

http://pydoc.org/2.3/urllib2.html



import urllib2

# set up authentication info
authinfo = urllib2.HTTPBasicAuthHandler()
authinfo.add_password('realm', 'host', 'username', 'password')

proxy_support = urllib2.ProxyHandler({"http" : "http://ahad-haam:3128"})

# build a new opener that adds authentication and caching FTP handlers
opener = urllib2.build_opener(proxy_support, authinfo, urllib2.CacheFTPHandler)

# install it
urllib2.install_opener(opener)

f = urllib2.urlopen('http://www.python.org/')
Abhineshwar Tomar 5 years, 3 months ago  # | flag

thats the thing about python, simple yet so powerful.

Add a comment

Sign in to comment