Welcome, guest | Sign In | My Account | Store | Cart

urllib2 for actions depending on http response codes (Python recipe) by Lee Harr
ActiveState Code (http://code.activestate.com/recipes/267197/)

2

I wanted to touch a particular web page (in order to open/close a database connection inside of Zope) so I came up with this module which uses urllib2 to make the web connection

I was not sure what a 'realm' was, so first I made the HTTPRealmFinder to find out what the realm is.

The HTTPinger calls my required page and acts according to the http return code.

Python, 92 lines
Download
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
# I wanted to touch a particular web page (in order to open/
# close a database connection inside of Zope) so I came
# up with this module which uses urllib2 to make the
# web connection
#
# I was not sure what a 'realm' was, so first I made the
# HTTPRealmFinder to find out what the realm is.
# The HTTPinger calls my required page and acts according
# to the http return code.


import urllib2
from urlparse import urlparse


class HTTPinger:
    def ping(self, url, webuser, webpass):
        scheme, domain, path, x1, x2, x3 = urlparse(url)
        
        finder = HTTPRealmFinder(url)
        realm = finder.get()
        
        handler = urllib2.HTTPBasicAuthHandler()
        handler.add_password(realm, domain, webuser, webpass)
        
        opener = urllib2.build_opener(handler)
        urllib2.install_opener(opener)
        
        try:
            urllib2.urlopen(url)
        except urllib2.HTTPError, e:
            if e.code == 401:
                print 'not authorized'
            elif e.code == 404:
                print 'not found'
            elif e.code == 503:
                print 'service unavailable'
            else:
                print 'unknown error: '
        else:
            print 'success'


class HTTPRealmFinderHandler(urllib2.HTTPBasicAuthHandler):
    def http_error_401(self, req, fp, code, msg, headers):
        realm_string = headers['www-authenticate']
        
        q1 = realm_string.find('"')
        q2 = realm_string.find('"', q1+1)
        realm = realm_string[q1+1:q2]
        
        self.realm = realm

        
class HTTPRealmFinder:
    def __init__(self, url):
        self.url = url
        scheme, domain, path, x1, x2, x3 = urlparse(url)
        
        handler = HTTPRealmFinderHandler()
        handler.add_password(None, domain, 'foo', 'bar')
        self.handler = handler
        
        opener = urllib2.build_opener(handler)
        urllib2.install_opener(opener)

    def ping(self, url):
        try:
            urllib2.urlopen(url)
        except urllib2.HTTPError, e:
            pass

    def get(self):
        self.ping(self.url)
        try:
            realm = self.handler.realm
        except AttributeError:
            realm = None
        
        return realm

    def prt(self):
        print self.get()

if __name__ == '__main__':
    pinger = HTTPinger()
    pinger.ping('https://example.com/nonexistent/path', 'username', 'password')
    pinger = HTTPinger()
    pinger.ping('https://example.com/basic/auth/protected/path', 'username', 'password')

    finder = HTTPRealmFinder('https://example.com/basic/auth/protected/path')
    finder.prt()

My first thought was to use httplib, but I could not see a way to do the basic HTTP auth with that, so I turned to urllib2.

There has been some discussion lately about augmenting the urllib2 examples, but it still was not very clear how to do what I needed to do. Hopefully this will help the next person.

http://python.org/doc/current/lib/module-httplib.html http://python.org/doc/current/lib/module-urllib2.html http://python.org/doc/current/lib/module-urlparse.html

Tags: web

2 comments

Ian Bicking 9 years, 3 months ago  # | flag

HTTP Basic auth. HTTP basic auth is implemented with a single header:

Authorization: Basic [base64-encoded-string]

The base64-encoded string is constructed like:

base64.encode('%s:%s' % (username, password))

urllib2.HTTPPasswordMgrWithDefaultRealm also does what you want, allowing you to use None as a generic realm when no other realm matches.

Rene Olsthoorn 8 years, 11 months ago  # | flag

Thanks Ian, it works. The following code, I use to access my E-Tech router HTML-page, which is authenticated:

import urllib2, base64

request = urllib2.Request('http://192.168.1.1/Status.htm')
base64string = base64.encodestring('%s:%s' % ('user', 'password'))[:-1]
request.add_header("Authorization", "Basic %s" % base64string)

htmlFile = urllib2.urlopen(request)
htmlData = htmlFile.read()
print htmlData
htmlFile.close()

Add a comment

Sign in to comment