Welcome, guest | Sign In | My Account | Store | Cart

This script allows how to transparently install a HTTP proxy (proxy HTTP 1.1, using CONNECT command) on all outgoing sockets.

I did that to bring TCP over HTTP to FTPlib, transparently. It should enable HTTP tunneling for all methods / modules that use the low-level socket API.

Python, 144 lines
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
#!/usr/bin/python
import socket

# Class that wraps a real socket and changes it to a HTTP tunnel whenever a connection is asked via the "connect" method
class ProxySock :
    
    def __init__(self, socket, proxy_host, proxy_port) : 

        # First, use the socket, without any change
        self.socket = socket

        # Create socket (use real one)
        self.proxy_host = proxy_host
        self.proxy_port = proxy_port

        # Copy attributes
        self.family = socket.family
        self.type = socket.type
        self.proto = socket.proto

    def connect(self, address) :

        # Store the real remote adress
        (self.host, self.port) = address
       
        # Try to connect to the proxy 
        for (family, socktype, proto, canonname, sockaddr) in socket.getaddrinfo(
            self.proxy_host, 
            self.proxy_port,
            0, 0, socket.SOL_TCP) :
            try:
                
                # Replace the socket by a connection to the proxy
                self.socket = socket.socket_formal(family, socktype, proto)
                self.socket.connect(sockaddr)
                    
            except socket.error, msg:
                if self.socket:
                    self.socket.close()
                self.socket = None
                continue
            break
        if not self.socket :
            raise socket.error, ms 
        
        # Ask him to create a tunnel connection to the target host/port
        self.socket.send(
                ("CONNECT %s:%d HTTP/1.1\r\n" + 
                "Host: %s:%d\r\n\r\n") % (self.host, self.port, self.host, self.port));

        # Get the response
        resp = self.socket.recv(4096)

        # Parse the response
        parts = resp.split()
        
        # Not 200 ?
        if parts[1] != "200" :
            raise Exception("Error response from Proxy server : %s" % resp)

    # Wrap all methods of inner socket, without any change
    def accept(self) :
        return self.socket.accept()

    def bind(self, *args) :
        return self.socket.bind(*args)
    
    def close(self) :
        return self.socket.close()
    
    def fileno(self) :
        return self.socket.fileno()

    
    def getsockname(self) :
        return self.socket.getsockname()
    
    def getsockopt(self, *args) :
        return self.socket.getsockopt(*args)
    
    def listen(self, *args) :
        return self.socket.listen(*args)
    
    def makefile(self, *args) :
        return self.socket.makefile(*args)
    
    def recv(self, *args) :
        return self.socket.recv(*args)
    
    def recvfrom(self, *args) :
        return self.socket.recvfrom(*args)

    def recvfrom_into(self, *args) :
        return self.socket.recvfrom_into(*args)
    
    def recv_into(self, *args) :
        return self.socket.recv_into(buffer, *args)
    
    def send(self, *args) :
        return self.socket.send(*args)
    
    def sendall(self, *args) :
        return self.socket.sendall(*args)
    
    def sendto(self, *args) :
        return self.socket.sendto(*args)
    
    def setblocking(self, *args) :
        return self.socket.setblocking(*args)
    
    def settimeout(self, *args) :
        return self.socket.settimeout(*args)
    
    def gettimeout(self) :
        return self.socket.gettimeout()
    
    def setsockopt(self, *args):
        return self.socket.setsockopt(*args)
    
    def shutdown(self, *args):
        return self.socket.shutdown(*args)

    # Return the (host, port) of the actual target, not the proxy gateway
    def getpeername(self) :
        return (self.host, self.port)

# Install a proxy, by changing the method socket.socket()
def setup_http_proxy(proxy_host, proxy_port) :

    # New socket constructor that returns a ProxySock, wrapping a real socket
    def socket_proxy(af, socktype, proto) :

        # Create a socket, old school :
        sock = socket.socket_formal(af, socktype, proto)

        # Wrap it within a proxy socket
        return ProxySock(
                sock,
                proxy_host, 
                proxy_port)

    # Replace the "socket" method by our custom one
    socket.socket_formal = socket.socket
    socket.socket = socket_proxy

Usage

Just call the setup_http_proxy method

setup_http_proxy("my.proxy", 8080)

Example

Here is a sample script that does FTP over HTTP

# Imports
from ftplib import FTP
from StringIO import StringIO

# Constants
PROXY_HOST="my.proxy"
PROXY_PORT=8080
FTP_HOST="my.ftp.site"
FTP_USER="anonymous"
FTP_PASS="anonymous"
FTP_FILE="/path.to/file.txt"
FTP_DIR="/path/to/folder/"

# Install the proxy on the "socket" module
setup_http_proxy(
        PROXY_HOST, 
        PROXY_PORT)

# Now, the libftp module can do FTP over HTTP !!, transaprently
ftp = FTP(
        FTP_HOST, 
        FTP_USER, 
        FTP_PASS)

# List content of a folder
print "Content of folder '%s' : %s" % (FTP_DIR, ftp.nlst(FTP_DIR))

# Get one file into buffer
buffer= StringIO()
ftp.retrlines("retr %s" % FTP_FILE, lambda line : buffer.write(line + "\n"))

# Print it
print "Content of file '%s' :\n%s" % (
        FTP_FILE, 
        buffer.getvalue()

Limitations, bugs

The next step should be to enable t black list of "no_proxy" hosts (like 'localhost' or '*.mydomain.com'). It should be fairly easy to do.

7 comments

test 12 years, 9 months ago  # | flag

I'm behind a proxy server right now. Usually I connect with Filezilla over HTTP\1.1 using CONNECT method, but for some tasks I need more functionality. When trying to connect using your module I get this exception:

Traceback (most recent call last):
  File "C:\Python27\MyScripts\ftp_client\main.py", line 19, in <module>
    ftp = FTP(FTP_HOST, FTP_USER, FTP_PASS)
  File "C:\Python27\lib\ftplib.py", line 117, in __init__
    self.connect(host)
  File "C:\Python27\lib\ftplib.py", line 133, in connect
    self.af = self.sock.family
AttributeError: ProxySock instance has no attribute 'family'

Is there a way to fix the problem? I'm using Python2.7

Raphaël Jolivet (author) 12 years, 9 months ago  # | flag

Ok I didn't copy the attribute from the wrapped socket. It worked fine for me.

Let's add some code in the ProxySock constructor : self.family = socket.family self.type = socket.type self.proto = socket.proto

I have updated the code. Please try again and tell if it works.

test 12 years, 9 months ago  # | flag

Now it works fine, thanks. Now I have all the instruments I need to work over an ftp

Raphaël Jolivet (author) 12 years, 9 months ago  # | flag

Thanks to you for your feedback. I'm glad to see it is useful to someone. That's the reason why I share code snipnets here.

ms4py 12 years, 5 months ago  # | flag

You should use a new-style class (class ProxySock(object)) with __getattr__ instead of copying attributes and wrapping methods:

def __getattr__(self, name):
    '''Automatically wrap methods and attributes for socket object.'''
    return getattr(self.socket, name)
fizejfi 10 years, 9 months ago  # | flag

Hello Raphael, My proxy needs an authentification with a username and a password. (different from the username and the password of the ftp I try to reach) How can I implement it with your solution ? Thanks

Radosław Hagno 7 years, 6 months ago  # | flag

Hello Raphael, I'm using Python 3.5 and I get TypeError was unhandled by user code Message: makefile() got an unexpected keyword argument 'encoding'

during creation of FTP.

Can you please help me fix it?