Welcome, guest | Sign In | My Account | Store | Cart
8

A scripted web client that will post data to a site as if from a form using ENCTYPE="multipart/form-data". This is typically used to upload files, but also gets around a server's (e.g. ASP's) limitation on the amount of data that can be accepted via a standard POST (application/x-www-form-urlencoded).

Python, 47 lines
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
import httplib, mimetypes

def post_multipart(host, selector, fields, files):
    """
    Post fields and files to an http host as multipart/form-data.
    fields is a sequence of (name, value) elements for regular form fields.
    files is a sequence of (name, filename, value) elements for data to be uploaded as files
    Return the server's response page.
    """
    content_type, body = encode_multipart_formdata(fields, files)
    h = httplib.HTTP(host)
    h.putrequest('POST', selector)
    h.putheader('content-type', content_type)
    h.putheader('content-length', str(len(body)))
    h.endheaders()
    h.send(body)
    errcode, errmsg, headers = h.getreply()
    return h.file.read()

def encode_multipart_formdata(fields, files):
    """
    fields is a sequence of (name, value) elements for regular form fields.
    files is a sequence of (name, filename, value) elements for data to be uploaded as files
    Return (content_type, body) ready for httplib.HTTP instance
    """
    BOUNDARY = '----------ThIs_Is_tHe_bouNdaRY_$'
    CRLF = '\r\n'
    L = []
    for (key, value) in fields:
        L.append('--' + BOUNDARY)
        L.append('Content-Disposition: form-data; name="%s"' % key)
        L.append('')
        L.append(value)
    for (key, filename, value) in files:
        L.append('--' + BOUNDARY)
        L.append('Content-Disposition: form-data; name="%s"; filename="%s"' % (key, filename))
        L.append('Content-Type: %s' % get_content_type(filename))
        L.append('')
        L.append(value)
    L.append('--' + BOUNDARY + '--')
    L.append('')
    body = CRLF.join(L)
    content_type = 'multipart/form-data; boundary=%s' % BOUNDARY
    return content_type, body

def get_content_type(filename):
    return mimetypes.guess_type(filename)[0] or 'application/octet-stream'

At Python 9, Moshe Zadka showed how to create the multipart-mime data using MimeWriter ( http://www.python9.org/p9-zadka.ppt ). His recipe worked just fine for me when I was talking to a Zope server, but triggered a cryptic error message when pointed at an ASP server that was using the COM file upload component from persits.com .

The main problem with MimeWriter is that it does not use '\r\n' for newlines, and the ASP server insisted on this. Other bits of persnicketiness on the part of the ASP included rejection of a mime message that had content-type headers in its regular form fields, and insistence on at least five dashes at the front of the boundary marker.

The function encode_multipart_formdata() shown here takes a more direct approach to creating the mime data, and fairly closely mimics the data sent by Internet Explorer 5.5.

24 comments

l-activestate.com 12 years, 6 months ago  # | flag

python mod. I made a wrapper to urllib2.urlopen() in order to support file uploading

http://fabien.seisen.org/python/

It uses boundary creation from mimetools and doesn't read the whole file in memory

import urllib2_file
import urllib2

data = {'name': 'value',
        'file':  open('/etc/services')
       }
urllib2.urlopen('http://site.com/script_upload.php', data)
Chris Green 12 years, 3 months ago  # | flag

using urls.

import urlparse

def posturl(url, fields, files):
    urlparts = urlparse.urlsplit(url)
    return post_multipart(urlparts[1], urlparts[2], fields,files)

This allows you to specify the form as a url and not worry about host and selector.

chris hoke 11 years, 10 months ago  # | flag

Update to use HTTPConnection. simple update to use HTTPConnection instead of HTTP for the recipe to simplify it and also to use HTTP 1.1.

Only replace first function of the recipe:

def post_multipart(host, selector, fields, files):
    content_type, body = encode_multipart_formdata(fields, files)
    h = httplib.HTTPConnection(host)
    headers = {
        'User-Agent': 'INSERT USERAGENTNAME',
        'Content-Type': content_type
        }
    h.request('POST', selector, body, headers)
    res = h.getresponse()
    return res.status, res.reason, res.read()

Should work as the original version.

chris hoke 11 years, 10 months ago  # | flag

extended return... new version does additionally return Status and Reason information. For exact same return of original version replace

res.status, res.reason, res.read()

with

res.read()
James Jurack 10 years, 4 months ago  # | flag

With cookie support on Python 2.4. Here's a version of your code that supports cookies with python 2.4's urllib2 and cookielib.

import httplib, mimetypes, mimetools, urllib2, cookielib

cj = cookielib.CookieJar()
opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cj))
urllib2.install_opener(opener)

def post_multipart(host, selector, fields, files):
    """
    Post fields and files to an http host as multipart/form-data.
    fields is a sequence of (name, value) elements for regular form fields.
    files is a sequence of (name, filename, value) elements for data to be uploaded as files
    Return the server's response page.
    """
    content_type, body = encode_multipart_formdata(fields, files)
    headers = {'Content-Type': content_type,
               'Content-Length': str(len(body))}
    r = urllib2.Request("http://%s%s" % (host, selector), body, headers)
    return urllib2.urlopen(r).read()

def encode_multipart_formdata(fields, files):
    """
    fields is a sequence of (name, value) elements for regular form fields.
    files is a sequence of (name, filename, value) elements for data to be uploaded as files
    Return (content_type, body) ready for httplib.HTTP instance
    """
    BOUNDARY = mimetools.choose_boundary()
    CRLF = '\r\n'
    L = []
    for (key, value) in fields:
        L.append('--' + BOUNDARY)
        L.append('Content-Disposition: form-data; name="%s"' % key)
        L.append('')
        L.append(value)
    for (key, filename, value) in files:
        L.append('--' + BOUNDARY)
        L.append('Content-Disposition: form-data; name="%s"; filename="%s"' % (key, filename))
        L.append('Content-Type: %s' % get_content_type(filename))
        L.append('')
        L.append(value)
    L.append('--' + BOUNDARY + '--')
    L.append('')
    body = CRLF.join(L)
    content_type = 'multipart/form-data; boundary=%s' % BOUNDARY
    return content_type, body

def get_content_type(filename):
    return mimetypes.guess_type(filename)[0] or 'application/octet-stream'
Will Holcomb 10 years, 3 months ago  # | flag

a less intrusive version using the urllib2 hierarchy. Here is the same basic idea, but using a class inherited into the BasicHandler hierarchy of urllib2. It has the advantage of leaving all the existing urllib2 functionality intact.

Example usage:

import MultipartPostHandler, urllib2, cookielib

cookies = cookielib.CookieJar()
opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cookies),
                                MultipartPostHandler.MultipartPostHandler)
params = { "username" : "bob", "password" : "riviera",
           "file" : open("filename", "rb") }
opener.open("http://wwww.bobsite.com/upload/", params)

The code is at: http://odin.himinbi.org/MultipartPostHandler.py

Brian Schneider 8 years, 11 months ago  # | flag

MultipartPostHandler didn't work for unicode files. I fixed it by reading in via StringIO class.

fix posted here:

http://peerit.blogspot.com/2007/07/multipartposthandler-doesnt-work-for.html

Thomas Guettler 8 years, 9 months ago  # | flag

Script which can be called from the command line. This script is based on this recipe and can be called from the shell:

http://fabien.seisen.org/python/urllib2_multipart.html

Tim Keating 8 years, 5 months ago  # | flag

Do NOT use this script verbatim. It relies on a deprecated backward-compatibility module in httplib that didn't work at all for me. When I switched to using the more modern HTTPConnection version (described by another commenter above) it worked correctly first time out of the box, so I strongly encourage you to use that instead.

Lee June 8 years ago  # | flag

how to deal this situation? I have a question in my case. I cannot find the solution, can anybody me out? thank.

Even for logout action, I have to supply the 'data' var as following

[code works]
import MultipartPostHandler, cookielib
cookies = cookielib.CookieJar()
opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cookies),
MultipartPostHandler.MultipartPostHandler)
urllib2.install_opener(opener)
data = urllib.urlencode({'usr':'myname','pwd':'mypwd',})
request = urllib2.Request('http://host/cgi-bin/bbslogin', data)
data=urllib2.open(request).read()
utmpnum, utmpkey, utmpuserid=ReadFromWeb(data)
data = urllib.urlencode({'utmpnum':utmpnum,'utmpkey':utmpkey,'utmpuserid':utmpuserid})
request = urllib2.Request('http://host/cgi-bin/bbslogout', data)
[/code works]

If I do not supply 'data', I was told "you are not logged in"

So, in this case, how can I supply all of utmpnum, utmpkey, utmpuserid and the attached file?

I have this code, but the response still says "you are not logged in"!

[code does not work]
(this is previous log in code)

bbs_att_url='http://host/cgi-bin/bbsdoupload'

data = {
'utmpnum':utmpnum,
'utmpkey':utmpkey,
'utmpuserid':utmpuserid,
'upfile' : open("myfile.ico", "rb") ,
}
data=urllib.urlencode(data)
request = urllib2.Request(bbs_att_url, data)
fd=urllib2.urlopen(request)
data=fd.read()
print data            #"you are not logged in" can be found
[/code does not work]
Robert Lujo 7 years, 11 months ago  # | flag

Take a look at recipe 576422: Python HTTP POST binary file upload with pycurl (http://code.activestate.com/recipes/576422/)

F. Steinel 7 years, 8 months ago  # | flag

See also Python Bug 3244, target => Python 2.7

İlkin Balkanay 6 years, 11 months ago  # | flag

I am trying to upload an image to flickr using flickr upload API. The recipe raises UnicodeDecodeError when trying to join '\r\n' and file.read() How can I construct the body of the post data if i want to post an image (binary file)?

İlkin Balkanay 6 years, 11 months ago  # | flag

body = CRLF.join([element.decode('string_escape') for element in L]) worked for me.

Stephen Gornick 6 years, 2 months ago  # | flag

Might want to take a look at: http://atlee.ca/software/poster

Sérgio M.B. 5 years, 1 month ago  # | flag

MultipartPostHandler.py is here http://pypi.python.org/pypi/MultipartPostHandler/0.1.0 but don't have Unicode files patch which it is here http://peerit.blogspot.com/2007/07/multipartposthandler-doesnt-work-for.html

It works very well thanks, but shouldn't be updated

Éric Araujo 4 years, 8 months ago  # | flag

We’re using a version of this recipe in distutils2/packaging, thanks!

itai 4 years, 6 months ago  # | flag

To make it possible to post binary files, change this line:

body = CRLF.join(L)

to this:

s = BytesIO()
for element in L:
    s.write(str(element))
    s.write(CRLF)
body = s.getvalue()
Sérgio M.B. 4 years, 4 months ago  # | flag

http://pypi.python.org/pypi/MultipartPostHandler2 , project created by be, with fix for utf-8 systems.

thjingkl steem 4 years, 3 months ago  # | flag

I am looking to make a program that would upload all the files in a specific directory to mixcloud. For that, you have to make a multipart/form-data post to https://api.mixcloud.com/upload/ along with the name and mp3 file. Can anyone help me, i'm new at programming.

Rangarajan 3 years, 5 months ago  # | flag

Hello! I actually need a python code that can upload files from a mobile sd card to PC hard disk at regular intervals. I already have http server i created running in my mobile that handles some web page calls by one of my android application. On the other hand. I need to send a get method call or something from a PC connected to the mobile via hotspot, where the server responds to it and sends image files present at some location in the SD card to some location in the computer. Both the PC calls and the apps html calls are independent but handled by the same http server code i wrote. I could not find any solution to this so for now I wrote a socket program that is doing this for me but I feel it would be nice if it was done over http. Can any one suggest me how this could be done?

Erik Telepovský 4 months ago  # | flag

Using requests library:

def post_multipart(host, selector, fields, files):
    import requests
    content_type, body = encode_multipart_formdata(fields, files)
    headers = {
        'content-type': content_type,
        'content-length': str(len(body))
    }
    r = requests.post(selector, data=body, headers=headers)
    return r.text

Add a comment

Sign in to comment