socket.recv -- three ways to turn it into recvall (Python)

2005-04-05T13:47:57-07:00

Python recipe 408859 by John Nielsen (network). Revision 5.

An issue with socket.recv is how to know when you are done receiving data. A TCP stream guarantees the bytes will not arrive out of order or be sent more than once. But you do not know the size of the data that will be sent to you. 100 bytes could be sent as group of 10 bytes or maybe in one shot. Ultimately, this means you have to use a loop in some fashion until you know it is done.

The basic recv returns an empty string when the socket is disconnected. From that you can build a simple loop that will work as long as the sender manages to disconnect the socket at the appropriate time. However, there could be situations where a local error will mask as a clean shutdown or maybe a close() is never called.

Three very basic methods are shown below that try to fix that problem. They use either a time-based, end marker, or size of payload method. Since you cannot be sure just what you are going to receive, you have to be careful that you get enough of a message to determine the size of payload or end marker.

I updated the recv_size method to allocate data in larger chunks if it gets a large stream of data, which can increase performance.

When to not just use socket.close() (Python)

2005-04-08T17:31:14-07:00

Python recipe 408997 by John Nielsen (network). Revision 5.

I have implemented a "broken" client/server to show how socket.shutdown can be more useful than a simple socket.close operation. (I rewrote this to hopefully be more clear)

The close operation is not atomic. It implicitly tries to send any remaining data in _addition_ to closing a descriptor. Splitting this close operation up with the aid of the shutdown command can help avoid bugs. It gives the server one final way to say, "something went wrong". The server would also know that the client did not end correctly, since the socket should remain open when the client finished sending data. For example, if the function exits unexpectedly and python closes the socket for you, the server would not be able to send any data back.

In the server below, the client and server have different ideas about what the end marker should be. The rev_end function is written so as to look for an end marker. And, as long as they agree it should work. The socket.shutdown is for when something goes wrong.

simplest useful (I hope!) thread pool example (Python)

2004-09-08T13:53:48-07:00

Python recipe 302746 by John Nielsen . Revision 6.

You can find examples on how to do threading, but they do not show off a thread pool. My goal was to get as small and simple as possible working thread pool example to show off the basic ideas without having extraneous things to understand. To show off the thread pool, I want stopping and starting of the threads to be explicit. This means the pool won't start until you are ready and will run forever until you are ready for it to stop. The main thread puts into the input queue and removes data from the output queue. The thread pool simply does the converse. Errors are also managed with another queue, so there is a clean distinction between errors and successful results.

simplest useful HTTPS with basic proxy authentication (Python)

2005-12-28T17:27:47-08:00

Python recipe 301740 by John Nielsen (network). Revision 4.

This is just about the most simple snippet of how to do proxy authentication with SSL using python. The current httplib only supports ssl through a proxy _without_ authentication. This example does basic proxy auth that a lot of proxy servers can support. This will at least give someone an idea of how to do it and then improve it and incorporate it however they want.

changing file attributes on windows (Python)

2004-09-03T11:54:59-07:00

Python recipe 303343 by John Nielsen (sysadmin).

The win32api module offers SetFileAttributes whiles allows you to make changes to a file in windows. You can set a file to be read only, archive, hidden, etc. The function is simple and convenient to use.

How to use string.Template from python 2.4 (Python)

2004-09-13T10:55:54-07:00

Python recipe 304004 by John Nielsen (text). Revision 3.

New to python 2.4 is the string.Template class. It is similar to the formatting python already had in which the % operator with strings was used in a recognizable sprintf style of formatting from C. In addition to format types, you also had a choice with a tuple or dictionary mapping. String.Template simplifies it to only the string format type and dictionary mapping. This makes string formatting easier to remember and use. This example shows you the basics of using the template and the difference with the old style.

simple ldap with python (Python)

2004-09-03T09:35:09-07:00

Python recipe 303336 by John Nielsen (network).

The ldap library at http://python-ldap.sourceforge.net wraps the Openldap C api. It can talk to various versions of ldap servers not just the Openldap servers. Note the use of the '_s' methods like search_s which all are synchronous. Here are some simple examples showing one how to use the library.

getting process information on windows (Python)

2004-09-03T11:10:53-07:00

Python recipe 303339 by John Nielsen (sysadmin).

To get process information for both NT and W2K (but not the 9x family) you can use the Performance Data Helper library(PDH). The is a simple example that shows you how to get a process list and their ids. It provides a convenient interface to performance information stored fundamentally in the registry. The basic process of using the PDH encompasses the following:

Get a list of all the objects you want
Get a list of the object's instances and data available for each instance: called 'items' or 'counters'
Get a group of performance data for each counter

In the case here we want the process object, the object's instances are it's list of processes, and the counter we want for the processes is 'ID Process'.

create an account in MS active directory (Python)

2004-09-03T13:58:34-07:00

Python recipe 303345 by John Nielsen (sysadmin).

This is the basic code to create an account in active directory that shows how to do things like set your own exension attributes, force a reset of the password when the user logs in, and set the home directory. It makes used of python's excellent COM support in win32com.

sorting -- old to new python 2.4 style (heapq,bisect,list.sort keywords,itemgetter) (Python)

2006-09-13T12:28:57-07:00

Python recipe 305304 by John Nielsen (algorithms). Revision 2.

Python 2.4 has added a new feature to handle the problem where you needed to do a sort based off of part of the data. In effect, it has simplified the Schwartzian Transform (which I learned many years ago from an perl article written by Randall Schwartz). The following code will start with the older style python sorting approaches, and then show the bisect and heapq modules, and finally the key,cmp,reverse methods newly added to sort. The sort method of the list is an in-place sort. There is also a new sorted() function which returns a copy of the list sorted for you.

pytunnel: tunneling w/threads (ssl thru proxy in this case) (Python)

2005-12-30T14:16:43-08:00

Python recipe 213238 by John Nielsen . Revision 4.

This code shows an implementation of tunneling. Though this code uses ssl as an example. It would not be hard to modify it to work for other situations as well.

The reason I use ssl as an example is because the standard python libraries do not support tunneling ssl through a proxy. Import pytunnel and give it a function w/the code you want to tunnel and your off.

For the latest code try: http://ftp.gnu.org/pub/savannah/cvs/pytunnel-cvs-latest.tar.gz

simple example to show off itertools.tee (Python)

2004-09-23T22:01:49-07:00

Python recipe 305588 by John Nielsen . Revision 7.

Itertools.tee offers an interesting way to "remember" things that have happened. Itertools.tee makes multiple iterators from one (if you still have an the original iterator you do not use it). When you advance iterator 1 but not iterator 2, iterator 2 stays behind. Which means, if you later advance iterator 2, it the goes forward through the same data.

In this example, I use iterator.tee to make 2 iterators, to allow an action to affect data that has been processed in the past. The first iterator, it_main, is what is used to process data normally in this case to do something like display an image selected. The second iterator, it_history, stays behind the first and only advances when a specific action arrives. In effect, it rolls forward through the data that it_main has already processed.

simple mx.ODBC example (Python)

2005-02-25T07:16:43-08:00

Python recipe 389535 by John Nielsen (database).

Mx.odbc is cross platform and very fast, I have used it to go through a few hundred thousand rows of an access database in seconds where pure ADO would take 30 min (or 3 min if you use the ADO type library). Here is a simple example of how to talk to a database, in this case an access file, get the columns of a table and get data from the table.

converting windows 64 bit time to python useable format (Python)

2004-09-03T13:05:30-07:00

Python recipe 303344 by John Nielsen (sysadmin).

In Win32 often you'll find time stored in 100-nanosecond intervals since January 1, 1600 UTC. It is stored in a 64-bit value which uses 2 32 bit parts to store the time. The following is a function that returns the time in the typical format the python time libraries use (seconds since 1970).

thread pool example #2 -- easy_pool class (Python)

2004-09-08T13:48:11-07:00

Python recipe 303108 by John Nielsen (threads). Revision 3.

I am trying to show how to have a thread pool building on the recipe in http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/302746. This is a python class that essentially makes a thread pool for a function you define. Like the earlier example, I want to show off the power of having a thread pool that you can stop and start at will. Interestingly, you can mimic more standard thread use with the pool -- which I show off in as little as 3 lines of simple code.

win32 impersonation (like sudo on unix) (Python)

2001-10-12T12:14:15-07:00

Python recipe 81402 by John Nielsen (sysadmin).

Sometimes it is convenient to make thread authenticated as another principle. For example, perhaps something should run temporarily w/administrative rights. This is especially useful if you do not want the issues of making a COM object or service(which are other ways to solve the problem).

creating a share on windows with python (Python)

2004-09-03T11:30:26-07:00

Python recipe 303341 by John Nielsen (sysadmin).

The following code is an example of how one would use python's win32net module to create a share on windows.

doctest, unittest, and python 2.4's cool doctest.DocFileSuite (Python)

2004-09-16T08:59:32-07:00

Python recipe 305292 by John Nielsen (debugging).

Doctest and unittest are like 2 extremes of testing your python code. One is simple and intuitive and the other is powerful and requires more formality. In Python 2.4, doctest, builds in some of the power of unittest, perhaps bridging the gap between the two. This example shows you how to use doctest and the doctest.DocFileSuite feature.

accessing MS exchange with webdav part 1 (Python)

2004-08-05T10:51:30-07:00

Python recipe 298338 by John Nielsen (sysadmin). Revision 3.

If you want to talk to a MS exchange server, webdav offers a convenient way to do so. The code below shows the very basics for searching, deleting, and creating.

Discover exchange in active directory (Python)

2004-09-04T14:53:41-07:00

Python recipe 109357 by John Nielsen (sysadmin). Revision 2.

Active directory is wordy and very detailed. It can be daunting to first figure out what you need to know to interface with it. Here is some simple code to discover information about the exchange environment you are in. This isn't a full discussion of how to manage exchange with active directory. I thought I'd get something out now and then plan to add that when I get time.

Most viewed recipes by John Nielsen