Popular recipes by John Nielsen

socket.recv -- three ways to turn it into recvall (Python)

2005-04-05T13:47:57-07:00

Python recipe 408859 by John Nielsen (network). Revision 5.

An issue with socket.recv is how to know when you are done receiving data. A TCP stream guarantees the bytes will not arrive out of order or be sent more than once. But you do not know the size of the data that will be sent to you. 100 bytes could be sent as group of 10 bytes or maybe in one shot. Ultimately, this means you have to use a loop in some fashion until you know it is done.

The basic recv returns an empty string when the socket is disconnected. From that you can build a simple loop that will work as long as the sender manages to disconnect the socket at the appropriate time. However, there could be situations where a local error will mask as a clean shutdown or maybe a close() is never called.

Three very basic methods are shown below that try to fix that problem. They use either a time-based, end marker, or size of payload method. Since you cannot be sure just what you are going to receive, you have to be careful that you get enough of a message to determine the size of payload or end marker.

I updated the recv_size method to allocate data in larger chunks if it gets a large stream of data, which can increase performance.

When to not just use socket.close() (Python)

2005-04-08T17:31:14-07:00

Python recipe 408997 by John Nielsen (network). Revision 5.

I have implemented a "broken" client/server to show how socket.shutdown can be more useful than a simple socket.close operation. (I rewrote this to hopefully be more clear)

The close operation is not atomic. It implicitly tries to send any remaining data in _addition_ to closing a descriptor. Splitting this close operation up with the aid of the shutdown command can help avoid bugs. It gives the server one final way to say, "something went wrong". The server would also know that the client did not end correctly, since the socket should remain open when the client finished sending data. For example, if the function exits unexpectedly and python closes the socket for you, the server would not be able to send any data back.

In the server below, the client and server have different ideas about what the end marker should be. The rev_end function is written so as to look for an end marker. And, as long as they agree it should work. The socket.shutdown is for when something goes wrong.

simple readlines in reverse w/deque (Python)

2006-08-04T19:09:16-07:00

Python recipe 496941 by John Nielsen (text). Revision 5.

This a very simple implementation for how to do a readlines in reverse. It is not optimized for performance (which often doesn't matter). I have a 2nd version that is faster by loading blocks of data into memory instead of character by character. Of course, the code then almost doubles in size. And finally a third version that is the fastest, using split.

thread pool example #2 -- easy_pool class (Python)

2004-09-08T13:48:11-07:00

Python recipe 303108 by John Nielsen (threads). Revision 3.

I am trying to show how to have a thread pool building on the recipe in http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/302746. This is a python class that essentially makes a thread pool for a function you define. Like the earlier example, I want to show off the power of having a thread pool that you can stop and start at will. Interestingly, you can mimic more standard thread use with the pool -- which I show off in as little as 3 lines of simple code.

simplest useful (I hope!) thread pool example (Python)

2004-09-08T13:53:48-07:00

Python recipe 302746 by John Nielsen . Revision 6.

You can find examples on how to do threading, but they do not show off a thread pool. My goal was to get as small and simple as possible working thread pool example to show off the basic ideas without having extraneous things to understand. To show off the thread pool, I want stopping and starting of the threads to be explicit. This means the pool won't start until you are ready and will run forever until you are ready for it to stop. The main thread puts into the input queue and removes data from the output queue. The thread pool simply does the converse. Errors are also managed with another queue, so there is a clean distinction between errors and successful results.

changing file attributes on windows (Python)

2004-09-03T11:54:59-07:00

Python recipe 303343 by John Nielsen (sysadmin).

The win32api module offers SetFileAttributes whiles allows you to make changes to a file in windows. You can set a file to be read only, archive, hidden, etc. The function is simple and convenient to use.

simplest useful HTTPS with basic proxy authentication (Python)

2005-12-28T17:27:47-08:00

Python recipe 301740 by John Nielsen (network). Revision 4.

This is just about the most simple snippet of how to do proxy authentication with SSL using python. The current httplib only supports ssl through a proxy _without_ authentication. This example does basic proxy auth that a lot of proxy servers can support. This will at least give someone an idea of how to do it and then improve it and incorporate it however they want.

simple mx.ODBC example (Python)

2005-02-25T07:16:43-08:00

Python recipe 389535 by John Nielsen (database).

Mx.odbc is cross platform and very fast, I have used it to go through a few hundred thousand rows of an access database in seconds where pure ADO would take 30 min (or 3 min if you use the ADO type library). Here is a simple example of how to talk to a database, in this case an access file, get the columns of a table and get data from the table.

recvall2 (another way to do a recvall like the helpful sendall) (Python)

2005-02-18T15:25:09-08:00

Python recipe 388041 by John Nielsen (network).

Socket.sendall is very handy for sending. It would be nice if there was a socket.recvall. Unfortunatelty, receiving data is hard. One way to to do a recvall, is to use timeouts that get reset if any amount of data arrives. Useful, if you know almost nothing about what you are receiving.

http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/213239

Another, if you can control the sender, is to use a sentinal or marker, and send when the end has arrived.

This example, shows a really simple way, to do that. The assumption is that you have a unique enough string as an end marker. You pass a socket to either of these functions. The sender takes on the end marker and the receiver looks for it.

create an account in MS active directory (Python)

2004-09-03T13:58:34-07:00

Python recipe 303345 by John Nielsen (sysadmin).

This is the basic code to create an account in active directory that shows how to do things like set your own exension attributes, force a reset of the password when the user logs in, and set the home directory. It makes used of python's excellent COM support in win32com.

getting process information on windows (Python)

2004-09-03T11:10:53-07:00

Python recipe 303339 by John Nielsen (sysadmin).

To get process information for both NT and W2K (but not the 9x family) you can use the Performance Data Helper library(PDH). The is a simple example that shows you how to get a process list and their ids. It provides a convenient interface to performance information stored fundamentally in the registry. The basic process of using the PDH encompasses the following:

Get a list of all the objects you want
Get a list of the object's instances and data available for each instance: called 'items' or 'counters'
Get a group of performance data for each counter

In the case here we want the process object, the object's instances are it's list of processes, and the counter we want for the processes is 'ID Process'.

creating a share on windows with python (Python)

2004-09-03T11:30:26-07:00

Python recipe 303341 by John Nielsen (sysadmin).

The following code is an example of how one would use python's win32net module to create a share on windows.

doctest, unittest, and python 2.4's cool doctest.DocFileSuite (Python)

2004-09-16T08:59:32-07:00

Python recipe 305292 by John Nielsen (debugging).

Doctest and unittest are like 2 extremes of testing your python code. One is simple and intuitive and the other is powerful and requires more formality. In Python 2.4, doctest, builds in some of the power of unittest, perhaps bridging the gap between the two. This example shows you how to use doctest and the doctest.DocFileSuite feature.

converting windows 64 bit time to python useable format (Python)

2004-09-03T13:05:30-07:00

Python recipe 303344 by John Nielsen (sysadmin).

In Win32 often you'll find time stored in 100-nanosecond intervals since January 1, 1600 UTC. It is stored in a 64-bit value which uses 2 32 bit parts to store the time. The following is a function that returns the time in the typical format the python time libraries use (seconds since 1970).

sorting -- old to new python 2.4 style (heapq,bisect,list.sort keywords,itemgetter) (Python)

2006-09-13T12:28:57-07:00

Python recipe 305304 by John Nielsen (algorithms). Revision 2.

Python 2.4 has added a new feature to handle the problem where you needed to do a sort based off of part of the data. In effect, it has simplified the Schwartzian Transform (which I learned many years ago from an perl article written by Randall Schwartz). The following code will start with the older style python sorting approaches, and then show the bisect and heapq modules, and finally the key,cmp,reverse methods newly added to sort. The sort method of the list is an in-place sort. There is also a new sorted() function which returns a copy of the list sorted for you.

Get attributes of an object in MS Active Directory (Python)

2004-09-03T14:41:32-07:00

Python recipe 303348 by John Nielsen (sysadmin).

Sometimes it is useful to know what attributes are available to you for an object in active directory. You cannot ask the object directly for that, instead you need to use the schema of the object. All of this is done with python's COM support using win32com. By default only attributes that have values are returned.

simple example to show off itertools.tee (Python)

2004-09-23T22:01:49-07:00

Python recipe 305588 by John Nielsen . Revision 7.

Itertools.tee offers an interesting way to "remember" things that have happened. Itertools.tee makes multiple iterators from one (if you still have an the original iterator you do not use it). When you advance iterator 1 but not iterator 2, iterator 2 stays behind. Which means, if you later advance iterator 2, it the goes forward through the same data.

In this example, I use iterator.tee to make 2 iterators, to allow an action to affect data that has been processed in the past. The first iterator, it_main, is what is used to process data normally in this case to do something like display an image selected. The second iterator, it_history, stays behind the first and only advances when a specific action arrives. In effect, it rolls forward through the data that it_main has already processed.

How to use string.Template from python 2.4 (Python)

2004-09-13T10:55:54-07:00

Python recipe 304004 by John Nielsen (text). Revision 3.

New to python 2.4 is the string.Template class. It is similar to the formatting python already had in which the % operator with strings was used in a recognizable sprintf style of formatting from C. In addition to format types, you also had a choice with a tuple or dictionary mapping. String.Template simplifies it to only the string format type and dictionary mapping. This makes string formatting easier to remember and use. This example shows you the basics of using the template and the difference with the old style.

get_date wrapper to datetime module (Python)

2004-09-04T07:35:42-07:00

Python recipe 303441 by John Nielsen (programs).

The datetime module only accepts inputs of time it understands. For example, the months given to it have to be in range of values 1-12. This wrapper works around that issue and enables you to move forward or backward more arbitrary units of time. It does that by changing the year, month, and day to fit the requirements of datetime.

A way to deal with checking for types (Python)

2004-09-28T11:17:30-07:00

Python recipe 305888 by John Nielsen . Revision 6.

Managing types is normally simple in python, since it does typing as late as possible during runtime. Sometimes the issues of type still rears it's head, especially among programmers used to the "type at compile time" variants.

You want to resist doing a naive approach like type(obj)==str, since you then ignore subclasses.

What this recipe does is make a distinction between objects that _are_ are certain type or it's subclass and objects that act good enough. Most of the time the difference will not matter. If it is a certain python type or in some cases like with a sequence, possible among a few python types, it returns a 1, if it is good enough it returns a -1, and finally it returns zero if it is not good enough.

Good enough is enabled by checking for attributes and callables instead of explicit type and sometimes by checking for success of certain actions. In cases where it can be good enough, you simply check for just a true value which both 1 and -1 will evaluate to.

I put in typical checks for many things ranging from generators to lists to file handles.