Latest recipes by John Nielsen

simple readlines in reverse w/deque (Python)

2006-08-04T19:09:16-07:00

Python recipe 496941 by John Nielsen (text). Revision 5.

This a very simple implementation for how to do a readlines in reverse. It is not optimized for performance (which often doesn't matter). I have a 2nd version that is faster by loading blocks of data into memory instead of character by character. Of course, the code then almost doubles in size. And finally a third version that is the fastest, using split.

When to not just use socket.close() (Python)

2005-04-08T17:31:14-07:00

Python recipe 408997 by John Nielsen (network). Revision 5.

I have implemented a "broken" client/server to show how socket.shutdown can be more useful than a simple socket.close operation. (I rewrote this to hopefully be more clear)

The close operation is not atomic. It implicitly tries to send any remaining data in _addition_ to closing a descriptor. Splitting this close operation up with the aid of the shutdown command can help avoid bugs. It gives the server one final way to say, "something went wrong". The server would also know that the client did not end correctly, since the socket should remain open when the client finished sending data. For example, if the function exits unexpectedly and python closes the socket for you, the server would not be able to send any data back.

In the server below, the client and server have different ideas about what the end marker should be. The rev_end function is written so as to look for an end marker. And, as long as they agree it should work. The socket.shutdown is for when something goes wrong.

socket.recv -- three ways to turn it into recvall (Python)

2005-04-05T13:47:57-07:00

Python recipe 408859 by John Nielsen (network). Revision 5.

An issue with socket.recv is how to know when you are done receiving data. A TCP stream guarantees the bytes will not arrive out of order or be sent more than once. But you do not know the size of the data that will be sent to you. 100 bytes could be sent as group of 10 bytes or maybe in one shot. Ultimately, this means you have to use a loop in some fashion until you know it is done.

The basic recv returns an empty string when the socket is disconnected. From that you can build a simple loop that will work as long as the sender manages to disconnect the socket at the appropriate time. However, there could be situations where a local error will mask as a clean shutdown or maybe a close() is never called.

Three very basic methods are shown below that try to fix that problem. They use either a time-based, end marker, or size of payload method. Since you cannot be sure just what you are going to receive, you have to be careful that you get enough of a message to determine the size of payload or end marker.

I updated the recv_size method to allocate data in larger chunks if it gets a large stream of data, which can increase performance.

simple mx.ODBC example (Python)

2005-02-25T07:16:43-08:00

Python recipe 389535 by John Nielsen (database).

Mx.odbc is cross platform and very fast, I have used it to go through a few hundred thousand rows of an access database in seconds where pure ADO would take 30 min (or 3 min if you use the ADO type library). Here is a simple example of how to talk to a database, in this case an access file, get the columns of a table and get data from the table.

recvall2 (another way to do a recvall like the helpful sendall) (Python)

2005-02-18T15:25:09-08:00

Python recipe 388041 by John Nielsen (network).

Socket.sendall is very handy for sending. It would be nice if there was a socket.recvall. Unfortunatelty, receiving data is hard. One way to to do a recvall, is to use timeouts that get reset if any amount of data arrives. Useful, if you know almost nothing about what you are receiving.

http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/213239

Another, if you can control the sender, is to use a sentinal or marker, and send when the end has arrived.

This example, shows a really simple way, to do that. The assumption is that you have a unique enough string as an end marker. You pass a socket to either of these functions. The sender takes on the end marker and the receiver looks for it.

A way to deal with checking for types (Python)

2004-09-28T11:17:30-07:00

Python recipe 305888 by John Nielsen . Revision 6.

Managing types is normally simple in python, since it does typing as late as possible during runtime. Sometimes the issues of type still rears it's head, especially among programmers used to the "type at compile time" variants.

You want to resist doing a naive approach like type(obj)==str, since you then ignore subclasses.

What this recipe does is make a distinction between objects that _are_ are certain type or it's subclass and objects that act good enough. Most of the time the difference will not matter. If it is a certain python type or in some cases like with a sequence, possible among a few python types, it returns a 1, if it is good enough it returns a -1, and finally it returns zero if it is not good enough.

Good enough is enabled by checking for attributes and callables instead of explicit type and sometimes by checking for success of certain actions. In cases where it can be good enough, you simply check for just a true value which both 1 and -1 will evaluate to.

I put in typical checks for many things ranging from generators to lists to file handles.

sorting part2 -- some performance considerations (Python)

2004-09-24T21:48:22-07:00

Python recipe 305779 by John Nielsen (algorithms). Revision 5.

Below you will find a simple python class that I wrote to test performance of various different sorting techniques. In this cas: list, heapq, and bisect. For printing out data, I make use of the very cool decimal module to limit errant floating point numbers with many digits. It helps me decide when I want to think about different types of sorts.

Python's list.sort is so good that generally you are not going to want to write your own sort, and instead use it and some of the new features recently added to it. However, with a list.sort you pay for sorting at retrieval time. A different type of sort using data structures to sort at creation time instead of retrieval can offer some performance characteristics, that may make one consider them instead.

simple example to show off itertools.tee (Python)

2004-09-23T22:01:49-07:00

Python recipe 305588 by John Nielsen . Revision 7.

Itertools.tee offers an interesting way to "remember" things that have happened. Itertools.tee makes multiple iterators from one (if you still have an the original iterator you do not use it). When you advance iterator 1 but not iterator 2, iterator 2 stays behind. Which means, if you later advance iterator 2, it the goes forward through the same data.

In this example, I use iterator.tee to make 2 iterators, to allow an action to affect data that has been processed in the past. The first iterator, it_main, is what is used to process data normally in this case to do something like display an image selected. The second iterator, it_history, stays behind the first and only advances when a specific action arrives. In effect, it rolls forward through the data that it_main has already processed.

using the operator module w/map (plus a bit on itertools and generator exp) (Python)

2004-09-22T12:39:17-07:00

Python recipe 305318 by John Nielsen (shortcuts). Revision 8.

These are just some simple examples of how you can leverage the operator module to help gain performance with something like map (it works great with sort too). There are times where techniques like this may be necessary. Generally, you'd want to avoid doing this simply because it makes python ugly and harder to debug.

sorting -- old to new python 2.4 style (heapq,bisect,list.sort keywords,itemgetter) (Python)

2006-09-13T12:28:57-07:00

Python recipe 305304 by John Nielsen (algorithms). Revision 2.

Python 2.4 has added a new feature to handle the problem where you needed to do a sort based off of part of the data. In effect, it has simplified the Schwartzian Transform (which I learned many years ago from an perl article written by Randall Schwartz). The following code will start with the older style python sorting approaches, and then show the bisect and heapq modules, and finally the key,cmp,reverse methods newly added to sort. The sort method of the list is an in-place sort. There is also a new sorted() function which returns a copy of the list sorted for you.

doctest, unittest, and python 2.4's cool doctest.DocFileSuite (Python)

2004-09-16T08:59:32-07:00

Python recipe 305292 by John Nielsen (debugging).

Doctest and unittest are like 2 extremes of testing your python code. One is simple and intuitive and the other is powerful and requires more formality. In Python 2.4, doctest, builds in some of the power of unittest, perhaps bridging the gap between the two. This example shows you how to use doctest and the doctest.DocFileSuite feature.

Basics of heapq w/python 2.4 additions (Python)

2004-09-24T18:22:34-07:00

Python recipe 305267 by John Nielsen (algorithms). Revision 7.

The following code shows why you would want to use a heap instead of a list and the basics of using heapq in python with some additions for python 2.4( I've cleaned this up from earlier to make it hopefully clearer.)

How to use String.Template from python 2.4 (Python)

2004-09-09T06:24:40-07:00

Python recipe 304005 by John Nielsen (text).

New to python 2.4 is the String.Template class. It is similar to the formatting python already had in which the % operator with strings was used in a recognizable sprintf style of formatting from C. In addition to format types, you also had a choice with a tuple or dictionary mapping. String.Template simplifies it to only the string format type and dictionary mapping. This makes string formatting easier to remember and use. This example shows you the basics of using the template and the difference with the old style.

How to use string.Template from python 2.4 (Python)

2004-09-13T10:55:54-07:00

Python recipe 304004 by John Nielsen (text). Revision 3.

New to python 2.4 is the string.Template class. It is similar to the formatting python already had in which the % operator with strings was used in a recognizable sprintf style of formatting from C. In addition to format types, you also had a choice with a tuple or dictionary mapping. String.Template simplifies it to only the string format type and dictionary mapping. This makes string formatting easier to remember and use. This example shows you the basics of using the template and the difference with the old style.

get_date wrapper to datetime module (Python)

2004-09-04T07:35:42-07:00

Python recipe 303441 by John Nielsen (programs).

The datetime module only accepts inputs of time it understands. For example, the months given to it have to be in range of values 1-12. This wrapper works around that issue and enables you to move forward or backward more arbitrary units of time. It does that by changing the year, month, and day to fit the requirements of datetime.

Get attributes of an object in MS Active Directory (Python)

2004-09-03T14:41:32-07:00

Python recipe 303348 by John Nielsen (sysadmin).

Sometimes it is useful to know what attributes are available to you for an object in active directory. You cannot ask the object directly for that, instead you need to use the schema of the object. All of this is done with python's COM support using win32com. By default only attributes that have values are returned.

create an account in MS active directory (Python)

2004-09-03T13:58:34-07:00

Python recipe 303345 by John Nielsen (sysadmin).

This is the basic code to create an account in active directory that shows how to do things like set your own exension attributes, force a reset of the password when the user logs in, and set the home directory. It makes used of python's excellent COM support in win32com.

converting windows 64 bit time to python useable format (Python)

2004-09-03T13:05:30-07:00

Python recipe 303344 by John Nielsen (sysadmin).

In Win32 often you'll find time stored in 100-nanosecond intervals since January 1, 1600 UTC. It is stored in a 64-bit value which uses 2 32 bit parts to store the time. The following is a function that returns the time in the typical format the python time libraries use (seconds since 1970).

changing file attributes on windows (Python)

2004-09-03T11:54:59-07:00

Python recipe 303343 by John Nielsen (sysadmin).

The win32api module offers SetFileAttributes whiles allows you to make changes to a file in windows. You can set a file to be read only, archive, hidden, etc. The function is simple and convenient to use.

creating a share on windows with python (Python)

2004-09-03T11:30:26-07:00

Python recipe 303341 by John Nielsen (sysadmin).

The following code is an example of how one would use python's win32net module to create a share on windows.