Getting the SHA-1 (or MD5) hash of a directory (Python recipe) by Stephen Akiki

ActiveState Code (http://code.activestate.com/recipes/576973/)

.. It's perfectly possible to do this for arbitrarily big directories while maintaining constant memory usage, providing you stick with one hashing algorithym (IMO sha1 is better, and combining the two mainly just disguises md5's failings with a tiny dash of sha1, like MSG on a chinese take-out meal)

like this:

import hashlib # and don't import md5 or sha1 -- they're deprecated in 2.6

finalhash = hashlib.sha1() # or md5; if you consider it sufficient for the problem.

# ...

# in the body of the loop over files:
#
# get the size of the file before opening it
f1size = os.path

# ... (actually open the file, as before)

while f1.tell() != f1size:
    # update the hash with the contents of the file,
    # 256k at a time
    finalhash.update (f1.read (0x40000))

You might also update the hash string for 'empty directory' too. However do you think it's really a good idea to use a 'magic value' like that? A simple

if not os.path.exists (directory):
    return -3

near the start of the function could avoid it.

You could also do the md5-hashes-into-sha1 thing with constant memory usage: calculate md5 sums and feed them into the sha1 object using update() one at a time.

Here is a version that won't run out of memory.

The checksum generated here is based purely on the contents of files found, processed in ascending sort order using the normalised file paths. This version does not account for changes in file ownership, permissions or stat information (although this might be useful to add in). The value of the checksum will change if renames affect the order in which the files are passed in or processed by the _update_checksum() internal function. This is usually fine for most use cases but be aware that YMMV.

#/usr/bin/env python
import hashlib
from os.path import normpath, walk, isdir, isfile, dirname, basename, \
    exists as path_exists, join as path_join

def path_checksum(paths):
    """
    Recursively calculates a checksum representing the contents of all files
    found with a sequence of file and/or directory paths.

    """
    if not hasattr(paths, '__iter__'):
        raise TypeError('sequence or iterable expected not %r!' % type(paths))

    def _update_checksum(checksum, dirname, filenames):
        for filename in sorted(filenames):
            path = path_join(dirname, filename)
            if isfile(path):
                print path
                fh = open(path, 'rb')
                while 1:
                    buf = fh.read(4096)
                    if not buf : break
                    checksum.update(buf)
                fh.close()

    chksum = hashlib.sha1()

    for path in sorted([normpath(f) for f in paths]):
        if path_exists(path):
            if isdir(path):
                walk(path, _update_checksum, chksum)
            elif isfile(path):
                _update_checksum(chksum, dirname(path), basename(path))

    return chksum.hexdigest()

if __name__ == '__main__':
    print path_checksum([r'/tmp', '/etc/hosts'])

Thanks guys, updated so it doesn't run out of memory.

I think its too simple: something changed but dunnow what :)

Better approach is calculating SHA1 against each file in directory.

Stable solution is cfv written in Python:

http://cfv.sf.net

These examples worked great for a file transfer QA project I was working on. David Moss's example (Third comment) seemed to work the best for my circumstances. I did notice that if you passed it a single filename in a list, to boinked. I added a few lines of code to re-listify the filename once passed in to the _update_checksum function. I tend to do things old-school and not very Pythonic, though. Is there a simpler way to get this to work with a single filename as well as what it already does for directories? My bigger project needs to create a pre-table before file transfers and then be able to do an individual file after each transfer.