Point this script at a folder or several folders and it will find and delete all duplicate files within the folders, leaving behind the first file found of any set of duplicates. It is designed to handle hundreds of thousands of files of any size at a time and to do so quickly. It was written to eliminate duplicates across several photo libraries that had been shared between users. As the script was a one-off to solve a very particular problem, there are no options nor is it refactoring into any kind of modules or reusable functions.
| Python |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 | #! /usr/bin/python
import os
import sys
import stat
import md5
filesBySize = {}
def walker(arg, dirname, fnames):
d = os.getcwd()
os.chdir(dirname)
try:
fnames.remove('Thumbs')
except ValueError:
pass
for f in fnames:
if not os.path.isfile(f):
continue
size = os.stat(f)[stat.ST_SIZE]
if size < 100:
continue
if filesBySize.has_key(size):
a = filesBySize[size]
else:
a = []
filesBySize[size] = a
a.append(os.path.join(dirname, f))
os.chdir(d)
for x in sys.argv[1:]:
print 'Scanning directory "%s"....' % x
os.path.walk(x, walker, filesBySize)
print 'Finding potential dupes...'
potentialDupes = []
potentialCount = 0
trueType = type(True)
sizes = filesBySize.keys()
sizes.sort()
for k in sizes:
inFiles = filesBySize[k]
outFiles = []
hashes = {}
if len(inFiles) is 1: continue
print 'Testing %d files of size %d...' % (len(inFiles), k)
for fileName in inFiles:
if not os.path.isfile(fileName):
continue
aFile = file(fileName, 'r')
hasher = md5.new(aFile.read(1024))
hashValue = hasher.digest()
if hashes.has_key(hashValue):
x = hashes[hashValue]
if type(x) is not trueType:
outFiles.append(hashes[hashValue])
hashes[hashValue] = True
outFiles.append(fileName)
else:
hashes[hashValue] = fileName
aFile.close()
if len(outFiles):
potentialDupes.append(outFiles)
potentialCount = potentialCount + len(outFiles)
del filesBySize
print 'Found %d sets of potential dupes...' % potentialCount
print 'Scanning for real dupes...'
dupes = []
for aSet in potentialDupes:
outFiles = []
hashes = {}
for fileName in aSet:
print 'Scanning file "%s"...' % fileName
aFile = file(fileName, 'r')
hasher = md5.new()
while True:
r = aFile.read(4096)
if not len(r):
break
hasher.update(r)
aFile.close()
hashValue = hasher.digest()
if hashes.has_key(hashValue):
if not len(outFiles):
outFiles.append(hashes[hashValue])
outFiles.append(fileName)
else:
hashes[hashValue] = fileName
if len(outFiles):
dupes.append(outFiles)
i = 0
for d in dupes:
print 'Original is %s' % d[0]
for f in d[1:]:
i = i + 1
print 'Deleting %s' % f
os.remove(f)
print
|
Discussion
The script uses a multipass approach to finding duplicate files. First, it walks all of the directories pass in and groups all files by size. In the next pass, the script walks each set of files of the same size and checksums the first 1024 bytes. Finally, the script walks each set of files that are the same size with the same hash of the first 1024 bytes and checksums each file in its entirety.
The very last step is to walk each set of files of the same length/hash and delete all but the first file in the set.
It ran against a 3.5 gigabyte set of files composed of about 120,000 files, of which there were about 50,000 duplicates, most of which were over 1 megabyte. The total run took about 2 minutes on a 1.33ghz G4 powerbook. Fast enough for me and fast enough without actually optimizing anything beyond the obvious.
Comments
Hard links? This is really cool, i was going to write something very similar. My application: I have made backups for the last ten years, and many times complete backups, which have now been copied on a single hdd for safety. Much of these files are identical. I wanted to hardlink them together to save the disk space. I can now just modify your app! thanks,
fslint. http://www.iol.ie/~padraiga/fslint/ does this, as well as other useful things (and in shorter code too, I believe). I haven't investigated the various file-compare optimizations of each system.
BTW, a common use of the fslint tools is to find dups on the same filesystem and replace them with hardlinks. If you don't care about the once-identical files being forever identical, you can avoid needless space waste.
Shortcuts In Windows. I love this, freed up 60G from our stuffed file server
We used
Where duplicates.txt was created with a slight modification of the main script.
Size of the scan. We freed up 80G on a 1.6TB SAN in about 3 hours, memory usage was fine throughout (I ran another freeware duplicate finder and it crashed twice)
73 Gigs freed ! Cool ! I had lots of mp3 duplicated, waiting forever to be correctly tagged so kept in different directories but then merged ..., plus backups ...
Here is the stupid patch to create hardlinks instead of deleting files.
Just for the record: This script has a very serious bug that leads to data loss: the "Scanning for real dupes..." step traverses potential duplicate sets defined as lists of names with identical file size. if you've got 4 files (a,b,c,d) of n bytes where a=b and c=d, only a will be left and b, c, and d will be deleted, thus losing the contents of the 2nd pair of duplicates.
This script: doublesdetector.py computes "SHA of files that have the same size, and group files by SHA". It simply finds duplicate files in a directory tree. It doesn't delete the duplicated files but it works fine for me (I use it to delete duplicate photos). It can probably be modified to add a delete files functionality.
Sign in to comment