Python has really nice logging framework, it has too a zipfile library in the default installation that makes you able to write compressed files.
Several of the logging handlers "rotate" files, by size, date, etc. Here is an example of handler class for the logging framework that, when the file is rotated, it will make a .zip of the old file.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47
import logging import logging.handlers import zipfile import sys, os, time, glob class TimedCompressedRotatingFileHandler(logging.handlers.TimedRotatingFileHandler): """ Extended version of TimedRotatingFileHandler that compress logs on rollover. by Angel Freire <cuerty at gmail dot com> """ def doRollover(self): """ do a rollover; in this case, a date/time stamp is appended to the filename when the rollover happens. However, you want the file to be named for the start of the interval, not the current time. If there is a backup count, then we have to get a list of matching filenames, sort them and remove the one with the oldest suffix. This method is a copy of the one in TimedRotatingFileHandler. Since it uses """ self.stream.close() # get the time that this sequence started at and make it a TimeTuple t = self.rolloverAt - self.interval timeTuple = time.localtime(t) dfn = self.baseFilename + "." + time.strftime(self.suffix, timeTuple) if os.path.exists(dfn): os.remove(dfn) os.rename(self.baseFilename, dfn) if self.backupCount > 0: # find the oldest log file and delete it s = glob.glob(self.baseFilename + ".20*") if len(s) > self.backupCount: s.sort() os.remove(s) #print "%s -> %s" % (self.baseFilename, dfn) if self.encoding: self.stream = codecs.open(self.baseFilename, 'w', self.encoding) else: self.stream = open(self.baseFilename, 'w') self.rolloverAt = self.rolloverAt + self.interval if os.path.exists(dfn + ".zip"): os.remove(dfn + ".zip") file = zipfile.ZipFile(dfn + ".zip", "w") file.write(dfn, os.path.basename(dfn), zipfile.ZIP_DEFLATED) file.close() os.remove(dfn)
I use this code in my services, normally they have a lot of traffic and generate tons of log. Since logs rotate every midnigth they are tons of files by week that they use to be zipped manually by an operator or by another script.
Now with this code it sends the debug straigth to a zipped file, saving time to operators.