Welcome, guest | Sign In | My Account | Store | Cart

TimedCompressedRotatingFileHandler (Python recipe) by Angel Freire
ActiveState Code (http://code.activestate.com/recipes/502265/)

Python has really nice logging framework, it has too a zipfile library in the default installation that makes you able to write compressed files.

Several of the logging handlers "rotate" files, by size, date, etc. Here is an example of handler class for the logging framework that, when the file is rotated, it will make a .zip of the old file.

Python, 47 lines
Download
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
import logging
import logging.handlers
import zipfile
import sys, os, time, glob

class TimedCompressedRotatingFileHandler(logging.handlers.TimedRotatingFileHandler):
    """
       Extended version of TimedRotatingFileHandler that compress logs on rollover.
       by Angel Freire <cuerty at gmail dot com>
    """
    def doRollover(self):
        """
        do a rollover; in this case, a date/time stamp is appended to the filename
        when the rollover happens.  However, you want the file to be named for the
        start of the interval, not the current time.  If there is a backup count,
        then we have to get a list of matching filenames, sort them and remove
        the one with the oldest suffix.

        This method is a copy of the one in TimedRotatingFileHandler. Since it uses
        
        """
        self.stream.close()
        # get the time that this sequence started at and make it a TimeTuple
        t = self.rolloverAt - self.interval
        timeTuple = time.localtime(t)
        dfn = self.baseFilename + "." + time.strftime(self.suffix, timeTuple)
        if os.path.exists(dfn):
            os.remove(dfn)
        os.rename(self.baseFilename, dfn)
        if self.backupCount > 0:
            # find the oldest log file and delete it
            s = glob.glob(self.baseFilename + ".20*")
            if len(s) > self.backupCount:
                s.sort()
                os.remove(s[0])
        #print "%s -> %s" % (self.baseFilename, dfn)
        if self.encoding:
            self.stream = codecs.open(self.baseFilename, 'w', self.encoding)
        else:
            self.stream = open(self.baseFilename, 'w')
        self.rolloverAt = self.rolloverAt + self.interval
        if os.path.exists(dfn + ".zip"):
            os.remove(dfn + ".zip")
        file = zipfile.ZipFile(dfn + ".zip", "w")
        file.write(dfn, os.path.basename(dfn), zipfile.ZIP_DEFLATED)
        file.close()
        os.remove(dfn)

I use this code in my services, normally they have a lot of traffic and generate tons of log. Since logs rotate every midnigth they are tons of files by week that they use to be zipped manually by an operator or by another script.

Now with this code it sends the debug straigth to a zipped file, saving time to operators.

Tags: debugging