Latest recipes tagged "logging"

JSON Formatted Logging (Python)

2016-05-27T01:07:42-07:00

Python recipe 580667 by Michael Blan Palmer (json, logging, python).

I have created a package that outputs JSON formatted lines to a log file. It can make use of the standard logging parameters and/or take custom input. The use of JSON in the log file allows for easy filtering and processing.

Set the logging level to every logger (Python)

2015-10-28T23:16:26-07:00

Python recipe 579119 by Tim McNamara (configuration, logging).

Python's logging allocates a name to every logger. That makes it hard to do something like, setting everything to logging.ERROR. Here's one way you might go about that:

Amazon SNS handler for the logging module (Python)

2014-11-06T17:32:26-08:00

Python recipe 578956 by Andrea Corbellini (aws, logging, sns). Revision 2.

This is a handler for the standard logging module that sends notifications to the Amazon Simple Notification Service.

You can use it like so:

logging.config.dictConfig({
    'version': 1,
    'disable_existing_loggers': False,
    'formatters': {
        'verbose': {
            'format': '%(levelname)s %(asctime)s %(module)s %(process)d '
                      '%(thread)d %(message)s',
        },
        'simple': {
            'format': '%(levelname)s %(message)s',
        },
    },
    'handlers': {
        'sns': {
            'level': 'INFO',
            'class': 'SNSHandler',
            'formatter': 'verbose',
            'topic_arn': 'YOUR SNS TOPIC ARN',
        },
    },
    'loggers': {
        'YOUR MODULE': {
            'handlers': ['sns'],
            'level': 'INFO',
            'propagate': True,
        },
    },
}

NondurableLogger class for use with concurrent.futures.ProcessPoolExecutor's submit and map methods (Python)

2014-02-10T17:50:59-08:00

Python recipe 578827 by Peter Santoro (logging, multiprocessing). Revision 2.

I needed a simple logging solution that I could use with with concurrent.futures.ProcessPoolExecutor and this is my initial recipe.

Print Logger Internals (Python)

2012-12-24T04:11:35-08:00

Python recipe 578389 by Dave Bailey (logging).

This helper function simplifies the difficult task of setting up and maintaining a logging system. Changes to a logging system can cause unanticipated consequences such as lost messages or duplicates. Debugging a logging hierarchy can be a tedious task. This function overrides the internal __repr__ functions of the internal classes and allows a print statement to generate the complete logger hierarchy with its associated internals. It allows easy debugging a logger and allows changes to be easily detected.

Get full caller name (package.module.function) (Python)

2012-11-30T21:54:46-08:00

Python recipe 578352 by anatoly techtonik (caller, debug, logging).

This function allows to get fully qualified name of the calling function. I expected this field to be available from logging module, but it is not here http://docs.python.org/2/library/logging.html#logrecord-attributes It might be that it is too expensive for performance or just doesn't play well in some situations. I don't know. I use it solely when debugging and it is convenient.

Also here: https://gist.github.com/2151727

Log watcher (tail -F *.log) (Python)

2014-04-04T15:54:03-07:00

Python recipe 577968 by Giampaolo Rodolà (color, log, logging, monitor, python, rotate, rotations, tail). Revision 10.

A python class which "watches" a directory and calls a callback(filename, lines) function every time one of the files being watched gets written, in real time.

Practically speaking, this can be compared to "tail -F *.log" UNIX command, but instead of having lines printed to stdout a python function gets called.

Similarly to tail, it takes care of "watching" new files which are created after initialization and "unwatching" those ones which are removed in the meantime. This means you'll be able to "follow" and support also rotating log files.

History

rev5 (2013-04-05):
- sizehint parameter
rev4 (2013-03-16):
- python 3 support (also dropped support for python <= 2.5)
- windows support
- unit tests
- main class can also be used as a context manager
rev3 (2012-01-13): initial release

Simple creation, configuration and installation of logging handlers (Python)

2011-06-01T19:05:40-07:00

Python recipe 577731 by Hank (logging).

A helper class and function to make it easy to configure logging, and an example of using it to send INFO to sys.stdout and WARNING or above to sys.stderr.

This is a simple port to Python 2 (I tested on 2.7) of Nick Coghlan's Python 3 recipe for doing the same thing.

Simple creation, configuration and installation of logging handlers (Python)

2011-06-01T12:13:21-07:00

Python recipe 577496 by Nick Coghlan (logging).

Creation of log message handler for the logging module is often a multi-step process, involving creation of the handler object, configuration of the message levels and formats, installation of any filters and then actual connection of the handler to the relevant logger object.

This helper function allows all of these things to be specified up front in a single function call, which then takes care of configuring the handler object appropriately.

Better quote module for bash shells (Python)

2010-12-03T09:16:45-08:00

Python recipe 577483 by Kevin L. Sitze (bash, command, commandline, escape, logging, quote, script, shell).

This Python module quotes a Python string so that it will be treated as a single argument to commands ran via os.system() (assuming bash is the underlying shell). In other words, this module makes arbitrary strings "command line safe" (for bash command lines anyway, YMMV if you're using Windows or one of the (less fine) posix shells).

easy logging with extras (Python)

2010-11-02T10:04:15-07:00

Python recipe 577448 by Bud P. Bruegger (logging).

The logging package offers the "extra" keyword argument in Logger.log to add user-defined attributes to the log record. LoggerAdaptors make it easy to use extras that are constant for a given logger; then simply use logger.debug, logger.info, ecc. But this won't support variable extra arguments.

The present recipe makes it easy to use "extra" attributes that are not constant but variables passed to the (modified) logging methods ('debug', 'info', ...)

logging support for python daemon (Python)

2010-10-25T15:22:18-07:00

Python recipe 577442 by Bud P. Bruegger (daemon, logging).

The recipe shows how to subclass python-daemon's [1] DaemonContext to add logging support. In particular, it is possible to ask to keep the files related to a list of loggers (loggers_preserve) open and to redirect stdout and stderr to a logger (e.g., one using a RotatingFileHandler).

[1] See http://pypi.python.org/pypi/python-daemon/

Tiny trace/debug-logging tool with smart event filtering (Python)

2011-04-29T02:23:37-07:00

Python recipe 577429 by Jan Kaliszewski (debugging, logging, trace). Revision 3.

Just import, call and log your selected call/return/exception/etc. events.
You can use a standard Python logger from the logging module.
Quite flexible tool and about 50 efective SLOC only (excluding the example script).
For debugging rather than production environments (programs noticeably slow down).

Multiprocess-safe logging file-handler + interprocess RLock (Python)

2010-09-22T17:30:10-07:00

Python recipe 577395 by Jan Kaliszewski (concurrency, file, file_lock, flock, handler, logging, process, rlock, threading, threadsafe). Revision 11.

A Python 2.x/3.x-compatibile multiprocess-safe logging file-handler (logging.FileHandler replacement, designed for logging to a single file from multiple independent processes) together with a simple interprocess recursive lock -- universal abstract classes + Unix/Linux implementation.

Update: It's is a deeply revised version. Especially, now it --

is Python 2.4, 2.5, 2.6, 3.1 -compatibile (previously Py>=2.6 was needed); probably works also with 2.7, 3.0 and 3.2 (but not tested if it does);
is multiprocess-safe as well as thread-safe (proviously thread safety within a process was missed);
is based on public interfaces only (previously FileHandler._open() was called and overriden);
implement full RLock instance interface, as documented for threading.RLock (previously non-blocking mode and context-manager interface were missing).

The module contains:

Unix/Linux-only example implementation (with flock-based locking): FLockRLock and FLockFileHandler classes.
universal abstract classes -- which may be useful at developing implementation for non-Unix platforms: MultiprocessRLock, MultiprocessFileHandler, LockedFileHandler,

Also a quick-and-dirty test was added.

It is still an alpha version -- I'll be very grateful for any feedback.

Further updates:

2010-09-20: Some corrections, especially: non-blocking mode bug in MultiprocessRLock.acquire() fixed; _test() function improved; plus fixes in the description below.
2010-09-22: _test() improved and moved to description section. Mistaken copyright-notice removed.

node.js quicklog method to log to a file (JavaScript)

2010-08-11T05:08:57-07:00

JavaScript recipe 577351 by Trent Mick (logging, nodejs).

I tend to have a "quicklog" method for a few of the languages I'm working in to do logging when stdout/stderr isn't necessarily available (GUI app) or convenient (lots of other output on stdout, etc.). My usage with nodejs was while working on the node REPL. Log output to stdout interfered with the REPL.

Python script main line for graceful exception handling and logging (Python)

2010-09-11T05:46:59-07:00

Python recipe 577258 by Trent Mick (error, logging, main, mainline, script). Revision 5.

This is a recipe is often use for the mainline of my Python scripts. With this recipe your Python script will:

gracefully handle Ctrl+C (i.e. KeyboardInterrupt)
log an error (using the logging module) for uncaught exceptions, importantly with the file and line number in your Python code where the exception was raised
gracefully ignore a closed output pipe (common when the user pipes your script through less and terminates that)
if your script logger is enabled for DEBUG level logging, a full traceback will be shown for an uncaught exception

Presumptions:

you have a global log variable a la:

import logging
log = logging.setLevel("scriptname")

your script's entry point is a function def main(argv): ...

Easy Exception, Error Handling and Debugging with Auto Logging (Python)

2010-04-06T20:40:05-07:00

Python recipe 577144 by AJ. Mayorga (debug, debugging, error_propagation, exception, logging). Revision 5.

Being new to Python and trying to chase where in my project exceptions were taking place always seems to take up a large amount of time. So I created this class that I now pretty use in all my projects to help me save time tracking exceptions and errors. its also has the ability to generate error logs or to output SQL query and data tuples for Database logging. Ive added sample code to give you an idea of how I normally use it. Hope this helps its saved me hours in tracking error/exception. Feed back would be great, mind you havent been Python all that long

Logging asserts (Python)

2010-03-01T03:22:43-08:00

Python recipe 577074 by d.schlabing (logging). Revision 2.

This tries to marry assert statements with logging. If you use asserts together with logging and the script stops because an assertion is not fulfilled, you will not find any information about this in the logs. Unless you do something like this:

meal = ["eggs", "bacon", "spam"]
try:
    assert "spam" not in meal, "But I don't like spam!"
except AssertionError:
    log.exception("But I don't like spam!")
    raise

With the proposed recipe a somewhat similar behaviour can be achieved by:

log_assert("spam" not in meal, "But I don't like spam!")

Use DebugView utility with standard python logging (Python)

2010-02-10T10:35:59-08:00

Python recipe 577040 by Christopher Prinos (debugging, debugview, logging, sysinternals, windows).

This is a custom logging.Handler class that lets you use standard logging calls to output messages to SysInternals' DebugView utility.

LoggingWebMonitor - a central logging server and monitor. (Python)

2010-02-02T01:56:42-08:00

Python recipe 577025 by Gabriel Genellina (client_server, debugging, distributed, logging, remote, sysadmin, web). Revision 3.

LoggingWebMonitor listens for log records sent from other processes running in the same box or network. Collects and saves them concurrently in a log file. Shows a summary web page with the latest N records received.