Python Logging: Reference and Examples

Python Logging: Reference and Examples

Last updated:
Table of Contents

All examples assume Python 3, unless otherwise noted

Here's a sample python project where you can see some of these examples with actual working code.

A Logger object defines the rules for which logs are handled. This is the object you call methods such as .warn() and .error() on.

A Handler object defines how to format the log messages and what to do with them (i.e. save to disk, show on STDOUT, send to a log server, etc).

Log to STDOUT using the root logger

The simplest way to log messages is to use the logging module itself

import logging

# you can call methods on the module object itself.
# this represents the root logger
logging.basicConfig(format='%(asctime)s %(levelname)s %(message)s')
logging.warn('this is a warning message, written to the root logger')

Output (in the STDOUT):

2018-02-18 17:23:37,532 WARNING this is a warning message, written to the root logger

Log to STDOUT with individual loggers

Multiple calls to getLogger() with the same name will always return a reference to the same Logger object.

In individual loggers (i.e. not the root logger) you need to set formatters via handlers.

Suppose the following file is located at rootmodule/foo.py

import logging

logger = logging.getLogger(__name__)

# add more information to the messages
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')

# stream handler prints errors to STDERR, which defaults to STDOUT
handler = logging.StreamHandler()
handler.setFormatter(formatter)

logger.addHandler(handler)
logger.error("Error message from rootmodule.foo")

OUTPUT (in the STDOUT):

2018-02-18 18:00:41,141 - rootmodule.foo - ERROR - Error message from rootmodule.foo

Log to a file

Suppose the following file is located at rootmodule/baz.py

import logging,os
import rootmodule

logger = logging.getLogger(__name__)

# add more information to the messages
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')

# create a file in the root module directory
output_file = os.path.dirname(rootmodule.__file__)+"/output.log"

handler = logging.FileHandler(output_file)
handler.setFormatter(formatter)

# by default, the level is WARN
logger.setLevel(logging.INFO)

logger.addHandler(handler)

logger.info("rootmodule.baz module loaded")

Add Stack Trace to error log messages

Use the traceback module.

Assume the following file is at rootmodule/abc.py:

import logging
import traceback

logger = logging.getLogger(__name__)

# add more information to the messages
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')

# stream handler prints errors to STDERR, which defaults to STDOUT
handler = logging.StreamHandler()
handler.setFormatter(formatter)

logger.addHandler(handler)

def abc():
    try:
        raise Exception("some exception")
    except Except as ex:
        # this calls sys.exc_info() behind the scenese,
        # to get the exception that's currently being handled
        tb = traceback.format_exc() 

        logger.error(tb)

This is what gets logged to STDOUT if you try and call abc():

2018-02-18 20:22:34,546 - rootmodule.abc - ERROR - Traceback (most recent call last):
  File "rootmodule/abc.py", line 17, in abc
    raise Exception("some exception")
Exception: some exception

Log rotation by size

This does not compress rotated files

It's the same as logging to file, but you use a different handler.

import logging,os
import rootmodule
from logging.handlers import RotatingFileHandler

logger = logging.getLogger(__name__)

formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')

# create a file in the root module directory
output_file = os.path.dirname(rootmodule.__file__)+"/output-rot.log"

# logs will be written to the output_file until it 
# reaches 1000000 bytes = 1MB, then a new file will be created
# and the old file will be renamed to output_rot.log.1
handler = RotatingFileHandler(output_file,maxBytes=1000000, backupCount=5)

handler.setFormatter(formatter)

logger.addHandler(handler)

logger.warn("module quux loaded")

Log rotation by size, with compression

Install this Python package: concurrent-log-handler

$ pip install concurrent-log-handler

Then use that handler. It behaves the same way as RotatingFileHandler, but it will compress the rotated logs to save space.

import logging,os
import rootmodule
from concurrent_log_handler import ConcurrentRotatingFileHandler

logger = logging.getLogger(__name__)

formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')

# create a file in the root module directory
output_file = os.path.dirname(rootmodule.__file__)+"/output-rot.log"

# same as the other example, but rorated files will be gzipped
handler = ConcurrentRotatingFileHandler(output_file,maxBytes=1000000, backupCount=5, use_gzip=True)

handler.setFormatter(formatter)

logger.addHandler(handler)

logger.warn("module loaded")

Log errors in in Flask

You just need to setup the correct handler in the initialization code and then you use the @app.errorhandler(Exception) decorator to register it, so that every Exception raised in the routes gets logged.

import logging
import os
import traceback

from concurrent_log_handler import ConcurrentRotatingFileHandler
from flask import Flask, request, jsonify, make_response

app = Flask(__name__)

#####################################################################################
################################### ROUTES ##########################################
#####################################################################################

@app.route('/sample', methods=['GET'])
def sample():
    # lots of code that may raise exceptions
    # ...
    # ...
    return make_response(jsonify({"message": "ok"}),200) 


@app.errorhandler(Exception)
def exceptions(e):
    tb = traceback.format_exc()

    app.logger.error('%s %s %s %s 5XX INTERNAL SERVER ERROR\n%s',
                     request.remote_addr,
                     request.method,
                     request.scheme,
                     request.full_path,
                     tb)

    # it's good practice not to include error details in the response
    resp = make_response("Internal Server Error", 500)

    return resp

#####################################################################################
############################### INTIALIZATION CODE ##################################
#####################################################################################

if __name__ == '__main__':

    output_file = os.path.abspath('application.log')

    # 10M = 1024*1000*10 bytes   
    handler = ConcurrentRotatingFileHandler(output_file, maxBytes=1024 * 1000 * 10, backupCount=5, use_gzip=True)

    formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
    handler.setFormatter(formatter)

    app.logger.addHandler(handler)

    # https://stackoverflow.com/a/20423005/436721
    app.logger.setLevel(logging.INFO)

    app.run(host='0.0.0.0', port=8080)

Django logging

This example was originally written by use Peter Baumgartner (ipmb) on Github.

Add the following to your settings.py file.

import logging.config
import os
from django.utils.log import DEFAULT_LOGGING

# Disable Django's logging setup
LOGGING_CONFIG = None

LOGLEVEL = os.environ.get('LOGLEVEL', 'info').upper()

logging.config.dictConfig({
    'version': 1,
    'disable_existing_loggers': False,
    'formatters': {
        'default': {
            # exact format is not important, this is the minimum information
            'format': '%(asctime)s %(name)-12s %(levelname)-8s %(message)s',
        },
        'django.server': DEFAULT_LOGGING['formatters']['django.server'],
    },
    'handlers': {
        # console logs to stderr
        'console': {
            'class': 'logging.StreamHandler',
            'formatter': 'default',
        },
        # Add Handler for Sentry for `warning` and above
        'sentry': {
            'level': 'WARNING',
            'class': 'raven.contrib.django.raven_compat.handlers.SentryHandler',
        },
        'django.server': DEFAULT_LOGGING['handlers']['django.server'],
    },
    'loggers': {
        # default for all undefined Python modules
        '': {
            'level': 'WARNING',
            'handlers': ['console', 'sentry'],
        },
        # Our application code
        'app': {
            'level': LOGLEVEL,
            'handlers': ['console', 'sentry'],
            # Avoid double logging because of root logger
            'propagate': False,
        },
        # Prevent noisy modules from logging to Sentry
        'noisy_module': {
            'level': 'ERROR',
            'handlers': ['console'],
            'propagate': False,
        },
        # Default runserver request logging
        'django.server': DEFAULT_LOGGING['loggers']['django.server'],
    },
})

References

python logging

Archive

Dialogue & Discussion