Python Logging: Reference and Examples

Last updated:
Table of Contents

All examples assume Python 3, unless otherwise noted

Here's a sample python project where you can see some of these examples with actual working code.

A Logger object defines the rules for which logs are handled. This is the object you call methods such as .warn() and .error() on.

A Handler object defines how to format the log messages and what to do with them (i.e. save to disk, show on STDOUT, send to a log server, etc).

Log to STDOUT using the root logger

The simplest way to log messages is to use the logging module itself

import logging

# you can call methods on the module object itself.
# this represents the root logger
logging.basicConfig(format='%(asctime)s %(levelname)s %(message)s')
logging.warn('this is a warning message, written to the root logger')

Output (in the STDOUT):

2018-02-18 17:23:37,532 WARNING this is a warning message, written to the root logger

Log to STDOUT with individual loggers

Multiple calls to getLogger() with the same name will always return a reference to the same Logger object.

In individual loggers (i.e. not the root logger) you need to set formatters via handlers.

Suppose the following file is located at rootmodule/foo.py

import logging

logger = logging.getLogger(__name__)

# add more information to the messages
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')

# stream handler prints errors to STDERR, which defaults to STDOUT
handler = logging.StreamHandler()
handler.setFormatter(formatter)

logger.addHandler(handler)
logger.error("Error message from rootmodule.foo")

OUTPUT (in the STDOUT):

2018-02-18 18:00:41,141 - rootmodule.foo - ERROR - Error message from rootmodule.foo

Log to a file

Suppose the following file is located at rootmodule/baz.py

import logging,os
import rootmodule

logger = logging.getLogger(__name__)

# add more information to the messages
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')

# create a file in the root module directory
output_file = os.path.dirname(rootmodule.__file__)+"/output.log"

handler = logging.FileHandler(output_file)
handler.setFormatter(formatter)

# by default, the level is WARN
logger.setLevel(logging.INFO)

logger.addHandler(handler)

logger.info("rootmodule.baz module loaded")

Add Stack Trace to error log messages

Use the traceback module.

Assume the following file is at rootmodule/abc.py:

import logging
import traceback

logger = logging.getLogger(__name__)

# add more information to the messages
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')

# stream handler prints errors to STDERR, which defaults to STDOUT
handler = logging.StreamHandler()
handler.setFormatter(formatter)

logger.addHandler(handler)

def abc():
    try:
        raise Exception("some exception")
    except Except as ex:
        # this calls sys.exc_info() behind the scenese,
        # to get the exception that's currently being handled
        tb = traceback.format_exc() 

        logger.error(tb) 

This is what gets logged to STDOUT if you try and call abc():

2018-02-18 20:22:34,546 - rootmodule.abc - ERROR - Traceback (most recent call last):
  File "rootmodule/abc.py", line 17, in abc
    raise Exception("some exception")
Exception: some exception

Log rotation by size

This does not compress rotated files

It's the same as logging to file, but you use a different handler.

import logging,os
import rootmodule
from logging.handlers import RotatingFileHandler

logger = logging.getLogger(__name__)

formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')

# create a file in the root module directory
output_file = os.path.dirname(rootmodule.__file__)+"/output-rot.log"

# logs will be written to the output_file until it 
# reaches 1000000 bytes = 1MB, then a new file will be created
# and the old file will be renamed to output_rot.log.1
handler = RotatingFileHandler(output_file,maxBytes=1000000, backupCount=5)

handler.setFormatter(formatter)

logger.addHandler(handler)

logger.warn("module quux loaded")

Log rotation by size, with compression

Install this Python package: concurrent-log-handler

$ pip install concurrent-log-handler

Then use that handler. It behaves the same way as RotatingFileHandler, but it will compress the rotated logs to save space.

import logging,os
import rootmodule
from concurrent_log_handler import ConcurrentRotatingFileHandler

logger = logging.getLogger(__name__)

formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')

# create a file in the root module directory
output_file = os.path.dirname(rootmodule.__file__)+"/output-rot.log"

# same as the other example, but rorated files will be gzipped
handler = ConcurrentRotatingFileHandler(output_file,maxBytes=1000000, backupCount=5, use_gzip=True)

handler.setFormatter(formatter)

logger.addHandler(handler)

logger.warn("module loaded")

Log errors in in Flask

You just need to setup the correct handler in the initialization code and then you use the @app.errorhandler(Exception) decorator to register it, so that every Exception raised in the routes gets logged.

import logging
import os
import traceback

from concurrent_log_handler import ConcurrentRotatingFileHandler
from flask import Flask, request, jsonify, make_response

app = Flask(__name__)

#####################################################################################
################################### ROUTES ##########################################
#####################################################################################

@app.route('/sample', methods=['GET'])
def sample():
    # lots of code that may raise exceptions
    # ...
    # ...
    return make_response(jsonify({"message": "ok"}),200) 


@app.errorhandler(Exception)
def exceptions(e):
    tb = traceback.format_exc()

    app.logger.error('%s %s %s %s 5XX INTERNAL SERVER ERROR\n%s',
                     request.remote_addr,
                     request.method,
                     request.scheme,
                     request.full_path,
                     tb)

    # it's good practice not to include error details in the response
    resp = make_response("Internal Server Error", 500)

    return resp

#####################################################################################
############################### INTIALIZATION CODE ##################################
#####################################################################################

if __name__ == '__main__':

    output_file = os.path.abspath('application.log')

    # 10M = 1024*1000*10 bytes   
    handler = ConcurrentRotatingFileHandler(output_file, maxBytes=1024 * 1000 * 10, backupCount=5, use_gzip=True)

    formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
    handler.setFormatter(formatter)

    app.logger.addHandler(handler)

    # https://stackoverflow.com/a/20423005/436721
    app.logger.setLevel(logging.INFO)

    app.run(host='0.0.0.0', port=8080)

Log rotation by time

TODO

References

Dialogue & Discussion