Python Logging: Reference and Examples

Last updated:
Table of Contents

All examples assume Python 3, unless otherwise noted

Here's a sample python project where you can see some of these examples with actual working code.

A Logger object defines the rules for which logs are handled. This is the object you call methods such as .warn() and .error() on.

A Handler object defines how to format the log messages and what to do with them (i.e. save to disk, show on STDOUT, send to a log server, etc).

Log to STDOUT using the root logger

The simplest way to log messages is to use the logging module itself

import logging

# you can call methods on the module object itself.
# this represents the root logger
logging.basicConfig(format='%(asctime)s %(levelname)s %(message)s')
logging.warn('this is a warning message, written to the root logger')

Output (in the STDOUT):

2018-02-18 17:23:37,532 WARNING this is a warning message, written to the root logger
Twitter Linkedin YC Hacker News Reddit

Log to STDOUT with individual loggers

Multiple calls to getLogger() with the same name will always return a reference to the same Logger object.

In individual loggers (i.e. not the root logger) you need to set formatters via handlers.

Suppose the following file is located at rootmodule/foo.py

import logging

logger = logging.getLogger(__name__)

# add more information to the messages
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')

# stream handler prints errors to STDERR, which defaults to STDOUT
handler = logging.StreamHandler()
handler.setFormatter(formatter)

logger.addHandler(handler)
logger.error("Error message from rootmodule.foo")

OUTPUT (in the STDOUT):

2018-02-18 18:00:41,141 - rootmodule.foo - ERROR - Error message from rootmodule.foo

Log to a file

Suppose the following file is located at rootmodule/baz.py

import logging,os
import rootmodule

logger = logging.getLogger(__name__)

# add more information to the messages
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')

# create a file in the root module directory
output_file = os.path.dirname(rootmodule.__file__)+"/output.log"

handler = logging.FileHandler(output_file)
handler.setFormatter(formatter)

# by default, the level is WARN
logger.setLevel(logging.INFO)

logger.addHandler(handler)

logger.info("rootmodule.baz module loaded")
Twitter Linkedin YC Hacker News Reddit

Add Stack Trace to error log messages

Use the traceback module.

Assume the following file is at rootmodule/abc.py:

import logging
import traceback

logger = logging.getLogger(__name__)

# add more information to the messages
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')

# stream handler prints errors to STDERR, which defaults to STDOUT
handler = logging.StreamHandler()
handler.setFormatter(formatter)

logger.addHandler(handler)

def abc():
    try:
        raise Exception("some exception")
    except Except as ex:
        # this calls sys.exc_info() behind the scenese,
        # to get the exception that's currently being handled
        tb = traceback.format_exc() 

        logger.error(tb) 

This is what gets logged to STDOUT if you try and call abc():

2018-02-18 20:22:34,546 - rootmodule.abc - ERROR - Traceback (most recent call last):
  File "rootmodule/abc.py", line 17, in abc
    raise Exception("some exception")
Exception: some exception

Log rotation by size

This does not compress rotated files

It's the same as logging to file, but you use a different handler.

import logging,os
import rootmodule
from logging.handlers import RotatingFileHandler

logger = logging.getLogger(__name__)

formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')

# create a file in the root module directory
output_file = os.path.dirname(rootmodule.__file__)+"/output-rot.log"

# logs will be written to the output_file until it 
# reaches 1000000 bytes = 1MB, then a new file will be created
# and the old file will be renamed to output_rot.log.1
handler = RotatingFileHandler(output_file,maxBytes=1000000, backupCount=5)

handler.setFormatter(formatter)

logger.addHandler(handler)

logger.warn("module quux loaded")
Twitter Linkedin YC Hacker News Reddit

Log rotation by size, with compression

Install this Python package: concurrent-log-handler

$ pip install concurrent-log-handler

Then use that handler. It behaves the same way as RotatingFileHandler, but it will compress the rotated logs to save space.

import logging,os
import rootmodule
from concurrent_log_handler import ConcurrentRotatingFileHandler

logger = logging.getLogger(__name__)

formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')

# create a file in the root module directory
output_file = os.path.dirname(rootmodule.__file__)+"/output-rot.log"

# same as the other example, but rorated files will be gzipped
handler = ConcurrentRotatingFileHandler(output_file,maxBytes=1000000, backupCount=5, use_gzip=True)

handler.setFormatter(formatter)

logger.addHandler(handler)

logger.warn("module loaded")

Log errors in in Flask

You just need to setup the correct handler in the initialization code and then you use the @app.errorhandler(Exception) decorator to register it, so that every Exception raised in the routes gets logged.

import logging
import os
import traceback

from concurrent_log_handler import ConcurrentRotatingFileHandler
from flask import Flask, request, jsonify, make_response

app = Flask(__name__)

#####################################################################################
################################### ROUTES ##########################################
#####################################################################################

@app.route('/sample', methods=['GET'])
def sample():
    # lots of code that may raise exceptions
    # ...
    # ...
    return make_response(jsonify({"message": "ok"}),200) 


@app.errorhandler(Exception)
def exceptions(e):
    tb = traceback.format_exc()

    app.logger.error('%s %s %s %s 5XX INTERNAL SERVER ERROR\n%s',
                     request.remote_addr,
                     request.method,
                     request.scheme,
                     request.full_path,
                     tb)

    # it's good practice not to include error details in the response
    resp = make_response("Internal Server Error", 500)

    return resp

#####################################################################################
############################### INTIALIZATION CODE ##################################
#####################################################################################

if __name__ == '__main__':

    output_file = os.path.abspath('application.log')

    # 10M = 1024*1000*10 bytes   
    handler = ConcurrentRotatingFileHandler(output_file, maxBytes=1024 * 1000 * 10, backupCount=5, use_gzip=True)

    formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
    handler.setFormatter(formatter)

    app.logger.addHandler(handler)

    # https://stackoverflow.com/a/20423005/436721
    app.logger.setLevel(logging.INFO)

    app.run(host='0.0.0.0', port=8080)

Log rotation by time

TODO

References

Dialogue & Discussion