Not this kind of logging

Why you’re doing logging in python wrong.

A very common convention I’ve seen over the last few years in utilizing python logging is to use the root logger:

import logging
logger = logging.getLogger()

This is wrong. Don’t do this. Instead of using the root logger — you should be using a specific logger, and leveraging logging “ancestry”.

Logging ancestry

In python’s logging module, ancestry is how loggers in functions, modules, and packages inherit configurations from a logger that’s higher up in the package hierarchy. Let’s look at the following package:

my-package/
├── src/
│   └── my_package
│       ├── sub_folder
│       │   ├── __init__.py
│       │   └── module_a.py
│       ├── module_b.py
│       ├── __init__.py
│       └── main.py
└── setup.py

In this package, we have a parent of my_package which will have subfolders and sub-modules, and the main module. If I were using the root logger, I can configure it to show the module, package, function, line, etc in my logging config. This is fine and dandy — but what if I have my own logging formats, and still need to see logs for other third-party packages that have used the root logger?

You’ve just overridden their configuration too — and if you’ve overridden them with a common logger name, or disabled the existing loggers, you’ve now lost your logging for other packages that you might need.

What’s in a name?

So what does this mean in practice and why is it important that the logging documentation recommends using the builtin __name__ variable? How does this make it easy to easily configure/utilize logging?

“The logger name hierarchy is analogous to the Python package hierarchy, and identical to it if you organise your loggers on a per-module basis using the recommended construction logging.getLogger(__name__).”

If I were to print(__name__) in module_a.py given the package structure above, the __name__ would show the stringmy_package.subfolder.module_a. This string actually has some special properties to the logging module. If a logger exists with the name my_package, the loggers my_package.subfolder and my_package.subfolder.module_a will both inherit it’s configuration because it is their ancestor.

If we have configured the logger my_package in the root folder’s __init__.py, this configuration will then be inherited by any other loggers that are created that have my_package. at the beginning of their logging name. If you don’t create a logger for an intermediary level, the logging module will create a placeholder for it.

NOTE: If you don’t configure the logger my_package in the __init__.py of my_package you may create placeholder loggers that have no configuration (i.e. default root logger configuration) — so any logging configuration you create may not take effect for all of your loggers, and you’ll have to go back and set their configuration yourself! Don’t bring yourself that pain!

How easy is it?

If you’ve defined your logger in the top level __init__.py you’ve made your life a lot easier — in any other area of the package you can create a logger that inherits the correct logging configuration with two lines of code:

import logging
logger = logging.getLogger(__name__)

This will give you logging that has specificity — so you’ll immediately see where the logging call came from within the package, not just the module or function name — this helps you easily identify where in the code your logging is coming from, which makes troubleshooting and identifying issues significantly faster, and following the logical pathways during an investigation significantly more clear.

Should I really NEVER use the root logger?

You should absolutely use the root logger if it’s useful — but you should definitely understand the consequences when you do, and why it’s useful. A great reason to change or use the root logger is if you need to understand what a third party package is doing as you’re using it.

Gunicorn is a great example where the logging swallows errors for validation issues when serving a FastAPI app — if you change the root logger level to DEBUG and configure it to use the stream logger, you will have changed it for all loggers — including the Gunicorn loggers! This lets you see all of the things it’s doing in addition to your own logging, including a validation error that would have otherwise been swallowed with no warnings!