Skip to main content
🛡️ Verified Technical Content: Written by Serhii Hrekov. | Last reviewed & updated in Git: July 21, 2026

Python Logging: The Complete Guide from basicConfig to Structured JSON Observability

· 11 min read
Serhii Hrekov
Serhii Hrekov
Senior Software Engineer & System Architect specializing in Python, Web Systems, Cloud Infrastructure & Automation

Logs are a primary source of truth for application health, performance audits, and runtime debugging. However, outputting unstructured plain text console strings limits searchability and scalability. To build modern, observable applications, you need a robust logging hierarchy, log rotation schemes, centralized configurations, and structured output formats.

This guide provides a comprehensive manual on Python logging, covering basic setup parameters, log rotation, dict-based configurations, JSON-structured output, contextual metadata injection, and performance best practices.

Logging Foundations and basicConfig​

The easiest way to initialize a global logging configuration for simple scripts or local development is via logging.basicConfig(). This sets the properties of the root logger, which acts as the top-level parent for all named loggers in your codebase.

Parameters​

  • level: The minimum severity threshold to process. Events below this level are ignored.
  • filename / filemode: If set, routes output to the specified file in append ('a') or overwrite ('w') mode. If omitted, logs default to console streams.
  • format / datefmt: Structures the log message template and configures timestamps.
  • stream: Specifies a destination output stream (e.g. sys.stdout or sys.stderr) when writing to standard output instead of files.

Formatter Placeholders​

Format templates are constructed using special placeholders:

  • %(asctime)s - Timestamp (e.g. 2025-12-14 07:05:00,123).
  • %(levelname)s - Severity name (INFO, WARNING, ERROR).
  • %(name)s - Name of the logger generating the event.
  • %(message)s - The actual logged text.
  • %(filename)s / %(lineno)d - File basename and source code line number.
import logging
import sys

logging.basicConfig(
    level=logging.DEBUG,
    stream=sys.stdout,
    format='%(asctime)s [%(levelname)s] (%(filename)s:%(lineno)d) %(message)s',
    datefmt='%Y-%m-%d %H:%M:%S'
)

logger = logging.getLogger("DevLogger")
logger.info("Local configuration active.")

Critical Gotcha: Single Execution​

basicConfig() configuration runs only once. If a handler has already been attached to the root logger-either by a previous call or by logging an event before configuration-subsequent calls to basicConfig() will be silently ignored. Always execute configuration routines at the absolute entry point of your application.

Stream and File Logging (Rotation for Production)​

In production, log files grow continuously. If left unmanaged, they can consume all available disk space. Log rotation automatically closes, archives, and replaces active files.

1. Manual FileHandler Attachment​

To log to a file while maintaining separate formatting or severity levels for the console, define handlers manually:

import logging
import sys

logger = logging.getLogger("AppService")
logger.setLevel(logging.DEBUG)

# 1. Create File Handler (Sends all DEBUG and above logs to the file)
file_handler = logging.FileHandler("app_debug.log", mode="a")
file_handler.setLevel(logging.DEBUG)
file_handler.setFormatter(logging.Formatter("%(asctime)s - %(levelname)s - %(message)s"))
logger.addHandler(file_handler)

# 2. Create Console Handler (Console only gets WARNING and above)
console_handler = logging.StreamHandler(sys.stdout)
console_handler.setLevel(logging.WARNING)
logger.addHandler(console_handler)

2. Size-Based Rotation​

The RotatingFileHandler triggers rotation once the log file reaches a specific size in bytes:

from logging.handlers import RotatingFileHandler
import logging

rotating_handler = RotatingFileHandler(
    "app.log",
    maxBytes=5 * 1024 * 1024, # 5 MB limits
    backupCount=5,            # Keep app.log.1, app.log.2, up to app.log.5
    encoding="utf-8"
)

3. Time-Based Rotation​

The TimedRotatingFileHandler rotates logs at predetermined intervals, such as hourly or daily:

from logging.handlers import TimedRotatingFileHandler

daily_handler = TimedRotatingFileHandler(
    "app_daily.log",
    when="midnight",  # Rotate daily at midnight
    interval=1,
    backupCount=7     # Keep a rolling 7-day archive
)

when intervals: 's' (seconds), 'm' (minutes), 'h' (hours), 'd' (days), 'midnight' (daily rotation).

Centralizing Configuration via dictConfig​

Manually configuring multiple loggers, handlers, and formats in code becomes verbose. Python provides logging.config.dictConfig() to load configurations from a centralized dictionary structure (often parsed from JSON or YAML files):

import logging.config

LOGGING_CONFIG = {
    'version': 1,
    'disable_existing_loggers': False,
    'formatters': {
        'standard': {
            'format': '%(asctime)s [%(levelname)s] %(name)s: %(message)s'
        }
    },
    'handlers': {
        'file_rotation': {
            'class': 'logging.handlers.TimedRotatingFileHandler',
            'filename': 'production.log',
            'when': 'midnight',
            'backupCount': 14,
            'formatter': 'standard',
        }
    },
    'loggers': {
        '': {  # Root logger
            'handlers': ['file_rotation'],
            'level': 'WARNING',
        }
    }
}

logging.config.dictConfig(LOGGING_CONFIG)

Structured Logging (JSON Output)​

Centralized logging engines (like Elasticsearch, Datadog, or Splunk) process structured log records more efficiently than plain text. Structured logging outputs events as key-value JSON pairs, eliminating the need for complex regular expression parsers.

Implementation with python-json-logger​

Install the JSON formatter helper library:

pip install python-json-logger

Include the JsonFormatter class directly in your dictConfig setup:

import logging.config
import sys

JSON_CONFIG = {
    'version': 1,
    'disable_existing_loggers': False,
    'formatters': {
        'json': {
            '()': 'pythonjsonlogger.jsonlogger.JsonFormatter',
            'format': '%(timestamp)s %(levelname)s %(name)s %(module)s %(lineno)d %(message)s'
        }
    },
    'handlers': {
        'json_console': {
            'class': 'logging.StreamHandler',
            'formatter': 'json',
            'stream': sys.stdout,
        }
    },
    'loggers': {
        '': {
            'handlers': ['json_console'],
            'level': 'INFO',
        }
    }
}

logging.config.dictConfig(JSON_CONFIG)
logger = logging.getLogger("AuthService")
logger.info("User session verified.")

JSON Output:

{"timestamp": "2025-12-15T12:00:00.123456", "levelname": "INFO", "name": "AuthService", "module": "auth", "lineno": 42, "message": "User session verified."}

Enriching Logs with Dynamic Context​

To trace user requests across multiple microservices, inject context properties (like user_id or trace_id) into your log formatters.

The extra Parameter​

Pass local runtime properties using the extra dictionary keyword. The JsonFormatter automatically appends these keys to the root level of the JSON output:

logger = logging.getLogger("OrderProcessor")

def process_checkout(user_id, order_id):
    context = {
        "user_id": user_id,
        "order_id": order_id,
        "environment": "production"
    }

    logger.info("Checkout completed.", extra=context)

JSON Output:

{
  "timestamp": "2025-12-15T12:05:00.111222",
  "levelname": "INFO",
  "name": "OrderProcessor",
  "message": "Checkout completed.",
  "user_id": 9982,
  "order_id": "ORD-77",
  "environment": "production"
}

Production Best Practices​

1. Never Log Directly to the Root Logger​

Always create named loggers using __name__ at the module level. This defines a clear package-based logging hierarchy, allowing you to filter logs by specific namespace dependencies:

# CORRECT
logger = logging.getLogger(__name__)

2. Defer String Interpolation (Lazy Formatting)​

Do not use f-strings or manual formatting when passing arguments to logging functions. Evaluating variables inside f-strings allocates memory and consumes CPU cycles even if the target logging level is disabled. Pass variables as arguments to defer formatting:

# INCORRECT: Formats string immediately
logger.debug(f"Calculated expensive metadata: {run_complex_fn()}")

# CORRECT: Defer formatting until level validation passes
logger.debug("Calculated expensive metadata: %s", run_complex_fn())

3. Segregate Runtime Errors from System Failures​

  • ERROR / CRITICAL: Reserve these for structural system failures that require immediate engineer attention (e.g. database connection failures or missing dependencies).
  • WARNING: Use for expected application flow deviations that are handled gracefully by the code (e.g. invalid user passwords or validation rejections).

4. Capture Exceptions with logger.exception()​

When handling caught exceptions inside an except block, use logger.exception(). This automatically appends the active traceback details (exc_info=True) to the log payload:

try:
    result = 10 / 0
except ZeroDivisionError:
    # Captures the full error stack trace automatically
    logger.exception("Failed to run arithmetic check.")

5. Reusable Library Loggers should leave Level at NOTSET​

If writing a reusable library, do not configure handlers or hardcode severity levels. Leave the default level at NOTSET (0) and attach a NullHandler. This gives the consuming application complete control over the library's log routing:

# In library/client.py
import logging

logger = logging.getLogger(__name__)
logger.addHandler(logging.NullHandler()) # Prevent "No handlers could be found" warning

Sources​