Structured Logging in Python: The Key to Observability

🪵 Structured Logging in Python: The Key to Observability

While the logging module is powerful, its default output is unstructured plain text, making it difficult for tools like ELK Stack or Splunk to search, aggregate, and analyze messages efficiently. Structured logging solves this by outputting logs in a standard format (usually JSON) where every event detail is an easily parsable key-value pair.

This article details how to integrate structured logging into a Python application using the popular library python-json-logger and how to leverage it for operational insight.

1. Why Structured Logs are Essential

Traditional logs are often inconsistent and require complex regular expressions to parse.

Feature	Plain Text Logs	Structured Logs (JSON)
Example	2025-12-14 ERROR Failed to connect (DB: prod)	{"timestamp": ..., "level": "ERROR", "message": "Failed to connect", "db_name": "prod"}
Machine Readability	Low (requires regex parsing)	High (direct key access)
Searchability	Difficult; searches must use fuzzy text matching.	Easy; search on specific keys (e.g., level:"ERROR" AND db_name:"prod").
Context	Contextual data (user ID, request ID) must be embedded in the message string.	Context is added as separate top-level keys.

2. Implementation: Using python-json-logger

The cleanest way to implement structured logging is by replacing the standard logging.Formatter with the jsonlogger.JsonFormatter in your dictConfig.

Installation

pip install python-json-logger

Code Example: Configuring the JSON Formatter

This configuration uses dictConfig to replace the standard formatter with the JSON version, automatically including standard log fields as JSON keys.

import logging.config
from pythonjsonlogger import jsonlogger
import sys

# 1. Define the custom JSON formatter class path
# We'll use the JsonFormatter that came with the package
JSON_FORMATTER_CLASS = 'pythonjsonlogger.jsonlogger.JsonFormatter'

LOGGING_CONFIG = {
    'version': 1,
    'disable_existing_loggers': False,
    'formatters': {
        'json': {
            '()': JSON_FORMATTER_CLASS,
            # Keys to include in every log record. Always include message and level.
            'format': '%(timestamp)s %(levelname)s %(name)s %(module)s %(lineno)d %(message)s'
        }
    },
    'handlers': {
        'json_console': {
            'class': 'logging.StreamHandler',
            'formatter': 'json', # Use the new JSON formatter
            'level': 'INFO',
            'stream': sys.stdout,
        },
    },
    'loggers': {
        '': {'handlers': ['json_console'], 'level': 'INFO', 'propagate': False}
    }
}

logging.config.dictConfig(LOGGING_CONFIG)
logger = logging.getLogger("DataService")

logger.info("Service initialized and listening for requests.")

Output (Single line, pretty-printed here for clarity):

{
  "timestamp": "2025-12-14T09:15:30.123456", 
  "levelname": "INFO", 
  "name": "DataService", 
  "module": "script_name", 
  "lineno": 27, 
  "message": "Service initialized and listening for requests."
}

3. Adding Dynamic Context to Structured Logs

The greatest benefit of structured logging is the easy addition of application-specific context using the extra dictionary (a best practice from the previous article). With the JsonFormatter, keys in extra automatically become top-level fields in the JSON output.

Code Example: Contextual Logging

logger = logging.getLogger("Transaction")
# Assumes JSON configuration is already loaded

def process_transaction(user_id, amount):
    # Contextual data we want to search on later
    context = {
        'user_id': user_id,
        'transaction_amount': amount,
        'region': 'EU'
    }
    
    try:
        # Simulate an error
        if user_id == 105:
            raise ValueError("Account suspended.")
        
        logger.info("Transaction approved.", extra=context)
        
    except Exception:
        # Use logger.exception() - the JsonFormatter automatically handles the traceback
        logger.exception("Transaction failed, initiating rollback.", extra=context)

Output (for user 105 failure):

{
  "timestamp": "...", 
  "levelname": "ERROR", 
  "name": "Transaction", 
  "message": "Transaction failed, initiating rollback.", 
  "user_id": 105,                       
  "transaction_amount": 100.0, 
  "region": "EU",
  "exc_info": "Traceback (most recent call last):\n..." 
}

4. Best Practice: Standardizing Keys

To ensure logs from different services are compatible for centralized searching, define and enforce a strict set of key names across all applications.

Recommended Key	Standard Value	Purpose
timestamp	ISO 8601	The universal time of the event.
host	Hostname/Pod ID	Where the event occurred.
name	Logger Name (__name__)	Source module or class.
service_name	Custom	The overall application name (e.g., OrderProcessor-v2).
trace_id	Unique ID	Used for tracing requests across multiple services.

---

Sources and Further Reading

1. Python JSON Logger Documentation - Installation and Usage
2. Logging Config with Python - A Practical dictConfig Example
3. The Benefits of Structured Logging (External Resource)
4. OpenTelemetry Standards for Trace/Context (For trace_id standardization)