Stripe’s Canonical Log Lines: A Smarter Approach to Structured Logging

Logging is an indispensable tool for debugging and understanding system behavior, especially in complex distributed systems. However, traditional logging practices often introduce their own set of challenges. Stripe, known for its robust engineering, has developed an innovative approach called “Canonical Log Lines” to overcome these common logging hurdles.

The Problem with Traditional Logging

While logging is fundamental, it comes with significant drawbacks:

• Excessive Text Volume: Systems generate an enormous amount of log data, making it akin to finding a “needle in a haystack” when trying to pinpoint specific issues.
• Slow Information Retrieval: Sifting through vast quantities of text data is inherently slow, hindering rapid incident response.
• Complex Querying: Log processing systems like Splunk often require intricate query syntax, demanding specialized knowledge to extract meaningful insights.

Evolution of Logging Practices

To address these challenges, logging has evolved:

Classic Log Traces

Initially, logs were simple text lines. For a single request, multiple log lines would be printed across various services. The primary challenge was correlating these disparate lines to understand the full lifecycle of a single request, especially when many requests were processed concurrently.

Structured Logging

Structured logging emerged as a solution, where log lines are augmented with key-value pairs. This allows for:

• Request ID Correlation: Attaching a request_id to every log line enables stitching together all events related to a specific request.
• Enhanced Queryability: Key-value pairs provide a structured way to query logs, for example, request_started | stats count by http_status or stats count P50(duration), P90(duration), P99(duration). This makes it easier to filter and aggregate data.

However, even with structured logging, a significant problem persists: information is often split across multiple log lines. This means that to answer complex questions or perform aggregations (e.g., “which users are being rate-limited the most?”), the log processing system still needs to parse, process, and aggregate a large volume of data, stitching together multiple events belonging to the same request. This process can be expensive and time-consuming.

Stripe’s Canonical Log Lines: The Solution

Stripe’s “Canonical Log Lines” approach tackles the limitations of structured logging by consolidating all critical information about a request into a single, comprehensive log line.

Core Concept

Instead of scattering information across many log lines, a single canonical log line is emitted at the end of a request. This line contains all relevant statistics and details pertaining to that request. Examples of information included are:

• Number of database queries fired
• HTTP path
• HTTP status code
• Request ID
• Rate quota information
• Method allocation count
• Other key statistics relevant to the request’s execution.

Benefits

This consolidated approach offers several advantages:

• Eliminates Stitching: For quick information retrieval, there’s no need for the log processing system to stitch multiple log lines together.
• Reduced Processing Overhead: The log processing system doesn’t have to parse and process a lot of distributed data to answer common questions.
• Efficient Aggregation and Querying: Canonical log lines can be indexed separately, making queries and aggregations significantly cheaper and faster. For example, finding the most rate-limited users becomes a simple query on a single log line type.
• Lightweight Payload: By combining common information into a small payload, it becomes easy to process and visualize.

Implementation Details

Stripe implemented canonical log lines with a thoughtful architecture:

1. Middleware Integration: The system is implemented within their application middleware. After every request completes, all relevant statistics are collected.
2. Consistent Naming Strategy: To prevent chaos and ensure data integrity, Stripe enforced a strict, consistent naming strategy for key-value pairs within these canonical log lines. This ensures that all teams and systems use standardized keys, preventing overlaps and maintaining consistency across the organization.
3. Asynchronous Logging Pipeline: The collected canonical log lines are pushed through an asynchronous pipeline:
   • API Server: Emits the log lines to stdout.
   • Fluent Bit: Captures the stdout logs.
   • Kafka: Fluent Bit pushes the captured logs asynchronously to Kafka.
   • Log Processing System: Picks up the logs from Kafka and indexes them into a specialized database.

This pipeline ensures that logging does not block the request path and that the data is reliably transported for processing and analysis.

Impact and Use Cases

The canonical log line system significantly simplifies observability:

• Simplified Queries: Queries become much simpler. For instance, to find which users are rate-limited the most, a query might look like canonical_log_line where rate_allowed is false | stats count by user_id.
• Developer Dashboards: The system enables the creation of lightweight, easy-to-build developer dashboards that render quick metrics and visualizations based on the consolidated data.

Conclusion

Stripe’s Canonical Log Lines represent a sophisticated evolution in logging practices. By consolidating critical request-level information into a single, structured log entry, they have dramatically improved the efficiency of debugging, monitoring, and analysis in their distributed systems. This approach reduces the complexity and cost associated with traditional logging, making it easier for engineers to gain insights and respond to incidents swiftly.