Trello's Migration from RabbitMQ to Kafka for Real-time WebSocket Updates

Trello, a popular Kanban board application, relies heavily on real-time updates to provide a seamless user experience. Imagine two users sharing a Trello board: if one person adds a to-do, it should instantly appear on the other person’s interface. This necessitates a robust real-time communication system, primarily driven by WebSockets.

This document dissects Trello’s journey in managing these real-time updates, specifically their migration from RabbitMQ to Kafka, highlighting the architectural evolution, challenges faced, and the significant benefits reaped. Trello’s path involved an initial move from Redis to RabbitMQ, and then a subsequent, more impactful, migration to Kafka.

Trello’s Real-time Update Architecture with RabbitMQ

Initially, Trello leveraged RabbitMQ to power its WebSocket-based real-time updates. The core use case involved sending updates to all relevant users subscribed to a particular board whenever an action (e.g., moving a card) occurred.

High-Level Flow

The general flow for real-time updates using RabbitMQ was as follows:

1. User Action: A user performs an action on the Trello UI.
2. API Call: An API call is made to the Trello backend.
3. Database Update: Changes are persisted in the database.
4. Event Publication: An event representing the change is pushed to a RabbitMQ cluster.
5. WebSocket Server Pull: WebSocket servers constantly pull these events from RabbitMQ.
6. Real-time Emission: WebSocket servers emit the updates to connected users who have subscribed to the relevant boards.

RabbitMQ Core Concepts

To understand Trello’s RabbitMQ architecture, it’s essential to grasp a few core RabbitMQ concepts:

• Queues: These are message buffers that hold messages until they are consumed by consumers.
• Exchanges: When a publisher sends a message to RabbitMQ, it doesn’t send it directly to a queue. Instead, it sends it to an exchange. The exchange’s role is to receive messages from producers and route them to appropriate queues based on predefined rules.
• Bindings: These are the routing rules that define how messages are routed from an exchange to a queue. A single queue can have multiple bindings.
• Routing Keys: Messages are published to an exchange with a routing key. This key is used by the exchange and its bindings to determine which queues should receive the message. For Trello, a board ID could serve as a routing key.
• Transient Queues: A special type of queue in RabbitMQ. If a queue is marked as transient, it is automatically deleted, along with all its bindings, as soon as the TCP connection from the client that created it is broken. This is crucial for dynamic, per-client subscriptions.

Detailed RabbitMQ Flow for Trello’s WebSockets

Here’s how Trello implemented real-time updates using these RabbitMQ concepts:

1. WebSocket Server Connection: When a WebSocket server starts, it establishes a connection to the RabbitMQ cluster.
2. Transient Queue Creation: For each WebSocket server instance, a transient queue is created on RabbitMQ.
3. User Subscription: When a user connects to a WebSocket server and subscribes to a specific entity (e.g., board:123), the WebSocket server acts on behalf of the user.
4. Binding Creation: The WebSocket server creates a binding on RabbitMQ for its transient queue. This binding specifies interest in a particular topic, such as board:123.
5. Update Publication: When an update occurs (e.g., a card moves on board:123), the Trello API server publishes a message to the RabbitMQ exchange with board:123 as the routing key.
6. Message Routing: The RabbitMQ exchange, based on the routing key and bindings, routes the message to all transient queues that have a binding for board:123.
7. Message Consumption: The WebSocket servers constantly pull messages from their respective transient queues.
8. Fan-out to Users: Upon receiving a message, the WebSocket server identifies which of its connected users are subscribed to board:123 and forwards the update to them in real-time.

Scaling RabbitMQ: The 15-Node Cluster Architecture

Trello’s RabbitMQ setup was a 15-node cluster designed for horizontal scalability and fault tolerance. It comprised two main types of clusters:

• Inbound Cluster: A single RabbitMQ cluster with 3 nodes. This cluster contained one primary exchange responsible for receiving all global updates from the Trello API servers.
• Outbound Clusters: Four separate RabbitMQ clusters, each containing 3 nodes. This totals 4 * 3 = 12 nodes.
  • These outbound clusters collectively hosted 16 outbound queues, distributed across them.

The total number of RabbitMQ nodes was 3 (inbound) + 12 (outbound) = 15 nodes.

Sharding Mechanism

To distribute messages efficiently across the 16 outbound queues, Trello employed a client-side sharding mechanism:

• When an update event was generated for a Trello board (e.g., T1), the Trello server would apply a hash function to the board ID (e.g., hash(T1) mod 16).
• This would yield a shard ID (0-15), corresponding to one of the 16 outbound queues (e.g., Q7).
• The message was then published to the inbound cluster, indicating its target shard.

Message Flow with Sharding and Shovel Plugin

1. API Server Publishes: The Trello API server pushes an update message to the inbound RabbitMQ cluster’s exchange, including the routing key (board ID) and the calculated shard ID.
2. Inbound Cluster Role: The inbound cluster’s primary role is to quickly accept messages and relay them to the appropriate outbound shard/queue.
3. RabbitMQ Shovel Plugin: To facilitate communication between the inbound cluster and the various outbound clusters, Trello utilized the RabbitMQ Shovel Plugin. This plugin connects a source cluster (inbound) to a destination cluster (outbound).
   • It consumes messages from a queue in the source cluster.
   • Identifies the routing information (shard ID).
   • Republishes the message to the corresponding queue in the correct outbound cluster.
4. Outbound Cluster Processing: Once a message arrives at the designated queue within an outbound cluster, the WebSocket servers connected to that cluster pull the message.
5. Fan-out to Users: The WebSocket server then forwards the message to the relevant connected users.

This architecture allowed for horizontal scaling by adding more outbound clusters and WebSocket servers.

graph TD
    subgraph Trello API Servers
        API[API Server]
    end

    subgraph RabbitMQ Inbound Cluster (3 Nodes)
        EX_IN[Exchange]
    end

    subgraph RabbitMQ Outbound Cluster 1 (3 Nodes)
        Q1_4[Queues Q1-Q4]
    end

    subgraph RabbitMQ Outbound Cluster 2 (3 Nodes)
        Q5_8[Queues Q5-Q8]
    end

    subgraph RabbitMQ Outbound Cluster 3 (3 Nodes)
        Q9_12[Queues Q9-Q12]
    end

    subgraph RabbitMQ Outbound Cluster 4 (3 Nodes)
        Q13_16[Queues Q13-Q16]
    end

    subgraph WebSocket Servers
        WS1[WebSocket Server 1]
        WS2[WebSocket Server 2]
        WS_N[WebSocket Server N]
    end

    subgraph End Users
        U1[User 1]
        U2[User 2]
        U_N[User N]
    end

    API -- Publishes event with routing key (Board ID) --> EX_IN
    EX_IN -- Shovel Plugin (Routes based on hash(Board ID) mod 16) --> Q1_4
    EX_IN -- Shovel Plugin --> Q5_8
    EX_IN -- Shovel Plugin --> Q9_12
    EX_IN -- Shovel Plugin --> Q13_16

    Q1_4 -- Pulls messages --> WS1
    Q5_8 -- Pulls messages --> WS1
    Q9_12 -- Pulls messages --> WS2
    Q13_16 -- Pulls messages --> WS_N

    WS1 -- Emits real-time updates --> U1
    WS1 -- Emits real-time updates --> U2
    WS_N -- Emits real-time updates --> U_N

    style EX_IN fill:#f9f,stroke:#333,stroke-width:2px
    style Q1_4 fill:#ccf,stroke:#333,stroke-width:2px
    style Q5_8 fill:#ccf,stroke:#333,stroke-width:2px
    style Q9_12 fill:#ccf,stroke:#333,stroke-width:2px
    style Q13_16 fill:#ccf,stroke:#333,stroke-width:2px
    style WS1 fill:#cfc,stroke:#333,stroke-width:2px
    style WS2 fill:#cfc,stroke:#333,stroke-width:2px
    style WS_N fill:#cfc,stroke:#333,stroke-width:2px

Problems with RabbitMQ at Scale

Despite the elaborate architecture, Trello encountered several significant challenges with RabbitMQ as they scaled:

1. Partition Handling: Managing partitions within RabbitMQ clusters proved to be complex and cumbersome.
2. Cluster Availability Issues: RabbitMQ clusters suffered from availability problems, primarily due to the split-brain problem.
   • In a split-brain scenario, two or more nodes in a cluster mistakenly believe they are the master, leading to conflicts and data inconsistencies.
   • The only reliable recovery mechanism from a split-brain was to reset the entire cluster.
   • A full cluster reset forced all connected clients (WebSocket servers) to reconnect, leading to significant downtime and a poor user experience. This made recovery from outages extremely expensive.
3. Transient Queue Overhead: The creation and deletion of transient queues and their associated bindings on every WebSocket server reconnect were slow and resource-intensive operations, adding to the system’s overhead.
4. Single Point of Failure: The inbound RabbitMQ cluster, despite being a 3-node cluster, could still represent a single point of failure or bottleneck for all incoming updates.

These issues prompted Trello to seek a more robust and scalable messaging solution, leading them to Kafka.

Migrating to Kafka: A Simplified and Resilient Architecture

Trello decided to migrate to Kafka to address the limitations of RabbitMQ, seeking benefits like:

• Reliable failovers.
• Guaranteed ordered message delivery.
• Lower latency.
• Higher availability.

Impact of the Migration

The move to Kafka yielded substantial improvements across multiple fronts:

• Memory Utilization: 33% less memory consumed.
• CPU Utilization: Twice the CPU utilization (indicating more efficient processing per core).
• Cost Reduction: A remarkable 5x reduction in infrastructure costs.
• Reduced Outages: Only 1 outage with Kafka compared to 4 with RabbitMQ over a comparable period.

These metrics clearly demonstrated a massive gain in efficiency, stability, and cost-effectiveness.

New Kafka-based Architecture

The Kafka-based architecture adopted a “pull” model, simplifying the overall system significantly.

graph TD
    subgraph Trello API Servers
        API_K[API Server]
    end

    subgraph Kafka Cluster
        K_TOPIC[Kafka Topic (Updates)]
    end

    subgraph Socket Master
        SM[Socket Master Process]
    end

    subgraph WebSocket Servers
        WS_C1[Socket Client (WebSocket Server 1)]
        WS_C2[Socket Client (WebSocket Server 2)]
        WS_CN[Socket Client (WebSocket Server N)]
    end

    subgraph End Users
        U_K1[User 1]
        U_K2[User 2]
        U_KN[User N]
    end

    API_K -- Pushes events --> K_TOPIC
    SM -- Pulls events, filters --> K_TOPIC
    SM -- Sends messages (TCP/HTTP) --> WS_C1
    SM -- Sends messages (TCP/HTTP) --> WS_C2
    SM -- Sends messages (TCP/HTTP) --> WS_CN

    WS_C1 -- Emits real-time updates --> U_K1
    WS_C2 -- Emits real-time updates --> U_K2
    WS_CN -- Emits real-time updates --> U_KN

    style K_TOPIC fill:#f9f,stroke:#333,stroke-width:2px
    style SM fill:#ccf,stroke:#333,stroke-width:2px
    style WS_C1 fill:#cfc,stroke:#333,stroke-width:2px
    style WS_C2 fill:#cfc,stroke:#333,stroke-width:2px
    style WS_CN fill:#cfc,stroke:#333,stroke-width:2px

The simplified flow is as follows:

1. Event Publication to Kafka: All update events from the Trello API servers are pushed directly to a Kafka topic.
2. Socket Master: A dedicated “Socket Master” process reads events from the Kafka topic.
   • The Socket Master’s role is to filter these events and identify which “Socket Clients” (WebSocket servers) should receive them.
3. Socket Master to Socket Client Communication: The Socket Master communicates with the relevant Socket Clients (WebSocket servers) via TCP or HTTP calls, relaying the filtered messages.
4. Socket Client Fan-out: Each Socket Client (WebSocket server) receives messages from the Socket Master. It then determines which of its connected users are subscribed to the specific topic (e.g., board:123) and forwards the real-time updates to them.

This architecture is inherently simpler and offers superior horizontal scalability. If more concurrent users or real-time communication capacity is needed, new Socket Client nodes can be added, which will receive updates from the Socket Master over TCP connections. The Socket Master, in turn, continues to pull data from Kafka. Kafka’s built-in guarantees for ordered delivery, fault tolerance, and high availability significantly simplify the overall system design.

Key Takeaways

Trello’s migration journey offers valuable lessons for designing and scaling real-time distributed systems:

• Infrastructure Cost Escalation: Day-zero solutions, while quick to implement, can become prohibitively expensive at scale. It’s crucial to continuously monitor and optimize infrastructure costs as systems grow.
• Batching for Efficiency: Whenever possible, batching operations can significantly minimize API calls and maximize throughput, leading to more efficient resource utilization.

This case study underscores the importance of understanding the trade-offs of different messaging systems and choosing the right tool for the job, especially as system requirements and scale evolve.