Implementing Redis-Style Transactions in a Custom Database (DiceDB)

Transactions are a cornerstone feature of any robust database, ensuring data integrity and atomicity. Redis, known for its performance and simplicity, implements transactions with a unique approach that deviates from traditional relational database models. This article delves into Redis’s transaction mechanics and details how to implement a similar system within a custom database, DiceDB, leveraging its single-threaded architecture.

Understanding Redis Transaction Behavior

Redis transactions are initiated, queued, and executed using a specific set of commands:

• MULTI: This command marks the beginning of a transaction. Once MULTI is issued, the client enters a transaction mode, indicated by (TX) in the CLI.
• Command Queuing: Any subsequent commands issued after MULTI are not executed immediately. Instead, they are queued for later execution. Redis responds with QUEUED for each command.
• EXEC: This command triggers the execution of all commands currently in the transaction queue. All queued commands are executed sequentially and atomically (from Redis’s perspective) in one shot. The responses for all commands are returned as an array.
• DISCARD: This command aborts the transaction. All commands currently in the queue are cleared, and the client exits transaction mode without executing any of them.

Key Characteristics of Redis Transactions

1. Atomicity (Simplified): While EXEC attempts to run all commands, Redis does not provide a rollback mechanism if a command fails during execution. If a command within the EXEC block fails, subsequent commands will still attempt to execute. Redis prioritizes simplicity and performance over complex transactional guarantees like full ACID compliance with rollbacks.
2. Single-Threaded Execution: Redis is inherently single-threaded. This design choice significantly simplifies transaction management. When an EXEC command is fired, all queued commands for that client are executed sequentially without interruption. During this execution, no other client’s commands are processed, effectively providing isolation and atomicity without explicit locking mechanisms.
3. Concurrent Queuing: Multiple clients can concurrently initiate MULTI and enqueue commands. Each client maintains its own isolated transaction queue. However, only one client’s EXEC command will be processed at a time due to Redis’s single-threaded nature. Other EXEC calls will wait their turn.
4. Error Handling: Attempting EXEC or DISCARD without first issuing MULTI results in an error.

Implementing Transactions in DiceDB

To replicate Redis’s transaction model in DiceDB, a significant shift from a stateless to a stateful architecture is required. Previously, DiceDB processed commands immediately upon receipt. Now, it must track the state of each connected client.

1. Client State Management

The core change involves associating a unique state with each connected client. This is achieved through:

• CoreClient Object: A new object, CoreClient, is introduced to encapsulate per-client state. It typically includes:

  • fileDescriptor (FD): The unique identifier for the client’s socket connection.
  • commandQueue (CQ): A list or array to store commands enqueued during a transaction.
  • isTransaction: A boolean flag indicating whether the client is currently in transaction mode.

• connectedClients Map: A global hash map (map[int]CoreClient) stores all active CoreClient objects, indexed by their file descriptor. This allows DiceDB to retrieve and update the state of any connected client.

Client Lifecycle Management:

• Client Connection: When a new client connects (i.e., a new socket connection is accepted and a file descriptor is obtained), a new CoreClient object is created with an empty commandQueue and isTransaction set to false. This object is then added to the connectedClients map using the file descriptor as the key.
• Command Reception: Before processing any command, DiceDB retrieves the CoreClient object corresponding to the client’s file descriptor from the connectedClients map. This ensures that the correct client state is always accessible.
• Client Disconnection: When a client disconnects, its CoreClient object is removed from the connectedClients map, and the associated file descriptor is closed.

2. Restructuring Command Execution

The existing command evaluation logic needs to be refactored to support queuing and transactional execution:

• executeCommand(command, client): This new helper function takes a command and the client object, executes the command against the database, and returns the raw byte response. Crucially, it does not write the response directly to a buffer.
• executeCommandToBuffer(command, buffer, client): This function wraps executeCommand. It calls executeCommand to get the byte response and then writes that response into a provided buffer. This separation is vital for EXEC, which needs to collect multiple responses before formatting them into a single array response.
• evalAndRespond (Main Command Loop): This function, which previously handled direct command execution, is now the central dispatcher. For each command received from a client:
  • It first checks the client.isTransaction flag.
  • If isTransaction is false: The command is executed immediately using executeCommandToBuffer, and the response is sent back to the client. This is the normal, non-transactional flow.
  • If isTransaction is true: DiceDB then checks if the command is EXEC or DISCARD (transaction control commands).
    • If the command is not EXEC or DISCARD: The command is appended to client.commandQueue, and the response QUEUED is sent back to the client.
    • If the command is EXEC or DISCARD: The respective transaction control logic is invoked.

3. Transaction Command Handlers

Dedicated handlers are implemented for MULTI, EXEC, and DISCARD:

• MULTI Handler: When MULTI is received, the client.isTransaction flag is set to true for that specific client. DiceDB then responds with OK.

• EXEC Handler: This is the most complex part. When EXEC is received:

  1. Pre-check: Verify client.isTransaction is true. If not, return an error (EXEC without MULTI).
  2. Execution Loop: Iterate through each command stored in client.commandQueue.
  3. Command Execution: For each queued command, call executeCommand to get its byte response.
  4. Response Collection: Collect all individual command responses.
  5. RESP Array Formatting: Format the collected responses into a single RESP (Redis Serialization Protocol) array. This involves prepending *<number_of_responses> and then concatenating each individual RESP-encoded response.
  6. Cleanup: After all commands are executed and responses collected, clear client.commandQueue and set client.isTransaction back to false.
  7. Response: Send the final RESP array response to the client.

• DISCARD Handler: When DISCARD is received:

  1. Pre-check: Verify client.isTransaction is true. If not, return an error (DISCARD without MULTI).
  2. Cleanup: Clear client.commandQueue and set client.isTransaction back to false.
  3. Response: Send an OK response to the client.

4. Concurrency and Atomicity in a Single-Threaded Model

DiceDB, like Redis, benefits greatly from its single-threaded nature in managing transactions:

• Implicit Atomicity: When an EXEC command is processed, the main event loop dedicates itself to executing all commands in that client’s queue. No other client’s commands or EXEC calls can interleave during this period. This inherently provides the