Distributed systems do not have a ‘shared clock’, so it becomes very difficult for two nodes to agree on what “now” means. To be honest, this is what makes distributed systems so interesting.
Also, this is where the “happened-before” relationship comes in handy and brings order to this chaos. Let’s dig slightly deeper…
If event A happened before event B, it means A could have causally influenced B. Two events with no such relationship are simply concurrent, and the system treats them as such. Here is a concrete example - I will use the classic names Alice and Bob and imagine them editing a shared document.
-
Alice reads the document (event A), makes a change (event B), and saves it.
-
Bob also reads the document (event C) before Alice’s save arrives.
-
Bob then saves his version (event D).
Now, without tracking the happened-before relationship, the system has no way of knowing that Bob’s read (C) missed Alice’s write (B). It will silently overwrite Alice’s changes.
With causal tracking, the system knows that event B happened before event D, so Bob’s write is based on stale data and should have seen B. It can detect this conflict and ask for a resolution instead of silently losing data.
By the way, this is exactly how systems like DynamoDB and Riak handle concurrent writes without data loss. There are a few practical ways to implement this:
- Lamport Clocks
Each node increments a counter on every event and updates it on message receipt. Pretty simple to implement.
- Vector Clocks
Each node tracks a counter per node. This captures true causal relationships and lets you detect concurrent writes precisely. Git’s merge detection works on a similar principle (kind of).
- Hybrid Logical Clocks (HLC)
Combines physical time with logical counters. Used in CockroachDB. Lets you reason about causality while staying close to wall-clock time, so you can efficiently serve time-based queries and maintain a consistent ordering without relying purely on physical clocks.
Of course, the choice depends on your consistency needs and the overhead you can afford. Vector clocks grow with the number of nodes, and HLCs are a good middle ground for most production systems.
By the way, causality does not solve all consistency problems, but it tells you what you cannot ignore: if A happened before B, your system must respect that order. Everything else is negotiable.
This is a rabbit hole in itself, but I hope this gives you enough of a kick to explore further :)