Distributed systems (or even entangled microservices) are difficult to debug, and to be honest, you almost always learn something new doing it. Some classic examples:
- zombie worker processing a job twice
- split-brain during leader election
- clock skew causing ‘causal violations’
- cascading timeout misconfiguration, and so many more
Now, to debug issues effectively, what I feel is that we need to hold and work with two mental models at once, and constantly context-switch between two very different perspectives - the global view and the local view.
The global view shows you the overall architecture - how services talk to each other, what the data flow looks like, and what guarantees the system is supposed to provide. This is the map. It tells you where everything should be.
The local view is where you actually live during an incident. You are looking at a single node’s state, its message queue, its retry logic, and its clock. You are asking: what does this component think is true right now?
What makes this interesting is that these two views do not always agree.
A service can believe it is healthy while the global system is only semi-functional. A network partition can make two nodes each think they are the leader. A queue consumer can be processing messages in a perfectly valid local order that violates the global processing guarantees you care about.
This is why distributed systems are hard to reason about. The bugs almost never live in a single component. They live in the gap between components :)
So, if you work with distributed systems, develop the habit of deliberately flipping between these views. When something breaks, first ask “what does the global state look like?” and then immediately follow with “what does each node think the global state looks like?”
The mismatch between those two answers is almost always where the problem hides.