Distributed systems are not just defined by what we put into them. They are equally defined by what we leave out. Let me give you 4 examples…
Take timeouts - A service with no timeout on an outbound HTTP call will wait indefinitely. Under load, all its worker threads pile up waiting, and the service effectively becomes unavailable. No one ‘decided’ to make it unavailable. They just forgot to decide how long to wait.
Or retries - If your service does not retry a failed downstream call, the data gap it creates can look like a bug in a completely unrelated service days later. What was omitted from the retry strategy is now an incident.
Take acknowledgements - When you fire a message to a queue and do not wait for an ack, you just chose to tolerate message loss. That choice is not written anywhere in your architecture diagram. It lives silently between the producer and the broker.
The same logic applies to back-pressure. If you do not model what happens when a consumer is too slow, the producer keeps going, memory climbs, and the system falls over. The crash was not caused by what was built. It was caused by what was not built.
Hence, while reviewing a distributed system design, we should ask:
- what happens when this message is lost,
- what happens when this call never returns,
- what happens if this node never comes back?
The answers you do not have are where the failures live.