How to design great systems? The answer is simple: design for failure. Great systems are not just built for happy paths, but they are designed to handle errors gracefully. I know this sounds like AI slop, but it is not …, coming back…
Assume and acknowledge that every component, from hardware to software, has the potential to fail. So, proactively consider all possible failure points for every single component, and see how your system reacts.
In any case, the system should not simply collapse; instead, it should be designed to limit and contain the damage while maintaining core functionality, and ideally, recover quickly with minimal disruption.
This is the framework that I have been applying while designing any and every system across my career. For me, it is all about failures :) I seem a pessimist that way, but for me it is not pessimism, but realism.