One of the most underrated qualities in system design is predictability, and it often matters more than raw performance.
Most systems have two stable modes: one where everything ticks along nicely, and one where the system has collapsed under load. The transition between these states is often sudden and painful.
The real challenge is not just handling load, but rejecting work fast enough to avoid entering the collapsed state in the first place. By the time you notice the warning signs - queue lengths spiking, latencies climbing - the expensive work has already been admitted, and it is too late.
A predictable system is one that does not degrade into weird timeouts, long tail latencies, or resource contention. When it gets too busy, it fails fast and fails cleanly by rejecting work early through mechanisms like admission control, backpressure, or rate limiting.
A prerequisite for building a predictable system is knowing one key number ahead of time - how much work every single operation will trigger.
Hence, build systems around well-defined, bounded units of work, with each request doing a clear and limited amount of work. No surprises. No unbounded fan-out. No hidden costs that only appear under pressure.
It takes time to appreciate this, but predictability may be the most valuable property a system can have. I learned this the hard way :) However, I now almost always prioritize predictability over everything.
More importantly, system behavior remains consistent, and everything that happens is far easier to reason about.