We tend to make decisions based on data that is easy to collect, not data that is most representative. This is availability bias, and it has a massive influence on how our systems are built.
A classic example: you profile your application and see a slow database query in the logs. It is visible, it is measurable, it is right there. So you optimize it. But the real bottleneck might be network latency or an upstream service timeout that is harder to instrument and therefore never gets investigated.
The data you do not have does not show up in your dashboards. So you end up optimizing for what you can see, not what actually matters.
This shows up in on-call decisions too. Incidents that trigger noisy alerts get attention. Silent degradations - the ones slowly eroding p99 latency or quietly increasing error rates below the alert threshold - go unnoticed until they become a larger problem.
The fix is not to collect more data, but rather to ask this before making a decision: “Does this data represent the actual problem, or is it just the data that was easy to get?”
The most crucial metric might be the one you are not measuring.