Most systems detect node or master failures using simple polling, and while this approach sounds straightforward, it has an interesting reliability issue…
The typical approach is to observe a node directly. This usually means pinging it, checking if a port is open, or running a lightweight query to confirm it is alive.
On paper, this seems fine, but all of these methods share the same weakness - what if the observer itself is wrong?
In a distributed setup, network glitches are normal. Temporary packet loss, routing hiccups, or partial network partitions can easily make a healthy node appear unreachable to the observer. The usual way to deal with this is to retry multiple times and declare failure after the n-th consecutive failure.
This creates a classic tradeoff.
If n is small (or polling happens frequently), failure detection becomes fast, but false positives increase. A short-lived network blip can trigger an unnecessary failover, which can sometimes be more disruptive than the original issue.
If n is large (or polling intervals are longer), false positives decrease, but real failures take longer to detect. That delay directly increases downtime.
But there is a more reliable way to think about this problem when you already have a cluster of nodes available.
Instead of relying on a single observer repeatedly polling a target node, you can allow multiple nodes in the cluster to independently perform health checks. The system then treats a node as failed only when a majority of observers agree that the node is unreachable.
This consensus-based approach reduces the risk of false positives caused by network partitioning. Even if one observer loses connectivity, the rest of the cluster can still provide an accurate view of system health.
Consensus is costly, so this approach is not the most cost-efficient. However, it can be very useful if your system is large enough and distributed across multiple geographies.