Imagine you want to monitor whether your MySQL master is down so that you can do a failover and promote a replica.
If you just rely on a prober that pings the master, it’s prone to failures - for example, network glitches between your prober and the master, which can lead to false positives. Also, you end up running multiple checks spaced out over time intervals, which reduces false alarms but slows down your response time.
So, a better approach is much simpler…
Instead of just checking if you (the prober) can reach the master, also monitor whether the replicas can see the master. Declare the master dead if both of the following conditions are met:
- the prober cannot reach the master
- the master’s replicas also cannot reach the master
When your prober sees that none of the replicas can reach the master (by observing the replication offsets), it can deduce that the replication topology is effectively broken anyway, and hence a failover is justified.
This is more reliable because you get consensus from distributed nodes that are actually part of the replication setup, making it far less prone to network glitches.
Simple, neat, and reliable.