526 views • Outage Dissections
Getting the service up is P0 during the outage, but that is not all. There are a few other things that we need to take care of once the issue is mitigated.
During the outage, the business logic or database must have crashed while handling in-flight requests; hence, there is a very high chance of the data becoming inconsistent.
For example: if the process crashed while transferring money from one account to another; it is possible that money got deducted from one account but did not credit to another.
Hence, once the outage is mitigated, the first thing to be done is to ensure that the data does not remain in an inconsistent state. Achieving this depends on the use case at hand and the tech stack involved.
Another place that needs attention is cache invalidation. If the outage sustains for a long time, there is a chance that some entries in the cache are not valid anymore.
Hence, depending on the use case, it is advised to go through the cache entries and delete the invalid once, ensuring the end-user sees a globally consistent view of the data.
Once the outage is mitigated and the inconsistencies are resolved, it is better if we take a look into existing alerting and monitoring practices so that we ensure that we get alerted at the right time.
In most cases, you would find that a few things could always be improved when it comes to monitoring the right things, and hence an outage is an opportunity to get alerting and monitoring improvements prioritized.
Every outage has a root cause, and it is super-important to fix the root cause to ensure that the over never happens again for the same reason. This requires us to go in-depth and take preventive measures to ensure complete closure.
For example, re-auditing all the INT32 columns and moving them to INT64 to ensure that we never repeat the outage due to integer overflow.
If you like what you read subscribe you can always subscribe to my newsletter and get the post delivered straight to your inbox. I write essays on various engineering topics and share it through my weekly newsletter.
466 views • 31 likes • 2022-07-06
Outages are inevitable; but we should design our architecture such that if a component is down, it should not lead to a ...
1101 views • 60 likes • 2022-07-01
Distributed Systems are prone to problems that seem very obscure. GitHub had an outage because a set of nodes in the Zoo...
1231 views • 83 likes • 2022-06-29
So, how are databases managed in production? When the master goes down, how a replica is chosen and promoted to be the n...
A set of courses designed to make you a better engineer and excel at your career; no-fluff, pure engineering.
Being a passionate engineer, I love to talk about a wide range of topics, but these are my personal favourites.
Arpit's Newsletter read by 17000+ engineers
🔥 Thrice a week, in your inbox, an essay about system design, distributed systems, microservices, programming languages internals, or a deep dive on some super-clever algorithm, or just a few tips on building highly scalable distributed systems.
Powered by this tech stack.