So, the outage is mitigated, now what?

526 views Outage Dissections

Getting the service up is P0 during the outage, but that is not all. There are a few other things that we need to take care of once the issue is mitigated.

Resolve Inconsistencies

During the outage, the business logic or database must have crashed while handling in-flight requests; hence, there is a very high chance of the data becoming inconsistent.

For example: if the process crashed while transferring money from one account to another; it is possible that money got deducted from one account but did not credit to another.

Hence, once the outage is mitigated, the first thing to be done is to ensure that the data does not remain in an inconsistent state. Achieving this depends on the use case at hand and the tech stack involved.

Invalidate the Cache

Another place that needs attention is cache invalidation. If the outage sustains for a long time, there is a chance that some entries in the cache are not valid anymore.

Hence, depending on the use case, it is advised to go through the cache entries and delete the invalid once, ensuring the end-user sees a globally consistent view of the data.

Audit alerting and monitoring

Once the outage is mitigated and the inconsistencies are resolved, it is better if we take a look into existing alerting and monitoring practices so that we ensure that we get alerted at the right time.

In most cases, you would find that a few things could always be improved when it comes to monitoring the right things, and hence an outage is an opportunity to get alerting and monitoring improvements prioritized.

Take preventive measures

Every outage has a root cause, and it is super-important to fix the root cause to ensure that the over never happens again for the same reason. This requires us to go in-depth and take preventive measures to ensure complete closure.

For example, re-auditing all the INT32 columns and moving them to INT64 to ensure that we never repeat the outage due to integer overflow.

Arpit Bhayani

Arpit's Newsletter

CS newsletter for the curious engineers

❤️ by 17000+ readers

If you like what you read subscribe you can always subscribe to my newsletter and get the post delivered straight to your inbox. I write essays on various engineering topics and share it through my weekly newsletter.

Other essays that you might like

So, the outage is mitigated, now what?

526 views 25 likes 2022-07-08

Outages happen and in such a tense situation, the main priority is to get the system back up, but is that it? Is everyth...

Control an outage by localizing the failures

466 views 31 likes 2022-07-06

Outages are inevitable; but we should design our architecture such that if a component is down, it should not lead to a ...

Dissecting GitHub Outage - Multiple Leaders in Zookeeper Cluster

1101 views 60 likes 2022-07-01

Distributed Systems are prone to problems that seem very obscure. GitHub had an outage because a set of nodes in the Zoo...

GitHub Outage - How databases are managed in production

1231 views 83 likes 2022-06-29

So, how are databases managed in production? When the master goes down, how a replica is chosen and promoted to be the n...

Be a better engineer

A set of courses designed to make you a better engineer and excel at your career; no-fluff, pure engineering.

System Design Masterclass

A masterclass that helps you become great at designing scalable, fault-tolerant, and highly available systems.

800+ learners

Details →

Designing Microservices

A free playlist to help you understand Microservices and their high-level patterns in depth.

17+ learners

Details →

GitHub Outage Dissections

A free playlist to help you learn core engineering from outages that happened at GitHub.

67+ learners

Details →

Hash Table Internals

A free playlist to help you understand the internal workings and construction of Hash Tables.

25+ learners

Details →

BitTorrent Internals

A free playlist to help you understand the algorithms and strategies that power P2P networks and BitTorrent.

42+ learners

Details →

Topics I talk about

Being a passionate engineer, I love to talk about a wide range of topics, but these are my personal favourites.

Arpit's Newsletter read by 17000+ engineers

🔥 Thrice a week, in your inbox, an essay about system design, distributed systems, microservices, programming languages internals, or a deep dive on some super-clever algorithm, or just a few tips on building highly scalable distributed systems.

  • v12.7.8
  • © Arpit Bhayani, 2022

Powered by this tech stack.