How to handle database outages?

Watch the video explanation ➔

Why a database goes down?

An unexpected heavy load on your database can lead to a process crash or a massive slowdown.

Before jumping to the potential short-term and long-term solutions, ensure you monitor the database well. CPU, Memory, Disk, and Connections are being closely monitored.

Short term solutions

  • Kill the queries that have been running for a long time
  • Quickly scale up your database if you have been seeing a consistent heavy usage
  • Check if the recent deployment is the culprit; if so, revert asap
  • Reboot the database will calm the storm and buy you some time

Long term solutions

  • Ensure the right set of indexes is in place
  • Tune your database default parameters to gain optimal performance
  • Check for the notorious N+1 Queries
  • Upgrade the database version to get the best that DB can offer
  • Evaluate the need for Horizontal scaling using Replicas and Sharding

Here's the video ⤵

Courses I teach

Alongside my daily work, I also teach some highly practical courses, with a no-fluff no-nonsense approach, that are designed to spark engineering curiosity and help you ace your career.


System Design Masterclass

A no-fluff masterclass that helps experienced engineers form the right intuition to design and implement highly scalable, fault-tolerant, extensible, and available systems.


Details →

System Design for Beginners

An in-depth and self-paced course for absolute beginners to become great at designing and implementing scalable, available, and extensible systems.


Details →

Redis Internals

A self-paced and hands-on course covering Redis internals - data structures, algorithms, and some core features by re-implementing them in Go.


Details →


Writings and Learnings

Blogs

Bookshelf

Papershelf


Arpit's Newsletter read by 80000+ engineers

Weekly essays on real-world system design, distributed systems, or a deep dive into some super-clever algorithm.