Changing something in staging can take down production ⚡
In an overly complicated microservices-laden infrastructure, it is not uncommon to take down production by changing something in staging. A few reasons from the top of my head are
- a shared config store that shares underlying storage
- an accidental dependency on the staging version of a service,
- serverless functions getting throttled at an account level
- or just some human error
With so many moving parts, it becomes really difficult to track down minute transitive dependencies. That’s how difficult distributed systems and microservices are.
Been there, seen that, done that.
⚡ I keep writing and sharing these engineering nuggets, so if you are keen on learning them, follow along. I have an entire playlist on Designing Microservices.
youtube.com/c/ArpitBhayani