Cloudflare had another outage, lasting 25 mins, on 5th Dec,

Cloudflare had another outage, lasting 25 mins, on 5th Dec, and this happened while they were patching their servers for a React vulnerability. Here’s what exactly happened…

• They were increasing how much of a request body their firewall could inspect from 128KB to 1MB.

• While rolling this out, an internal testing tool started breaking. Since it wasn’t needed for live traffic, they decided to just turn it off for now.

• Here’s where things went wrong: they turned off the tool using a global config system that pushes changes instantly to the entire network, with no gradual rollout.

• This triggered, or rather uncovered, a bug.

• Cloudflare’s rulesets system has actions like “block”, “log”, “skip”, and “execute”. The “execute” action triggers the evaluation of another ruleset. Their internal testing system uses this to run test rules before public release.

• They never applied a killswitch to an “execute” rule before. This assumed the rule_result.execute object existed, but it didn’t, so Lua threw an error when trying to access a nil value.

• This caused HTTP 500 errors for customers on Cloudflare’s older FL1 proxy… and took down 28% of their traffic.

The RCA is linked below. Do give it a read.