Netflix reduced connection churn in its API gateway by 10x.
Connection churn is a massive problem with proxies like API gateways, and it effectively means the consistent opening and closing of TCP connections. Under heavy traffic, this leads to increased CPU utilization and latency.
To reduce connection churn, Netflix used two things
- H2 multiplexing - allowing multiple requests over a single TCP connection.
- Subsetting - spread traffic across shards, each with its own connection pool.
This not only reduced the number of TCP connections needed to open and close but also improved scalability i.e., with fewer connections to manage, Zuul (their API Gateway) was able to handle more traffic without sacrificing performance.
They have written an amazing blog going deeper into the actual optimization and algorithm. Definitely one of the most fascinating articles I’ve read recently - link in the comments.