Fun fact: Pinterest spent three months chasing a network bug that turned out to be caused by a container they were not even using.
Their training jobs, running on k8s, kept crashing with intermittent network loss. The system logs pointed to AWS network driver resets - a self-healing mechanism that kicks in when a network thread does not get CPU time for 5 seconds straight.
They tried the usual fixes - huge pages, jemalloc, CPU affinity, etc, but nothing worked. The fix came from profiling. Kubelet was spiking to 6.5% CPU (versus its usual 1%) right before every crash.
The kernel was tracking about 70,000 memory cgroups. Only 240 were actually in use. The root cause was a leftover default.
Their GPU machines used the AWS Deep Learning AMI, which auto-starts an Amazon ECS agent systemd unit. They were running Kubernetes, not ECS, so the agent had no cluster to join and kept crashing and restarting, silently leaking zombie memcgs for days.
So, the fix was just one line - disable the ECS agent systemd unit in the base image.
One key learning from this - base OS images carry more baggage than you would expect. It is worth auditing what systemd units actually run on your machines.