![\"Techstrong](\"https:\/\/www.newsbeep.com\/ca\/wp-content\/uploads\/2026\/03\/Techstrong-Gang-Youtube-PodcastV2-770.png\")
<\/a><\/p>\nWhen Netflix upgraded its container runtime, from Docker to the open source containerd<\/a>, it noticed some nodes were stalling out when operations scaled up.<\/p>\n This was a major problem. When a Netflix user decides what to play on Netflix, it triggers hundreds of containers. So it was quite important to ruthlessly tack down even the tiniest performance bottleneck.<\/p>\n The culprit turned out to be the way Netflix initialized containers. But the bug only showed itself in certain cases. With a bit of sleuthing, Netflix engineers found the delay was worse on some CPU architectures more than others, according to a blog post.\u00a0<\/p>\n \u201cUnderstanding and optimizing both the software stack and the hardware it runs on is key to delivering seamless user experiences at Netflix scale,\u201d wrote Netflix Senior Software Engineer Andrew Halaney<\/a> and Netflix Senior Performance Engineer Harshad Sane<\/a>, in an engineering blog post,\u00a0 \u201cMount Mayhem at Netflix: Scaling Containers on Modern CPUs.<\/a>\u201c<\/p>\n Long Boot Times<\/p>\n Like most Kubernetes operations, a Netflix application will scale until it maxes out a node, then it procures a new instance and continues scaling on that one.<\/p>\n After about 100 containers, however, Netflix was finding that their servers were starting to slow<\/a>.\u00a0 A health check that read the mount table (a list of everything mounted on a server) would take 30 seconds or longer to complete. This was problematic in that Linux has the CPU dedicate itself to checking if the health check was complete, delaying everything else in processing queue.<\/p>\n Too Many UIDs<\/p>\n Under the older Docker setup, containers typically shared the host\u2019s user ID (UID) pool. When Netflix migrated to containerd\u2014the runtime responsible for executing the Kubelet\u2019s management tasks\u2014it shifted to a more secure User Namespace model. In this setup, each container is isolated with its own unique UID range.<\/p>\n However, this approach required the kernel to individually \u2018idmap\u2019 every layer of a container image to that specific range. For a multi-layered image, this meant a massive spike in kernel calls just to instantiate a single container.<\/p>\n Containerd, when assembling a container\u2019s root filesystem, is very needy with its requests for kernel-level locks, dominates the CPU\u2019s time \u2014 especially for containers with more than 50 layers.<\/p>\n \u201cIf a node is starting many containers at once, every CPU ends up busy trying to execute these mounts,\u201d the pair of Netflix engineers wrote. \u201cAny system trying to quickly set up many containers is prone to this, and this is a function of the number of layers in the container image.\u201d<\/p>\n Spinning up 100 containers, for instance, would require 20,200 mounts, each requiring system calls to the kernel!<\/p>\n Multi-Core Engineering<\/p>\n Netflix is not alone in suffering from this issue.\u00a0 Meta engineers, working with thousands of containers for AI inferencing, have bemoaned the performance drops<\/a> that came along with managing containers with thousands of UIDs.<\/p>\n In fact, this has been a common system design issue for multi-core processing. Too often, data structures become the bottleneck, argued one widely-cited paper<\/a> for the 2008 USENIX Operating System and Design conference. Those researchers also found<\/a> that many commonly-used applications \u2014 including Exim, memcached, Apache, PostgreSQL, and MapReduce \u2014 also suffer from kernel-level bottlenecks in multi-core servers.<\/p>\n Linux kernel developer Christian Brauner, creator of the next-generation Virtual File System mounting<\/a>, has\u00a0long argued<\/a> that the mount() system call is broken for modern containers, suggesting that file-descriptor-based mounting could be used instead.<\/p>\n The CPU Bottleneck<\/p>\n But it gets even weirder when the engineering team looked at the difference between the hardware running these containers!<\/p>\n Predominately, these timeouts were happening with the Intel Xeon-based AWS r5.metal<\/a> instances (with the Intel Skylake\/Cascade Lake architecture with 96 virtual CPUs). They happened far less frequently on either the later 7th generation Intel m7i.metal-24xl<\/a> or the AMD EPYC -based m7a.24xlarge<\/a>, which were also used on the Netflix Kubernetes deployments.<\/p>\n The engineering team developed a microbenchmark<\/a> to compare the lock contention across different multi-core systems. They found that the older mesh architecture used in r5.metal chips was the bottleneck. The design struggled to synchronize the global mount lock across multiple cores, leading to massive cache-line contention.<\/p>\n The other AWS instances use a distributed architecture, where multiple cores each have their own local last-level cache. Lock contention is more rare in these designs.<\/p>\n \u201cCentralized cache management amplified cache contention while distributed cache design smoothly scaled under load,\u201d the engineers concluded.<\/p>\n The Fix Goes Upstream<\/p>\n The Netflix engineering team tackled the issue at the software level, namely by reducing the number of kernel system calls the containerd was making.<\/p>\n They created a pull request<\/a> that changed how containerd did global lock usage. This was made possible by Linux kernel 6.3, released in April 2023, which introduced support for recursive binds in\u00a0mount\u2019s rbind option.<\/p>\n Instead of requiring global locks to mount each layer, containerd simply performed one single recursive bind mount<\/a> (idmap) of the entire parent directory where all those layers reside.<\/p>\n \u201cThis makes the number of mount operations go from O(n) to O(1) per container, where n is the number of layers in the image,\u201d the engineers wrote.<\/p>\n