Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Containers: Namespaces, Cgroups, and the Union Filesystem

Overview

A container looks like one technology (“Docker”) and is actually a composition of four independent kernel primitives: namespaces, which restrict what a process can see; cgroups, which restrict what it can consume; a union filesystem, which assembles an image from content-addressed layers; and an OCI runtime, which turns a declarative spec into the clone, setns, and cgroup-file-write calls that wire it all together. Orchestrators such as Kubernetes sit on top and treat this bundle as a schedulable unit, translating resources.requests and limits into cgroup controller parameters. The engineering power of containers comes precisely from the orthogonality of the primitives — you can share a network namespace but isolate memory, or isolate a filesystem view without limiting CPU — and every production incident around containers is a story about one axis being isolated while the team assumed another was.

This page maps the joints: how namespaces and cgroups divide the isolation problem between view and resources, how image layers and the union filesystem make that composition cheap and distributable, and how the runtime and orchestrator compose the primitives without losing their semantics. The failure analysis is where the connection earns its keep, because two failure modes — the cgroup OOM kill and cgroup CPU throttling — happen inside kernel accounting structures the application cannot see, and they masquerade as application bugs: random SIGKILLs, unexplained tail-latency spikes, GC pauses that appear only in production. Understanding where each primitive ends and the next begins is the difference between tuning a container and exorcising it.

The Subsystems, Independently

Namespaces: isolation of the view. A namespace wraps a global kernel resource so that processes inside it see a private instance. The main kinds: pid (private process tree — PID 1 inside, arbitrary outside), net (private interfaces, routes, iptables, sockets), mnt (private mount table — the basis of “this container has its own root filesystem”), uts (hostname), ipc (System V/POSIX IPC objects), user (uid/gid remapping — root inside maps to unprivileged outside), cgroup (a virtualized view of /proc/self/cgroup), and time (since Linux 5.6). Namespaces change visibility only: they cost no memory accounting, no CPU limits, nothing except the wrapped resource. A process in six namespaces is still an ordinary task scheduled by the same CFS/EEVDF scheduler as everything else on the box. See ../../os/containers/namespaces.md and ../../os/kernel-advanced/namespaces-cgroups.md.

Cgroups: isolation of resources. A cgroup is a node in a hierarchy to which resource controllers apply limits, weights, and accounting: memory.max (hard cap — exceed and the kernel OOM-kills inside the cgroup), memory.high (soft throttle — allocation slows as the kernel reclaims aggressively), cpu.max (quota/period — a hard CPU ceiling in slices), cpu.weight (relative share when competing), io.max, pids.max. Cgroup v2 unified the once-separate hierarchies so one process sits in one cgroup for all controllers. Crucially, memory cgroups charge page cache to the container that touched it — a container that merely reads big files occupies memory the kernel counts against it, reclaiming is per-cgroup, and eviction pressure from one workload no longer poisons another’s cache. See ../../os/containers/cgroups.md.

Union filesystems and image layers. An image is a stack of read-only layers plus a writable top layer, exposed as one merged view by overlayfs. Each layer is a tarball of filesystem deltas, content-addressed by digest; identical layers are shared between images and between all containers using them, and a container’s first write to a file from a lower layer triggers a copy-up into the writable layer. This is the Git model applied to root filesystems — hash the content, share the objects, pay copy-on-write on mutation — exactly the principle of ./git-cas.md, which is why image pulls dedupe, registries can serve layer ranges, and disk usage stays sublinear across hundreds of containers on one host. See ../../os/containers/docker.md.

OCI runtime and the orchestrator. The Open Container Initiative defines a config format and a runtime contract; runc is the reference implementation that performs the actual composition: clone() with the requested namespace flags, write the requested cgroup files, pivot_root into the overlayfs mount, drop capabilities per the spec. A container engine (containerd, CRI-O) supervises runtimes; a kubelet drives the engine through the CRI. Kubernetes adds the composition semantics that individual primitives lack: a Pod shares one net namespace among its containers (so localhost is a shared bus between app and sidecar) while keeping mnt/pid private; limits map to memory.max/cpu.max, requests to scheduler weights; and the node’s kubelet enforces eviction orderings before the kernel’s OOM killer has to be consulted at all.

The Shared Principle

Isolation is two orthogonal axes — view (namespaces) and resources (cgroups) — and every “container” property is a composition decision about which axis to share at which boundary. The OCI runtime is nothing but a translator from declarative intent to (namespace flags, cgroup values, mount table, capabilities) tuples; a Pod is the statement “share the view axis across these processes, isolate the resource axis per container”; a sandboxed sidecar is the statement “share the network view but give the sidecar its own CPU weight.” Once stated this way, the design space is legible: VMs isolate both axes at the hardware level (../../os/advanced/virtualization.md), processes share both axes, and containers are the continuum between — see ../misconceptions/containers-are-vms.md for the full audit of where the VM analogy dies.

A second shared idea: cgroups are a reclamation system that must decide “who is dead” from incomplete information — the same epistemic problem as garbage collection (./gc-memory-management.md). The kernel cannot know that your 900 MiB Java heap is supposed to be 900 MiB; it sees a cgroup at memory.max and picks a victim by oom_score. memory.high, PSI pressure signals, and kubelet eviction are the early-warning approximations, and their tuning is exactly GC-tuning logic: trade detection latency against false kills.

Where the Coupling HOLDS and BREAKS

JointCoupling that HOLDSWhere it BREAKS
Namespaces ↔ cgroupsOrthogonal: any view can pair with any resource grant; the OCI spec composes them cleanlyThey interact invisibly: a pids.max or net namespace limit manifests as an application-level error (thread creation failure, port bind failure) that looks nothing like “resource limit”
Image layers ↔ runtimeLayers are immutable and content-addressed, so the same image produces identical filesystem state anywhereCopy-up latency and metadata-heavy layers (thousands of small files) make cold-start I/O-bound; two containers sharing a layer share page cache but NOT memory limits — accounting is per-cgroup
Orchestrator ↔ kernelrequests/limits map cleanly to cgroup parameters; the scheduler can only promise what cgroups can enforceLimits that are too clever bite back: a cpu.max quota causes throttling even when the node is idle, and a memory.limit 1 MiB above steady state converts into a periodic OOM kill loop
Kernel OOM ↔ applicationOOM kill is scoped to the cgroup; the blast radius is one container, not the nodeThe victim is chosen by kernel heuristics (oom_score), so the process killed is often not the leaking one in multi-process containers (JVM + sidecar + shell); the app experiences it as a mystery SIGKILL with no stack trace

The deepest break: the kernel enforces, but the kernel does not explain. When memory.max fires, the evidence is a kernel log line and a dead PID — the application’s own telemetry may show nothing (it died before flushing), and the orchestrator shows “OOMKilled, exit 137.” Cross-boundary forensics requires correlating cgroup event files, PSI, and GC logs — three observability systems owned by three different layers of the stack (see ./observability-pipeline.md for why telemetry crossing subsystem boundaries is where pipelines get expensive).

Failure Propagation: OOM Kills and Throttling Versus SLOs

Cgroup OOM versus application OOM. When a container crosses memory.max, the kernel invokes the OOM killer within that cgroup and SIGKILLs the highest-scoring task. This differs from application-level memory exhaustion in three propagating ways. First, no exception is thrown: malloc gets replaced by a signal, so try/catch, heap dumps, and error middleware are all bypassed — failure lands on the restart counter, not the error counter, and dashboards show it as flapping pods rather than a memory bug. Second, the victim is a heuristic choice: in a pod with a JVM (large heap), a Go sidecar, and a shell script, the kernel’s score may pick the JVM even though the sidecar leaked — the leak survives the kill. Third, page cache is charged too: a container streaming large files can push itself over memory.max with cache it doesn’t need, killing the heap-owning process. The mitigation stack moves detection earlier: memory.high to slow allocations before the cliff, PSI-based userspace OOM daemons (systemd-oomd) to kill by pressure trend rather than hard limit, and JVM MaxRAMPercentage to keep native+heap under the limit. See ../../os/modern/oom-killer.md.

CPU throttling versus latency SLOs. cpu.max is a hard ceiling in time slices: 200000 100000 means “at most 200 ms of CPU per 100 ms period” — 2 CPUs, checked per period. A bursty single-threaded request path that exceeds its per-period share is frozen until the next period: up to 100 ms of added tail latency per throttled slice, even on a node that is 90% idle. This is the mechanism behind the classic production mystery — “p99 is terrible in Kubernetes but fine on the VM, and CPU utilization is low”: utilization aggregates over the whole period, throttling bites within it. The design lever is to prefer cpu.weight (fair sharing under contention only, free when idle) over hard quotas for latency-sensitive work, to set quota ≥ observed p99 burst demand rather than average demand, and to know that throttling is per-cgroup: one greedy container cannot steal another’s weight, but one container with a miscalculated quota punishes only itself.

A Concrete Worked Example

The throttled tail, with numbers. A service has cpu.max = 150000 100000 (1.5 CPUs) and handles requests needing 10 ms of CPU. Sustained capacity: 1.5 CPU → 150 request-CPU-ms per 100 ms period → ~15 req/s of single-threaded work before each period is exhausted. Traffic reaches 25 req/s: within the first ~6 ms of each period the quota is spent; the remaining ~94 ms of the period every in-flight request is frozen — p99 latency gains a ~94 ms plateau that no application profiling will ever find, because the pause happens between instructions, in the scheduler. Utilization dashboards show 100% of quota (1.5 CPUs of a 16-CPU node = 9% node utilization). Fix options: raise quota to 3 CPUs (utilization still under 20%, throttle gone), or drop the quota and set cpu.weight so the service shares fairly only under node contention. The arithmetic exposes the invariant: a hard quota buys isolation by buying latency cliffs; a weight buys fairness with no cost when idle.

The OOM loop, with numbers. JVM pod with memory.limit = 2 GiB, heap -Xmx1800m (1.76 GiB), metaspace+threads+native+code cache ~250 MiB, page cache from log streaming ~150 MiB. Steady state sits at ~1.95 GiB of a 2 GiB budget. A 10% traffic increase pushes heap into a GC-heavy phase: live-set grows 100 MiB, total crosses memory.max, the kernel kills the JVM (highest oom_score), the pod restarts, replays its work queue — and the cycle repeats every ~25 minutes. The connection-map fix: notice that three uncoordinated subsystems (JVM heap sizing, native overhead, page-cache accounting) each assumed they owned the memory, and the cgroup is the only place where their sum becomes visible. Budget at the cgroup boundary: heap 60% of limit, MaxRAMPercentage=60, monitor cgroup memory.events (oom kill counter) and PSI, alert on memory.high crossings — not on OOM kills, which are the late signal.

When This Connection Is Useful

  • Problem-solving heuristics. Any “works on the VM, misbehaves in the container” report decomposes along the four primitives: which view differs (namespaces: DNS, localhost, mounts), which limit differs (cgroups: quota, memory), which filesystem behavior differs (overlayfs: copy-up, inode counts, fcntl locking on some layered filesystems), which composition differs (Pod-shared netns, sidecars). Test each axis in isolation before blaming the application.
  • Interview answers. “How do containers work?” has a junior answer (Docker) and a senior answer: namespaces give the view, cgroups give the budget, overlayfs gives cheap reproducible images, runc composes them, and the orchestrator adds scheduling and liveness semantics on top — each with its own failure propagation. Follow-ups about “why is my pod OOMKilled when RSS is below the limit” (page cache charging) and “why is p99 bad at low CPU” (throttling periods) separate mechanism knowledge from vocabulary.
  • Engineering decisions. Choose weights over hard CPU quotas for latency-sensitive services; set memory.high at ~90% of memory.max as an early-warning throttle; budget heap/native/page-cache at the cgroup boundary, not per-subsystem; keep images small in layers (metadata-heavy layers dominate cold-start), not just in bytes; alert on cgroup pressure and memory.events, not only on restarts — see ../../os/containers/kubernetes.md for the orchestrator-side enforcement chain.

Interview Questions

  1. Explain the division of labor between namespaces and cgroups. Give one production failure that each primitive can cause independently, and one that only their composition can cause.
  2. A pod with a single-threaded request path and cpu.max set to 1.5 CPUs shows p99 latency ~100 ms worse than the same binary on an uncontained VM, while node CPU is 90% idle. Walk through the mechanism and give two remedies with their trade-offs.
  3. Why can a container be OOM-killed even though its application heap usage is well under the memory limit? Name the accounting components involved and the early-warning signals you would alert on instead.
  4. Compare a container and a virtual machine along the two isolation axes. Which properties of a VM can a container never replicate, and which VM properties do modern containers falsely appear to have?
  5. Kubernetes schedules a Pod whose containers share one network namespace but have separate memory limits. What coupling does this create between the containers, and how would you size the limits knowing that coupling?

Key Takeaways

  • Containers are a composition of orthogonal primitives — namespaces (view), cgroups (resources), union filesystems (images), OCI runtime (composition) — and every container property is a choice about which axis to share at which boundary.
  • Cgroup memory accounting includes page cache; the OOM killer chooses victims by heuristic within the cgroup, so a cgroup OOM arrives as a mystery SIGKILL, bypasses all application error handling, and often kills the wrong process.
  • Hard CPU quotas are time-sliced: throttling freezes tasks until the next period (up to 100 ms default), producing tail-latency plateaus invisible to utilization dashboards; prefer cpu.weight for latency-sensitive work and reserve quotas for batch.
  • Image layers are content-addressed copy-on-write — the Git model applied to root filesystems — which buys dedupe and reproducibility at the cost of copy-up latency and layer-metadata cold-start costs.
  • Move failure detection up the ladder: memory.high and PSI pressure give early, tunable signals; the hard-limit OOM kill is the late, heuristic, noisy one — alert on pressure trends, not just restarts.

Cross-References