Understanding Cgroups: The Foundation of Container Resource Management

It wasn’t until I hit specific issues on my Raspberry Pi 5 cluster during an OS upgrade, that I truly understood why cgroups are the unsung hero of modern cloud-native computing. If you’re running containers, or just curious about Linux resource management, here’s what I’ve learned about this technology.

What Are Cgroups, Really?

Control Groups (cgroups) are a Linux kernel feature that acts like a resource manager for your processes. Think of them as the facilities manager of a large apartment building – they decide who gets how much water pressure, electricity, and heating. In computing terms, cgroups control how much CPU, memory, disk I/O, and network bandwidth each process (or group of processes) can use.

The beauty of cgroups is their hierarchical nature. Just like folders on your filesystem, you can create groups within groups, each with their own resource limits. A parent group’s limits apply to all its children, making it perfect for managing complex applications with multiple components.

My Raspberry Pi Wake-Up Call

My first real encounter with cgroups configuration came when setting up Kubernetes on a Raspberry Pi 5. Unlike x86 systems where cgroups “just work,” the Pi (ARM) requires explicit enablement. After struggling with pods failing to enforce resource limits, I discovered the issue: cgroups v2 wasn’t properly enabled on the Pi’s kernel.

The fix was adding this to /boot/cmdline.txt:

cgroup_enable=cpuset cgroup_enable=memory cgroup_memory=1

Cgroups v1 vs v2: The Unified Hierarchy

One source of confusion in my learning was the two versions of cgroups. Here’s what I’ve figured out:

Cgroups v1 uses separate hierarchies for different resources:

• /sys/fs/cgroup/memory/ for memory limits
• /sys/fs/cgroup/cpu/ for CPU limits
• Multiple controllers, each with its own hierarchy

Cgroups v2 unifies everything into a single hierarchy:

• One location: /sys/fs/cgroup/
• All controllers managed together
• Better for containerized workloads

Ubuntu 24.04 and newer default to cgroups v2, which is what Kubernetes prefers. You can check your system with:

stat -fc %T /sys/fs/cgroup/
# "cgroup2fs" means v2, "tmpfs" means v1

How Kubernetes Leverages Cgroups

Every time you set resource requests and limits in a Kubernetes pod spec, you’re actually configuring cgroups:

resources:
  requests:
    memory: "64Mi"
    cpu: "250m"
  limits:
    memory: "128Mi"
    cpu: "500m"

Here’s what happens behind the scenes:

1. kubelet receives the pod specification
2. The container runtime (like containerd) creates a new cgroup for the pod
3. Each container gets its own sub-cgroup with the specified limits
4. The kernel enforces these limits at the system level

This is why a container that tries to use more memory than its limit gets OOMKilled – the kernel’s cgroup controller steps in and says “that’s enough!”

Practical Resource Management Patterns

Through trial and error, I’ve developed some patterns for resource management:

Memory: Hard Limits, Soft Landings

Memory limits are enforced immediately – exceed them and your process dies. I’ve learned to:

• Set requests at the actual usage level
• Keep limits 20-30% higher than typical peak usage
• Monitor with kubectl top pods to find the sweet spot

CPU: The Flexible Resource

Unlike memory, CPU limits are more forgiving. A process can’t use more CPU than allocated, but it won’t be killed for trying. My approach:

• Set requests based on baseline usage
• Use higher limits to allow for burst processing
• Remember that 1 CPU = 1000m (millicores) in Kubernetes

Debugging Cgroup Issues

When things go wrong (and they will), here are the tools I use:

# View cgroup hierarchy
systemd-cgls

# Check a specific process's cgroup
cat /proc/<PID>/cgroup

# Monitor real-time resource usage
systemd-cgtop

# For containers, check the runtime
crictl stats

One memorable debugging session involved a pod constantly getting OOMKilled despite having plenty of cluster memory available. The issue? The node’s cgroup hierarchy had stale entries from previous pod runs, effectively reducing available memory. A node restart cleared it up, but it taught me to monitor cgroup health, not just cluster metrics.

Beyond Containers: Systemd and Desktop Linux

While I discovered cgroups through Kubernetes, they’re not exclusive to containers. Systemd uses cgroups extensively for service management. Every systemd service runs in its own cgroup, which you can configure:

[Service]
# Limit service to 512MB memory
MemoryLimit=512M
# Limit to 50% of one CPU
CPUQuota=50%

This is particularly useful for resource-hungry services in an environment where you’re running multiple services on limited hardware.

Lessons Learned and Future Exploration

My journey with cgroups taught me several key lessons:

1. Infrastructure isn’t magic – Understanding the kernel features that enable container orchestration makes you a better operator
2. Resource limits need headroom – Setting limits too close to actual usage leads to instability
3. Different architectures have different requirements – What works on x86 might need tweaking on ARM
4. Monitoring is crucial – You can’t manage what you don’t measure

Looking ahead, I’m exploring:

• Using cgroups v2’s PSI (Pressure Stall Information) for better autoscaling – (See Facebooks Take Here.)
• Fine-tuning I/O limits for database workloads

Wrapping Up

Cgroups might be invisible most of the time, but they’re the foundation that makes modern container orchestration possible. Whether you’re running a simple Docker container or managing a complex Kubernetes cluster, understanding cgroups helps you:

• Debug resource-related issues more effectively
• Design better resource allocation strategies
• Optimize application performance within constraints

Next time Kubernetes tells you a pod was evicted or OOMKilled, you’ll know exactly what’s happening under the hood. And if you’re setting up a Raspberry Pi cluster, remember to enable those cgroups!

P.S. Big thanks to (Farid Zakaria for their recent deep deep dive into cgroups.)[https://fzakaria.com/2025/05/26/linux-cgroup-from-first-principles]