KVM vs Container Virtualization: CPU Pinning, Memory Overhead & I/O Isolation

A KVM guest and a container can run the same web stack, but they share the host in different ways. A guest boots its own kernel on virtual hardware; a container is a group of ordinary processes that the host kernel fences off with namespaces and cgroups. That difference decides how CPU time is handed out, where memory goes, how disk requests travel, what one busy tenant can do to another, and what an attacker has to break to get out.

This guide takes each resource in turn, with the libvirt and cgroup v2 settings that control it. Measure on your own hardware before relying on any of them.

Key takeaways

• Pin by physical core, not by thread. Put a guest's vCPUs on sibling hyperthreads of the same cores, and keep QEMU's emulator and I/O threads elsewhere.
• isolcpus is a blunt tool. It is fixed until reboot; cgroup cpusets do most of the same job at runtime.
• A guest's memory overhead is mostly a second kernel and a second page cache. Hugepages cut page-table cost but rule out overcommit.
• A container's CPU limit is a time budget. cpu.max quotas can stall a multi-threaded process mid-request.
• The security boundaries differ in kind. Escaping a guest means breaking KVM or QEMU; escaping a container means breaking the shared kernel.

Side by side

CPU: pinning, isolation and NUMA

Read the topology first

Each vCPU is a thread of the QEMU process. Unpinned, the host scheduler moves those threads between cores, which costs cache warmth and, on a NUMA host, can put a vCPU on a different socket from the guest's memory.

lscpu -e=CPU,NODE,SOCKET,CORE
cat /sys/devices/system/cpu/cpu2/topology/thread_siblings_list
numactl --hardware

The snippets below assume CPUs 2 and 10 are two hyperthreads of one physical core, as are 3 and 11, all on NUMA node 0. Check thread_siblings_list on your host rather than assuming siblings have adjacent numbers.

Pin vCPUs, the emulator and I/O threads

<vcpu placement='static'>4</vcpu>
<iothreads>1</iothreads>
<cputune>
  <vcpupin vcpu='0' cpuset='2'/>
  <vcpupin vcpu='1' cpuset='10'/>
  <vcpupin vcpu='2' cpuset='3'/>
  <vcpupin vcpu='3' cpuset='11'/>
  <emulatorpin cpuset='0,8'/>
  <iothreadpin iothread='1' cpuset='1,9'/>
</cputune>
<cpu mode='host-passthrough'>
  <topology sockets='1' dies='1' cores='2' threads='2'/>
</cpu>

1. vCPU pairs land on sibling threads, and the guest is told it has two cores of two threads, which matches what it gets. Its scheduler can then place work sensibly.
2. Whole cores belong to one guest. Tenants on sibling threads of one core share its L1 and L2 caches and execution units: a performance problem and, given the history of SMT side-channel bugs, a security one.
3. QEMU's own threads stay out of the way. emulatorpin covers the threads doing device emulation and housekeeping; iothreadpin covers the I/O thread the disks use below. Both sit on housekeeping cores so they never preempt a vCPU.

On a running guest, use virsh vcpupin web01 0 2 and virsh emulatorpin web01 0,8, and check with virsh vcpuinfo web01.

isolcpus and nohz_full trade-offs

Pinning keeps vCPUs on their cores; it does not keep other work off them. Boot parameters can:

isolcpus=domain,managed_irq,2-3,10-11 nohz_full=2-3,10-11 rcu_nocbs=2-3,10-11 irqaffinity=0-1,8-9

isolcpus removes CPUs from scheduler load balancing, nohz_full stops the periodic tick on a CPU with exactly one runnable task, rcu_nocbs moves RCU callbacks elsewhere and irqaffinity sets the default interrupt CPUs.

The costs: isolcpus is fixed until reboot, and the kernel documentation marks it deprecated in favour of cpusets. Threads pinned to an isolated range are not balanced within it, so several can pile onto one CPU while the others idle. nohz_full only helps when one task owns the CPU, adds work to every kernel entry and exit there (so an exit-heavy guest can get slower), and needs at least one housekeeping CPU.

For most hosting workloads, pinning plus cpusets is the better default: confine host services to housekeeping CPUs with a cpuset or systemd's AllowedCPUs=, and keep the boot parameters for latency needs you have measured.

NUMA placement

Memory on the other socket is slower to reach, so keep a guest's vCPUs and memory on one node:

<numatune>
  <memory mode='strict' nodeset='0'/>
</numatune>

strict fails the allocation instead of quietly using the remote node; a guest that will not start is easier to diagnose than one that is mysteriously slow. Check with numastat -p $(pgrep -f 'guest=web01'). A guest larger than one node needs a matching virtual layout with <numa> cells inside <cpu>.

Memory: where the overhead goes

What a guest costs

• Its own kernel and base services, loaded separately by every guest.
• Its own page cache. If the host also caches the disk image, file data sits in RAM twice. cache='none' on the disk opens the image with O_DIRECT and takes the host cache out of the path.
• The QEMU process. Measure it: ps -o rss= -p $(pgrep -f 'guest=web01') minus the guest's RAM is a rough figure, and it grows with the number of emulated devices.
• Host page tables for the guest's memory.

As worked arithmetic: if each guest's kernel and base services take 200 MiB, twenty guests spend 4 GiB on twenty copies of the same thing. Twenty containers share one kernel, and their page cache is the host's. Shared pages are charged to the cgroup that first touched them, which explains a container's limit filling with another container's files.

Hugepages

With 4 KiB pages, a 16 GiB guest needs 16 GiB / 4 KiB = 4,194,304 last-level page-table entries; at 8 bytes each, 32 MiB of page tables for one guest. With 2 MiB pages it needs 8,192 entries; with 1 GiB pages, 16.

The bigger gain is the TLB. With hardware-assisted paging, a TLB miss in a guest walks both the guest's and the host's tables: with four levels each, up to 24 memory references for one translation. TLB reach is entries times page size, so each entry covers 512 times more memory with 2 MiB pages than with 4 KiB pages.

Reserve 1 GiB pages at boot, when contiguous memory is still available, then back the guest with them:

default_hugepagesz=1G hugepagesz=1G hugepages=32

<memoryBacking>
  <hugepages>
    <page size='1' unit='GiB'/>
  </hugepages>
  <locked/>
</memoryBacking>

Reserved hugepages cannot be used for anything else, swapped or deduplicated, and a guest that needs more than are free on its NUMA node will not start. Hugepages and overcommit do not mix.

Ballooning and KSM

A virtio balloon asks the guest kernel to hand pages back. Kernel Samepage Merging (KSM) merges identical pages across guests, trading CPU for memory, and merging between tenants creates a timing side channel: a tenant can detect whether a page it holds is shared. Leave KSM off between tenants who do not trust each other.

Container memory limits

echo "+cpu +cpuset +memory +io +pids" > /sys/fs/cgroup/cgroup.subtree_control
mkdir /sys/fs/cgroup/tenant-a
echo 3G  > /sys/fs/cgroup/tenant-a/memory.high
echo 4G  > /sys/fs/cgroup/tenant-a/memory.max
echo 0   > /sys/fs/cgroup/tenant-a/memory.swap.max
echo 1   > /sys/fs/cgroup/tenant-a/memory.oom.group
echo 512 > /sys/fs/cgroup/tenant-a/pids.max

Above memory.high the kernel reclaims and slows the group; at memory.max the OOM killer acts inside it. Setting memory.high lower gives a warning zone instead of a sudden kill, and memory.oom.group kills the group together so an application is not left half-alive. pids.max stops one fork loop exhausting the host's process table. Docker's --memory=4g --memory-swap=4g sets the hard limit with no swap.

Disk I/O

virtio-blk or virtio-scsi

Both are paravirtual: the guest passes requests through shared-memory queues instead of driving emulated hardware.

• virtio-blk is the simpler path with less work per request. Each disk is its own PCI device, seen as /dev/vdX.
• virtio-scsi puts many disks behind one controller, seen as /dev/sdX, and supports SCSI passthrough and persistent reservations for clustering.

Use virtio-blk for a few fast disks and virtio-scsi for many disks or SCSI semantics. Both support multiple queues and pass discard through, so guest TRIM reaches thin-provisioned storage.

io_uring and native AIO

<disk type='block' device='disk'>
  <driver name='qemu' type='raw' cache='none' io='native' discard='unmap' iothread='1' queues='4'/>
  <source dev='/dev/vg0/web01-root'/>
  <target dev='vda' bus='virtio'/>
</disk>

io sets how QEMU submits requests to the host. threads uses a worker pool and is the most compatible. native uses Linux AIO and needs cache='none', because Linux AIO is only truly asynchronous with O_DIRECT. io_uring works with or without direct I/O, generally needs fewer system calls per request, and requires QEMU 5.0 and libvirt 6.3 or later.

io_uring has also been a frequent source of kernel bugs, and since kernel 6.6 the kernel.io_uring_disabled sysctl lets administrators restrict it. If your hardening policy does, use native.

I/O limits for containers

A container's requests go straight to the host block layer, so the limits sit on its cgroup:

lsblk -d -o NAME,MAJ:MIN
echo "259:0 rbps=104857600 wbps=52428800 riops=4000 wiops=2000" > /sys/fs/cgroup/tenant-a/io.max

That caps reads at 100 MiB/s, writes at 50 MiB/s and IOPS as stated on device 259:0. io.weight gives proportional shares instead, but only works with the BFQ scheduler or the io.cost controller. Unlike cgroup v1, v2 charges buffered writes to the cgroup that dirtied the pages, on filesystems with cgroup writeback support such as ext4 and XFS.

Container CPU limits and stalls

echo "200000 100000" > /sys/fs/cgroup/tenant-a/cpu.max
echo 4-7 > /sys/fs/cgroup/tenant-a/cpuset.cpus
echo 0   > /sys/fs/cgroup/tenant-a/cpuset.mems

cpu.max is quota and period in microseconds: 200 ms of CPU per 100 ms window, two CPUs' worth. Docker's --cpus=2 and a Kubernetes CPU limit of 2 set the same thing.

Now the arithmetic. Eight busy threads use 200 ms of CPU in 200 / 8 = 25 ms of wall-clock time, and the group is throttled for the remaining 75 ms of the period. A request arriving then waits up to 75 ms, though average CPU use looks modest. Rising nr_throttled and throttled_usec in cpu.stat confirm it. Fixes: size thread pools to the quota, use cpu.weight instead of a hard cap, or give the container its own cores with cpuset.cpus. In Kubernetes, the kubelet's static CPU manager policy gives exclusive cores to Guaranteed pods requesting whole CPUs.

Noisy neighbours

Some shared resources sit outside both cgroups and pinning:

• Last-level cache and memory bandwidth. A tenant streaming through memory evicts everyone's cache lines. Intel RDT and AMD's equivalent, exposed through resctrl, can partition them on supporting hardware.
• Kernel-wide tables. Containers share one kernel's connection-tracking table and dentry and inode caches.
• Interrupt and packet processing, which runs in softirq context on whichever CPU took the interrupt and is not always charged to the tenant that caused it.
• The storage device, whose internal queues and garbage collection are shared whatever the host-side limits say.

Inside a guest, contention shows as steal time: time the vCPU was ready but the host ran something else. Containers have no steal time; read cpu.stat and the pressure files cpu.pressure, memory.pressure and io.pressure, which report the share of time tasks waited on each resource.

The security boundary

KVM

To escape a guest, an attacker must exploit KVM's handling of VM exits or QEMU's device emulation in host userspace. Shrink that surface:

• Give guests virtio devices only and remove emulated devices they do not need.
• Run QEMU unprivileged, confined by SELinux (sVirt) or AppArmor, with its seccomp sandbox on. libvirt does this by default on most distributions; confirm it on yours.
• Patch host microcode and kernel promptly. Speculative-execution and SMT bugs cross the VM boundary in hardware, which is why cores should not be split between tenants.

Containers

Every container shares the host kernel, so one reachable kernel bug can be an escape. Layer the defences:

• Run as non-root and, where possible, with user namespaces so container root maps to an unprivileged host user. Rootless Podman does this by default; recent Kubernetes releases support hostUsers: false.
• Keep seccomp on. Docker applies a default profile; in Kubernetes set seccompProfile: RuntimeDefault.
• Drop all capabilities and add back only what is needed, set no-new-privileges, and use a read-only root filesystem.
• Never mount the container runtime's socket into a container: access to it is root on the host.

For untrusting tenants who still want container workflows, sandboxed runtimes move the boundary back towards hardware: gVisor handles system calls in a user-space kernel, while Kata Containers and Firecracker run each workload in a lightweight VM.

Failure modes and troubleshooting

Checklist

1. Map topology with lscpu -e and thread_siblings_list before pinning.
2. Pin vCPUs by whole core, mirror it in <cpu>, and keep emulator and I/O threads on housekeeping cores.
3. Keep each guest on one NUMA node with strict <numatune>.
4. Use isolcpus and nohz_full only for measured latency needs.
5. Use hugepages for large, long-lived guests, knowing they cannot be overcommitted.
6. Use cache='none' with io='native' or io='io_uring', per your io_uring policy.
7. For containers, set memory.high below memory.max, set pids.max, and watch cpu.stat.
8. Enable BFQ or io.cost before relying on io.weight.
9. Keep KSM off and never split a physical core between untrusting tenants.
10. Containers: non-root, user namespaces, seccomp, capabilities dropped, no runtime socket.

Closing

Neither model is simply faster or safer. A KVM guest buys a separate kernel and a hardware-backed boundary, paid for with a second kernel's memory and an extra hop on every disk request. A container gets near-native access to the host, and its isolation is only as strong as the shared kernel and the limits you set. Choose per workload, set limits deliberately, and watch the counters that show when they bite.

To see how your own site's page speed holds up, the free speed audit is a quick place to start.