Repository navigation
ateom microvm: drop privileged: true, using atelet device plugin for KVM - #1254
Merged
Haven Xia (HavenXia) merged 4 commits intoAug 27, 2026
Conversation
A container's device access is gated by the cgroup v2 device controller, which the runtime programs to deny everything by default. That check runs before DAC, so a hostPath mount, a supplemental group, or any capability are all insufficient for /dev/kvm: cloud-hypervisor fails VmCreate with EPERM. `privileged: true` works only because it swaps the filter for allow-all, handing over every other device on the node as well. Advertise the devices as extended resources instead. atelet already runs on every node, so it hosts the plugin rather than adding a second DaemonSet: it registers with kubelet, and on allocation returns the device node, which kubelet turns into both the node and a matching cgroup allow rule. A micro-VM worker then gets default-deny plus exactly /dev/kvm and /dev/net/tun, with no privileged. Only devices actually present are advertised, so the resource request also places the pod: a gVisor-only node has no /dev/kvm and never receives a micro-VM worker. That replaces the ate.dev/sandboxClass nodeSelector, which was a hand-applied label that could disagree with the hardware in either direction. The taint toleration stays, since extended resources constrain where a pod fits but repel nothing. Detection reads the node's /dev mounted read-only at /host/dev, because atelet's own /dev is the container's and does not show /dev/kvm; seeing the nodes is all it grants, as atelet holds no capabilities and its own cgroup filter still denies opening them.
Drop the last `privileged: true`. The micro-VM worker now runs unprivileged with ALL capabilities dropped and an explicit set added back, and AppArmor Unconfined — both classes mount inside the worker, which the default profile denies. The capability set is the gVisor one plus FSETID and DAC_READ_SEARCH, both of which virtiofsd needs: it re-applies a fixed set to its sandboxed child (FSETID for the setgid bits it copies, DAC_READ_SEARCH to open files by handle for --migration-mode find-paths), and the kernel refuses to raise a capability the bounding set omits, so without them virtiofsd exits with "can't apply the child capabilities" and the VM never gets its virtio-fs device. The rest covers the same worker-side work as gVisor: sharing /run/kata-containers and remounting /proc/sys (SYS_ADMIN), building the actor veth and tap (NET_ADMIN/NET_RAW), and unpacking image rootfs as root (DAC_OVERRIDE/FOWNER/CHOWN/MKNOD). Unlike gVisor, this class also gives up the runtime's default seccomp profile, so that virtiofsd can keep its own sandbox: that sandbox unshares mount, pid and net namespaces and pivot_root()s, which the default profile denies whatever capabilities the worker holds. The trade favors containing the guest, since virtiofsd parses requests from it and so its namespaces are part of the guest-to-host boundary, while the pod-wide profile mostly confines ateom, which we trust. Both guest-facing processes also confine themselves regardless: virtiofsd installs its own seccomp filter and drops to nine capabilities with an empty bounding set, and cloud-hypervisor applies per-thread filters. /dev/kvm and /dev/net/tun are not capabilities and do not come from privileged either; they are requested as extended resources from atelet's device plugin.
Micro-VM worker pods now reach KVM-capable nodes through the device atelet advertises for them, so this hand-applied label no longer places anything. Removing it leaves one source of truth: either the node has the device or it does not, which cannot disagree with the hardware the way a label can. The taint of the same name is a different mechanism and stays, along with the toleration for it. Advertising a device attracts these pods to capable nodes but repels nothing else from them, so a cluster reserving nested-virt nodes still needs the taint to keep other workloads off.
This was referenced Aug 27, 2026
Benjamin Elder (BenTheElder)
marked this pull request as ready for review
August 27, 2026 05:54
|
check if bind mounting work for /dev/kvm too, it does for /dev/tun https://github.com/google/sam/blob/1fc706fc2a302bc19983caa22f8caf5c238f0597/site/content/docs/user/running-agents.md#in-a-kubernetes-pod |
Benjamin Elder (BenTheElder)
changed the title
privileged: true, using atelet device plugin for KVM/TUNprivileged: true, using atelet device plugin for KVM
Collaborator
Author
No it does not work for /dev/kvm, the cgroup device controller will block it. You have to either be privileged or be specifically granted access to the device. This is why kata ships a device-plugin for KVM as well. I assumed for tun as well, which turns out to be in the default allowlist ... |
Collaborator
Author
|
Need to double check if that's the same in say, crun. |
Collaborator
Author
Only devices the container runtime denies by default need a device plugin. /dev/net/tun is in runc's default cgroup device allow-list, so a worker opens it from an ordinary bind mount; creating the tap is then a capability question (CAP_NET_ADMIN, which the worker already holds) rather than a device one. /dev/kvm is not in that list and still needs the grant: verified that a bind-mounted /dev/kvm is refused even with the worker's full capability set, and even when opening a device node the container makes itself, because the cgroup program matches on device number rather than on the path. Advertising tun as an extended resource also made placement depend on it, so a node that had yet to load the tun module would refuse a worker that needs nothing from it.
Benjamin Elder (BenTheElder)
force-pushed
the
ateom-microvm-less-privileged
branch
from
August 27, 2026 06:26
d77edc0 to
b20f125
Compare
Haven Xia (HavenXia)
approved these changes
Aug 27, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is the companion to #496
In order to grant access to
/dev/kvmwe have to either:An NRI plugin is highly privileged in it's own right for all pods on the host and is difficult to ship portably at the moment.
DRA is promising, but not enough functionality is GA yet at our current 1.35+ target.
Device plugin fits reasonably well. We do wind up publishing an ~arbitrarily high limit, which has some cost in kubelet memory, but otherwise is relatively clean.
This approach is what kata uses currently. Their device plugin is not available unbundled, and we anyhow have a per-node daemonset.
atlet is taught to sniff if /dev/kvm appears on the host at all, so we can also stop using the manually labeled nodes for microVM class and instead schedule to the KVM + TUN devices on nodes that advertise them.
Later we can migrate to device plugin by using
I implemented that already, but I don't think it's worth merging at the moment. We would want consumable capacity to be on by default. We can migrate later without changing the pod spec by using
extendedResourceName.main...BenTheElder:substrate:ateom-microvm-dra
NOTE: I confirmed with upstream that device plugin is not going anywhere despite being "v1beta1", it's GA in all but name. It won't receive new features but we don't really need anyhow. We'll move to DRA down the line.
By doing this, we can drop
privileged: truefrom the uVM ateom pods.We can also drop the
ate.dev/sandboxClassnode label hacks, reducing friction to deploy.