[Martin Heberling (Bickor)]

This is an exploratory PR based on an AKS interoperability investigation. I am opening it as a draft to discuss whether hypervisor selection belongs at controller scope or should move into SandboxConfig / WorkerPool compatibility metadata.

Summary

• keep /dev/kvm as the default MicroVM hypervisor device
• allow ate-controller to generate workers using /dev/mshv
• restrict MSHV workers to explicitly labeled x86-64 nodes
• reject arbitrary device paths before they become privileged hostPath mounts
• document backend and snapshot compatibility constraints

Why

AKS KataVmIsolation nodes use an Azure Linux Microsoft Hypervisor root-partition image. These nodes expose /dev/mshv, not /dev/kvm, so the existing hard-coded KVM worker pod cannot start even though Cloud Hypervisor is built with MSHV support.

The new controller setting is available as either:

--microvm-hypervisor-device=/dev/mshv
ATE_MICROVM_HYPERVISOR_DEVICE=/dev/mshv

Only /dev/kvm and /dev/mshv are accepted. MSHV workers also require:

ate.dev/sandboxClass=microvm
ate.dev/microvm-hypervisor=mshv
kubernetes.io/arch=amd64

AKS validation

Validated on:

• AKS Kubernetes 1.36.2
• Standard_D8s_v5
• KataVmIsolation
• AKSAzureLinux-V3katagen2-202608.06.1
• host kernel 6.6.137.mshv2-1.azl3
• Cloud Hypervisor v53.0
• Kata 4.0 guest assets
• virtiofsd v1.14.0

Using the real ateom-microvm image and native gRPC API, the following succeeded through /dev/mshv:

1. RunWorkload
2. Cloud Hypervisor VM boot
3. kata-agent connection over vsock
4. actor container creation and HTTP readiness
5. full CheckpointWorkload
6. full RestoreWorkload
7. guest-agent resource metrics after restore

The checkpoint contained base-id, config.json, memory-ranges, rootfs-upper.tar, and state.json. An in-memory counter returned 1 before checkpoint and 2 after restore, confirming memory continuity.

This did not use the host's Kata containerd shim. The outer worker remained a normal runc pod; ateom-microvm launched Cloud Hypervisor and drove snapshot/restore directly.

Current AKS limitation

This PR only addresses the MicroVM hypervisor device. A complete stock Substrate deployment is still blocked on managed AKS because Substrate currently requires the PodCertificateRequest and ClusterTrustBundle APIs for internal mTLS, and AKS 1.35/1.36 do not expose them. AKS integration also needs dedicated storage, registry-auth, DNS, and deployment overlays.

Compatibility

Hypervisor selection is currently controller-wide. KVM and MSHV pools or snapshots must not be mixed, and the setting should not be changed while snapshots exist. Snapshot portability across the backends is not guaranteed.

Testing

• go test -short ./...
• go test -race ./cmd/atecontroller
• focused controller KVM/MSHV tests with -race
• go test ./cmd/ateom-microvm/...
• go vet ./cmd/atecontroller/...
• hack/verify/go-generate.sh
• hack/verify/gofmt.sh
• hack/verify/golangci-lint.sh
• Appropriate documentation changes included

[google-cla]

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

[Martin Heberling (Bickor)]

Hey Benjamin Elder (@BenTheElder) I was planning on leaving this as draft, but if you're interested in merging I'm happy to do any improvements you suggest! The PR was made in the context of this linkedin post.

[Benjamin Elder (BenTheElder)]

So I've been working on #1254 which we can probably build on top of :-)

I need to learn more about the mshv requirement.

[Huy Pham (hdp617)]

Martin Heberling (@Bickor) What's the status of this PR? Are you still working on it?

Benjamin Elder (@BenTheElder) Is it a GA blocker for Azure support? If yes, I can help with this; I have access to AKS.

[Benjamin Elder (BenTheElder)]

Benjamin Elder (@BenTheElder) Is it a GA blocker for Azure support? If yes, I can help with this; I have access to AKS.

#1207 is in, we should look at how to handle this now.

Feel free to take a look, there are other more pressing microsoft blockers though, azure even with gvisor has #1827 in particular. I've been in discussions about those.

[Martin Heberling (Bickor)]

This was just a prototype. Closing in favor of Anish's efforts: #2689