Skip to content

Support runtime: cloud-hypervisor for enclaves #66639

Closed

Description

@lpcox

Summary

AWF supports Cloud Hypervisor microVMs as an enclave executor (enclaves[].runtime: "cloud-hypervisor", preview). gh-aw can't select it yet: the frontmatter schema on main only allows runtime: docker for both script and agent enclave entries.

We want to validate Cloud Hypervisor enclaves end to end through a real compiled gh-aw workflow and real mcpg. Today that is only possible through AWF's standalone probe, which uses a mock gateway. This issue covers the compiler changes that real workflow needs.

Current behavior

Compiling a copy of gh-aw-firewall's working smoke-enclave-build-test.md (Docker enclave, real mcpg v0.4.29, passes every 12 hours) with one added line:

enclaves:
  - script:
    runtime: cloud-hypervisor
    repos:
      - repo: github/gh-aw
        sensitivity: internal

fails with gh-aw v0.91.4:

error: Unknown property: script. ... 'script' belongs under 'enclaves', ...

The root cause is pkg/parser/schemas/main_workflow_schema.json: under enclaves.items.oneOf[*].properties.runtime, the value is {"enum": ["docker"]}.

Required changes

0. Mark as experimental (required)

runtime: cloud-hypervisor for enclaves is a preview in AWF and must be clearly experimental in gh-aw too:

  • Schema: start the runtime description with ⚠️ Experimental., matching existing experimental fields such as upload-code-coverage, and note that it requires GitHub-hosted Ubuntu x86_64 KVM runners.
  • Compile-time warning: add Using experimental feature: enclaves cloud-hypervisor runtime to the experimental-warning table in pkg/workflow/compiler_validators.go. Emit it whenever any enclave entry selects cloud-hypervisor, and add a test in the style of lsp_experimental_warning_test.go.
  • Docs: label the option experimental/preview in the enclave reference docs. State that it fails closed on unsupported hosts with no Docker fallback, and that its behavior and config may change without a deprecation period.
  • Optional: gate it behind a features: flag, so using it is an explicit opt-in, in line with AWF's own required cloudHypervisor.previewEnabled: true.

1. Schema and validation

  • Allow cloud-hypervisor in enclaves[].runtime for both script and agent entries.
  • Reject the combinations AWF rejects (see AWF docs/awf-config-spec.md §14), so they fail at compile time rather than at runtime:
    • image overrides together with runtime: cloud-hypervisor.
    • Mixing container and VM runtimes across enclave entries.
    • dynamic entries (only static script/agent are in scope).
    • Combining with sandbox.agent.runtime of sbx or cloud-hypervisor, or with a Docker host path prefix.
    • For agent: AWF also requires the API proxy, a configured provider route, and model, and does not allow DinD.

2. AWF config emission

  • Pass runtime: "cloud-hypervisor" through on each enclave entry.

  • Emit a top-level cloudHypervisor block. The primary agent stays on Docker: an enclave-only selection keeps container.containerRuntime: "docker".

  • Field values for the cloudHypervisor block:

    Field Value
    previewEnabled true
    mountPolicy "workspace-only"
    artifactReleaseTag the pinned AWF version, e.g. v0.28.47
    cloudHypervisorBinary, kernelPath, rootfsPath, supervisorPath, artifactManifestPath, artifactManifestBundlePath from the bundle setup step outputs
  • Reference config (what AWF's own acceptance harness generates): scripts/ci/cloud-hypervisor-enclave-live-smoke.js makeConfig() and scripts/ci/cloud-hypervisor-enclave-environment-probe.js makeProbeConfig() in github/gh-aw-firewall.

3. Setup steps (when any enclave uses Cloud Hypervisor)

  • Reuse the existing primary-agent steps from pkg/workflow/cloud_hypervisor_install.go:
    • generateCloudHypervisorKVMAccessStep
    • generateCloudHypervisorHostPreflightStep
    • generateCloudHypervisorBundleSetupStep
  • New: enclave rootfs artifacts.
    • Each AWF release publishes setup-cloud-hypervisor-enclave-artifacts.sh (usage: <release-tag> [cache-root]) alongside the enclave rootfs and its attested manifest.
    • The script exports these variables, which the AWF host reads (src/cloud-hypervisor/host-enclave-executor.ts):
      • AWF_CLOUD_HYPERVISOR_ENCLAVE_SCRIPT_ROOTFS
      • AWF_CLOUD_HYPERVISOR_ENCLAVE_AGENT_ROOTFS
      • AWF_CLOUD_HYPERVISOR_ENCLAVE_MANIFEST
      • AWF_CLOUD_HYPERVISOR_ENCLAVE_MANIFEST_BUNDLE
    • They must reach the AWF process environment.
  • Runner: GitHub-hosted Ubuntu x86_64 with KVM (e.g. ubuntu-24.04). Unsupported hosts fail closed in AWF; there is no fallback.

Acceptance

  • Compiling emits the experimental warning, and the schema and docs label the option experimental.
  • A copy of gh-aw-firewall's smoke-enclave-build-test.md with runtime: cloud-hypervisor compiles.
  • The compiled workflow runs on ubuntu-24.04 with real mcpg.
  • It produces exactly one successful enclave_run_script invocation in the enclave audit log.

gh-aw-firewall will add that smoke workflow once this lands.

Context

AWF has completed host, storage, artifact-trust and broker preflight on live KVM runners for v0.28.47. The remaining gap is a real end-to-end run through mcpg.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions