Skip to content

[Workflow Health] P0: cloud-hypervisor "--version exited with code undefined" — 29 workflows affected across 3 engines #61528

Closed as not planned
Closed as not planned
[Workflow Health] P0: cloud-hypervisor "--version exited with code undefined" — 29 workflows affected across 3 engines#61528
Labels
cookieIssue Monster Loves Cookies!priority-p0type-failureworkflow-health

Description

@github-actions
github-actions
bot
opened on Sep 17, 2026

P0: cloud-hypervisor "--version exited with code undefined" — engine-agnostic sandbox regression

Signature (from failed job logs):

[ERROR] Fatal error: Error: "/run/awf-cloud-hypervisor/trusted-artifacts/run-XXXXXX/cloud-hypervisor --version" exited with code undefined:

followed shortly after by Set output 'model_not_supported_error' — this downstream flag is a
misleading side effect of the crash, not evidence of a genuine model-resolution failure. The
real failure is the cloud-hypervisor preflight version check inside the sandbox.

Scope: 29 open [aw] <workflow> failed issues currently match this exact log signature (as of
2026-09-17T04:39Z), first observed 2026-09-15T22:25Z (#61219), with a sharp acceleration overnight
2026-09-16→17 (18 of the 29 filed in the preceding ~8 hours). Confirmed across copilot, codex, and
claude engines:

Likely root cause location: the compiled sandbox config (awf-config.json, embedded in each
.lock.yml) sets:

"cloudHypervisor":{"previewEnabled":true,"mountPolicy":"workspace-and-tool-cache","vcpuCount":2,"memoryMib":4096}

This is a newly-enabled preview sandbox runtime path (awf-cloud-hypervisor trusted-artifacts
binary) that appears to fail its own --version preflight check on the runner, aborting the agent
step before the actual engine (copilot/codex/claude) ever starts.

Suggested fix directions:

  1. Disable cloudHypervisor.previewEnabled (roll back to the previous non-preview sandbox path)
    until the preflight crash is root-caused, or
  2. Pin/upgrade the gh-aw-firewall release providing the cloud-hypervisor trusted-artifact binary
    to a version where --version succeeds on the current runner image, or
  3. Make the --version preflight check non-fatal (fall back to the previous sandbox runtime) rather
    than aborting the whole agent run.

Impact: Actively failing runs across at least 29 distinct workflows spanning all 3 supported
engines in under 24 hours, and creating false-positive model_not_supported_error signals that risk
being misattributed to the (separate, pre-existing) gpt-5.3-codex model config issue tracked
previously under #61030 (now closed via expires: 1d).

DO NOT file further individual per-workflow issues for this exact signature — consolidate here.

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

  • api.github.com

[!TIP]
api.github.com is blocked because GitHub API access uses the built-in GitHub tools by default. Instead of adding api.github.com to network.allowed, use tools.github.mode: gh-proxy for direct pre-authenticated GitHub CLI access without requiring network access to api.github.com:

tools:
  github:
    mode: gh-proxy

See GitHub Tools for more information on gh-proxy mode.

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "api.github.com"

See Network Configuration for more information.

Generated by 🏥 Workflow Health Manager - Meta-Orchestrator · copilot · auto · 117.9 AIC · ⌖ 13.8 AIC · ⊞ 12.6K · ◷

  • expires on Sep 17, 2026, 8:44 PM UTC-08:00

Activity

  1. github-actions
    added
    workflow-health
    priority-p0
    type-failure
    cookieIssue Monster Loves Cookies!
    on Sep 17, 2026
  2. github-actions
    added a parent issue on Sep 17, 2026

  3. github-actions commented on Sep 17, 2026

    @github-actions
    github-actionsbot
    on Sep 17, 2026 – with GitHub Actions
    ContributorAuthor

    🍪 Issue Monster selected this for Copilot

    I’ve identified this issue as a good candidate for automated resolution and requested assignment to the Copilot coding agent.

    If assignment succeeds, the Copilot coding agent will analyze the issue and create a pull request with the fix.

    Om nom nom! 🍪

    🍪 Om nom nom by Issue Monster · pi · gpt54 · 21.5 AIC · ⊞ 12.8K · ◷

  4. github-actions commented on Sep 17, 2026

    @github-actions
    github-actionsbot
    on Sep 17, 2026 – with GitHub Actions
    ContributorAuthor

    🍪 Issue Monster selected this for Copilot

    I've identified this issue as a good candidate for automated resolution and requested assignment to the Copilot coding agent.

    If assignment succeeds, the Copilot coding agent will analyze the issue and create a pull request with the fix.

    Om nom nom! 🍪

    🍪 Om nom nom by Issue Monster · pi · gpt54 · 11.9 AIC · ⊞ 12.8K · ◷

  5. github-actions commented on Sep 17, 2026

    @github-actions
    github-actionsbot
    on Sep 17, 2026 – with GitHub Actions
    ContributorAuthor

    🍪 Issue Monster selected this for Copilot

    I’ve identified this issue as a good candidate for automated resolution and requested assignment to the Copilot coding agent.

    If assignment succeeds, the Copilot coding agent will analyze the issue and create a pull request with the fix.

    Om nom nom! 🍪

    🍪 Om nom nom by Issue Monster · pi · gpt54 · 6.5 AIC · ⊞ 12.8K · ◷

  6. github-actions commented on Sep 17, 2026

    @github-actions
    github-actionsbot
    on Sep 17, 2026 – with GitHub Actions
    ContributorAuthor

    🍪 Issue Monster selected this for Copilot

    I've identified this issue as a good candidate for automated resolution and requested assignment to the Copilot coding agent.

    If assignment succeeds, the Copilot coding agent will analyze the issue and create a pull request with the fix.

    Om nom nom! 🍪

    🍪 Om nom nom by Issue Monster · pi · gpt54 · 7.86 AIC · ⊞ 12.5K · ◷

  7. github-actions commented on Sep 18, 2026

    @github-actions
    github-actionsbot
    on Sep 18, 2026 – with GitHub Actions
    ContributorAuthor

    Update — 2026-09-18T04:36Z: root cause mutated to EACCES, incident still active

    The consolidated cloud-hypervisor sandbox regression is not resolved — it has changed failure
    signature from exited with code undefined to a permission error on the same trusted-artifact
    binary:

    [ERROR] Fatal error: Error: Unable to execute "/run/awf-cloud-hypervisor/trusted-artifacts/run-XXXXXX/cloud-hypervisor --version";
verify the trusted Cloud Hypervisor artifact exists, is executable, and is complete: code=EACCES;
Command failed with EACCES: /run/awf-cloud-hypervisor/trusted-artifacts/run-XXXXXX/cloud-hypervisor --version

    Confirmed directly from job logs of two of this dashboard's failing-workflows.json entries:

    • LintMonster run 35300195220 (2026-09-18T02:39Z) — agent job failed with the EACCES signature above.
    • Daily Firewall Logs Collector and Reporter run 35299593898 (2026-09-18T02:30Z) — same signature.

    Scope: is:open search for "cloud-hypervisor --version" EACCES in body currently matches
    30 open [aw] <name> failed issues, earliest at 2026-09-17T14:59Z (#61627), i.e. the crash mode
    changed roughly 10 hours after this tracker was filed and is still ongoing as of this run
    (2026-09-18T04:36Z). The original "exited with code undefined" signature this issue was filed for
    is now down to near-zero fresh occurrences — this EACCES form has effectively replaced it as the
    dominant failure mode from the same subsystem.

    Assessment: Same root cause family (the awf-cloud-hypervisor trusted-artifact preflight
    --version check), but now failing with EACCES (permission denied executing the binary) rather
    than an undefined exit code. This looks consistent with the trusted-artifact extraction/permission
    step not marking the binary executable, or a runner/tmpfs mount-option (noexec) issue, rather than
    a binary-startup crash.

    Recommendation: Do not file new per-workflow issues for this signature either — continue
    consolidating here. Suggest checking the trusted-artifact download/extraction step for
    chmod +x on cloud-hypervisor (and awf-supervisor/virtiofsd) after unpacking, and verifying
    /run/awf-cloud-hypervisor/trusted-artifacts is not mounted noexec.

    Filed by Workflow Health Manager — no new issue created; this is an update to the existing P0.

    Warning

    Firewall blocked 1 domain

    The following domain was blocked by the firewall during workflow execution:

    • api.github.com

    [!TIP]
    api.github.com is blocked because GitHub API access uses the built-in GitHub tools by default. Instead of adding api.github.com to network.allowed, use tools.github.mode: gh-proxy for direct pre-authenticated GitHub CLI access without requiring network access to api.github.com:

    tools:
  github:
    mode: gh-proxy

    See GitHub Tools for more information on gh-proxy mode.

    To allow these domains, add them to the network.allowed list in your workflow frontmatter:

    network:
  allowed:
    - defaults
    - "api.github.com"

    See Network Configuration for more information.

    Generated by 🏥 Workflow Health Manager - Meta-Orchestrator · copilot · auto · 80.5 AIC · ⌖ 15.8 AIC · ⊞ 12.8K · ◷

  8. github-actions commented on Sep 18, 2026

    @github-actions
    github-actionsbot
    on Sep 18, 2026 – with GitHub Actions
    ContributorAuthor

    This issue was automatically closed because it expired on 2026-09-18T04:44:57.158Z.

    Closed by Workflow

  9. github-actions
    closed this as not plannedon Sep 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    cookieIssue Monster Loves Cookies!priority-p0type-failureworkflow-health

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions