🩺 Runner Doctor UpdateRunner Doctor knowledge-base update: D14 retry-fix follow-up (2026-09-19 scan) #8844

Summary

• Scan window: issues/PRs updated on or after 2026-09-19 in github/gh-aw-firewall.
• Items reviewed: ~35 issues/PRs across ARC/DinD, self-hosted, GHES/GHEC, and runtime search terms, cross-checked against the live catalog files.
• New lessons found: 1 (an update to the existing D14 entry; PR Fix Cloud Hypervisor trusted artifact EACCES on hosted runners #8835, the current EACCES-fix attempt for the same failure family, is still open/unmerged and is therefore intentionally not documented as fixed yet).

Proposed knowledge-base changes

File: .github/workflows/shared/self-hosted-failure-modes.md

The D14 row ("cloud-hypervisor --version" exited with code undefined under --container-runtime cloud-hypervisor) was last updated by PR #8756 (merged 2026-09-19), which added citations for #8727/#8728 alongside the #8622 baseline fix. Since then, PR #8801 ("Retry transient Cloud Hypervisor version-probe failures in preflight", merged 2026-09-20, closing issue #8767) landed a further fix that is not yet reflected:

• runCloudHypervisorPreflight now wraps the cloud-hypervisor/virtiofsd --version probes in a bounded retry (runVersionWithRetry, 3 attempts, 250 ms apart), since the artifact's SHA-256 digest is already verified before the probe runs — a single-attempt failure at that point reflects a transient exec/host hiccup, not a corrupt or wrong binary.
• Deterministic errors (ENOENT, EACCES, EISDIR, ENOTDIR, ENOEXEC — via a structured error.code or a code=XXX message fallback) are still fail-fast/no-retry, since they will not resolve on their own.
• Each retry attempt logs a logger.warn with the attempt count and underlying error for visibility.

Proposed edit to the D14 row's Fix / flag column (append after the existing #8622/#8727/#8728 text):

Further fixed in AWF (PR #8801, merged 2026-09-20): the --version probe for both the Cloud Hypervisor and virtiofsd binaries is now retried up to 3 times (250 ms apart) before failing, since the staged artifact's digest is already trusted by this point and a single failure is more likely a transient exec hiccup. Deterministic errors (ENOENT/EACCES/EISDIR/ENOTDIR/ENOEXEC) skip the retry and fail closed immediately. Upgrade AWF to include #8801.

Proposed edit to the D14 row's Probe column (append):

; on AWF including #8801, a single transient probe failure no longer aborts the run — check agent/preflight logs for Cloud Hypervisor version probe failed for "<path>" (attempt N/3) warnings indicating the retry engaged, versus an immediate failure for the fail-fast error codes above

Proposed edit to the D14 row's Citations column: append , github/gh-aw-firewall#8767, github/gh-aw-firewall#8801.

Note on a still-open, related item: issue #8834 and its (currently unmerged) fix PR #8835 target a different symptom in the same code area — spawn .../cloud-hypervisor EACCES on hosted runners, proposing to relocate trusted-artifact staging from /run/awf-cloud-hypervisor/trusted-artifacts to /var/lib/awf-cloud-hypervisor/trusted-artifacts to avoid a noexec-mounted /run. Since PR #8835 is still open (mergeable_state: blocked, not merged), this is not proposed as a catalog change yet — flagging it here so the next scan window picks it up once merged.

Proposed doctor changes

File: .github/workflows/self-hosted-runner-doctor.md

• Error-string quick-lookup (~line 116): keep the existing "cloud-hypervisor --version" exited with code undefined" → D14 (...) entry, but append the same github/gh-aw-firewall#8801-based retry note as above so the lookup and full playbook stay consistent.
• Playbook paragraph for D14 (~line 238): append the same "Further fixed in AWF (PR Retry transient Cloud Hypervisor version-probe failures in preflight #8801...)" sentence and citation as described above.

Proposed portable agent changes

File: .github/agents/self-hosted-runner-doctor.md

Apply the identical edits in all embedded locations that mirror the shared/workflow files:

• Error-string lookup near the top (~line 87)
• Playbook paragraph for D14 (~line 215)
• Embedded catalog table row for D14 (~line 371)
• Embedded error-string lookup table row for D14 (~line 416)

Each should receive the same "Further fixed in AWF (PR #8801...)" addition and citation appends as the shared/workflow files, keeping the portable copy in sync per the doctor's own sync requirement.

Source issues and PRs

• [aw] sbx, gVisor & Cloud Hypervisor Documentation Updater failed #8767 — "[aw] sbx, gVisor & Cloud Hypervisor Documentation Updater failed" (closed by Retry transient Cloud Hypervisor version-probe failures in preflight #8801)
• Retry transient Cloud Hypervisor version-probe failures in preflight #8801 — "Retry transient Cloud Hypervisor version-probe failures in preflight" (merged 2026-09-20)
• Fix Cloud Hypervisor trusted-artifact EACCES failures on hosted runners #8834 — "Fix Cloud Hypervisor trusted-artifact EACCES failures on hosted runners" (open; tracked for a future scan once its fix PR merges)
• Fix Cloud Hypervisor trusted artifact EACCES on hosted runners #8835 — "Fix Cloud Hypervisor trusted artifact EACCES on hosted runners" (open, unmerged — not yet actionable for the catalog)

All other candidate items reviewed in this window — PR #8747/#8748 (closed unmerged, superseded by #8801), PR #8756 (the prior runner-doctor update itself), PR #8663 (unrelated security-hardening batch, no runner-specific lesson), PR #8800/#8815/#8799/#8802/#8785/#8779/#8818 (NVX/refactor/test-only changes with no self-hosted-runner-environment lesson), and PR #8733 (README wording fix) — were either already captured, superseded, or not genuine non-hosted-runner lessons, so they are skipped per the de-duplication and scope rules.

Generated by Runner Doctor Updater · copilot · auto · 141.1 AIC · ⊞ 4.3K · ◷

• expires on Oct 21, 2026, 4:41 PM UTC