Fix flaky CI: fast test scratch (/ci) + UFFD orphan reaping + transient vsock retry

[rgarcia]

Addresses the flaky failures on the self-hosted Linux runner (deft-kernel-dev) — instance ... did not reach Running within 45s, vm.restore ... Client.Timeout exceeded, vsock ... EOF — which are environmental (disk-I/O contention) plus a test-harness orphan leak, not product bugs.

Root cause (proven)

The integration tests are heavily I/O-bound on VM snapshot/fork/restore, all of which flow through $TMPDIR. On the runner, /tmp is bind-mounted onto the encrypted RAID5 data volume (data_crypt), which delivers ~158 MB/s idle and ~23 MB/s under concurrent CI load. A controlled cgroup io.max experiment confirmed causation: throttling I/O reproduces the failures with a clean dose-response (10 MB/s → 6/6 fail), while a CPU-throttle control passes 6/6. Separately, when a test is SIGKILLed/times out during a standby/fork/restore transition, its Firecracker guest is leaked and — once its UFFD pager exits — spins a vCPU at 100% forever, compounding host contention.

Changes

1. Run tests on a fast scratch disk — test.yml sets TMPDIR=/ci, a dedicated unencrypted OS-SSD scratch (~400–500 MB/s, isolated from data_crypt/Docker/prod) provisioned by the dev_ci_scratch Ansible role (kernel/infra #616, merged + applied to deft). Measured under live load: /ci 406 MB/s vs /tmp 23 MB/s. Makefile passes TMPDIR through the test-linux sudo env (otherwise sudo strips it). /ci is shorter than /tmp, so it stays under the 108-char AF_UNIX sun_path limit for VMM control sockets (unlike a /tmp/hci$N prefix).
2. Reap orphaned guests in test cleanup — cleanupOrphanedProcesses now also scans /proc for firecracker/cloud-hypervisor/hypeman-uffd-pager whose cmdline is under the test's t.TempDir(), killing guests before pagers so an orphaned UFFD guest is never left to spin. Catches the PID-nil window where meta.HypervisorPID is cleared during standby/fork/restore.
3. Tolerate transient post-restore exec blips — assertGuestFile/requireRunningSleepInstance use execCommandWithRetry, retrying only transient vsock/gRPC connection errors (EOF/Unavailable/closing), never real assertion failures.

(Fix 4, a production "kill guest if its pager dies" guard, is left as a documented TODO — it needs a real pager-health watchdog and is risky to do narrowly; the test-side reaper mitigates CI today.)

Dependency / rollout

TMPDIR=/ci requires kernel/infra #616 applied on the runner (done — /ci exists, hourly reaper active). Safe ordering: this can merge now since /ci is already live.

Testing

gofmt/vet/build pass. FCUFFD passed 3/3 across reruns pre-/ci. Fork-test on /ci vs /tmp at idle: both pass, /ci faster (22.6s vs 25.5s median); the under-load win is carried by the cgroup experiment + the 406-vs-23 MB/s throughput measurement (the heavy-load window didn't naturally recur during the fork-test run).

🤖 Generated with Claude Code

---

Note

Low Risk
Changes are confined to CI workflow, Makefile env passthrough, and test-only helpers; production hypervisor behavior is unchanged aside from log-only QEMU diagnostics.

Overview
Targets flaky Linux CI on self-hosted KVM runners by moving VM-heavy test I/O off slow /tmp and hardening the integration test harness.

CI / Makefile: test.yml sets TMPDIR=/ci and co-locates HYPEMAN_TEST_PREWARM_DIR on /ci (same filesystem as per-test scratch). Adds HYPEMAN_TEST_REFLINK_STRICT=1. Makefile forwards TMPDIR through sudo env in test-linux.

Test harness: New acquireHeavyIO caps parallel VM/snapshot integration tests (default 4). cleanupOrphanedProcesses also scans /proc for firecracker/cloud-hypervisor/qemu/UFFD pager processes under the test data dir (guests before pagers). execCommandWithRetry / isTransientExecError retry transient vsock/gRPC exec failures after restore. Reflink assertions gated via assertCopyReflinkedGated and HYPEMAN_TEST_REFLINK_STRICT. Shared image builds serialized per ref in testsupport. TestFCUFFDOneShotLifecycle skipped (KERNEL-1354).

Diagnostics: QEMU startup failures log vsock guest-CID conflicts via /proc scan (diagnostics.go). UFFD pager-loss production guard documented as TODO in hypervisor_linux.go.

^{Reviewed by Cursor Bugbot for commit a681add. Bugbot is set up for automated code reviews on this repo. Configure here.}

[firetiger-agent]

Created a monitoring plan for this PR.

What this PR does: Fixes a CI test flake where leaked Firecracker/UFFD guests from aborted tests were permanently burning CPU on the shared runner, compounding I/O contention and causing more flakes. This is a test-infrastructure fix with no production code changes.

Intended effect:

• Test workflow pass rate: baseline 50–69% failure (week of Jun 1–8); confirmed improving if failure rate drops and holds below 50% over the next 7 days after merge.
• Orphan reap log lines: "Cleaning up orphaned guest/uffd pager process" messages should appear in test output on flaky runs instead of silent CPU leaks that degrade the runner.

Risks:

• Reaper over-broad match — /proc cmdline substring scan could in theory match a sibling test's guest; alert if Test failure rate stays above 60% for 3+ consecutive days post-merge and logs show unexpected instance-not-found or mid-test SIGKILL errors.
• Retry masking real failures — isTransientExecError string matching could hide a genuine assertion failure that produces a matching error message; alert if new failures surface only as context deadline exceeded on tests expected to fail fast.

Status updates will be posted automatically on this PR as monitoring progresses.

View monitor

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Autofix Details

Bugbot Autofix prepared a fix for the issue found in the latest run.

• ✅ Fixed: Data-dir substring kills sibling tests
  • Updated cmdline directory matching to require a path-boundary after the matched dir and added coverage proving sibling prefix paths like /0010 no longer match /001.

Or push these changes by commenting:

@cursor push 7779d03804

Preview (7779d03804)

diff --git a/lib/instances/manager_test.go b/lib/instances/manager_test.go
--- a/lib/instances/manager_test.go
+++ b/lib/instances/manager_test.go
@@ -306,22 +306,75 @@
 
 // cmdlineReferencesDir reports whether any argv token is, or lives under, dir.
 func cmdlineReferencesDir(argv []string, dir string) bool {
+	dir = filepath.Clean(dir)
 	for _, tok := range argv {
 		tok = strings.TrimSpace(tok)
 		if tok == "" {
 			continue
 		}
 		// Tokens may embed the path inside larger strings (e.g. qemu's
-		// "socket,...,path=/tmp/.../qmp.sock"), so a substring check on the
-		// cleaned prefix is the robust match. The prefix is a unique per-test
-		// t.TempDir() root, so substring matching cannot collide across tests.
-		if strings.Contains(tok, dir) {
-			return true
+		// "socket,...,path=/tmp/.../qmp.sock"), so scan for occurrences and
+		// require a path boundary at the end of the matched directory.
+		for start := 0; start < len(tok); {
+			idx := strings.Index(tok[start:], dir)
+			if idx < 0 {
+				break
+			}
+			idx += start
+			end := idx + len(dir)
+			if end == len(tok) || (end < len(tok) && os.IsPathSeparator(tok[end])) {
+				return true
+			}
+			start = idx + 1
 		}
 	}
 	return false
 }
 
+func TestCmdlineReferencesDir(t *testing.T) {
+	t.Parallel()
+
+	dir := filepath.Clean(filepath.Join(string(os.PathSeparator), "tmp", "hypeman", "001"))
+
+	tests := []struct {
+		name string
+		argv []string
+		want bool
+	}{
+		{
+			name: "exact token",
+			argv: []string{"qemu-system-x86_64", dir},
+			want: true,
+		},
+		{
+			name: "embedded under dir",
+			argv: []string{
+				"cloud-hypervisor",
+				"--api-sock",
+				"socket,id=qmp,path=" + filepath.Join(dir, "instances", "vm-1", "qmp.sock"),
+			},
+			want: true,
+		},
+		{
+			name: "sibling dir with prefix is not a match",
+			argv: []string{
+				"cloud-hypervisor",
+				"--api-sock",
+				"socket,id=qmp,path=" + filepath.Join(string(os.PathSeparator), "tmp", "hypeman", "0010", "instances", "vm-1", "qmp.sock"),
+			},
+			want: false,
+		},
+	}
+
+	for _, tt := range tests {
+		tt := tt
+		t.Run(tt.name, func(t *testing.T) {
+			t.Parallel()
+			assert.Equal(t, tt.want, cmdlineReferencesDir(tt.argv, dir))
+		})
+	}
+}
+
 func isGuestProcess(exe string) bool {
 	switch {
 	case exe == "firecracker":

_{You can send follow-ups to the cloud agent here.}

[hiroTamada]

reviewed end to end — solid work, the flake-class breakdown is convincing and I verified the risky bits locally: no double-acquire deadlock in the heavyIO semaphore (all helper/caller paths acquire exactly once), and the salted CID derivation is assign-only at every recompute site (snapshot.go:326, snapshot.go:480, fork.go:362, fork.go:550), so process restarts are safe. approving, with one question and two smaller things:

question

• lib/instances/create.go:57-78 — the commit message attributes the qemu "vhost-vsock: unable to set guest cid: Address already in use" flake to weak-hash collisions, but the old 8-char hash on random cuid2 IDs collides at essentially ideal 2^-32 rates (simulated 200k IDs: 6 collisions vs ~4.7 expected for a perfect 32-bit hash), which makes an observable collision rate hard to explain that way. meanwhile standby forks inherit the snapshot CID verbatim (snapshot.go:478), so consecutive warm-chain links share a CID, and a slow-dying prior hypervisor produces exactly that error — a teardown race the salt doesn't address. did TestQEMUWarmForkChain stop flaking specifically, or just CI overall? (the derivation change is good hygiene either way — this is about whether that flake class is actually closed or just rarer now that /ci speeds up teardown.)

portability

• lib/instances/snapshot_integration_scenario_test.go:111 — assertCopyReflinked runs unconditionally and hard-fails on non-reflink filesystems (tmpfs/ext4 /tmp, i.e. most contributor machines). pre-existing, but this PR codifies the reflink-capable-TMPDIR assumption into runner infra only. consider gating it on the existing probeReflinkSupport (firecracker_test.go:1172) like the disk-usage assertion already does, with an env strict mode set in CI so the runner still catches real reflink regressions.

nit

• .github/workflows/test.yml:156-162 — the comment documents internal runner disk topology (RAID5/LUKS layout, ansible role name, throughput numbers) in a public repo; something like "fast scratch fs provisioned on the runner; must be reflink-capable, kept short for AF_UNIX sun_path" carries what a reader needs.

[rgarcia]

@hiroTamada You're right — the 8-char hash on random cuid2 IDs collides at ~ideal 2^-32 rates, so weak-hash collision can't explain an observable flake; I overstated that in the commit message. The salt's only real motivation was cross-process CID uniqueness across concurrent runners, but as you point out that doesn't touch the actual mechanism — the warm-chain CID handoff (snapshot.go:478) where a slow-dying prior hypervisor still holds the CID (a teardown race). So I've rolled the salt back (b4929d3) and instead added log-only instrumentation (cb3596a): on a guest cid … in use failure it logs the attempted CID and scans /proc for the holder (PID/PPID/orphan-flag + tmp dir), so the next CI hit tells us definitively whether it's the prior chain link, a leaked orphan, or a concurrent run — then we can fix the real race. And to your question directly: I can't attribute the disappearance to the CID change — TestQEMUWarmForkChain stopped recurring, but that's confounded with the heavyIO cap, the reaper, and /ci speeding teardown, which is exactly why I'm instrumenting rather than calling that flake class closed.

(Other two are addressed in a681add: assertCopyReflinked is now gated on probeReflinkSupport with a CI strict env HYPEMAN_TEST_REFLINK_STRICT=1 — contributor machines on ext4/tmpfs skip it, the runner still enforces it; and the test.yml comment is genericized to drop the internal disk topology.)

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Bugbot Autofix prepared a fix for the issue found in the latest run.

• ✅ Fixed: Reflink strict env not forwarded
  • Added HYPEMAN_TEST_REFLINK_STRICT to both sudo env invocations in test-linux so reflink strict mode reaches go test in CI.

Or push these changes by commenting:

@cursor push f8da96f52d

Preview (f8da96f52d)

diff --git a/Makefile b/Makefile
--- a/Makefile
+++ b/Makefile
@@ -305,6 +305,7 @@
 			"HYPEMAN_UFFD_SYSTEMD_INSTANCE_PREFIX=$${HYPEMAN_UFFD_SYSTEMD_INSTANCE_PREFIX:-}" \
 			"HYPEMAN_TEST_PREWARM_DIR=$${HYPEMAN_TEST_PREWARM_DIR:-}" \
 			"HYPEMAN_TEST_PREWARM_STRICT=$${HYPEMAN_TEST_PREWARM_STRICT:-}" \
+			"HYPEMAN_TEST_REFLINK_STRICT=$${HYPEMAN_TEST_REFLINK_STRICT:-}" \
 			"HYPEMAN_TEST_REGISTRY=$${HYPEMAN_TEST_REGISTRY:-}" \
 			go test -tags containers_image_openpgp -run=$(TEST) $$VERBOSE_FLAG -timeout=$(TEST_TIMEOUT) ./...; \
 	else \
@@ -314,6 +315,7 @@
 			"HYPEMAN_UFFD_SYSTEMD_INSTANCE_PREFIX=$${HYPEMAN_UFFD_SYSTEMD_INSTANCE_PREFIX:-}" \
 			"HYPEMAN_TEST_PREWARM_DIR=$${HYPEMAN_TEST_PREWARM_DIR:-}" \
 			"HYPEMAN_TEST_PREWARM_STRICT=$${HYPEMAN_TEST_PREWARM_STRICT:-}" \
+			"HYPEMAN_TEST_REFLINK_STRICT=$${HYPEMAN_TEST_REFLINK_STRICT:-}" \
 			"HYPEMAN_TEST_REGISTRY=$${HYPEMAN_TEST_REGISTRY:-}" \
 			go test -tags containers_image_openpgp $$VERBOSE_FLAG -timeout=$(TEST_TIMEOUT) ./...; \
 	fi

_{You can send follow-ups to the cloud agent here.}

^{Reviewed by Cursor Bugbot for commit a681add. Configure here.}

[cursor]

Reflink strict env not forwarded

Low Severity

CI sets HYPEMAN_TEST_REFLINK_STRICT=1, but test-linux runs go test under sudo env with only an explicit variable list. Unlike TMPDIR and HYPEMAN_TEST_PREWARM_STRICT, reflink strict is not forwarded, so reflinkStrict() is always false under make test and gated fork reflink checks can be skipped when the probe fails.

Additional Locations (1)

• .github/workflows/test.yml#L151-L153

 

^{Reviewed by Cursor Bugbot for commit a681add. Configure here.}