Improve CI setup, reliability, and failure reporting

[ymichael]

Human comments

What was wrong

The October 9 audit found CI failures from FUSE mirrors, fresh fork dependency resolution, and storage-test teardown racing retained trash removal. Warm jobs still spent heavily on installation. Linux and Windows forwarded test shard/worker arguments into prerequisite builds, giving identical builds different cache hashes; one server regression test also waited for the full 30-second production timeout. Existing reports and replay diagnostics obscured where time and failures came from. Further investigation reproduced replay notifications escaping during cursor writes, numeric PID assertions confusing replacement processes with stopped children, and a final autosave assertion running before query observers settled.

What changed

• Install FUSE with isolated apt indexes, strict refresh errors, an installed-package fast path, and bounded mirror fallback.
• Exclude mobile dependencies only when the selected tests do not need them, retaining prerequisite and full-run coverage.
• Resolve standalone fork dependencies to lockfiles, install with npm ci, and retain manifests, locks, SDK tarball, and failure stages as artifacts.
• Wait for retained storage cleanup before tearing down its fixture. Include event counts, child state, stderr, and bounded grammar violation details in replay stalls.
• Pass Linux/Windows shard selection and Linux's two-worker limit only to test tasks. Declare the worker variable on all Turbo test overrides and support it explicitly in the two Vitest 3 packages.
• Shorten the liveness test using real capped timers in its child process, preserving production timeout and timer-reference behavior.
• Report first-attempt outcomes, step timings, and recovered failures. Parse workflow YAML structurally in the concurrency regression check.
• Keep replay events gated during incomplete cursor writes. Check retained child handles or completed-exit callbacks instead of historical numeric PIDs. Wait for the final autosave observer update without weakening assertions.
• Give the two observed subprocess-heavy CI fixture failures explicit 30-second budgets and bound fixture subprocesses. Document every remaining investigated failure group and its disposition.

How you verified

• CI with replay/PID fixes passed at 5596e9d1b2: 40 successful jobs and one normally skipped. Final CI also passed at 450c58f4fa: 40 successful jobs and one normally skipped, including the autosave assertion and fixture-budget follow-ups.
• Turbo dry runs across all 104 test tasks: worker count changes only test hashes, shard selection changes only app/server test hashes, and prerequisite hashes stay unchanged. CI artifacts confirm equal SDK type-build hashes across Linux server shards and distinct test hashes, with no forwarded arguments.
• Real Vitest discovery preserves complete, disjoint coverage: 339 server and 606 app files on both platform matrices. Vitest 3 and 4 resolve Linux's two-worker limit.
• Local follow-ups passed 25 selection/sequencing tests, all 42 tunnel tests and tunnel typechecks, seven parity tests, the liveness test, and two workflow tests. Relevant typechecks and formatting/diff checks passed.
• The liveness test still fails when the original unref bug is reintroduced; on Windows it fell from 30.5s to 0.72s. The rejected-event diagnostic failed before its fix and passes afterward. Removing the workflow concurrency limit still fails its YAML-based check. All temporary mutations were restored.
• Controlled Windows cache benchmark: two runners with reversed old/new order and initially empty task caches. Given shard 1's cache, shard 2 averaged 74.01s before versus 52.46s after (21.55s/29.1% saved); SDK type builds changed from 20.3–21.3s misses to hits while shard 2 tests still executed. All eight invocations passed. This excludes setup/cache transfer and is conditional reuse, not an every-run latency saving; normal jobs prefer their own caches and parallel cold jobs cannot share unfinished outputs.
• Earlier validation covered 46 CI-tooling tests and 24 storage tests, delayed-trash-removal fail-before/pass-after proof, actual FUSE installation on Ubuntu 22.04 amd64 and 24.04 arm64, and a real standalone Pi fork install/typecheck/test/build. Simulated npm resolution failure retained its evidence and remained blocking.

Further local verification passed 38 activation/service tests, 32 runtime lifecycle tests, 59 provider-parity tests, nine protocol tests, four ACP guidance tests, 30 CI fixture tests, and all 20 plugin-settings tests; relevant lint/typechecks passed. The replay regression fails against the original child. Controlled delayed query notifications reproduce the original autosave assertion failure; the fix passed 30/30 repetitions. Temporary instrumentation was removed.

Fresh Windows install benchmark: three separate fresh four-vCPU runners per arm, same Node/pnpm/lockfile, no dependency cache. Full installs averaged 51.403s; excluding mobile averaged 36.264s (15.138s/29.5% saved). This applies only to eligible broad installs, not the whole workflow.

Platform repetitions: Linux app shard passed three uncached runs (76 files/694 assertions each), and all 28 Git-diff tests passed three times. Windows passed five uncached repetitions each of provider parity (59 tests), activation/service (38 tests), and runtime lifecycle (32 tests).

Historical Git ENOTEMPTY, Linux worker exit, and the exact ACP/Codex replay incidents remain unexplained; passing repetitions do not prove them resolved. Playgrounds/Interactive Answers belong to unmerged feature PRs; a reproduced state-readiness candidate is preserved separately rather than importing those features here. Other historical failures are classified in docs/ci-performance.md, including already-fixed and branch-only defects. The intentionally forced provider-model check is unchanged. No merge is authorized.

AGENT GENERATED