Repository navigation
CI: Playwright browser install can hang unbounded and eat a shard's entire 20-minute budget — the 3-attempt retry never fires #1807
Description
Symptom — maximally confusing: a cancelled run, no failing test, no results file
PR #1805, run 32068185803, job Run tests (shard 3) (95507532540):
21:03:02 Run npx --yes "playwright-core@1.62.1" install --with-deps --only-shell chromium
21:03:18 Installing dependencies... ← last output
21:22:19 ##[error]The operation was canceled.
Job duration 20m25s against timeout-minutes: 20 (.github/workflows/dotnet-test.yml:502). At cleanup:
Terminate orphan process: pid (2645) (npm exec playwright-core@1.62.1 install --with-deps --only-shell chromium)
grep: test/test-results.log: No such file or directory
Zero tests ran. Every other job in the run was green. The PR surfaced as COMPLETED/CANCELLED with Consolidate test results failing — no failing test to look at, no .trx, nothing naming a cause. It costs a full CI cycle each time and reads like a mystery.
Root cause: a retry loop with no per-attempt timeout is a single unbounded attempt
.github/workflows/dotnet-test.yml:604-620 is well-built in every other respect — 3 attempts, backoff, no continue-on-error, no input-shaped if:, and a deliberate ::error::… this is an INFRA failure, not a test failure.:
for attempt in 1 2 3; do
if npx --yes "playwright-core@${PLAYWRIGHT_VERSION}" install --with-deps --only-shell chromium; then
ok=1; break
fi
delay=$((10 * attempt)); sleep $delay
done🚨 The retry only triggers when the command FAILS. It does nothing when the command HANGS. Attempt 1 stalled for 19 minutes, attempts 2 and 3 never ran, and that carefully-worded infra error never printed — the job cap fired first and rewrote the whole thing as "cancelled".
--with-deps shells out to apt-get, and the shell download is a CDN fetch: an unbounded network fetch inside a bounded job. Same family as the Docker-Hub-pull CI red we have hit before.
Fix — two shapes
- Bound each attempt (smallest, makes the existing retry real). CI runners are Linux, where
timeoutexists (unlike our macOS dev hosts):A hang then becomes a failed attempt → the retry fires → and if all three fail, the intendedif timeout 240 npx --yes "playwright-core@${PLAYWRIGHT_VERSION}" install --with-deps --only-shell chromium; then
::error::prints and the shard reddens for the right reason. - Pre-bake the browser into the runner image (removes the fetch from the hot path entirely).
PLAYWRIGHT_VERSIONis already kept in step withdeploy/base-images/portal-ai/Dockerfile, so the version pin exists.
These compose — (1) is worth doing regardless, because it converts an unattributable cancellation into a named infra failure.
🚨 Likely shared root with Systemorph/MeshWeaver.Education
Education is measuring CI flakiness right now and reports ~50% of today's runs needed a re-run, with bootstrap retries the single largest contributor at 11% — and it runs the same npx playwright install. If this is one root cause across both repos, fixing it here is worth far more than the ~30 s the browser-cache lever was measured to be worth there. Cross-linking so the two are not investigated twice.
Not a flake to re-run away
Recorded because this is easy to mis-triage: the diff on #1805 touches only the LSP diagnostics path and cannot reach a step that runs before any test starts. Re-running only the failed jobs was legitimate here under the "harness itself crashed / infrastructure error" exemption — but nothing should be merged on a cancelled run.
Activity
rbuergi commentedon Aug 17, 2026
Checked on MeshWeaver.Education, as asked — the shared root does not apply there, but the shape does
Direct answer to "does your bootstrap step's retries bound each attempt?": Education has no retry around
playwright installat all. All four call sites are the bare form, with no wrapper and no timeout:So Education cannot be hitting your defect (a retry that only fires on failure, never on a hang) — there is no retry to mis-fire. It does share the underlying unbounded network fetch inside a bounded job: a hang here would silently eat the shard's
timeout-minutes: 45and surface ascancelled, exactly as yours did.And the "11%" you were passed was mine, and it is wrong — it is also a different mechanism
Please don't carry that number: I over-counted, and the corrected figures from all 47 shard logs of 2026-08-17 are
More importantly, Education's "bootstrap retries" have nothing to do with
playwright install. They are Playwright's ownretries: 1re-running the bootstrap test afterawaitTypeLive('Store/Plugin', '/Store')exhausts a 300 s deadline (both instances: runs 517 shard 1 and 526 shard 3). Different layer, different cause. Shared-root hypothesis ruled out for Education — you can stop treating it as corroboration.Measured split of the step, which bears on your fix #2
36 observations, 33–62 s, median 40 s, no hang seen. From the timestamps inside one step:
npm ciplaywright install --with-deps→ apt system deps (Installing dependencies…)So for the "pre-bake the browser into the runner image" fix: it removes the ~15 s CDN fetch, not the 21 s apt step — unless the image also carries Playwright's system dependencies, in which case
--with-depscan drop toinstalland you get both. Worth baking both or the win is a third of what it looks like.That also corrects the "~30 s browser-cache lever" figure attributed to me: I measured the cacheable browser download at ~15 s, and I did not implement it — at ~15 s off a 14.5 min shard against a 115 MB × N cache-quota cost, it did not clear the bar on my side. Your fix #1 (
timeoutinside the retry) is a correctness fix and is worth far more than either; notetimeoutexists on the Linux runners but not on the macOS dev hosts, which is exactly the trap that makes it easy to "verify" locally and have it silently do nothing.rbuergi commentedon Aug 17, 2026
Second occurrence tonight — same shard, same signature, different PR. Recording it because one instance is an anecdote and two in ninety minutes is a rate.
PR #1813, run 32074664424,
Run tests (shard 3):Job duration 20m24s against
timeout-minutes: 20.grep: test/test-results.log: No such file or directory— zero tests ran, again. Orphan reaped at cleanup:npm exec playwright-core@1.62.1 install --with-deps --only-shell chromium.Compare the first occurrence (#1805, run 32068185803):
20m25s, hang from 21:03:18, same orphan, same missing results file. Both landed on shard 3, ~90 minutes apart.What the recurrence adds
::error::… INFRA failuremessage printed neither time. That message has now failed to appear on both occasions it was written for.CANCELLEDwith no failing test and no.trx— the most expensive possible failure mode to triage, because there is nothing to look at.Bumps the priority of the one-line fix
timeout 240around the attempt inside the existing loop converts both of these into a failed attempt → retry → and, if all three fail, the named infra error that was supposed to print. Two shards' worth of CI time tonight would have been ~12 minutes instead of ~40, and both would have been self-describing.Re-running the failed jobs on #1813 under the harness-error exemption (the harness never started; no test executed, so there is no flake being buried).
rbuergi commentedon Aug 17, 2026
Third occurrence tonight — PR #1779, shard 2
Same signature as PR #1805 (shard 3) and PR #1813 (shard 3): the job is cancelled at the 20-minute cap, zero tests execute, no
.trxis produced, and the orphanednpm exec playwright-core … installis reaped at cleanup. Three PRs, three different diffs, none of which can reach a test — an LSP import-scope change, a layout/i18n change, and a CD-workflow change that touches no C# at all.That makes tonight's total roughly an hour of CI spent on the least attributable failure shape available:
CANCELLED, no failing test, nothing to triage, and the step's own::error::… INFRA failure, not a test failuremessage printing on none of the three — because the retry loop catches failure, and this is a hang.Worth stating plainly: #1815 is now on the critical path. With three hits in under three hours, every PR in flight has a material chance of losing a full cycle to it, including #1815's own runs. It should merge ahead of the queue rather than take its turn in it.
Re-ran under the harness-error exemption: no test executed, so there is no flaky test being retried away.
rbuergi commentedon Aug 19, 2026
Fixed across two merged PRs:
timeoutkills only its direct child, so playwright'ssudo-spawnedapt-getsurvived as an orphan holding/var/lib/apt/lists/lockand blocked attempts 2 and 3 with its own predecessor. The blocker was privilege, not process-group topology —kill(2)silently skips members the caller may not signal.Verified on a real runner: step wall clock 70s → 23s, phase-split confirmed (
0 upgraded, 9 newly installed).