fix(#585): stop laundering Docker-contention flakes in the WebKit browser-matrix spike - #624
Merged
Merged
Conversation
…wser-matrix spike Root-cause research confirmed the one failing browser-matrix cell recorded during ADR-0005's Phase 0 spike (current-altinity-stable/same-origin/webkit) is Docker-contention flakiness specific to this sandbox's amd64-emulated 4-container boot order, not a genuine WebKit/@clickhouse/client-web incompatibility (0/38 reproductions across a narrow rerun, a full 4-row webkit rerun, and a rerun under artificial CPU stress). The evidence's blank "Failure detail" column was itself a retrofit-ordering artifact: the capture code was added after the one live run that produced the failure, and results.json was never regenerated since. playwright.config.js now retries (retries: 2, trace: 'retain-on-failure'), and run-matrix.mjs's evidence generator records a passed-after-retry cell as its own 'flaky' status with full per-spec detail attached, never silently folded into a clean 'passed' (laundering the retry) nor left blank like a 'failed' with no detail. Also closes three adjacent observability gaps: spike-server.mjs now logs its own caught errors to stderr instead of discarding them; run-matrix.mjs threads Playwright's top-level pw.errors[] into whole-project/webServer-level failures; captured error text has ANSI escape codes stripped at capture time. Does not regenerate docs/evidence/585/ or touch ADR-0005's Rejected decision — two other hard gates fail independently (the confirmed 24.8.x server-matrix incompatibility, and the LOC-based deletion estimate). Evidence regeneration with this fix in place is a separate follow-up. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MqDRfSfy6w1FJnFibYY44F
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
Fixes the WebKit browser-matrix flake found during ADR-0005's Phase 0 spike (#585). Root-cause research (separate from this PR) found:
current-altinity-stable/same-origin/webkit — 15/16 cells passed) reproduces 0/38 times across a narrow rerun, a full 4-row webkit rerun, and a rerun under artificial 16-way CPU stress. This is Docker-contention flakiness specific to this sandbox's amd64-emulated 4-container simultaneous boot, not a genuine WebKit/@clickhouse/client-webincompatibility.collectBrowserFailureDetailwas added ~2h15m after the one live run that produced the failure, anddocs/evidence/585/results.json'sgeneratedAtnever changed since — the field simply predates the data.This PR lands the fix and closes three adjacent observability gaps found live during that investigation, so a future real failure isn't as unrecoverable as this one was. It does not regenerate
docs/evidence/585/or change ADR-0005's Rejected status — two other hard gates fail independently (confirmed live 24.8.x server-matrix incompatibility; the LOC-based net-deletion estimate). Evidence regeneration with this fix in place is a separate follow-up.Changes
tests/spike/clickhouse-client/playwright.config.js:retries: 2+trace: 'retain-on-failure'.tests/spike/clickhouse-client/run-matrix.mjs:collectBrowserFailureDetailnow keys off each test's own attempt/status instead of the spec-levelokboolean (previously a retry-then-pass silently laundered into a clean pass, losing the signal).classifyBrowserMatrixCellreports a passed-after-retry cell as its own'flaky'status with full detail attached — never folded into'passed'or left blank like'failed'. Both gate computations (browser matrix,selectEarliestPassingVersion) treat'flaky'as a cleared gate. ANSI stripped from captured errors at capture time. Playwright's top-levelpw.errors[]now wired into whole-project/webServer-level failures.tests/spike/clickhouse-client/spike-server.mjs: both previously-silent error catches now log to stderr (never to the client, never headers/bodies).tests/spike/clickhouse-client/validate-evidence.mjs: a'flaky'status still requires a backing detail record, same rigor as a real failure.tests/spike/clickhouse-client/run-matrix.test.ts: new tests for all of the above.CHANGELOG.md:[Unreleased]entry.Testing
npm run check:types,npm run check:arch,npm run check:schemas,npm run check:examples,npm test(100/97.14/100/100 coverage),npm run build— all green on this branch.npm run test:client-spike) — green.current-altinity-stableClickHouse container under Playwright — got1 flaky, 8 passed. Confirmed the fixed evidence code turns that into{ status: 'flaky', failureDetail: [...] }with real per-spec detail, not a blank. The probe was reverted before committing (not part of this diff).Checklist
npm testpasses (the per-file coverage gate is non-negotiable)npm run buildsucceeds (single-filedist/sql.html)tests/spike/clickhouse-client/, a test-owned comparison harness; nothing undersrc/changedCHANGELOG.md([Unreleased]) updateddocs/evidence/585/and ADR-0005 are deliberately not touched by this PR; regeneration is a tracked follow-up, not silently implied by this fixPart of #585 (does not close it — Phase 2-4 remain gated on a new decision per the issue's own
## Phasesnote).🤖 Generated with Claude Code
https://claude.ai/code/session_01MqDRfSfy6w1FJnFibYY44F