Skip to content

fix(#585): stop laundering Docker-contention flakes in the WebKit browser-matrix spike - #624

Merged
BorisTyshkevich merged 1 commit into
mainfrom
fix/585-webkit-browser-matrix-flake
Aug 6, 2026
Merged

fix(#585): stop laundering Docker-contention flakes in the WebKit browser-matrix spike#624
BorisTyshkevich merged 1 commit into
mainfrom
fix/585-webkit-browser-matrix-flake

Conversation

@BorisTyshkevich

Copy link
Copy Markdown
Collaborator

What & why

Fixes the WebKit browser-matrix flake found during ADR-0005's Phase 0 spike (#585). Root-cause research (separate from this PR) found:

  • The one failing cell (current-altinity-stable/same-origin/webkit — 15/16 cells passed) reproduces 0/38 times across a narrow rerun, a full 4-row webkit rerun, and a rerun under artificial 16-way CPU stress. This is Docker-contention flakiness specific to this sandbox's amd64-emulated 4-container simultaneous boot, not a genuine WebKit/@clickhouse/client-web incompatibility.
  • The evidence's blank "Failure detail" column was itself a retrofit-ordering artifact: collectBrowserFailureDetail was added ~2h15m after the one live run that produced the failure, and docs/evidence/585/results.json's generatedAt never changed since — the field simply predates the data.

This PR lands the fix and closes three adjacent observability gaps found live during that investigation, so a future real failure isn't as unrecoverable as this one was. It does not regenerate docs/evidence/585/ or change ADR-0005's Rejected status — two other hard gates fail independently (confirmed live 24.8.x server-matrix incompatibility; the LOC-based net-deletion estimate). Evidence regeneration with this fix in place is a separate follow-up.

Changes

  • tests/spike/clickhouse-client/playwright.config.js: retries: 2 + trace: 'retain-on-failure'.
  • tests/spike/clickhouse-client/run-matrix.mjs: collectBrowserFailureDetail now keys off each test's own attempt/status instead of the spec-level ok boolean (previously a retry-then-pass silently laundered into a clean pass, losing the signal). classifyBrowserMatrixCell reports a passed-after-retry cell as its own 'flaky' status with full detail attached — never folded into 'passed' or left blank like 'failed'. Both gate computations (browser matrix, selectEarliestPassingVersion) treat 'flaky' as a cleared gate. ANSI stripped from captured errors at capture time. Playwright's top-level pw.errors[] now wired into whole-project/webServer-level failures.
  • tests/spike/clickhouse-client/spike-server.mjs: both previously-silent error catches now log to stderr (never to the client, never headers/bodies).
  • tests/spike/clickhouse-client/validate-evidence.mjs: a 'flaky' status still requires a backing detail record, same rigor as a real failure.
  • tests/spike/clickhouse-client/run-matrix.test.ts: new tests for all of the above.
  • CHANGELOG.md: [Unreleased] entry.

Testing

  • npm run check:types, npm run check:arch, npm run check:schemas, npm run check:examples, npm test (100/97.14/100/100 coverage), npm run build — all green on this branch.
  • The spike's own dedicated test suite (npm run test:client-spike) — green.
  • Live verification: injected a real one-shot failure into the exact flaky cell's test, ran it against a real Docker current-altinity-stable ClickHouse container under Playwright — got 1 flaky, 8 passed. Confirmed the fixed evidence code turns that into { status: 'flaky', failureDetail: [...] } with real per-spec detail, not a blank. The probe was reverted before committing (not part of this diff).

Checklist

  • npm test passes (the per-file coverage gate is non-negotiable)
  • Tests added/updated in the same change as the code
  • npm run build succeeds (single-file dist/sql.html)
  • Layers kept honest — this is confined entirely to tests/spike/clickhouse-client/, a test-owned comparison harness; nothing under src/ changed
  • No new runtime dependency
  • CHANGELOG.md ([Unreleased]) updated
  • Reconciled affected tracked work — docs/evidence/585/ and ADR-0005 are deliberately not touched by this PR; regeneration is a tracked follow-up, not silently implied by this fix

Part of #585 (does not close it — Phase 2-4 remain gated on a new decision per the issue's own ## Phases note).

🤖 Generated with Claude Code

https://claude.ai/code/session_01MqDRfSfy6w1FJnFibYY44F

…wser-matrix spike

Root-cause research confirmed the one failing browser-matrix cell recorded
during ADR-0005's Phase 0 spike (current-altinity-stable/same-origin/webkit)
is Docker-contention flakiness specific to this sandbox's amd64-emulated
4-container boot order, not a genuine WebKit/@clickhouse/client-web
incompatibility (0/38 reproductions across a narrow rerun, a full 4-row
webkit rerun, and a rerun under artificial CPU stress). The evidence's blank
"Failure detail" column was itself a retrofit-ordering artifact: the capture
code was added after the one live run that produced the failure, and
results.json was never regenerated since.

playwright.config.js now retries (retries: 2, trace: 'retain-on-failure'),
and run-matrix.mjs's evidence generator records a passed-after-retry cell as
its own 'flaky' status with full per-spec detail attached, never silently
folded into a clean 'passed' (laundering the retry) nor left blank like a
'failed' with no detail. Also closes three adjacent observability gaps:
spike-server.mjs now logs its own caught errors to stderr instead of
discarding them; run-matrix.mjs threads Playwright's top-level pw.errors[]
into whole-project/webServer-level failures; captured error text has ANSI
escape codes stripped at capture time.

Does not regenerate docs/evidence/585/ or touch ADR-0005's Rejected
decision — two other hard gates fail independently (the confirmed 24.8.x
server-matrix incompatibility, and the LOC-based deletion estimate).
Evidence regeneration with this fix in place is a separate follow-up.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MqDRfSfy6w1FJnFibYY44F
@BorisTyshkevich
BorisTyshkevich merged commit 6250625 into main Aug 6, 2026
8 checks passed
@BorisTyshkevich
BorisTyshkevich deleted the fix/585-webkit-browser-matrix-flake branch August 6, 2026 14:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant