ci(#564): run the Playwright suite on pull requests (Chromium, path-gated) - #566
Merged
Conversation
The e2e job was gated to tags, schedule and manual dispatch, so it reported `skipped` on every pull request and every push to `main` while the overall run still reported success. A check that never runs is not a check. The cost was concrete: two `tile-open-workbench` specs have been failing deterministically on chromium, firefox AND webkit, and nothing surfaced it. Today's `bug`/`low` batch (#554-#558, #561-#563) all merged "green"; two of those PRs changed or added e2e specs that CI never executed. The failures only appeared when the workflow was dispatched by hand (6 failed / 568 passed on `main`), and a second dispatch at f68861c showed the identical 6 failures, so they pre-date that batch. Filed as #565. Pull requests now run **Chromium only**, path-gated on anything that can move rendered geometry (`src/**` — which is where `styles.css` lives — plus `tests/e2e/**`, `playwright.config.js`, `build/**`, `schemas/**`, the manifests and the workflows). Both #565 failures reproduce on Chromium, so one engine is enough to have caught them, and it is cheap enough to sit on every relevant PR. Tags, nightly and manual dispatch keep the full three-engine matrix: engine-specific breakage is real here, and that is what a release has to clear. The `gate` job needed no change — it already fails on any `failure` among its needs and tolerates a legitimately skipped job. #565's two specs are quarantined with `test.fail()` so this gate is meaningful from the first run rather than red on arrival. Deliberately not `skip`/`fixme`: the specs keep executing, so when the underlying bug is fixed they report "expected to fail, but passed" and the quarantine has to be removed. Their assertions encode #535's intended widen semantics and must not be rewritten to match the broken output. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GiubaoqEuBzAyo5C4P8Vqr
7 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
Closes #564. The Playwright suite was gated to tags, schedule and manual dispatch, so the
e2ejob reportedskippedon every pull request and every push tomain— while the overall run reported success. A check that never runs is not a check.The cost was concrete, not hypothetical. Today's
bug/lowbatch (#554–#558, #561–#563) all merged on green CI; two of those PRs changed and added e2e specs that CI never executed. Dispatching the workflow by hand againstmaingave the first real signal in that range — 6 failed / 568 passed, the same twotile-open-workbenchspecs failing on chromium, firefox and webkit. A second dispatch atf68861cproduced the identical 6 failures, so they pre-date the batch. Filed as #565.The change
Pull requests now run Chromium only, path-gated on anything that can move rendered geometry:
Tags, nightly and manual dispatch keep the full three-engine matrix. That split is deliberate: both #565 failures reproduce on Chromium, so one engine would have caught them and is cheap enough to sit on every relevant PR — but engine-specific breakage is real in this repo (WebKit focus and IndexedDB behaviour especially), and that is what a release has to clear.
gateneeded no change: it already fails on anyfailureamong its needs and tolerates a legitimately skipped job.Quarantining #565 so the gate is useful immediately
Turning on a gate against a red baseline just blocks everything, so #565's two specs are marked
test.fail().Deliberately not
test.skip/test.fixme: those stop exercising the path, and nobody would notice when the bug is fixed. Withtest.fail()the specs keep running, and the day the underlying defect is repaired they report "expected to fail, but passed" — which is the signal to delete the quarantine. Both call sites say so, and say explicitly that the assertions encode #535's intended widen semantics and must not be rewritten to match the broken output.Verification
Locally,
npx playwright test tests/e2e/tile-open-workbench.spec.js --project=chromium→ 21 passed, exit 0 (the two quarantined specs register as expected failures rather than errors).The real proof is this PR itself: it touches
.github/workflows/**andtests/e2e/**, so the path filter matches and thee2ejob should run here instead of reportingskipped— which is exactly the behaviour being added. Please confirm that job appears and is green before merging.Checklist
npm testpasses (the per-file coverage gate is non-negotiable) — unaffected; nosrc/or unit-test changenpm run buildsucceeds (single-filedist/sql.html) — unaffectedsrc/core/, network insrc/net/(injected fetch), DOM insrc/ui/CHANGELOG.md([Unreleased]) updated if behavior or the deployed surface changed — n/a, CI-only, no shipped-artifact change🤖 Generated with Claude Code
https://claude.ai/code/session_01GiubaoqEuBzAyo5C4P8Vqr