Skip to content

ci(#564): run the Playwright suite on pull requests (Chromium, path-gated) - #566

Merged
BorisTyshkevich merged 1 commit into
mainfrom
ci/e2e-on-pull-requests-564
Jul 29, 2026
Merged

ci(#564): run the Playwright suite on pull requests (Chromium, path-gated)#566
BorisTyshkevich merged 1 commit into
mainfrom
ci/e2e-on-pull-requests-564

Conversation

@BorisTyshkevich

Copy link
Copy Markdown
Collaborator

What & why

Closes #564. The Playwright suite was gated to tags, schedule and manual dispatch, so the e2e job reported skipped on every pull request and every push to main — while the overall run reported success. A check that never runs is not a check.

The cost was concrete, not hypothetical. Today's bug/low batch (#554#558, #561#563) all merged on green CI; two of those PRs changed and added e2e specs that CI never executed. Dispatching the workflow by hand against main gave the first real signal in that range — 6 failed / 568 passed, the same two tile-open-workbench specs failing on chromium, firefox and webkit. A second dispatch at f68861c produced the identical 6 failures, so they pre-date the batch. Filed as #565.

The change

Pull requests now run Chromium only, path-gated on anything that can move rendered geometry:

src/**            ← styles.css lives here; the single biggest reason for this filter
schemas/**  tests/e2e/**  playwright.config.js  build/**
package.json  package-lock.json  .github/workflows/**

Tags, nightly and manual dispatch keep the full three-engine matrix. That split is deliberate: both #565 failures reproduce on Chromium, so one engine would have caught them and is cheap enough to sit on every relevant PR — but engine-specific breakage is real in this repo (WebKit focus and IndexedDB behaviour especially), and that is what a release has to clear.

gate needed no change: it already fails on any failure among its needs and tolerates a legitimately skipped job.

Quarantining #565 so the gate is useful immediately

Turning on a gate against a red baseline just blocks everything, so #565's two specs are marked test.fail().

Deliberately not test.skip/test.fixme: those stop exercising the path, and nobody would notice when the bug is fixed. With test.fail() the specs keep running, and the day the underlying defect is repaired they report "expected to fail, but passed" — which is the signal to delete the quarantine. Both call sites say so, and say explicitly that the assertions encode #535's intended widen semantics and must not be rewritten to match the broken output.

Verification

Locally, npx playwright test tests/e2e/tile-open-workbench.spec.js --project=chromium21 passed, exit 0 (the two quarantined specs register as expected failures rather than errors).

The real proof is this PR itself: it touches .github/workflows/** and tests/e2e/**, so the path filter matches and the e2e job should run here instead of reporting skipped — which is exactly the behaviour being added. Please confirm that job appears and is green before merging.

Checklist

  • npm test passes (the per-file coverage gate is non-negotiable) — unaffected; no src/ or unit-test change
  • Tests added/updated in the same change as the code
  • npm run build succeeds (single-file dist/sql.html) — unaffected
  • Layers kept honest: pure logic in src/core/, network in src/net/ (injected fetch), DOM in src/ui/
  • No new runtime dependency
  • README / CHANGELOG.md ([Unreleased]) updated if behavior or the deployed surface changed — n/a, CI-only, no shipped-artifact change
  • Reconciled affected tracked work (roadmap Roadmap to 1.0.0 #68, the issue body, ADR/CHANGELOG) if this change reshaped it — Two tile-widen e2e specs are stale against the grafana-grid@2 placement shape #565 records the quarantine and its exit condition

🤖 Generated with Claude Code

https://claude.ai/code/session_01GiubaoqEuBzAyo5C4P8Vqr

The e2e job was gated to tags, schedule and manual dispatch, so it reported
`skipped` on every pull request and every push to `main` while the overall
run still reported success. A check that never runs is not a check.

The cost was concrete: two `tile-open-workbench` specs have been failing
deterministically on chromium, firefox AND webkit, and nothing surfaced it.
Today's `bug`/`low` batch (#554-#558, #561-#563) all merged "green"; two of
those PRs changed or added e2e specs that CI never executed. The failures only
appeared when the workflow was dispatched by hand (6 failed / 568 passed on
`main`), and a second dispatch at f68861c showed the identical 6 failures, so
they pre-date that batch. Filed as #565.

Pull requests now run **Chromium only**, path-gated on anything that can move
rendered geometry (`src/**` — which is where `styles.css` lives — plus
`tests/e2e/**`, `playwright.config.js`, `build/**`, `schemas/**`, the
manifests and the workflows). Both #565 failures reproduce on Chromium, so one
engine is enough to have caught them, and it is cheap enough to sit on every
relevant PR. Tags, nightly and manual dispatch keep the full three-engine
matrix: engine-specific breakage is real here, and that is what a release has
to clear.

The `gate` job needed no change — it already fails on any `failure` among its
needs and tolerates a legitimately skipped job.

#565's two specs are quarantined with `test.fail()` so this gate is meaningful
from the first run rather than red on arrival. Deliberately not `skip`/`fixme`:
the specs keep executing, so when the underlying bug is fixed they report
"expected to fail, but passed" and the quarantine has to be removed. Their
assertions encode #535's intended widen semantics and must not be rewritten to
match the broken output.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GiubaoqEuBzAyo5C4P8Vqr
@BorisTyshkevich
BorisTyshkevich merged commit 879ff4e into main Jul 29, 2026
8 checks passed
@BorisTyshkevich
BorisTyshkevich deleted the ci/e2e-on-pull-requests-564 branch August 6, 2026 15:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Playwright e2e never runs on PRs or main — green CI does not mean the browser suite passed

1 participant