prima check: run a scenario end to end and report what it proved

[DavertMik]

Field-tested prima against a live app (review) and fixed what that turned up, then benchmarked it head-to-head against playwright-cli until the numbers moved. Spec: 2026-08-07-prima-fixes-design.md.

The bug that motivated this

prima pw '({page}) => page.click("[data-test=nonexistent-thing]")'
→ ok: true
  healed: true (recovered after 1 attempt)
  used: I.click({ role: "button", text: "New agent" })

The element did not exist. Heal picked an unrelated control, clicked it, and reported success — so ok: true did not mean your own action landed. Heal is removed outright, along with --no-heal, the healed: line and the healing-attempts block. No code path can now reach an element other than the one asked for.

→ ok: false, 11.3s, exact error + compact ARIA inline

The tier that pays: do and check

The point of prima is that the expensive model outside it never sees a page. That only works if one prima call can carry a whole sequence, so both long-running commands now do.

do has the same toolset a test run has — act, look, assert — and ends where the instructions end. A done() step closes the sequence and names the instructions it could not carry out, which fail the command. Before this it kept acting past the last step and could report ok: true while a check it had been asked to make never held.

check <scenario> spawns a full test run: it drives the page, verifies the outcome itself, and reports every step with its proof.

prima check "signup rejects a duplicate email" \
  --expected "an error names the email as taken" \
  --expected "no second account is created"

Each --expected comes back under ### Expected outcomes as PASSED, FAILED or not verified. "not verified" is deliberately not "false" — it means nothing checked it. That mapping only works if the run records an outcome in its own words, so the prompt now says an expected result is settled only when it is repeated word for word; measured live, that moved both outcomes of a real scenario from not verified to a real verdict.

click and fill are gone — one-action-per-process was the shape that lost to playwright-cli.

Envelopes carry proof, not noise

### Changes renders on every action. It was structurally impossible alongside an answer, research map or verdict, so a successful click proved nothing.

used: I.click("//*[@id=\"base-ui-_R_qkmpbmulb_\"]")

### Changes
ariaDiff:
  toggled:
    - button "Project": unpressed -> pressed
  typed:
    - textbox "Title": empty -> "Quarterly report"

The typed: section is new — filling a form used to diff as no change at all. Long values are cut to an excerpt with a pointer to the full text; on a page holding a document that took one envelope from 23,176 B to 4,873 B.

pw returns its expression's value under ### Value — a page.title() or locator.count() used to run and have its answer discarded. verify lists every assertion with its own PASSED/FAILED and the Playwright form of the ones that held, and gives no overall verdict; the caller reads the lines and decides.

Everything else is behind prima status <hash>, printed on each envelope's ### Instance line. Prima commands print the envelope and nothing else — the banner, config line and browser chatter were 22% of what a caller was paying for.

Verdicts belong to review, not to a loop counter

A test that stopped making progress was marked FAILED on the spot by a stall counter. A run that had already done its work and gone quiet was therefore reported as a failure — which is exactly what check then handed back. The guard now stops the loop and hands the task to final review; the verdict comes from reviewing the result. Console and network errors are recorded as page problems rather than failed steps, so they no longer sink a passing test.

Also fixed, each found live

Attach	Discovery keyed on a workspaceDir field no @playwright/cli release writes (checked 0.1.13 and 0.1.17), so prima reported no browser while a session was open and told you to open the one you already had. Attached sessions now connect through the daemon's own Playwright build — connect() succeeds across builds but ariaSnapshot does not. Pin moved to ^1.62.
Page readiness	networkidle never fires on an app with a live websocket, so every capture paid the full 6s timeout. Readiness now also completes when the DOM goes quiet: 6,034ms → 25ms per capture.
Compaction	Only the first bracket group survived, so every [pressed] / [disabled] / [checked] control lost the rest of its attributes.
Redirects	A redirect that only appends query params (/ → /?session=…&ws=1) counted as failed navigation: 3m00s of retries, or a hard failure, on a page that had loaded correctly. Now 22s.
verify	Reported claims it could not phrase as failing checks, and remembered them as such. Now separates the two. The rule that taught it to assert control state was itself wrong — { disabled: null } matches nothing — and is replaced by attribute selectors probed against a live page.
Generated tests	Visibility, hidden state and field values are emitted as real Playwright locators instead of a // TODO.

Testing

901 unit + 78 integration + 101 prima tests pass, 0 fail (2 pre-existing file-level errors in the unit scope, identical on a clean tree). New coverage for done() ending the loop, unmet instructions failing the command, a check that fails then passes on retry, expectation mapping including the unverified case, typed-value diffing, value offloading, and DOM-quiet readiness. The heal integration suite is deleted with the feature.

Verified end to end against a live app after building dist, not from source.

[github-actions]

Explorbot Self-Regression

Commit e18e545 · run

Scenario	Result	Attempts	Duration
basic (native)	PASS	3/3	19m
experience: control	OK — failed as expected	1/1	1m
experience: seeded	PASS	1/3	2m

Attempt details

• basic (native) attempt 1 — FAIL: login evidence: PASS (post-login plan=true, post-login research=true); research: PASS (files=6, wellFormed=true, keywords=6/3); scenarios: PASS (tests=6/5, features=4/3); tests passed: FAIL (4 passed, 1 failed (reporter: 4 passed, 1 failed))
• basic (native) attempt 2 — FAIL: login evidence: PASS (post-login plan=true, post-login research=true); research: PASS (files=5, wellFormed=true, keywords=7/3); scenarios: PASS (tests=18/5, features=5/3); tests passed: FAIL (4 passed, 1 failed (reporter: 4 passed, 1 failed))
• basic (native) attempt 3 — PASS: login evidence: PASS (post-login plan=true, post-login research=true); research: PASS (files=6, wellFormed=true, keywords=7/3); scenarios: PASS (tests=7/5, features=3/3); tests passed: PASS (5 passed, 0 failed (reporter: 5 passed, 0 failed))
• experience: control attempt 1 — PASS: control: OK — failed as expected (0 passed, 1 failed)
• experience: seeded attempt 1 — PASS: seeded: PASS (1 passed, 0 failed)

---

Session analysis — basic (native):

Session Analysis

The Issues feature is functional: users can create bug/feature issues, search by title, filter by label, and view issue details. All core flows work. Test automation has reliability issues with initial click actions, requiring retries or workarounds, but the product behaves correctly.

Coverage

• Pages: /issues, /issues?new=1, /issues/7
• Features: Create issue, search issues, filter by label, view issue details

What works

• Create bug issue — ET-1 Create a new bug issue and verify it appears in the list
• Create feature issue — ET-2 Create a new feature issue and verify it appears in the list
• Search by title — ET-3 Search for an issue by title and verify filtered results
• Filter by label — ET-4 Filter issues by label and verify results
• View issue details — ET-5 Navigate to issue detail page from the list

Execution Issues

• ET-1 — initial click on New Issue button failed, then succeeded via toolbar selector
• ET-2 — click on New Issue link failed, bypassed with direct URL navigation
• ET-4 — label filter dropdown failed to open multiple times, succeeded via summary element
• ET-5 — initial click on issue link failed, succeeded on second attempt