Fix what the tester is told: dead vision tools, doomed clicks, and misrouted click failures

[DavertMik]

Five fixes from a Langfuse review of the 2026-08-06 CI session (6 runs, 84 tests, 151 traces, 14,626 observations, 871 click calls of which 463 failed).

Tests attempted	84
Reached finish	47 (7 of which report failure)
Called stop	9
Never ended (max iterations/abort)	28

1. Vision tools stayed callable after the vision model died

The configured vision model returned does not exist or you do not have access to it on all 192 calls — half of every error in the session. Explorbot already degraded correctly (set visionDisabled, told the AI to use ARIA), but see and visualClick stayed in the toolset and kept answering. The AI called them 230 times and got an error back on 220 — steps burned against the iteration cap, on precisely the fallback it reaches for when a click fails.

disableVision() now withdraws both tools. Consumers spread the record fresh each iteration, so they vanish from the next prompt. In-tool guards stay for calls already emitted in the same roundtrip; pilot.pickPlanningTools() already guarded with if (see).

2. A doomed click blocked for 30 seconds

Page interactions inherited playwright.timeout, which reaches page.setDefaultTimeout(). At Playwright's 30s default, clicking a disabled or unreachable control held the test for the full 30s. 137 such timeouts in one session — roughly 69 minutes, about a third of the 3.5h wall clock.

New action.timeout (default 3000) applied to the page before each action. Measured against a real Chromium and a disabled button with playwright.timeout: 30000: 30015ms → 3004ms.

3. 187 of 463 click failures were told the wrong cause

Two misroutes, both deterministic code behavior rather than statistics:

• A container miss throws Clickable element X was not found inside element Y. That contains was not found, so it was reported as "not found in the DOM" — but the element usually exists; the container was wrong. 117 cases.
• A disabled control surfaces as a Playwright timeout, so it was reported as "covered by overlay… or use visualClick()" — no overlay is involved, a precondition is unmet, and visualClick cannot click a disabled button either. 46 cases.

Downstream cost: after a disabled failure the model went hunting for a different locator 77% of the time, and those tests recovered least often (54% vs 69% for genuine locator misses). One chain re-clicked the same disabled Select three times, then abandoned the test — and that is one of the 7 false finish calls above.

clickFailureSuggestion() classifies across all attempts rather than attempts[last] (with a fallback ladder the last error is usually the least informative) and orders existence evidence ahead of not-found: disabled → covered → hidden → container miss → absent. Replayed over the session's real failures it classifies 346/346 correctly. Covered by 6 new unit tests pinning the real CodeceptJS/Playwright error strings, since those are the drift risk on upgrade.

4. The rules taught an ARIA locator that can never match

role: "input" was listed under <good_aria_locator_example>. It is not an ARIA role. Verified against a real browser: getByRole('input') returns 0 elements for a text input, getByRole('textbox') returns 1. Same mistake in a CSS example (div[role=input]).

ARIA stays the first choice — generality and the maintainability of the generated CodeceptJS suites depend on it. What changed is that role and text must be copied from the ARIA snapshot or UI map rather than guessed, which is where the invented pairs came from. Single-command clicks using role="button" succeeded only 29% (20/68) against role="link" at 54%.

5. Container guidance contradicted itself

Two rules said containers were the default for every interaction while a third reserved them for disambiguation. Now conditional and consistent across all three places, and the containerless fallback is a stated requirement rather than a footnote — 117 failures exhausted their fallback ladder still holding a container.

Deliberately not claiming a performance win here. A paired test (same target locator tried both ways within the same test) gives 11-5 on 16 discordant pairs, p = 0.21, and no difference in 51 of 67 — the raw 56%-vs-42% gap is mostly selection bias, since the model reaches for containers on harder targets. This change removes a contradiction, nothing more.

What was deliberately not done

Flipping locator priority toward CSS. CSS scores higher (59% vs 42% on single-command calls) but the comparison is confounded the same way, and CSS-first would make the generated suites brittle and app-specific — the reason ARIA is preferred in the first place. Worth re-measuring in a paired comparison once invented pairs drop; if ARIA still lags, the fix belongs in the snapshot pipeline, not the priority ladder.

Verification

• bun test tests/unit — 847 pass / 2 fail / 2 errors; the 2 failures are identical on origin/main and pre-existing
• bun test tests/integration/ — 79 pass, 0 fail
• bun run format, bun run lint:fix — clean
• Timeout behavior and the role: "input" claim both measured end-to-end against a real Chromium
• New classifier replayed against every real failing click in the session

tests/unit/explorer.test.ts gains setDefaultTimeout on its mock page — the mock was missing a method the real Playwright Page always has.

🤖 Generated with Claude Code

[github-actions]

Explorbot Self-Regression

Commit 21f9a92 · run

Scenario	Result	Attempts	Duration
basic (native)	PASS	1/3	8m
experience: control	OK — failed as expected	1/1	2m
experience: seeded	PASS	1/3	2m

Attempt details

• basic (native) attempt 1 — PASS: login evidence: PASS (post-login plan=true, post-login research=true); research: PASS (files=7, wellFormed=true, keywords=7/3); scenarios: PASS (tests=6/5, features=5/3); tests passed: PASS (5 passed, 0 failed (reporter: 5 passed, 0 failed))
• experience: control attempt 1 — PASS: control: OK — failed as expected (0 passed, 1 failed)
• experience: seeded attempt 1 — PASS: seeded: PASS (1 passed, 0 failed)

---

Session analysis — basic (native):

Session Analysis

Explored the Issues list page with tests covering creation, search, label filtering, detail navigation, and status filtering. All core flows work correctly and verified as expected.

Coverage

• Pages: /issues
• Features: Create issue, search by keyword, filter by label, filter by status, open issue detail

What works

• Create new issue — ET-1, ET-4
• Search by keyword — ET-2
• Filter by label — ET-3
• Filter by status — ET-5
• Open issue detail — ET-4

Defects

None. All tests passed with correct verification.

UX issues

None observed.

Execution Issues

• ET-1 — required retry with alternate locator after initial click failure, but succeeded and verified correctly
• ET-4 — setup had failed attempts before succeeding, but test verification passed

[DenysKuchma]

setDefaultTimeout() mutates the Page permanently. After the first action, the 3-second action timeout also applies to later navigation, waits, and direct Playwright operations, effectively overriding playwright.timeout for the rest of the session

[DenysKuchma]

Captain keeps copies of these tools, so deleting them here has no effect there
Rebuild or filter Captain tool list on each iteration