[Lever lane] Reduce mock Gateway UI scenario failures and rerun waste #1922

Activity

1. shanselman

   added

   leverLever delivery tracking issueLever delivery tracking issue

   on Oct 29, 2026
2. shanselman

   self-assigned this

   on Oct 29, 2026
3. github-project-automation

   added this to Windows Companion Appon Oct 29, 2026
4. github-project-automation

   moved this to Backlog in Windows Companion Appon Oct 29, 2026
5. shanselman

   added a parent issue on Oct 29, 2026

   • [Lever] Reduce slow CI time-to-green without weakening validation #1919
6. clawsweeper

   added

   P2Normal priority bug or improvement with limited blast radius.Normal priority bug or improvement with limited blast radius.

   impact:otherThis issue has meaningful maintainer-visible impact outside the owned taxonomy.This issue has meaningful maintainer-visible impact outside the owned taxonomy.

   issue-rating: 🦪 silver shellfishThin issue quality; more reproduction proof or environment detail is needed.Thin issue quality; more reproduction proof or environment detail is needed.

   on Oct 29, 2026
7. clawsweeper commented on Oct 29, 2026

   

   clawsweeperBot

   on Oct 29, 2026 – with ClawSweeper

   More actions

   Codex review: this still needs some work.

   Summary
   Keep this issue open: it is a collaborator-owned experiment awaiting baseline evidence, and current main does not establish that its reliability goal has been met.

   Reproducibility: no. current failure was reproduced during this read-only review; the related visibility report supplies historical reproduction steps, but this lane still needs an exact recent failure and its artifacts.

   Next step
   Inspect exact failed-run logs and artifacts, reproduce an in-scope defect, and wait for the parent baseline watermark before evaluating a candidate.

   Review details

   Best possible solution:

   Retain only a reproduced fixture repair that preserves assertions and meets the parent experiment's reliability, duration, and budget criteria.

   Do we have a high-confidence way to reproduce the issue?

   No current failure was reproduced during this read-only review; the related visibility report supplies historical reproduction steps, but this lane still needs an exact recent failure and its artifacts.

   Is this the best way to solve the issue?

   A single measured fixture repair is appropriate after failure classification; retries, skips, or weaker visibility assertions would invalidate the stated goal.

   AGENTS.md: found and applied where relevant.

   Remaining risk / open question:

   • The reported recent CI failures are not tied to exact run IDs or inspected failure artifacts in this issue.
   • The intermittent visibility failure has no established mechanism, and its repair may require files outside this lane's ownership.
   • No completed baseline or matched reliability comparison is recorded.

   Codex review notes: model internal, reasoning medium; reviewed against c9d11df25d1b.

   Label changes

   Label changes:

   • add P2: Fixture reliability and CI rerun costs warrant normal-priority investigation without evidence of an urgent product regression.
   • add impact:other: The reported impact is unreliable automated acceptance coverage and wasted CI runner capacity.
   • add issue-rating: 🦪 silver shellfish: Current issue advisory state selects this label.

   Label justifications:

   • P2: Fixture reliability and CI rerun costs warrant normal-priority investigation without evidence of an urgent product regression.
   • impact:other: The reported impact is unreliable automated acceptance coverage and wasted CI runner capacity.

   Evidence reviewed

   What I checked:

   • Collaborator-owned bounded experiment: The issue author association is COLLABORATOR; its body requires reproduction before changes, limits ownership to the runner and fixture implementation, and records that it is waiting for the baseline watermark.
   • Parent experiment remains pending: [Lever] Reduce slow CI time-to-green without weakening validation #1919 requires a six-run same-SHA PR baseline, matched comparisons, preserved validation, and a shared runner-minute cap; its discussion contains no baseline result.
   • Current runner preserves strict failure admission: The runner propagates failed test processes and rejects missing reports, zero tests, and selected tests that did not all pass; the workflow invokes it with fixture screenshots. (scripts/test-gateway-fixture.ps1:49, c9d11df25d1b)
   • Related repairs do not prove the experiment complete: Merged chore(ci): run mock Gateway UI scenarios in pull requests #1905 repairs placement discovery and enables strict CI fixture execution; fix(tests): fixture UI checks fail on missing placement RPC and intermittent wide-window visibility #1899 explicitly retains investigation of intermittent wide-window visibility after passing runs. (tests/OpenClaw.TestSupport/Gateway/GatewayScenario.cs:264, 1912df7e2f97)
   • Verified feature history: GitHub commit patches identify shanselman as the author of the original fixture runner and karkarl as the author of its current category selection; local history also shows shared contributions to fixture scenarios and process hosting. (scripts/test-gateway-fixture.ps1:63, 1912df7e2f97)

   Likely related people:

   • Karen: Raw commit 1912df7 adds scripts/test-gateway-fixture.ps1:63 relative to its recorded parents. This identifies author metadata, not feature responsibility or a PR merger. (role: source-line author; confidence: high; commits: 1912df7e2f97; files: scripts/test-gateway-fixture.ps1)

   How this review workflow works

   • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
   • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
   • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
   • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
   • Maintainers can also comment @clawsweeper review to request a fresh review only.
   • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge; those execution lanes are retired.
   • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

   Reviewed October 29, 2026, 5:49 PM ET / 21:49 UTC.
8. shanselman commented on Oct 29, 2026

   

   shanselman

   on Oct 29, 2026

   CollaboratorAuthor

   More actions

   Coordinator update (Copilot lever agent, not a person). Local reproduction on unchanged main c9d11df: the Chat / settingsOrigin:true case fails at GatewayFixtureSettingsModalUiTests.cs:153, matching the CI signature; a separate Channels / false case timed out at line 102 and is tracked as its own finding. Ownership check: no open PR or other lane touches that test file, so this lane may edit it to add draft-checkpoint instrumentation only. Constraints: keep the final empty-draft expectation and all cases, no unbounded sleeps, no product-file edits (many open PRs touch those), no PR or CI until the A/A watermark. Parent: #1919 (Reduce slow CI time-to-green without weakening validation).
9. shanselman commented on Nov 3, 2026

   

   shanselman

   on Nov 3, 2026

   CollaboratorAuthor

   More actions

   Closing this lane as resolved by merged #1941 (fix(tests): modal smoke intermittently loses its seeded draft), with the corresponding draft-readiness coverage carried into current main by #1929 (refactor: remove the Bindings page). The old same-SHA control showed the failure on 4/6 attempts, but that is not today's failure rate. The fix adds an explicit readiness checkpoint rather than retrying, skipping, or weakening the assertion. Repeated current-main proof and timing are tracked by the refreshed parent experiment, not this lane.
10. shanselman

    closed this as completedon Nov 3, 2026
11. github-actions

    locked as resolved and limited conversation to collaborators on Nov 5, 2026