Skip to content

fix: CI E2E shards reject invalid test reports - #1806

Closed
bkudiess wants to merge 4 commits into
bkudiess-w2a-upstream-draftfrom
bkudiess-w2e-e2e-result-admission
Closed

bkudiess wants to merge 4 commits into
bkudiess-w2a-upstream-draftfrom
bkudiess-w2e-e2e-result-admission

Conversation

@bkudiess

@bkudiess bkudiess commented Oct 21, 2026 •

Copy link
Copy Markdown
Collaborator

Depends on #1805 (fix: fixture smoke rejects invalid test reports). Target remains bkudiess-w2a-upstream-draft.

What Problem This Solves

Fixes: CI E2E shards can accept stale or internally inconsistent test reports when the command exits successfully.

User Impact

User impact: CI operators get current-invocation admission, executed/skipped counts and the actual shard/report path. Reports live under TestResults/E2E/<invocation-id>; recursive uploads retain them and sibling Gateway provenance.

Why This Change Was Made

Reuse shared strict admission while preserving ordinary CI MXC skip policy, every matching named SSH proof, selectors, native exit status and recursive artifacts. The prepared integration adopts #1831 (fix(ci): keep progressing E2E fixtures alive under the hang guard): actual stdout/stderr activity, including partial lines, renews the existing 600-second idle guard; --blame retains sequence diagnostics without memory-dump flags. The 20-minute test-step and 25-minute job limits remain unchanged. TRX and console evidence stay together beneath each invocation GUID. Controlled harnesses intercept the process-helper boundary rather than accidentally launching real dotnet.

Evidence

Current published head, 2026-10-23: fa14c500db3a5e81c1c5ad5f8b45f1d13773a025, tree ceefec433029cd8ecb6d38f37f56f8ec11ceab9b, on public parent d507c86b8d7b990b7ebacd09c625b4e683ff13ce (tree 527be21d155115c378d8e21a13d0fec857af6266). The canonical branch and PR head were verified after one exact-lease push over bca32068433d5b6141b3c0643e062d89bc17be42. Current-head CI, run 39975588653, attempt 1 is in progress; checkout/report admission and full new-head proof are pending.

Mechanical stack sync only: replayed just eb52e7dba34d78321a4c85af88321d07bc6e7e12..bca32068 onto the fixture-fix parent. All four middle-layer patches, commit messages and authors are exactly preserved; range-diff reports four equivalent commits. The parent-relative three-file layer patch is byte-identical. The entire tree delta from old bca32068 is exactly the incoming two fixture test files, 41 added lines: GatewayFixtureProtocolTests.cs and GatewayScenario.cs. No new main refresh, history folding or local execution occurred. The coordinator reports the incoming correction passed 102 focused tests and Opus found no P0-P3 findings; that is parent-scoped evidence, not full new-head proof or a new run by this layer.

Not merge-ready: retained bca32068 local scripts passed 152 checks / 93 controlled invocations, all five policy cases and the real process-lifecycle scenarios. Its full build and full Tray passed, but its full Shared failed at 670 ms against the unchanged <500 ms scaling assertion. One authorized instrumented full-Shared diagnostic passed scaling but failed a separate telemetry wait; it does not clear that failure or establish green uninstrumented proof. Source and binaries were restored and verified at the old source identity. None of those executions or old binaries is relabeled as fa14c500 proof.

Required landing constraint: SQUASH-ONLY for this middle PR. The user chose to retain the exact reviewed commit history and require squash merge when landing PR #1806. This is not merge authorization. Keep this PR draft, unmerged, without auto-merge; retain native stack 1807 / 2431235 in order 1805/1806/1808.

The coordinator's same Opus 5.5 assembled review found no runtime/integration defects in the bca32068 integration. Its sole P3 history concern is addressed by the user-selected squash-only landing constraint, not a history rewrite. Exact replay correspondence retains that recorded review without claiming a new review run. This is source-review evidence, not a passing validation gate. No fresh ClawSweeper verdict, request, resolution or ready-state change is claimed.

Retained earlier source reviews and publication history

Prior publication: bca32068433d5b6141b3c0643e062d89bc17be42, tree c18fdd7845a0e75c1056027198bc1d483d6e0bb4, on eb52e7dba34d78321a4c85af88321d07bc6e7e12 and pinned main 8ca63eb746bfafa8a31c743922b5350b839749f5, was published over 1acad8a33cef976100591d8c326ad643bebb6318. Its completed CI failure and local evidence remain below under their actual identities.

Earlier publication, 2026-10-22: head 1acad8a33cef976100591d8c326ad643bebb6318, comparison base b1f5886bfaf3ac3ed5dc8fb837ec1bcba43ff00c. Normal-pushed the reviewed test-only matrix change. Run 39755977941, attempt 1, finished FAILED: network/revocation E2E and one UI test failed. Focused contracts passed; local execution at that old head was pending. Stack order and draft state were unchanged.

Matrix correction and static review at 1acad8a3: each shard retains success, missing-report, mixed-skip and native-exit cases with its argument/filter/results-path wiring. Setup alone runs the remaining generic/path variants and retains all named MXC/SSH obligations. Its 33 expanded invocations are unchanged; each recovery shard goes from 14 to 4. Assertions, fixture Debug/Release, wrong-root/split-parent, child environment, owned cleanup and all five policy cases were preserved. The coordinator's Opus 5.5 review accepted the exact 26437a23..1acad8a3 delta and combined stack correction: both actionable P3 notes fixed, no new P0-P3 findings or coverage loss. That review was static only. Subsequent hosted execution at 1acad8a3 independently passed 152 checks / 93 controlled invocations and all five hang-guard cases; local execution at that old head was pending.

Dependency refresh at prior head 26437a23: #1824 (fix(setup): install a Node runtime supported by the Gateway) landed upstream as 51f18c1222c996fb6fa2eb4d7e8d668492103a0e at 06:52:54 UTC. This stack includes that repair rather than duplicating it. At that refresh, all five stack commits were patch-equivalent by range-diff, and the six changed files plus shared validator were byte-identical. Existing source reviews retain their recorded identities; they are not relabeled as new runs. The successful prior-head CI is retained below under its actual source and checkout identities.

Independent shared autoreview (Codex, GPT-6.1 Sol, medium, P0-P3) covered all six changed files and their full workflow/validator context at cumulative head e748ccd341f9c182a1c2088c314a676e34b8d5e9 against main e4d79798bd59af7242c75b3fb98911f27adf3817. It found one integration defect: the upstream hang-guard test harness still emitted minimal TRX incompatible with strict E2E admission. That finding superseded earlier limited-context no-findings assessments.

The correction in c4bcb3b7db1e3e6f204992d73fbf4b9c4fe8994a makes the test fixture emit contract-valid reports, requires admitted counts on positive cases and preserves the intended MXC missing-proof rejection. Production admission remains strict. A separate parent-relative corrective review completed with no P0-P3 findings and confirmed all five hang-guard cases remain intact. Subsequent CI passed the corrected workflow contract on those recorded upper heads; native validation remains pending. Source review alone is not merge readiness. The accessibility/DevBuild layer was restacked with an equivalent patch.

At the earlier 1acad8a3 update, CI was red, local execution was held and retained diagnostic binaries still needed a verified rebuild. The later bca32068 rebuild and validation outcomes are recorded below; none explains or erases the earlier failures. ClawSweeper disposition remains pending.

Change Type

  • Bug fix
  • Feature
  • Refactor
  • Docs or instructions
  • Tests or validation
  • Security hardening
  • Chore or infrastructure

Scope

  • Tray or WinUI UX
  • Windows node capability
  • Local MCP or winnode
  • Gateway, connection, or pairing
  • Setup or onboarding
  • Permissions, privacy, or security
  • Tests, CI, or docs

Required proof pools

  • none: test-report admission and CI wiring change, not a product/native contract requiring a custom pool. Controlled contracts do not establish native fixture, accessibility, Gateway or formal MXC behavior.

Validation

Current fa14c500: correspondence only, execution pending

git range-diff eb52e7dba34d78321a4c85af88321d07bc6e7e12..bca32068433d5b6141b3c0643e062d89bc17be42 d507c86b8d7b990b7ebacd09c625b4e683ff13ce..fa14c500db3a5e81c1c5ad5f8b45f1d13773a025 reported four equivalent commits. Independent full-index patch, raw commit-message and author comparison also passed. git diff --check d507c86b8d7b990b7ebacd09c625b4e683ff13ce fa14c500db3a5e81c1c5ad5f8b45f1d13773a025 passed. The combined tree changed only by the incoming parent fixture patch; these are static correspondence checks, not tests.

No local builds, tests, GUI proof or diagnostics were run for this sync. The incoming parent's reported 102 focused passes and Opus review do not establish complete validation of this head. Run 39975588653 is the newly triggered current-head CI, not a rerun of the old failure. Automatic ClawSweeper dispatch 39975583064 succeeded; dispatch is not a review verdict and no manual review request was made.

Historical bca32068: source-bound local execution

Commands below ran on Windows ARM64 with fresh isolated settings/roaming/local roots and OPENCLAW_REPO_ROOT set to the task worktree. Tests used Debug/net10.0 with no explicit RID, preserving the native default. $sharedResults, $trayResults and $diagnosticResults denote distinct private absolute result directories; only those location values are redacted. All times are UTC on 2026-10-22. Actual leaf counts, unique execution IDs and embedded report windows were inspected.

Exact command Actual result and window
pwsh -NoProfile -File .\scripts\test-proof-evidence.ps1 PASS, exit 0: 152 checks / 93 controlled invocations, 15:36:33.8025749–15:39:11.2188574.
pwsh -NoProfile -File .\scripts\test-ci-workflow-contract.ps1 PASS, exit 0: all 5 policy cases plus upstream exit/argument/output fidelity, partial-line progress, silent-root termination, live-root tree termination and exited-root inherited-pipe/reader-settlement scenarios, 15:39:13.4190384–15:40:38.7870036. The process suite ran through this command, not as an extra repeated suite.
pwsh -NoProfile -File .\build.ps1 -NoTrustRepository PASS, exit 0: all five targets (WinUI, Shared, Cli, WinNodeCli, SetupEngine), 16:46:31.9388475–16:51:20.8933419.
dotnet test .\tests\OpenClaw.Shared.Tests\OpenClaw.Shared.Tests.csproj --no-restore -c Debug --logger "trx;LogFileName=shared.trx" --results-directory $sharedResults FAIL, exit 1: 4,658 passed / 1 failed / 25 skipped, 16:51:21.7888065–16:54:01.1801922.
dotnet build .\tests\OpenClaw.Tray.Tests\OpenClaw.Tray.Tests.csproj --no-restore --no-incremental -c Debug --verbosity normal PASS, exit 0: explicitly replaced old diagnostic outputs; 0 warnings/errors, 16:56:51.8832949–16:57:06.2139813.
dotnet test .\tests\OpenClaw.Tray.Tests\OpenClaw.Tray.Tests.csproj --no-restore -c Debug --logger "trx;LogFileName=tray.trx" --results-directory $trayResults PASS, exit 0: 4,474 passed / 0 failed / 0 skipped, 16:57:09.2312473–16:59:00.0246570. No --no-build shortcut.

The focused scripts ran on an uncommitted integration snapshot at c5f61135 plus the recorded delta, then were committed as bca32068; exact file hashes establish source correspondence, not a new post-commit run. The subsequent build/Shared/Tray commands used clean committed bca32068. The reviewed matrix remains 33 setup scenarios (14 generic/path and 19 expanded named/grouped), four per recovery shard, with the other controlled contracts retained.

Unmodified-source blocker: MarkdownParserFuzzTests.Build_ManyListItems_ScalingCurve reported 10k=117ms 20k=247ms 30k=670ms; the original strict 30k < 500ms assertion failed. Test and parser source match the prepared parent; compiled PDB/source and DLL/PDB identity were checked. Source equality and a parent's pass do not establish a cause or waive this failure. Execution stopped at Shared; the independent Tray rebuild/full suite proceeded only under a separate authorization.

Tray's four corrected prerequisite theory cases all passed. WizardConsoleTailTests.IsolatedTailSurfacesPostStartRpcFailure passed in 2.1243561 s and QueuedSend_InFlightAckWithoutLifecycle_RequeuesAndRetriesSameIdempotencyKey in 0.2481773 s. These passes do not explain their earlier failures. All 479 Tray PDB documents matched current source and the rebuilt DLL/PDB identity matched before and after the full test.

Shared's 25 ordinary skips comprise 17 integration-disabled cases, seven explicitly gated native-MXC cases and one NVIDIA-GPU-unavailable case. None is claimed as complete custom proof.

One bounded diagnostic, not a green closeout

Temporary instrumentation was confined to the scaling method and its local helpers. Original input strings/sizes, one warm plus one measured parser call per size, order, Stopwatch interval, concurrency and <500ms assertion remained unchanged. Probes stayed outside the measured interval and output was buffered until all six calls finished.

Exact command Actual result and window
dotnet test .\tests\OpenClaw.Shared.Tests\OpenClaw.Shared.Tests.csproj --no-restore -c Debug --logger "trx;LogFileName=diagnostic-shared.trx" --results-directory $diagnosticResults DIAGNOSTIC FAIL, exit 1: 4,658 passed / 1 failed / 25 skipped, 19:59:05.3953078–20:01:42.8105444. Scaling passed at 28/59/79 ms, but McpHttpServerTelemetryTests.ShutdownWhileWaitingForHandler_RecordsShutdownNotBusyOrTimeout failed with Timed out waiting for telemetry.
dotnet build .\tests\OpenClaw.Shared.Tests\OpenClaw.Shared.Tests.csproj --no-restore --no-incremental -c Debug --verbosity normal PASS after byte-for-byte restoration, exit 0/0 warnings/errors, 20:01:47.7056706–20:02:32.2405069. All 162 PDB documents and DLL/PDB identity matched restored source. No uninstrumented test followed.

The diagnostic did not reproduce 670 ms or explain it. Thread CPU readings were coarse; GC observations were process-wide, and overlapping test windows establish correlation only. Instrumentation can perturb timings even outside the original measured interval. The separate telemetry failure is retained without a cause or flakiness claim. Original source is restored and the worktree is clean; original, diagnostic and restored binary identities are preserved separately, not relabeled as one execution.

Private-receipt identities and restored binary correspondence

Private raw logs and artifacts were retained locally, not uploaded as public proof links. SHA-256 identities:

  • Unmodified failed Shared TRX: 462F10F8FC2C238416C834FAC3009FA1794A79E2C59C5DBF39A0AFCDBAF2EC57.
  • Passing Tray TRX: D780583535339136109A7344660A4C29CCB62A0CF99B4C34404492B17A2014D2.
  • Instrumented Shared TRX: 0F4D120BA5AE08C6FE309BA6937D018BBDD035DA55ECDB893D4C3A1D424F681F.
  • Restored scaling source: 5291446DEDDCA7E96E12B9B3E123C900CDFD94A6EAC041C42759FCADDF825163.
  • Verified rebuilt Tray DLL: 49B74C47E0CCF3039A7293A857D5CD415E9FFBBE3529A422931C6213ABA0DF6C.
  • Post-diagnostic restored Shared DLL: 533AD9A6AC33C2BE3FCA5F5F18AD08C124A4A793D929587953CE2242101DE43B; PDB: 322AABEF986B3BAA39C97431B89EF39184D59088D3E86C34E45B6BCD7C026334.

All 1,938 tracked source hashes were restored or unchanged. The restored Shared DLL is not asserted byte-identical to the original baseline, and rebuilding is not a new passing test run.

Historical hosted CI for bca32068: run 39849023229, attempt 1 finished FAILED. Actual checkout was 022a2ba17449d333a21208c718f3413d33f48972, tree 74b20cbef70a6f5a1e57c70f2d20d534e9260d2f. It added six design/docs/instruction files beyond bca32068; relevant scripts/code/tests/build inputs matched, but the entire tree was not equal. Prior read-only collection admitted raw reports, execution windows and all three invocation-GUID consoles.

  • Hosted focused contracts passed 152/93, all five policy cases and the actual-process lifecycle scenarios. Shared passed 4,676 with eight ordinary skips; Tray passed 4,474 with no skips. These hosted passes do not clear the separate local failures.
  • Network E2E passed 2/2 and revocation E2E passed 1/1. Both retained and admitted reports alongside console evidence beneath their invocation GUIDs.
  • Setup/connect failed at the 600-second output-idle guard. The last setup journal event started wsl --install; the completed artifact retained its GUID console but no TRX. Named MXC/SSH outcomes were unavailable, not passes or skips. No underlying process/EOF or cleanup cause was established.
  • UI reported 508 passed / one failed / 33 skipped: NativeFallback_GatewayMoreRejectsAnOverlappingContinue again timed out waiting for the Gateway More button. Accessibility did not run. CI Gate received both UI and setup failures and correctly failed.

Those E2E executions resolved Gateway 2026.10.6; the earlier 2026.10.5 evidence is not substituted. Automatic ClawSweeper dispatch 39849016677 succeeded, but was not review approval. No current-head fix or flakiness claim follows from these historical results, and no diagnostic was restarted for this sync.

Retained earlier hosted execution and local preparation history

Historical 1acad8a3: test-only matrix correction

Run 39755977941, attempt 1 finished FAILED. PR head 1acad8a33cef976100591d8c326ad643bebb6318 was tested through merge checkout 7de6200b4feb69a405923fc6c7fa044856a5701d. Checkout logs in all six test jobs agree; its tree b13ea9b936171c6ce150375204fe526798d04342 exactly equals the PR-head tree. Twelve downloaded TRX reports were reconciled with actual leaves, counters and their executed-step windows.

Check Historical 1acad8a3 observed result
Focused contracts ./scripts/test-proof-evidence.ps1: PASS, 152 checks / 93 controlled invocations, logged 09:33:02.3355169 UTC. ./scripts/test-ci-workflow-contract.ps1: PASS, all 5 hang-guard cases, logged 09:33:35.8523257 UTC.
Shared / Tray PASS: 4,676 passed / 8 skipped; 4,474 passed / 0 skipped.
Connection / WinNode CLI PASS: 1,672 passed / 1 skipped; 129 passed.
SetupEngine / Tray integration / Dev containment PASS: 2,425 passed / 1 skipped; 55 passed / 3 skipped; 31 passed.
Setup/connect E2E PASS: 43 passed / 7 permitted skips. All three named SSH obligations passed; both named MXC obligations were explicitly skipped because an MXC-capable runner was not enabled.
Network E2E / Revocation E2E FAIL: both aborted at the 600-second Blame inactivity guard with zero executed cases, not failed test-body assertions.
Ordinary WinUI / Functional UI FAIL: 508 passed / 1 failed / 33 skipped; Functional UI passed 19/19. Accessibility did not run after the UI failure.
x64 publish smoke / x64 and ARM64 MSIX / bundle PASS; these do not replace the required native local full-build command.
CI Gate FAIL.

The network journal ends at wsl --install command start with no recorded completion. Revocation's journal shows wizard completion and continued Windows-node context injection immediately before termination. These are distinct observed boundaries; neither establishes why initialization was slow or whether an unfinished operation would succeed.

The separate UI failure is NativeSharedFlowTests.NativeFallback_GatewayMoreRejectsAnOverlappingContinue: Timed out waiting for Gateway More button loaded., at NativeSharedFlowTests.cs:533 via TestSupport.cs:94. No root cause or relationship to the E2E guard is established. No test or CI rerun was requested to replace these failures.

Prior-head hosted execution after Node prerequisite repair

Run 39736849196, attempt 1 finished SUCCESS. PR head: 26437a23780ec1b7d86354053a7e7d2821f71c0b. Actual tested merge checkout: aab3e7b974ed9c2d7bf26609d6c3181f545b9539. These identities come from checkout logs, not an assumption that CI ran the PR head directly.

Check Actual result
Report/evidence contracts PASS: 172 checks / 113 controlled runner invocations.
Workflow contracts and hang-guard fixture PASS: all 5 cases.
Shared / Tray PASS: 4,676 passed / 8 permitted skips and 4,474 passed / 0 skips.
Connection / WinNode CLI PASS: 1,672 passed / 1 skip; 129 passed / 0 skips.
Setup/connect E2E PASS: 43 passed / 0 failed / 7 permitted skips.
Network E2E PASS: 2/2.
Revocation E2E PASS: 1/1.
Ordinary WinUI / Accessibility PASS: 509 passed / 33 permitted skips; 23/23.
Dev containment PASS: 31/31.
CI Gate PASS.

At that Node-repair snapshot, all nine E2E setup logs across the three stack runs explicitly reported Gateway Node runtime: v24.22.1 and successful CLI installation. The old Node-version rejection was resolved in those executions. The seven setup skips remain MXC-capable-runner exclusions, not formal containment proof.

The new E2E runner successfully admitted real reports on all three shards. Setup, network and revocation output reconciles with actual downloaded TRX result rows and counters.

Hosted success does not replace the repository-required native local full build, Shared and Tray validation.

All four multiline-PowerShell variants and all six reasoning-picker variants passed on each recorded pre-dedup head. These passes do not explain or erase the earlier capture-marker failure or UI testhost crash.

Retained earlier CI and failure history

Previous-head hosted execution: 2026-10-22, before Node prerequisite refresh

Run 39733581583, attempt 1 finished FAILED. PR head: c4bcb3b7db1e3e6f204992d73fbf4b9c4fe8994a. Actual job checkout: synthetic merge 0a955f08f7d410c1cd2109a962ba6d51d2586a01, retained separately from the PR head.

Check Actual result
Focused report/evidence contracts PASS: 172 checks / 113 controlled runner invocations.
Workflow contracts and E2E hang-guard fixture PASS: all 5 hang-guard cases.
Shared suite FAIL: 4,675 passed / 1 failed / 8 skipped.
Tray suite PASS: 4,474 passed / 0 failed / 0 skipped.
Hosted x64 publish smoke, x64/ARM64 MSIX artifacts and bundle PASS. These are not the required local full-build command.
Setup/connect E2E FAIL: 20 passed / 23 failed / 7 skipped.
Network / revocation E2E FAIL: 0/2 and 0/1 passed/failed.

In the previous-head runs, all three E2E lanes were blocked before Gateway test bodies: production setup pinned Node 24.19.0, while the official installer requires 24.22.1+ or 26.12.1+. This production pin is unchanged by this stack. The existing repair is #1824 (fix(setup): install a Node runtime supported by the Gateway); it is now merged upstream and included through the base refresh, not duplicated in this stack. Setup/connect reported 20 passed / 23 failed / 7 skipped, network 0 passed / 2 failed, and revocation 0 passed / 1 failed. Those recorded CI Gates were correctly red. The completed post-repair outcomes are recorded above.

Shared testing failed in ApprovedMultilinePowerShell_ReachesRunnerVerbatim(prompt: True, unsupportedWindows: False): actual output appended \n[truncated]. Actual counts: 4,675 passed / 1 failed / 8 skipped. Connection/CLI steps did not run afterward. The C# owner is unchanged by this stack; the precise capture-interruption cause is unproved.

The original malformed-TRX fast-validation failures remain in canceled/superseded runs 39732990766 and 39732990363. Replacement runs pass the repaired gate; canceled runs are not counted as passes. No assertions, hang deadlines or skip policy were weakened.

Historical local preparation checks before bca32068

Command / check Result
git diff 26437a23780ec1b7d86354053a7e7d2821f71c0b..1acad8a33cef976100591d8c326ad643bebb6318 --check PASS, no whitespace errors. Static comparison confirms only case selection changed; assertions and fixture code outside that block are identical.
git diff b1f5886bfaf3ac3ed5dc8fb837ec1bcba43ff00c..26437a23780ec1b7d86354053a7e7d2821f71c0b --check Historical PASS at prior head.
Stack git range-diff and exact changed-file comparison across the prior Node-prerequisite refresh Equivalent implementation patches; unchanged runner/validator bytes at that refresh. Deliberate conflict repair and duplicate cleanup preceded source review.
pwsh -NoProfile -File ./scripts/test-proof-evidence.ps1 Pending locally at 1acad8a3; current-head hosted 152/93 passed separately, as recorded above.
pwsh -NoProfile -File ./scripts/test-ci-workflow-contract.ps1 Pending locally at 1acad8a3; all five hang-guard cases are statically unchanged.
./build.ps1 Was pending locally at the old public head: heavy execution was held for the occupied lane.
dotnet test ./tests/OpenClaw.Shared.Tests/OpenClaw.Shared.Tests.csproj --no-restore Was pending locally at the old public head. Required rebuild/restore checks and nonzero fresh TRX execution were still needed.
dotnet test ./tests/OpenClaw.Tray.Tests/OpenClaw.Tray.Tests.csproj --no-restore Was pending locally at the old public head. Explicit verified rebuild was needed to replace diagnostic outputs; isolated settings required.
Retained historical validation, not current-head proof

Recorded Windows ARM64 final-source closeout corresponding to old head b081589f068765d1264c8b5d7449877acf2a8a96 passed: focused contracts 173 checks / 114 controlled invocations, workflow contracts, full build, Shared 4,450 passed / 35 skipped / 0 failed, Tray 4,257 / 0 skipped / 0 failed. Execution preceded the metadata-only commit; the author recorded source correspondence, not a fresh committed-head run. Initial fresh-worktree --no-restore attempts failed with NETSDK1004; restoration and full closeout followed. Location and mapped-PSDrive cases failed before their respective corrections. Parent scaling failures are not explained or cleared by this layer.

Historical commands were pwsh -NoProfile -File ./scripts/test-proof-evidence.ps1, ./build.ps1, and the Shared/Tray commands above with fresh TRX logger/results destinations. The prior #1805 (fix: fixture smoke rejects invalid test reports) closeout used -NoTrustRepository for the full build and omitted --no-restore for tests. Ordinary allowed skips are not complete custom proof.

Later unmodified local f296ea5127c50a1be198cc318d584c528bad207f passed build/Shared (4,653 passed / 25 skipped) but failed Tray (4,467 passed / 1 failed) at WizardConsoleTailTests.IsolatedTailSurfacesPostStartRpcFailure's five-second wait. One separately authorized instrumented full-Tray diagnostic passed that target but failed the unchanged OpenClawChatDataProviderTests.QueuedSend_InFlightAckWithoutLifecycle_RequeuesAndRetriesSameIdempotencyKey assertion (4,467 passed / 1 failed). Source was restored, but diagnostic binaries then remained pending rebuild. The later verified bca32068 Tray rebuild is recorded above. Neither earlier execution is proof for 26437a23, 1acad8a3 or bca32068; neither establishes a fix or root cause for those failures.

Real Behavior Proof

Current fa14c500 limit: only the mechanical replay and fixture-only inherited tree delta were verified in this sync. New-head runtime proof is pending. Existing reviewed middle-layer source is equivalent, but the inherited test fixtures changed and historical runtime results are not new-head execution.

Retained bca32068 evidence: exact-source controlled runner execution passed 152 checks / 93 invocations, preserving failure admission, named MXC/SSH obligations and invocation-local console evidence. The workflow suite exercised actual subprocess output-idle lifecycle scenarios. These are controlled script/process contracts, not real Gateway or formal MXC proof. Full local build and Tray passed; the required unmodified Shared suite failed, and the diagnostic did not clear it.

Historical 1acad8a3 proof: run 39755977941 at merge checkout 7de6200b4feb69a405923fc6c7fa044856a5701d (tree equal to 1acad8a3) admitted the real setup/connect report with 43 passes and 7 permitted MXC skips. All three named SSH proofs passed. Network and revocation correctly returned native failure after zero-case inactivity aborts; they did not manufacture passing reports. Their scenario proof remains incomplete, and ordinary MXC skips are not formal no-skip proof.

Retained 26437a23 proof below: its runner was unchanged by the subsequent 1acad8a3 test-only matrix correction, but bca32068 now integrates the new upstream process owner. This older all-shard success is not proof for that integration and does not clear later failures.

  • Environment tested: GitHub-hosted Windows and real WSL Gateway, Node 24.22.1.
  • PR head or commit tested: 26437a23780ec1b7d86354053a7e7d2821f71c0b; actual merge checkout aab3e7b974ed9c2d7bf26609d6c3181f545b9539.
  • Exact steps or command run (workflow pwsh shell):
    • ./scripts/Invoke-CiE2e.ps1 -Name setup-connect -Filter "FullyQualifiedName~OpenClaw.E2ETests.Setup.SetupAndConnectTests|FullyQualifiedName~OpenClaw.E2ETests.Setup.MxcSetupAndConnectTests|FullyQualifiedName~OpenClaw.E2ETests.Setup.MirroredWslPortLeaseTests|FullyQualifiedName~OpenClaw.E2ETests.Setup.SshOwnershipAdversarialProofTests|FullyQualifiedName~OpenClaw.E2ETests.Setup.TrayExecutableResolutionTests"
    • ./scripts/Invoke-CiE2e.ps1 -Name network-recovery -Filter FullyQualifiedName~OpenClaw.E2ETests.Setup.NetworkRecoveryTests
    • ./scripts/Invoke-CiE2e.ps1 -Name revocation-recovery -Filter FullyQualifiedName~OpenClaw.E2ETests.Setup.RevocationAndRecoveryTests
  • Evidence after fix: setup admission reported discovered=50, executed=43, passed=43, failed=0, skipped=7; network admission reported 2/2 passing with no skips; revocation admission reported 1/1 passing with no skips. Matching TRX rows/counters were inspected.
  • Observed result: real successful E2E report admission on all shards. Ordinary setup skips are explicitly qualified.
  • Screenshot or artifact links verified? (Yes/No/N/A): Yes. Job links and matching downloaded TRX/setup artifacts were inspected. No screenshot claim.
  • Not verified or blocked: fa14c500 full local/hosted CI, fixture/runtime and ClawSweeper disposition; retained unmodified bca32068 Shared scaling failure and separately recorded diagnostic telemetry failure. Prior hosted UI/E2E failures remain unresolved history. Historical build/Tray passes and incoming parent-focused proof are not full validation of this head; no formal MXC/no-skip proof claimed.

Security Impact

  • New permissions or capabilities? (Yes/No): No
  • Secrets or tokens handling changed? (Yes/No): No
  • New or changed network calls? (Yes/No): No
  • Command or tool execution surface changed? (Yes/No): No new execution capability.
  • Data access scope changed? (Yes/No): No
  • If any answer is Yes, explain the risk and mitigation: N/A. Existing execution and owned-cleanup boundaries remain intact.

Compatibility and Migration

  • Backward compatible? (Yes/No): Yes for valid reports and existing defaults. Invalid or zero-execution reports intentionally fail.
  • Config or environment changes? (Yes/No): No new environment requirements.
  • Migration needed? (Yes/No): No.
  • If yes, list the exact upgrade steps: N/A. Use emitted invocation paths instead of assuming fixed report paths.

Review Conversations

  • I replied to or resolved every bot review conversation addressed by this PR.
  • I left unresolved only conversations that still need maintainer judgment.

Earlier conversation handling remains in the retained history; current-head ClawSweeper disposition is pending. No fresh review request or resolution is claimed. Required future landing: PR #1806 must be squash-merged; retain the native stack workflow and obtain separate merge authorization.

@clawsweeper

clawsweeper Bot commented Oct 21, 2026 •

Copy link
Copy Markdown

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

ClawSweeper review complete

ClawSweeper finished reviewing this revision. The review result is being finalized.

View the workflow run.

@bkudiess
bkudiess added this pull request to stack #1807 October 21, 2026 17:12
@clawsweeper clawsweeper Bot added P2 Normal priority bug or improvement with limited blast radius. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. labels Oct 21, 2026
@clawsweeper

clawsweeper Bot commented Oct 21, 2026

Copy link
Copy Markdown

Codex review: needs changes before merge.

What this changes

The PR gives each CI E2E invocation a fresh report directory and validates report consistency before accepting shard results.

Example: The network-recovery command exits 0 but leaves only yesterday's report with executed=12.

  • Before: The fixed-path parser can accept the stale report.
  • After: The runner requires a fresh report in its new invocation directory and rejects the missing current report.

Review scores

Measure Result What it means
Overall readiness 🦐 gold shrimp (3/6) Useful, focused and source-correct, with substantial controlled regression evidence but limited directly inspectable runtime proof.
Proof confidence 🦐 gold shrimp (3/6) This collaborator-authored PR is outside the ordinary contributor proof gate; current-head Windows ARM64 evidence reports controlled runner regressions with synthetic reports, while hosted E2E success metadata supplies no inspectable runtime trace because log retrieval was blocked.
Patch quality 🐚 platinum hermit (4/6) No actionable review findings were identified.

Product

Kind: Bug fix · Worth it: Yes · Fix scope: Complete
User problem: CI operators can receive successful E2E results from stale or internally inconsistent test reports.
Reason: The collaborator-authored direction strengthens established report semantics while preserving shard selection and ordinary MXC versus mandatory SSH skip policy.

Merge readiness

⛔ Needs changes before merge - 1 item remains

Keep open: this collaborator-authored PR repairs report admission still missing on main, with no definite introduced defect found; its dependent draft, reported conflict, and unresolved CI failure prevent landing.

Priority: P2
Reviewed head: b081589f068765d1264c8b5d7449877acf2a8a96

Before merge

  • Complete next step - Resolve the reported conflict in stack order, retain main's hang-detection arguments and update the exact-argument regression accordingly, then inspect the failing Tray UI job and establish its cause before landing.

Findings

None.

Tests

  • Missing end-to-end proof: Supplied output exercises synthetic TRX through a controlled dotnet boundary; direct logs showing the unmodified hosted runner consuming real reports were inaccessible.
Agent review details

How this fits together

GitHub Actions invokes the PowerShell E2E runner with a shard selector; dotnet produces TRX reports that the runner validates before reporting success and uploading recursive artifacts.

flowchart LR
  A[GitHub Actions shard] --> B[E2E runner]
  B --> C[Dotnet test]
  C --> D[Invocation TRX]
  D --> E[Shared report validator]
  E --> F[Named proof checks]
  F --> G[Shard outcome]
Loading

Technical review

Best possible solution:

Integrate the shared-validator cutover while preserving named-proof policy, recursive artifacts, and main's newer hang-detection behavior.

Do we have a high-confidence way to reproduce the issue?

Current main can accept an old report with a positive executed counter after a successful command, establishing the failure path by source inspection; no target code was run during this review.

Is this the best way to solve the issue?

Reusing the existing strict validator fixes admission at the runner boundary and removes the weaker parser without creating another validation owner.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning medium; reviewed against 4dea62313040.

Provenance checked

  • E2E runner admission and named obligations keeps the original intent (969fe6d: Split existing CI E2E lanes conservatively while preserving actual execution and named MXC/SSH obligations.)
  • Shared admission and regression coverage keeps the original intent (3977f82: Reject stale or incomplete validation through invocation-specific directories and shared strict report admission while retaining permitted ordinary CI skips.)
  • Parent fixture regression harness keeps the original intent (bf3b60e: Exercise fixture report admission with a controlled launch boundary; this PR appends separate E2E runner coverage without changing parent behavior.)

Testing

Proof path: in-process harness.

Security

None.

Evidence

What I checked:

  • Introduced change: The pinned introduction replaces counter-only admission with New-TestInvocation and Read-TestResultContract, preserves native failure exit codes, and checks all matching named proof results. (scripts/Invoke-CiE2e.ps1:22, b081589f0687)
  • Current main still needs admission repair: Fetched main still reads the fixed TRX path and trusts its executed counter without checking freshness or reconciling leaf results. (scripts/Invoke-CiE2e.ps1:37, 4dea62313040)
  • Established admission contract: The existing shared owner validates timestamps, XML namespace, identities, counters, leaf outcomes, execution and skip policy; improve: make agent guidance and validation more reliable #1724 (improve: make agent guidance and validation more reliable) established this behavior. (scripts/TestResultContract.ps1:17, 3977f8206ed4)
  • Artifact compatibility: The existing CI caller uploads TestResults/E2E recursively, retaining invocation reports and sibling Gateway provenance without a competing report consumer. (.github/workflows/ci.yml:846, b081589f0687)
  • Complete discussion and supplied validation: The complete body matches supplied SHA256 9f8e2d90c6351cf19c79cb03d1d4503ab912ad7755de05428cf01aa0e56703e0. It reports 173 checks including 114 controlled runner invocations, passing build/Shared/Tray validation, and explicitly distinguishes parent validation from this head. Both discussion comments are bot status messages. (b081589f0687)
  • Hosted results and inspection limit: Exact-head checks show all three E2E shards passing and Tray UI testing failing, with CI Gate also failing. Job logs redirected to a blocked Azure storage domain, so the UI failure could not be attributed to the diff or independently established as external. (b081589f0687)

Likely related people:

  • Scott Hanselman: Raw commit 969fe6d adds scripts/Invoke-CiE2e.ps1:29 relative to its recorded parents. This identifies author metadata, not feature responsibility or a PR merger. (role: source-line author; confidence: high; commits: 969fe6dad230; files: scripts/Invoke-CiE2e.ps1)
  • Barbara Kudiess: Raw commit 3977f82 adds scripts/TestResultContract.ps1:17 relative to its recorded parents. This identifies author metadata, not feature responsibility or a PR merger. (role: source-line author; confidence: high; commits: 3977f8206ed4; files: scripts/TestResultContract.ps1)

Review metrics

Metric Value Why it matters
Runner and regression scope Runner +49/-36 lines; tests +216/-2 lines Production changes reuse the existing validator; added controlled cases exercise shard wiring, rejection paths, named obligations and artifact locations.

Labels

Label changes:

  • add P2: Incorrect E2E report admission undermines CI reliability without establishing an urgent product-runtime regression.
  • add rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🦐 gold shrimp and patch quality is 🐚 platinum hermit.
  • add status: ⏳ waiting on author: ClawSweeper has contributor-facing work open and is waiting for author action.

Label justifications:

  • P2: Incorrect E2E report admission undermines CI reliability without establishing an urgent product-runtime regression.
  • rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🦐 gold shrimp and patch quality is 🐚 platinum hermit.
  • status: ⏳ waiting on author: ClawSweeper has contributor-facing work open and is waiting for author action.

Rating scale

6/6 🦀 challenger crab · 5/6 🦞 diamond lobster · 4/6 🐚 platinum hermit · 3/6 🦐 gold shrimp · 2/6 🦪 silver shellfish · 1/6 🧂 unranked krab. Overall follows the weaker of proof and patch quality; ✨ marks media proof (a screenshot, video, or linked artifact) that directly shows the changed behavior.

Workflow

ClawSweeper edits this one comment on every review. Comment @clawsweeper re-review for a fresh review only. Repair and merge execution lanes are retired.

Reviewed October 21, 2026, 2:03 PM ET / 18:03 UTC.

@roboclaw-bot roboclaw-bot added the status: 🚢 actively landing A maintainer or agent is actively driving this item through implementation, validation, or merge. label Oct 22, 2026
@roboclaw-bot
roboclaw-bot force-pushed the bkudiess-w2e-e2e-result-admission branch from b081589 to 0a26aac Compare October 22, 2026 06:17
@roboclaw-bot

roboclaw-bot commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

Preparation follow-up at current head 26437a23780ec1b7d86354053a7e7d2821f71c0b.

Conflicts and scoped cleanup are published. Independent full-context review identified an invalid legacy TRX fixture; that fixture was corrected without weakening production admission, and its corrective review found no P0-P3 issue. The stack includes upstream Node-runtime repair #1824 (fix(setup): install a Node runtime supported by the Gateway).

Latest Build and Test, attempt 1: SUCCESS. Actual checkout: aab3e7b974ed9c2d7bf26609d6c3181f545b9539. Controlled contracts passed 172 checks / 113 runner invocations; all five hang-guard cases passed.

Real E2E report admission passed on setup (43 passed / 7 permitted skips), network (2/2) and revocation (1/1).

Node 24.22.1 installation was verified. All earlier failing multiline-PowerShell/reasoning-picker variants passed in the new run, without claiming their previous failure causes were fixed.

Remaining required native local build/Shared/Tray validation needs an authorized Windows host. Actual fixture-smoke runtime proof remains absent. All three PRs remain drafts; nothing was merged or auto-merged. Earlier failures, cancellations and source identities remain in the PR body.

@roboclaw-bot roboclaw-bot removed the status: 🚢 actively landing A maintainer or agent is actively driving this item through implementation, validation, or merge. label Oct 22, 2026
@roboclaw-bot roboclaw-bot added the status: 🚢 actively landing A maintainer or agent is actively driving this item through implementation, validation, or merge. label Oct 22, 2026
@roboclaw-bot
roboclaw-bot force-pushed the bkudiess-w2e-e2e-result-admission branch from c4bcb3b to 26437a2 Compare October 22, 2026 06:53
@roboclaw-bot roboclaw-bot removed the status: 🚢 actively landing A maintainer or agent is actively driving this item through implementation, validation, or merge. label Oct 22, 2026
@bkudiess bkudiess added the status: 🚢 actively landing A maintainer or agent is actively driving this item through implementation, validation, or merge. label Oct 22, 2026
@bkudiess
bkudiess force-pushed the bkudiess-w2e-e2e-result-admission branch from 1acad8a to bca3206 Compare October 22, 2026 20:27
@bkudiess bkudiess added status: 🚢 actively landing A maintainer or agent is actively driving this item through implementation, validation, or merge. and removed status: 🚢 actively landing A maintainer or agent is actively driving this item through implementation, validation, or merge. labels Oct 22, 2026
Copilot and others added 4 commits October 23, 2026 11:05
Use invocation-specific strict TRX admission while preserving CI skip distinctions, named MXC and SSH obligations, and report diagnostics. Cover malformed reports, artifact roots, and PowerShell provider locations with controlled runner regressions.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 94b03923-ddf0-49eb-bc89-922312a24b6d
Co-authored-by: bkudiess <7658216+bkudiess@users.noreply.github.com>
Co-authored-by: bkudiess <7658216+bkudiess@users.noreply.github.com>
Exercise common admission and path variants once on setup-connect while retaining per-shard routing, mixed-skip, missing-report and native-exit checks. Keep named MXC/SSH, hang, fixture environment and cleanup assertions unchanged.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 94b03923-ddf0-49eb-bc89-922312a24b6d
Intercept the process-helper boundary beside an unchanged copied runner and validator. Preserve report mutations, named proofs, environment and path assertions while requiring invocation-local console evidence and the existing idle deadline.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 94b03923-ddf0-49eb-bc89-922312a24b6d
@bkudiess
bkudiess force-pushed the bkudiess-w2e-e2e-result-admission branch from bca3206 to fa14c50 Compare October 23, 2026 15:07
@bkudiess bkudiess closed this Oct 24, 2026
@bkudiess bkudiess removed the status: 🚢 actively landing A maintainer or agent is actively driving this item through implementation, validation, or merge. label Oct 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

P2 Normal priority bug or improvement with limited blast radius. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants