[bkudiess]

What Problem This Solves

Developers need repeatable real-app contract tests that run on native ARM64 and x64 without confusing product failures with unavailable or unsafe test conditions.

User Impact

User impact: no shipping UI or product behavior changes. Developers get an opt-in eight-case Windows app suite with isolated profiles, architecture-specific automation-tool pinning, and explicit proof results.

The suite requires a visible, unlocked desktop and no concurrent input or shared-profile writers. Provision the matching pinned lvt executable explicitly; the runner does not install it.

Why This Change Was Made

Run the real Companion app against deterministic synthetic loopback Gateways and drive the tested actions through UI Automation. Cover navigation, preference persistence in both directions, credential-free/unavailable startup, streaming, cancellation recovery, and switching saved Gateways.

The test-only harness adds controlled protocol barriers, independently identifiable peers, owned process lifetimes, and strict report/capture admission. It uses the official MCP client SDK for the external driver. The default smoke lane and existing cron/chat-send fixture scenarios remain supported.

ARM64 and x64 require different pinned lvt payloads. The runner, adapter, ownership evidence and payload snapshots now select the matching native executable hash.

Evidence

Native ARM64 local execution: 1/1 preflight and 8/8 journeys passed, zero failures or skips. All 16 owned-window screenshots were visually reviewed. Every case reported unchanged real-profile metadata and successful cleanup; source/payload freeze checks passed and all ten driver-session receipts recorded the native ARM64 pin.

This is source-equivalent pre-commit proof, not a post-commit rebuild/run. The run recorded base HEAD dc9f6345821075362580bd598cc32e4f2aa6a18a plus SHA256 values for all 45 changed files. Those hashes were verified unchanged before committing, and the final history merge preserved the exact validated source tree (d883d76eedbb5be83e40c39c3a5223c5a490c266). Do not treat that correspondence as current-head binary provenance or complete proof-pool acceptance.

At the publication check, main had advanced to 127016568ef2af52a28922bcc3ce1e2af42a0218. The PR merge base remains dc9f6345, its 45-file candidate diff is unchanged, and the new upstream commit touches no candidate-changed paths. The candidate has not been updated or revalidated against that newer main commit.

Artifacts are retained locally in the authoring session: runner output, preflight/journey TRX, suite-summary.json, before/after source and payload snapshots, per-case reports, capture receipts and screenshots. They have not been uploaded or linked publicly.

Independent pre-publication Copilot rubber-duck review of 04f8794f found no P0/P1 blockers in the inspected harness and UI-journey code. This was one reasoning pass, with unverified model identity and no helper-owned isolation or structured machine status. Chat protocol tests and several unit-test files were sampled rather than exhaustively reviewed; the finalization regression script was not inspected directly.

Non-blocking follow-up: the sentence below the coverage matrix in docs/TEST_COVERAGE.md still describes C01/C04, while the eight required cases span C01-C04. The matrix and enforced case inventory describe the broader subset. The reviewer also noted that offline pin-literal agreement is not independent binary provenance; the ARM64 executable checksum was separately derived from the digest-verified published archive and checked in the live receipts. No hosted bot review has been requested.

Change Type

• Bug fix
• Feature
• Refactor
• Docs or instructions
• Tests or validation
• Security hardening
• Chore or infrastructure

Scope

• Tray or WinUI UX
• Windows node capability
• Local MCP or winnode
• Gateway, connection, or pairing
• Setup or onboarding
• Permissions, privacy, or security
• Tests, CI, or docs

Required proof pools

• windows-11-arm64: native ARM64 driver admission, test-host and real-app execution. Focused local checks passed; complete pool acceptance and current-head binary proof remain outstanding.
• windows-winui-interactive: eight opt-in real-app UI journeys and owned-window captures. Source-equivalent local proof passed; this is not a claim that the full interactive pool ran.

Validation

All results below apply to the source-equivalent local tree described above.

Exact command	Result
.\build.ps1	Passed
dotnet build .\tests\OpenClaw.Tray.UITests\OpenClaw.Tray.UITests.csproj -c Debug -r win-arm64 -p:Platform=ARM64 -p:DevBuild=false -m:1 -p:UseSharedCompilation=false	Passed; one existing generated-code CS0618 warning
dotnet test .\tests\OpenClaw.Shared.Tests\OpenClaw.Shared.Tests.csproj --no-restore --logger "console;verbosity=minimal" -m:1 -p:UseSharedCompilation=false	4,343 passed, 0 failed, 35 skipped
dotnet test .\tests\OpenClaw.Tray.Tests\OpenClaw.Tray.Tests.csproj --no-restore --logger "console;verbosity=minimal" -m:1 -p:UseSharedCompilation=false	4,083 passed, 0 failed, 0 skipped
.\scripts\test-contract-lvt-release.ps1	Passed: x64/ARM64 pins, native default and four unsupported architectures
.\scripts\test-contract-journey-results.ps1	Passed
.\scripts\test-contract-proof-finalization.ps1	Passed: valid x64/ARM64 fixtures; 26 invalid final-gate conditions rejected
.\scripts\validate-docs.ps1	Passed
git diff --check HEAD	Passed

Focused harness command:

dotnet test .\tests\OpenClaw.Tray.IntegrationTests\OpenClaw.Tray.IntegrationTests.csproj --no-restore -c Debug -r win-arm64 -p:Platform=ARM64 -p:DevBuild=false --filter "FullyQualifiedName~LvtProtocolTests|FullyQualifiedName~LvtOwnedTargetTests|FullyQualifiedName~OwnedStdioMcpClientTests|FullyQualifiedName~ContractJourneyReportTests|FullyQualifiedName~ProfileMetadataSnapshotTests|FullyQualifiedName~GatewayFixtureProfileTests|FullyQualifiedName~GatewayFixtureRunTests" --logger "console;verbosity=minimal" -m:1 -p:UseSharedCompilation=false

Result: 109 passed, 0 failed, 0 skipped. Final broad-suite results were recorded in the validation agent's console/session, not persisted as separate Shared/Tray TRX artifacts. Live-journey TRX reports are retained locally.

The new native-driver regression failed before the ARM64 fix with PlatformNotSupportedException and passed afterward through the checksum guard.

Real Behavior Proof

• Environment tested: native Windows ARM64, .NET SDK 10.0.401, Debug app with non-Dev release identity marker, pinned ARM64 lvt v0.5.0. No attached debugger.
• PR head or commit tested: published candidate 04f8794ff06e486c8d17c52dd74325a7b673eb45; local execution preceded the commits, with source-hash/tree correspondence described under Evidence. Post-commit/current-head rebuild and execution are not verified.
• Exact steps or command run:

$env:OPENCLAW_REPO_ROOT = (Get-Location).Path
.\scripts\test-gateway-fixture.ps1 `
  -AppPath '.\src\OpenClaw.Tray.WinUI\bin\ARM64\Debug\net10.0-windows10.0.22621.0\win-arm64\OpenClaw.Tray.WinUI.exe' `
  -TestAssemblyPath '.\tests\OpenClaw.Tray.UITests\bin\ARM64\Debug\net10.0-windows10.0.22621.0\win-arm64\OpenClaw.Tray.UITests.dll' `
  -Suite Journeys `
  -LvtPath '<explicit absolute path to the verified ARM64 v0.5.0 lvt.exe>' `
  -ArtifactsDirectory '<explicit absolute path to the evidence directory>'

Only the two machine-local tool/evidence paths are replaced with placeholders.

• Evidence after fix, copied from the live runner:

VSTest version 18.7.0 (arm64)
Passed!  - Failed:     0, Passed:     1, Skipped:     0, Total:     1, Duration: 18 s - OpenClaw.Tray.UITests.dll (net10.0)
Passed!  - Failed:     0, Passed:     8, Skipped:     0, Total:     8, Duration: 1 m 31 s - OpenClaw.Tray.UITests.dll (net10.0)
Case/TRX admission passed for 8 required cases. Final qualification is still pending.
Suite qualification: passed. Metadata changes do not identify the process that wrote them.

• Observed result: all eight declared behaviors passed; metadata isolation, cleanup and final integrity gates passed. Captures show both persisted preference directions, disconnected credential-free startup, pending/streamed/final chat, cancellation and subsequent recovery, and bravo-specific identity/history after a Gateway switch.
• Screenshot or artifact links verified? No. All 16 local screenshots were inspected, but no public uploads or artifact links are claimed.
• Not verified or blocked: native x64 live execution; production Release runtime; current-head rebuilt binaries and rerun; complete named proof pools; CI results. The unavailable-Gateway capture shows Connecting/loading and does not prove a settled offline/error presentation. Synthetic peers do not establish real provider, WSL setup, installer/MSIX, MXC, or full C01-C08 contract coverage.

Security Impact

• New permissions or capabilities? No production permissions or capabilities.
• Secrets or tokens handling changed? Yes, test-only: additional generated synthetic peer credentials and isolated profiles; no installed settings, tokens or identity files are copied.
• New or changed network calls? Yes, test-only: synthetic loopback WebSocket/MCP traffic. No provider or real Gateway forwarding.
• Command or tool execution surface changed? Yes, opt-in test-only: run an explicitly provisioned, architecture-pinned lvt.exe over stdio MCP with --allow-input, attached only to the owned app process/window lifetime.
• Data access scope changed? Yes, test-only: read real roaming/local OpenClaw profile file metadata to detect shared writers; evidence does not include their file contents.
• If any answer is Yes, explain the risk and mitigation: keep every child profile isolated; redact generated credentials; reject changed/unknown driver binaries, incomplete or ambiguous UI observations, shared-profile changes, missing evidence and unsuccessful cleanup. Desktop automation is opt-in and must not overlap human input or other automation.

Compatibility and Migration

• Backward compatible? Yes: existing smoke defaults and production code are unchanged.
• Config or environment changes? Yes, test invocation only: use -Suite Journeys, explicit -LvtPath, optionally explicit -TestAssemblyPath, matching native app/test builds, and the runner's opt-in child environment.
• Migration needed? No.
• If yes, list the exact upgrade steps: provision and checksum-verify the complete matching lvt v0.5.0 archive; follow docs/GATEWAY_FIXTURE_TESTING.md for native build and isolated execution. No OpenClaw user-profile migration is needed.

Review Conversations

• I replied to or resolved every bot review conversation addressed by this PR.
• I left unresolved only conversations that still need maintainer judgment.

[clawsweeper]

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

ClawSweeper review complete

ClawSweeper finished reviewing this revision. The review result is being finalized.

View the workflow run.

[clawsweeper]

Codex review: needs changes before merge.

What this changes

This PR adds eight opt-in native Windows app journeys with synthetic Gateways, a pinned UI Automation driver, and evidence qualification.

Example: Change Show tool calls from off to on, then restart Companion.

• Before: The fixture suite has no dedicated journey verifying this preference in both directions across restart.
• After: A journey toggles the real control, checks persistence, restarts the owned app, and verifies the restored control.

Review scores

Measure	Result	What it means
Overall readiness	🦪 silver shellfish (2/6)	The testing direction is worthwhile, but a concrete admission defect remains and the supplied qualification evidence is only partially reviewable.
Proof confidence	🦪 silver shellfish (2/6)	Real behavior proof is not required for maintainer- or bot-authored pull requests.
Patch quality	🦐 gold shrimp (3/6)	1 actionable review finding remain.

Product

Kind: Test only · Worth it: Yes · Fix scope: Complete
User problem: Developers need repeatable native app checks that distinguish product failures from unsafe or unavailable test conditions.
Reason: The collaborator-authored direction is approved, and the bounded opt-in journeys add persistence, cancellation, startup, and peer-routing coverage beyond the existing browse smoke.

Merge readiness

⛔ Needs changes before merge - 2 items remain

Keep this PR open: it adds useful, owner-approved coverage absent from main, but its preflight opt-in introduces a test-selection regression.

Priority: P2
Reviewed head: 04f8794ff06e486c8d17c52dd74325a7b673eb45

Before merge

• Gate the preflight on the contract-journey opt-in (P2) - With OPENCLAW_RUN_GATEWAY_FIXTURE_UI=1 and contract journeys disabled, an unfiltered Tray.UITests run now selects this test because GatewayFixtureUiFact checks only the ordinary fixture flag. It then throws immediately because OPENCLAW_CONTRACT_REPORTS_DIR is supplied only by the journey runner. Existing fixture users therefore acquire a failing journey prerequisite without opting into that lane. Use a journey-specific fact attribute or equivalent discovery-time gate for this preflight, and verify ordinary fixture opt-in skips it while the journey runner still executes it.
• Complete next step - Inspect the failed Tray UI job's TRX or logs to determine whether its failure is introduced by this PR; the available annotation exposes only exit code 1.

Findings

• [P2] Gate the preflight on the contract-journey opt-in — tests/OpenClaw.Tray.UITests/Journeys/ContractJourneyPreflightTests.cs:10-11

Tests

• Missing end-to-end proof: The reported ARM64 journey run lacks reviewable captures and source/payload correspondence artifacts, and native x64 execution remains unverified.

Agent review details

How this fits together

The test runner launches isolated Companion instances against loopback Gateway fixtures, drives their native controls through lvt, and produces case reports, captures, and a qualification summary.

flowchart LR
Runner[Test runner] --> Fixture[Gateway fixture host]
Fixture --> App[Isolated Companion]
App --> Peers[Synthetic Gateway peers]
Runner --> Driver[Pinned lvt driver]
Driver --> App
Fixture --> Evidence[Reports and captures]
Evidence --> Qualification[Qualification gate]

Loading

Technical review

Best possible solution:

Keep one explicitly opted-in journey lane that qualifies isolated real-app behavior without changing ordinary fixture-suite admission.

Do we have a high-confidence way to reproduce the issue?

Source shows that enabling ordinary fixture UI tests without journey variables selects the new preflight and throws for its missing report directory; no target code, builds, or tests were run during this read-only review.

Is this the best way to solve the issue?

Extending the existing fixture owners is appropriate, and the deterministic barriers and separate qualification results serve distinct risks without introducing another production path.

Full review comments:

• [P2] Gate the preflight on the contract-journey opt-in — tests/OpenClaw.Tray.UITests/Journeys/ContractJourneyPreflightTests.cs:10-11
  With OPENCLAW_RUN_GATEWAY_FIXTURE_UI=1 and contract journeys disabled, an unfiltered Tray.UITests run now selects this test because GatewayFixtureUiFact checks only the ordinary fixture flag. It then throws immediately because OPENCLAW_CONTRACT_REPORTS_DIR is supplied only by the journey runner. Existing fixture users therefore acquire a failing journey prerequisite without opting into that lane. Use a journey-specific fact attribute or equivalent discovery-time gate for this preflight, and verify ordinary fixture opt-in skips it while the journey runner still executes it.
  Confidence: 0.98

Overall correctness: patch is incorrect
Overall confidence: 0.91

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning medium; reviewed against 127016568ef2.

Provenance checked

• Gateway scenarios and server keeps the original intent (3e74082: Provide deterministic populated-app testing without real Gateway provisioning, preserve read-only browse defaults, and keep full system E2E separate.)
• Fixture app readiness, lifecycle, and reporting keeps the original intent (4895ed5: Support real-app recovery proof with isolated setup roots and retain scenario-specific lifecycle ownership.)
• Fixture smoke runner keeps the original intent (3e74082: Keep desktop execution opt-in and reject skipped or zero-test runs instead of treating them as proof.)
• Duplicate UI-test project reference keeps the original intent (3a58bf3: Workspace feature integration carried an additional TestSupport reference; this PR removes the duplicate while retaining the existing reference.)

Testing

Proof path: shipped entry point.

Security

None.

Evidence

What I checked:

• Pinned introduction and current-main boundary: Reviewed merge-base-to-head hunks across the 45 introduced files. Current main has no ContractJourney implementation, and the verified synthetic merge preserves all five base-only files. (04f8794ff06e)
• Journey preflight uses the ordinary fixture opt-in: GatewayFixtureUiFact checks OPENCLAW_RUN_GATEWAY_FIXTURE_UI, whereas the preflight immediately requires OPENCLAW_CONTRACT_REPORTS_DIR; the journey cases separately check OPENCLAW_RUN_CONTRACT_JOURNEYS. (tests/OpenClaw.Tray.UITests/Journeys/ContractJourneyPreflightTests.cs:10, 04f8794ff06e)
• Full proof statement inspected: Read the complete REST body, matching supplied SHA256 95bc1ea99128241fee0185cbd1b045b8ebe95b419659fafc16e33083ba17bc65, plus both comments and reviews. It supplies copied ARM64 runner output for one preflight and eight passing journeys, while explicitly withholding local captures and source/payload manifests and disclaiming current-head binary provenance. (04f8794ff06e)
• Existing fixture ownership and intent: feat: explore and test the app without a live Gateway #1439 introduced the isolated real-app fixture and explicitly deferred canned streaming and multiple saved Gateways; fix(chat): clear stale setup warnings after recovery #1624 subsequently extended its app lifecycle and capture support. (tests/OpenClaw.GatewayFixtureHost/GatewayFixtureRun.cs:58, 127016568ef2)
• Required UI check failed, cause unavailable: The exact-head UI, functional, and accessibility job failed in Run Tray UI Tests: https://github.com/openclaw/openclaw-windows-node/actions/runs/38785972281/job/116599515294. Annotations expose only exit code 1; downloading logs was blocked at the Azure blob host, so attribution remains unresolved. (.github/workflows/ci.yml:680, 04f8794ff06e)
• Repository policy applied: Read complete root and docs AGENTS.md files; docs is the only nested policy governing changed paths. Applied test, documentation, ownership, and proof guidance; the explicit single-reviewer and read-only instructions override helper execution and validation runs. No maintainer-notes directory exists. (AGENTS.md:1, 04f8794ff06e)

Likely related people:

• Scott Hanselman: Raw commit 3e74082 adds tests/OpenClaw.TestSupport/Gateway/FixtureGatewayServer.cs:17 relative to its recorded parents. This identifies author metadata, not feature responsibility or a PR merger. (role: source-line author; confidence: high; commits: 3e740827aeb1; files: tests/OpenClaw.TestSupport/Gateway/FixtureGatewayServer.cs)
• Barbara Kudiess: Raw commit 4895ed5 adds tests/OpenClaw.GatewayFixtureHost/GatewayFixtureRun.cs:42 relative to its recorded parents. This identifies author metadata, not feature responsibility or a PR merger. (role: source-line author; confidence: high; commits: 4895ed50e39e; files: tests/OpenClaw.GatewayFixtureHost/GatewayFixtureRun.cs)

Labels

Label changes:

• add P2: Useful developer validation infrastructure with a limited test-selection defect and no shipping application changes.
• add rating: 🦪 silver shellfish: Overall readiness is 🦪 silver shellfish; proof is 🦪 silver shellfish and patch quality is 🦐 gold shrimp.
• add status: ⏳ waiting on author: ClawSweeper has contributor-facing work open and is waiting for author action.

Label justifications:

• P2: Useful developer validation infrastructure with a limited test-selection defect and no shipping application changes.
• rating: 🦪 silver shellfish: Overall readiness is 🦪 silver shellfish; proof is 🦪 silver shellfish and patch quality is 🦐 gold shrimp.
• status: ⏳ waiting on author: ClawSweeper has contributor-facing work open and is waiting for author action.

Rating scale

6/6 🦀 challenger crab · 5/6 🦞 diamond lobster · 4/6 🐚 platinum hermit · 3/6 🦐 gold shrimp · 2/6 🦪 silver shellfish · 1/6 🧂 unranked krab. Overall follows the weaker of proof and patch quality; ✨ marks media proof (a screenshot, video, or linked artifact) that directly shows the changed behavior.

Workflow

ClawSweeper edits this one comment on every review. Comment @clawsweeper re-review for a fresh review only. Repair and merge execution lanes are retired.

Reviewed October 15, 2026, 2:43 PM ET / 18:43 UTC.

[shanselman]

Hi @bkudiess, I'm a GitHub Copilot triage bot helping with repository housekeeping, not Scott. Thanks for the native app contract-testing work. Do you plan to continue this against current main? Some fixture UI coverage has since landed, so it would help to know which additional journeys you still want to bring over. If the remaining work has moved elsewhere or is no longer planned, you're welcome to close this draft with a pointer. Otherwise, a short status note is plenty. No automatic closure.