Repository navigation
feat(macos): let the agent use the localhost preview with approval - #2143
Merged
Merged
Conversation
Add the localhost_preview tool (inspect, screenshot, navigate, click, fill, select, scroll). The engine offers it only to clients that advertise clientCapabilities._meta.rustcode.preview and forwards each call as the _rustcode/preview ACP extension request, bounded by a timeout and turn cancellation; only an explicit ok answer is success. Screenshots are attached after the tool batch as image markers. The macOS app runs actions in the visible preview after the user allows access for one chat and origin, shows a revoke button, and ends grants on revoke, project switch or engine restart. Agent actions stay on the approved localhost origin, external navigation, popups and downloads are blocked, and page dialogs never stall an action. Harden the preview lifecycle (#2140): cancel downloads, keep WebKit free of retaining script message handlers, and add web view lifecycle tests plus an opt-in soak workload with recorded measurements. Refs #2138 Refs #2140 Claude-Session: https://claude.ai/code/session_01FuxRryDVUZFYjWLpkFxZ2b
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #2138
Refs #2140
Summary
The agent can now use the macOS localhost preview through a new
localhost_previewtool:inspect,screenshot,navigate,click,fill,selectandscroll. It only works after the user approves it, and the user sees every action in the preview.Engine (
rustcode/engine/src/preview.rs,acp/preview.rs)clientCapabilities._meta.rustcode.preview: 1at initialize, which only the macOS app does. Without that, the tool is left out of the native tool list and the system prompt, so the TUI, the desktop shell and other ACP clients never see it. Plan mode, subagents and compact prompts don't get it either. Calling it anyway fails and does nothing._rustcode/previewACP extension request with the session id. The engine waits at most 150 s and stops waiting as soon as the turn is cancelled. Only an explicit"ok": truecounts as success. A denial, revocation, timeout, transport error or malformed answer becomes an error result, and a timeout or cancellation says the outcome is unknown.marker. Models without image input get it described by thevision_modelfallback.App (
RustCodeKit/PreviewAgent.swift,RustCode/PreviewAgentBridge.swift)revoked.withPreviewTimeoutreturns at the limit even if a page script never finishes). The scripts run in WebKit's isolated.defaultClientworld throughcallAsyncJavaScript. There is no script message handler and no action that runs arbitrary JS. Password values are never returned, and cross-origin iframes are not read.alert/confirmare answered without a modal (so they can't stall an action), and downloads are cancelled (for the user too). The panel opens when the agent asks, the element acted on gets a brief outline, and a badge shows the current action.#2140: lifecycle audit and measurements
Code audit: the controller owns at most one
WKWebView. On a project switch it invalidates the KVO observations, clears the delegates (WebKit holds them weakly anyway), removes the view from its host and drops it. The KVO closures capture the controller weakly. The new bridge adds noWKScriptMessageHandler, so nothing inside WebKit retains the controller. I found no leak. Hardening: downloads are now cancelled explicitly (decidePolicyFor navigationResponseplusshouldPerformDownload).Regression tests (new
RustCodeTeststarget that tests the app module against real WebKit and a 127.0.0.1 test server):I checked that the first two fail when a web view is deliberately retained.
Soak workload (
PreviewSoakTests, opt-in withRUSTCODE_PREVIEW_SOAK=1). The page has 5k DOM rows, a 60 Hz timer, a canvas rAF loop and 200k objects allocated per load. Run on macOS 27.0.1 (26A434), Apple silicon:30 switches used 31 WebContent processes. None of the old ones were still running and no old web view was alive. Reloading the same page grows its WebContent process until it levels off at about 616 MB (JavaScriptCore keeps discarded heaps), and a project switch releases it completely. The app process stays flat. Numbers and method are in
docs/preview.md.What #2140 still needs (so this PR only refs it):
No pause/unload option was added, because nothing measured justifies one.
Docs
docs/preview.mdis new and covers the capability, protocol, approval and scope rules, limits and the lifecycle measurements.docs/acp.mdanddocs/install.mdare updated.Verification
cargo fmt --all -- --check,cargo check --workspace --tests,cargo test --workspaceall pass. New engine tests cover: tool hidden without the capability (and in plan mode or subagents), capability version parsing, malformed operations never forwarded, request params, success formatting, denial/error-code mapping, malformed answers never counted as success, transport error, timeout (paused clock), cancellation, and saving plus attaching a screenshot.swift buildandswift testinmacos/pass. The 13 new RustCodeKit tests cover parsing, origin and navigation scope, grant/deny/revoke, project change invalidation, one prompt at a time, approval timeout and the action timeout. The 9 app tests drive real WebKit: inspect, fill, select, click, scroll, not-found and invalid selectors, a page dialog during an action, an external link blocked, out-of-origin navigate refused, deny/revoke/project switch, no page, malformed requests, and a screenshot.https://claude.ai/code/session_01FuxRryDVUZFYjWLpkFxZ2b