Skip to content

feat(macos): let the agent use the localhost preview with approval - #2143

Merged
LHagfoss merged 2 commits into
mainfrom
feat/macos-agent-preview
Oct 19, 2026
Merged

LHagfoss merged 2 commits into
mainfrom
feat/macos-agent-preview

Conversation

@LHagfoss

Copy link
Copy Markdown
Owner

Closes #2138
Refs #2140

Summary

The agent can now use the macOS localhost preview through a new localhost_preview tool: inspect, screenshot, navigate, click, fill, select and scroll. It only works after the user approves it, and the user sees every action in the preview.

Engine (rustcode/engine/src/preview.rs, acp/preview.rs)

  • The tool is offered only when the client sends clientCapabilities._meta.rustcode.preview: 1 at initialize, which only the macOS app does. Without that, the tool is left out of the native tool list and the system prompt, so the TUI, the desktop shell and other ACP clients never see it. Plan mode, subagents and compact prompts don't get it either. Calling it anyway fails and does nothing.
  • Arguments are checked strictly before anything is sent: unknown keys, missing selectors and non-loopback navigate targets are rejected. Each call goes to the client as the _rustcode/preview ACP extension request with the session id. The engine waits at most 150 s and stops waiting as soon as the turn is cancelled. Only an explicit "ok": true counts as success. A denial, revocation, timeout, transport error or malformed answer becomes an error result, and a timeout or cancellation says the outcome is unknown.
  • Screenshots: tool results are text-only in every provider protocol, so a screenshot is saved as an image attachment and added after the tool batch as a user message marked as coming from the tool, using the ![image](file://…) marker. Models without image input get it described by the vision_model fallback.

App (RustCodeKit/PreviewAgent.swift, RustCode/PreviewAgentBridge.swift)

  • The first request from a chat shows a prompt in the preview panel with Deny and "Allow for This Chat". The grant covers one chat, one project and one origin, and is kept in memory only. An unanswered prompt is denied after 120 s, and only one prompt is open at a time. While any chat has access, an Agent button in the toolbar revokes it. Grants also end on a project switch or an engine restart, and a prompt that is still open is answered revoked.
  • Every action is checked against the grant, the project and the page origin, both before it runs and after any wait. Actions run one at a time with a 20 s limit (withPreviewTimeout returns at the limit even if a page script never finishes). The scripts run in WebKit's isolated .defaultClient world through callAsyncJavaScript. There is no script message handler and no action that runs arbitrary JS. Password values are never returned, and cross-origin iframes are not read.
  • While the agent is acting, links and popups to other sites are blocked and reported instead of being opened in the browser, alert/confirm are answered without a modal (so they can't stall an action), and downloads are cancelled (for the user too). The panel opens when the agent asks, the element acted on gets a brief outline, and a badge shows the current action.
  • The inbound loop does not wait for the approval: the request runs in its own Task.

#2140: lifecycle audit and measurements

Code audit: the controller owns at most one WKWebView. On a project switch it invalidates the KVO observations, clears the delegates (WebKit holds them weakly anyway), removes the view from its host and drops it. The KVO closures capture the controller weakly. The new bridge adds no WKScriptMessageHandler, so nothing inside WebKit retains the controller. I found no leak. Hardening: downloads are now cancelled explicitly (decidePolicyFor navigationResponse plus shouldPerformDownload).

Regression tests (new RustCodeTests target that tests the app module against real WebKit and a 127.0.0.1 test server):

  • the web view is released after a project switch
  • 5 switches keep at most one web view
  • the controller and its web view are released together after agent use

I checked that the first two fail when a web view is deliberately retained.

Soak workload (PreviewSoakTests, opt-in with RUSTCODE_PREVIEW_SOAK=1). The page has 5k DOM rows, a 60 Hz timer, a canvas rAF loop and 200k objects allocated per load. Run on macOS 27.0.1 (26A434), Apple silicon:

Phase Test process RSS WebContent RSS
before any web view 62.9 MB –
loaded 91.8 MB 78.0 MB
+30 s in window / +30 s hidden 91.6 / 91.5 MB 77.9 / 77.7 MB
25 / 50 / 75 / 100 reloads 82.3 / 82.8 / 81.6 / 81.8 MB 565.9 / 615.7 / 616.0 / 616.7 MB
10 / 20 / 30 project switches 87.8 / 88.1 / 88.2 MB ~75.8 MB (a fresh process each time)

30 switches used 31 WebContent processes. None of the old ones were still running and no old web view was alive. Reloading the same page grows its WebContent process until it levels off at about 616 MB (JavaScriptCore keeps discarded heaps), and a project switch releases it completely. The app process stays flat. Numbers and method are in docs/preview.md.

What #2140 still needs (so this PR only refs it):

  • Hours-long Instruments runs in the real app.
  • A WebSocket/HMR dev server in the workload.
  • A trustworthy visible-vs-hidden CPU comparison. The test window isn't frontmost, so WebKit may already treat the page as occluded: CPU barely moved in either state.

No pause/unload option was added, because nothing measured justifies one.

Docs

docs/preview.md is new and covers the capability, protocol, approval and scope rules, limits and the lifecycle measurements. docs/acp.md and docs/install.md are updated.

Verification

  • cargo fmt --all -- --check, cargo check --workspace --tests, cargo test --workspace all pass. New engine tests cover: tool hidden without the capability (and in plan mode or subagents), capability version parsing, malformed operations never forwarded, request params, success formatting, denial/error-code mapping, malformed answers never counted as success, transport error, timeout (paused clock), cancellation, and saving plus attaching a screenshot.
  • swift build and swift test in macos/ pass. The 13 new RustCodeKit tests cover parsing, origin and navigation scope, grant/deny/revoke, project change invalidation, one prompt at a time, approval timeout and the action timeout. The 9 app tests drive real WebKit: inspect, fill, select, click, scroll, not-found and invalid selectors, a page dialog during an action, an external link blocked, out-of-origin navigate refused, deny/revoke/project switch, no page, malformed requests, and a screenshot.
  • Not verified: an end-to-end run in the packaged app with a live model.

https://claude.ai/code/session_01FuxRryDVUZFYjWLpkFxZ2b

Add the localhost_preview tool (inspect, screenshot, navigate, click,
fill, select, scroll). The engine offers it only to clients that
advertise clientCapabilities._meta.rustcode.preview and forwards each
call as the _rustcode/preview ACP extension request, bounded by a
timeout and turn cancellation; only an explicit ok answer is success.
Screenshots are attached after the tool batch as image markers.

The macOS app runs actions in the visible preview after the user allows
access for one chat and origin, shows a revoke button, and ends grants
on revoke, project switch or engine restart. Agent actions stay on the
approved localhost origin, external navigation, popups and downloads
are blocked, and page dialogs never stall an action.

Harden the preview lifecycle (#2140): cancel downloads, keep WebKit
free of retaining script message handlers, and add web view lifecycle
tests plus an opt-in soak workload with recorded measurements.

Refs #2138
Refs #2140

Claude-Session: https://claude.ai/code/session_01FuxRryDVUZFYjWLpkFxZ2b
@github-actions github-actions Bot added the feature New feature or request label Oct 19, 2026
@LHagfoss
LHagfoss merged commit b5e1f7a into main Oct 19, 2026
8 checks passed
@LHagfoss
LHagfoss deleted the feat/macos-agent-preview branch October 19, 2026 13:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

feature New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(macos): let the agent use the localhost preview with approval

1 participant