[juliusmarminge]

Computer-use PiP could show unrelated desktop pixels after a failed window capture, flash while replacing images, and remain open after a turn ended. Capture now stays bound to the selected window, retries transient failures, decodes replacement frames before painting them to a persistent canvas, and dismisses the computer preview when the turn settles. Windows packaged apps now retain their own names and native icons instead of appearing as ApplicationFrameHost.exe.

The subscriber-driven loop caps active capture at 4 FPS, leaves idle time after slow captures, suppresses identical frames, and drops static-window polling to 1 FPS after three unchanged captures. A changed frame restores the normal rate; renewed motion can take up to one second to detect. Drivers advertising display_only use an observation-only capture that skips action-registry PNG decoding and snapshot replacement. Older drivers keep the typed screenshot path. No SDK ABI change is required.

Stacked on #16532.

Windows integration

Window identities are keyed by PID and window handle, so Calculator and Settings remain distinct when Windows hosts them in one process. PiP captions and icons travel with the captured frame, display-only inspections do not redirect the preview, and native icons resolve packaged-app AUMIDs through their manifests. Duplicate detection includes app identity as well as image bytes and window metadata.

The temporary Cua Driver 0.34.1 CREATE_NO_WINDOW source patch and regression test are included in patches/cua-driver-0.34.1-windows-console.patch. T3CODE_CUA_DRIVER_SDK_LIBRARY optionally stages a locally rebuilt Windows SDK DLL into both the desktop host and server sidecar. This override is opt-in; normal release builds still use the published SDK.

Windows test build 9007 was exercised interactively: Calculator operations stayed in the background, its name/icon matched the preview, closing it showed an unavailable state, and Theo confirmed no flashing. After integration with the newer stack head: 133 focused tests pass, one is skipped, and two packaging tests fail identically on the parent commit (Linux cargo executable lookup on Windows and the cross-architecture native probe). Server/web typechecks and targeted lint pass. The combined stack has not received another packaged interactive pass.

Windows integration by GPT-6.1-Sol through the Codex harness.

Cua Driver 0.34.1

The native Linux fixes this PR used to carry as a source patch shipped upstream in Cua Driver 0.34.1 (trycua/cua#4895): the Wayland opt-in reaches embedded hosts, GNOME windows are captured from their own actor with compositor downscaling (WinRects API 10), the GNOME helper reuses one session-bus connection, GNOME's changing PNG Creation Time chunk is stripped, and get_window_state accepts display_only on Linux, macOS, and Windows with the same schema, refusal, and response fields.

The stack now pins 0.34.1 from #16527 (SDK, platform packages, and the release checksum table), and this PR packages Linux like macOS and Windows: the published driver archive is bundled, and the SDK's native library comes from npm. The source patch, scripts/build-cua-linux.sh, SDK library replacement, the Linux CLI's bundled-driver lookup, and the extra Rust toolchain and X11/Wayland dev packages in CI are removed.

One piece of the old patch is not upstream: it hid GNOME's stage-wide agent cursor during window-targeted actions and after each action ended. With 0.34.1 that cursor follows upstream behavior.

Evidence from the patched build

Measured on the patched 0.34.0 build, whose native changes 0.34.1 now ships. 0.34.1 itself has not been reprofiled.

A covered Helium window benchmark measured median native capture at 61.1 ms normally versus 38.5 ms display-only (19 measured calls per mode after warmup). For static content, normal 4 FPS capture used 14.3% driver CPU / 13.3% GNOME CPU / 4.8% device GPU busy; display-only adaptive polling used 1.8% / 6.9% / 3.8%, with 23 captures and one changed frame over 20.8 seconds. For animated content, capture stayed at 4 FPS while driver CPU fell from 14.8% to 6.1%. The persistent D-Bus transport reduced median native capture from 381 ms to 58 ms and raised capture rate from 1.29 to 4 FPS.

Installed build 2757: the covered Helium window appears correctly in the actual PiP. A 30-second static run captured 33 frames and averaged 2.65% device GPU busy and 41.7% tracked CPU (down from 7.3%/62.2% on 2755); an animated run captured 119 frames (3.96/sec) at 9.48% GPU and 62.4% CPU (down from 13.0%/72.9%). Closing PiP stopped capture completely: zero calls for 20 seconds. Tracked CPU includes T3 processes, GNOME, Cua, and Helium; one core = 100%.

Validation

• Focused tests: apps/server/src/cua, apps/desktop/src/cua, packages/shared/src/cuaDriverHostOptions.test.ts, scripts/build-desktop-artifact.test.ts, scripts/lib/cli-external-packages.test.ts: 166 pass. The one failure, "skips the primary native probe for cross-architecture Windows payloads", also fails on main.
• Typecheck clean for scripts, server, shared, and desktop; targeted lint clean.
• The upstream 0.34.1 linux-x86_64-binary archive matches the pinned SHA-256 on Linux x64, reports cua-driver 0.34.1, and its get_window_state schema advertises display_only with the description our capability probe matches.

Still draft: before/after video evidence is outstanding, and an installed 0.34.1 build has not been exercised on GNOME.

Implemented with GPT-6 Astra through the Codex harness; the 0.34.1 bump and removal of the source patch by Claude Opus 5.5 through Claude Code.

🤖 Generated with Claude Code

Application mentions

The composer can now find installed applications from the selected environment when typing @, including Helium on Windows. Selections retain the environment and launch identity, render as native app chips, and tell providers to resolve the current window before interacting. The inventory uses Cua Driver's cross-platform listApps API and stays disabled when Computer Use is off. Native icons retain a fallback until the image loads and when loading fails.

Validation: 184 focused tests pass across app inventory, reference projection, composer editing, icon loading, and authorization. Server/web typechecks passed. Windows build 9009 was exercised with an @helium mention: the provider received the correct executable and environment, launched Helium, navigated releases and discussions, and closed its test window. macOS/Linux inventory behavior has fixture coverage but has not been exercised interactively.

Application mentions implemented with GPT-6.1-Sol through the Codex harness.

[github-actions]

Thread transfer impact

✅ Thread transfer remains within every enforced ceiling.

ℹ️ No successful main baseline artifact is available yet. This run establishes the initial measurement.

Provider	Metric	Main baseline	This PR	Impact	PR ceiling
Codex	Total thread wire	—	5.0 KiB	—	6.8 KiB	✅
Codex	Thread snapshot wire	—	3.8 KiB	—	4.9 KiB	✅
Codex	Live turn WebSocket wire	—	1.2 KiB	—	2.0 KiB	✅
Codex	Live turn WebSocket decoded	—	20.9 KiB	—	29.3 KiB	✅
Codex	Live turn messages	—	2	—	8	✅
Claude	Total thread wire	—	5.0 KiB	—	6.8 KiB	✅
Claude	Thread snapshot wire	—	3.8 KiB	—	4.9 KiB	✅
Claude	Live turn WebSocket wire	—	1.2 KiB	—	2.0 KiB	✅
Claude	Live turn WebSocket decoded	—	21.2 KiB	—	29.3 KiB	✅
Claude	Live turn messages	—	2	—	8	✅

Baseline: unavailable · PR result: 5261335 · Source CI: success

Scenario and decoded snapshot size

10 historical turns, 5 command tools per turn, 878.9 KiB retained MCP result per historical turn, and a 1.05 MiB retained result in the measured turn.

• Codex decoded thread snapshot: 108.5 KiB
• Claude decoded thread snapshot: 108.8 KiB

Updated in place by a trusted workflow. PR artifacts are strictly validated and never executed.

[juliusmarminge]

terminal-preview-24 feasibility: blocked on the native stack and final-driver verification; no changes pushed.

The actual head is t3code/cua/linux-pip, based on t3code/cua/6-browser-tabs. The Cua services are absent from current main, so this draft cannot be rebuilt as a standalone main-based fix without bringing in its parent stack, which is outside this batch.

The final native pass needs:

• Installed 0.34.1 on GNOME Wayland: cover the selected target window; its preview must contain only that window. Confirm static content drops to 1 FPS, motion resumes 4 FPS, and closing the preview or settling the turn stops capture calls and closes lingering previews.
• A normal packaged Windows build without T3CODE_CUA_DRIVER_SDK_LIBRARY: start the driver and check for console flashes. The source patch is applied through the override path, so the release artifact needs its own proof. Exercise Calculator and Settings identities/icons and the unavailable state after closing each app.
• macOS: exercise the display-only capture path and app mentions/permission prompts. App enumeration currently acquires the driver from a read-scoped RPC; decide whether first-use downloads/permission prompts are intended when typing @.

Keep the app-mention feature separate from the preview lifecycle fix when the parent stack is ready. Capture the outstanding before/after recording against the final installed driver. No duplicate with #18217 or #8469 was found.