feat(desktop): Computer History for macOS, Windows, and Linux by sheehanmunim · Pull Request #6668 · pingdotgg/t3code

sheehanmunim · 2026-08-15T07:38:48Z

Summary

Adds opt-in Computer History: accessibility interaction capture (not screenshots) → local segment store → summarized memories agents can reference, with optional Codex skysight mirror.
Desktop owns the recorder daemon (macOS AX via Swift MCP; Windows UI Automation + Linux AT-SPI/EWMH via Rust MCP); server summarizes on an interval and injects recent history into provider context.
Settings UI under Computer History (enable / pause / mirror / clear / timeline), plus Linux FRONTMOST sampling fixes and a Linux Xvfb+Openbox smoke harness.

Depends on #6537 (mac/computer-use). This branch is stacked on that work — for a Computer History–only diff, compare against mac/computer-use.

Demo

computer-use-demo.mp4

Settings → Computer Use:

Test plan

macOS Dev — enable persists; daemon running; events collected; summaries + Codex mirror
Windows Blade (interactive session) — Notepad/Calculator sample.frontmost events; memory written; Dev coexists with Nightly (phase=running, events continuing)
Linux (Colima Ubuntu aarch64) — build t3-desktop-mcp; Xvfb + Openbox records sample.frontmost for XTerm (reconfirmed after PR open)
Shared store unit tests — packages/shared/src/computerHistory/store.test.ts (6/6 passing)
Review Settings → Computer History in desktop Dev and confirm Accessibility / UI Automation prompts as needed
Confirm disabling stops the daemon and clears injection from new agent turns

Note

Add Computer History recording and Desktop MCP server for macOS, Windows, and Linux

Introduces a new Rust-based Desktop MCP server (native/t3-desktop-mcp-rs) for Windows and Linux, and a Swift-based server (native/t3-desktop-mcp) for macOS, exposing accessibility-driven computer-use tools over stdio JSON-RPC.
Adds a Computer History daemon that records frontmost app and website activity into JSONL segment files, with filtering, pausing, and periodic summarization (10-minute and 6-hour memory levels).
Integrates Computer History context into all AI provider adapters (Claude, Codex, Cursor, Grok) so active session prompts include recent activity when the feature is enabled.
Adds a Chrome extension (native/t3-chrome-extension) with a native messaging bridge that connects the browser to the desktop MCP server, enabling browser control tools.
Adds /settings/computer-use and /settings/computer-history routes and settings UI, with IPC channels for permissions querying, history status/timeline, patching settings, and clearing history.
Adds a platform agent cursor overlay (Windows, Linux, macOS) that shows a visual pointer during desktop tool invocations and fades after completion.
Extends CanonicalRequestType and approval handling across providers to support tool_approval and permissions_approval request kinds, with distinct UI labels and timeline icons.
Risk: The daemon runs as a background process sampling accessibility state every ~2 seconds; on unsupported display servers (e.g. Wayland with no XWayland) capture operations return errors rather than crashing, but some tools may be silently unavailable.

^{Macroscope summarized d9b242d.}

Note

High Risk
Spawns native binaries with desktop automation, continuous local activity recording when enabled, and broad provider/MCP integration across auth-adjacent approval flows.

Overview
Computer Use and Computer History are wired end-to-end: settings pages, desktop IPC, a spawned t3-desktop-mcp recorder daemon, and optional injection of recent history into agent turns.

Computer Use adds settings for enabling desktop control, agent cursor overlay, and Chrome browser control. The desktop app exposes macOS accessibility/screen-recording status and Chrome extension detection via IPC. When enabled, Claude, Codex, Cursor, and Grok sessions attach a stdio t3-desktop MCP server (alongside existing t3-code), resolved per session from desktopControl settings. Codex advertises form elicitation; the runtime maps MCP permission and generic tool approvals into permissions_approval / tool_approval with matching UI in the composer and timeline. macOS Info.plist gains NSAppleEventsUsageDescription so Automation prompts work for spawned MCP tools.

Computer History adds a ComputerHistoryManager that writes control.json, spawns computer-history on the desktop MCP binary at bootstrap (if already enabled) and stops it cleanly on quit. Server-side background work syncs control and summarizes segments every minute; Codex turns can append loaded history to developer instructions. Settings cover enable/pause, Codex mirror, privacy exclusions, timeline clear/reveal/delete, with status polls that do not respawn the daemon after disable.

^{Reviewed by Cursor Bugbot for commit e4775dc. Bugbot is set up for automated code reviews on this repo. Configure here.}

Note

Replaces #6579. Same commits/HEAD intent; opened fresh because Macroscope UI Consistency on #6579 was stuck failing with retained findings that no longer match HEAD (check kept claiming only native commit e1f85f3 changed web UI).

Made with Cursor

Note

Replaces #6664 (which replaced #6579). Same branch tip intent. Opened fresh because Macroscope Effect Service Conventions on #6664 retained a stale finding against deleted apps/desktop/src/computerHistory/manager.ts while UI Consistency, Correctness, and Bugbot were already green with 0 open review threads.

Made with Cursor

coderabbitai · 2026-08-15T07:38:59Z

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: cbdecddf-b740-4913-8216-43bfa76250cf

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

macroscopeapp

One convention issue found: a compatibility re-export shim was added for the new Computer History service module. Everything else in the touched Effect service code (namespace subpath imports, inline Context.Service interface, make + layer exports, environment-acquired dependencies, Effect.catchTags) matches the conventions.

^{Posted via Macroscope — Effect Service Conventions}

macroscopeapp · 2026-08-15T07:45:57Z

Approvability

Verdict: Needs human review

1 blocking correctness issue found. Diff is too large for automated approval analysis. A human reviewer should evaluate this PR.

^{You can customize Macroscope's approvability policy. Learn more.}

Match website filters against page host/path only, drop cross-window browser context fallback, keep AppleScript agent windows across brief AX gaps, and exact-match numeric app names that are not live PIDs. Co-authored-by: Cursor <cursoragent@cursor.com>

Strip query/fragment per URL candidate so title punctuation like ? or # cannot truncate the real page URL before host extraction. Co-authored-by: Cursor <cursoragent@cursor.com>

Keep exclusions working for http://[::1] and <https://host> haystacks. Co-authored-by: Cursor <cursoragent@cursor.com>

Outline/link URLs must not drive includeOnly or exclude matching. Co-authored-by: Cursor <cursoragent@cursor.com>

Prevent includeOnly path needles from matching substring paths on unrelated hosts. Co-authored-by: Cursor <cursoragent@cursor.com>

Preserve absolute path needles, accept full-URL rules, match localhost host/path filters, and require path segment boundaries. Co-authored-by: Cursor <cursoragent@cursor.com>

example.com/ and https://example.com match every page on that host, and trailing slashes no longer break full-URL path filters. Co-authored-by: Cursor <cursoragent@cursor.com>

Trim leading and trailing slashes so //private matches across platforms. Co-authored-by: Cursor <cursoragent@cursor.com>

Address open High/Medium review threads: nonblocking bridge writes, focused-window titles and segment rotation for History, owner-only agent-cursor sockets, no WAYLAND_DISPLAY env mutation, Wayland type_text/X11 cookie checks, private-mode and outline URL hardening, create-first bridge binding, and inert browser when disabled. Co-authored-by: Cursor <cursoragent@cursor.com>

Use MSG_DONTWAIT instead of O_NONBLOCK so NativeHost reads stay blocking, revalidate the bridge fd under writeLock, stop fabricating FRONTMOST apps, propagate Shift release errors, and never resurrect enabled from control.json. Co-authored-by: Cursor <cursoragent@cursor.com>

Codex already received recent history on sendTurn; load the same block for Claude, Cursor, and Grok so enabled History works regardless of harness. Co-authored-by: Cursor <cursoragent@cursor.com>

Stop the server from rewriting control.json, treat intentional History stops as stopped, and harden shared Computer Use paths (socket ownership, Int traps, AppleScript escaping, scroll/click targeting, bridge writes, AT-SPI retry). Co-authored-by: Cursor <cursoragent@cursor.com>

Use a transparent circular badge (no black square crop) and rounded-full clipping so the Computer Use settings row matches the pointer favicon. Co-authored-by: Cursor <cursoragent@cursor.com>

Point the Computer Use settings icon at the same soft-glow cursor-112 artwork as the desktop overlay, on a transparent canvas with no rounded crop. Co-authored-by: Cursor <cursoragent@cursor.com>

Match Computer Use tip so the lavender radial glow does not square-crop at settings row size. Co-authored-by: Cursor <cursoragent@cursor.com>

Keep the original pointer art and fade the glow into the badge rim so Settings and Chrome tab favicons no longer show a nested square crop. Co-authored-by: Cursor <cursoragent@cursor.com>

…ters Reject overlapping cross-window element drags, refuse symlink bridge dirs, recognize Safari TP/Edge Canary for private sticky, and ignore empty app-filter needles. Co-authored-by: Cursor <cursoragent@cursor.com>

Map press_key, harden Linux/Windows click/type paths, clear stale Chrome window cache, escape app-list brackets, recognize Mozilla Firefox, and honor in-memory mirror settings on clear/status. Co-authored-by: Cursor <cursoragent@cursor.com>

Only clear the daemon child after confirmed exit (or failed signal delivery), so a timed-out stop cannot report success while capture continues. Co-authored-by: Cursor <cursoragent@cursor.com>

macroscopeapp · 2026-08-16T17:24:12Z

+    persisted: ComputerHistorySettings,
+    patch: Partial<ComputerHistorySettings>,
+  ): Promise<ComputerHistorySettings> => {
+    const base = lastMergedSettings ?? persisted;


🟠 High computerHistory/ComputerHistoryManager.ts:328

After the first patch, mergePatchSettingsImpl and the getStatus, clear, and removeMemory handlers keep using lastMergedSettings instead of newly persisted settings. A later partial patch therefore resurrects obsolete fields and can report or apply incorrect enabled, paused, filter, or Codex-mirroring values until restart. Reconcile or invalidate the optimistic snapshot against current persisted settings before using it.

🤖 Copy this AI Prompt to have your agent fix this:

In file @apps/desktop/src/computerHistory/ComputerHistoryManager.ts around line 328: After the first patch, `mergePatchSettingsImpl` and the `getStatus`, `clear`, and `removeMemory` handlers keep using `lastMergedSettings` instead of newly persisted settings. A later partial patch therefore resurrects obsolete fields and can report or apply incorrect `enabled`, `paused`, filter, or Codex-mirroring values until restart. Reconcile or invalidate the optimistic snapshot against current persisted settings before using it.

…ail-closed Close app-list ids at the first unescaped bracket, and only fall back from AT-SPI to X11 when there is no match — not when the query is ambiguous. Co-authored-by: Cursor <cursoragent@cursor.com>

cursor

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit d9b242d. Configure here.}

cursor · 2026-08-16T17:29:43Z

+        timer = setTimeout(() => {
+          // Failsafe if the OS never emits `exit` after SIGKILL.
+          finish();
+        }, 3_000);


Stop clears before exit

Medium Severity

The new SIGKILL path can call finish() (via !signaled or the 3s failsafe) and clear state.stopping before the child’s exit event runs. Disable then writes stopped via writeStoppedStatus, but a late exit handler still sees state.stopping === null and overwrites status with unavailable, so an intentional stop can look like a daemon crash and surface lastError even when Computer History is disabled.

Additional Locations (2)

apps/desktop/src/computerHistory/ComputerHistoryManager.ts#L257-L263

apps/desktop/src/computerHistory/ComputerHistoryManager.ts#L301-L319

^{Reviewed by Cursor Bugbot for commit d9b242d. Configure here.}

github-actions Bot added vouch:unvouched PR author is not yet trusted in the VOUCHED list. size:XXL 1,000+ changed lines (additions + deletions). labels Aug 15, 2026

macroscopeapp Bot reviewed Aug 15, 2026

View reviewed changes

Comment thread apps/desktop/src/computerHistory/manager.ts Outdated

macroscopeapp Bot reviewed Aug 15, 2026

View reviewed changes

Comment thread native/t3-desktop-mcp/Sources/main.swift Outdated

macroscopeapp Bot reviewed Aug 15, 2026

View reviewed changes

Comment thread native/t3-desktop-mcp/Sources/ComputerHistory.swift

macroscopeapp Bot reviewed Aug 15, 2026

View reviewed changes

Comment thread native/t3-desktop-mcp/Sources/ComputerHistory.swift Outdated

Comment thread native/t3-desktop-mcp/Sources/ComputerHistory.swift Outdated

cursor Bot reviewed Aug 15, 2026

View reviewed changes

Comment thread native/t3-desktop-mcp/Sources/main.swift

Comment thread native/t3-desktop-mcp/Sources/main.swift

cursor Bot reviewed Aug 15, 2026

View reviewed changes

Comment thread native/t3-desktop-mcp-rs/src/history.rs

macroscopeapp Bot reviewed Aug 15, 2026

View reviewed changes

Comment thread native/t3-desktop-mcp-rs/src/history.rs Outdated

Comment thread native/t3-desktop-mcp-rs/src/history.rs Outdated

cursor Bot reviewed Aug 15, 2026

View reviewed changes

Comment thread native/t3-desktop-mcp-rs/src/history.rs

macroscopeapp Bot reviewed Aug 15, 2026

View reviewed changes

Comment thread native/t3-desktop-mcp-rs/src/history.rs Outdated

cursor Bot reviewed Aug 15, 2026

View reviewed changes

Comment thread native/t3-desktop-mcp-rs/src/history.rs

macroscopeapp Bot reviewed Aug 15, 2026

View reviewed changes

Comment thread native/t3-desktop-mcp/Sources/ComputerHistory.swift Outdated

Comment thread native/t3-desktop-mcp/Sources/ComputerHistory.swift Outdated

Comment thread native/t3-desktop-mcp/Sources/ComputerHistory.swift Outdated

cursor Bot reviewed Aug 15, 2026

View reviewed changes

Comment thread native/t3-desktop-mcp-rs/src/history.rs Outdated

Comment thread native/t3-desktop-mcp-rs/src/history.rs

cursor Bot reviewed Aug 15, 2026

View reviewed changes

Comment thread native/t3-desktop-mcp-rs/src/history.rs Outdated

Comment thread native/t3-desktop-mcp-rs/src/history.rs Outdated

cursor Bot reviewed Aug 15, 2026

View reviewed changes

Comment thread native/t3-desktop-mcp-rs/src/history.rs

sheehanmunim force-pushed the feat/computer-history-v3 branch from 02174c1 to c04c7d9 Compare August 15, 2026 16:38

macroscopeapp Bot reviewed Aug 15, 2026

View reviewed changes

cursor Bot reviewed Aug 15, 2026

View reviewed changes

Comment thread apps/desktop/src/computerHistory/ComputerHistoryManager.ts

Comment thread apps/server/src/provider/Layers/CodexAdapter.ts

macroscopeapp Bot reviewed Aug 15, 2026

View reviewed changes

cursor Bot reviewed Aug 15, 2026

View reviewed changes

Comment thread native/t3-desktop-mcp/Sources/BrowserBridge.swift

macroscopeapp Bot reviewed Aug 15, 2026

View reviewed changes

Comment thread native/t3-desktop-mcp-rs/src/platform/linux.rs

sheehanmunim force-pushed the feat/computer-history-v3 branch from 12f47ca to 4609222 Compare August 16, 2026 06:30

cursor Bot reviewed Aug 16, 2026

View reviewed changes

Comment thread apps/server/src/computerHistory/service.ts Outdated

Comment thread apps/desktop/src/computerHistory/ComputerHistoryManager.ts

macroscopeapp Bot reviewed Aug 16, 2026

View reviewed changes

Comment thread native/t3-desktop-mcp/Sources/main.swift Outdated

Comment thread native/t3-desktop-mcp-rs/src/history.rs

Comment thread native/t3-desktop-mcp-rs/src/browser.rs Outdated

Comment thread native/t3-desktop-mcp-rs/src/history.rs Outdated

cursor Bot reviewed Aug 16, 2026

View reviewed changes

Comment thread native/t3-desktop-mcp-rs/src/history.rs

sheehanmunim and others added 17 commits August 16, 2026 13:08

fix(history): match website filters on URL tokens, not titles

0557b1a

Strip query/fragment per URL candidate so title punctuation like ? or # cannot truncate the real page URL before host extraction. Co-authored-by: Cursor <cursoragent@cursor.com>

fix(history): parse IPv6 and angle-bracket URL hosts for filters

3c8ff7a

Keep exclusions working for http://[::1] and <https://host> haystacks. Co-authored-by: Cursor <cursoragent@cursor.com>

fix(history): use only the first URL token for site filters

9a78a17

Outline/link URLs must not drive includeOnly or exclude matching. Co-authored-by: Cursor <cursoragent@cursor.com>

fix(history): require host match for host/path website needles

7d3bdc7

Prevent includeOnly path needles from matching substring paths on unrelated hosts. Co-authored-by: Cursor <cursoragent@cursor.com>

fix(history): harden website path filters for /path, URLs, localhost

37cf7f9

Preserve absolute path needles, accept full-URL rules, match localhost host/path filters, and require path segment boundaries. Co-authored-by: Cursor <cursoragent@cursor.com>

fix(history): treat origin filters as whole-host; normalize path slashes

70cb219

example.com/ and https://example.com match every page on that host, and trailing slashes no longer break full-URL path filters. Co-authored-by: Cursor <cursoragent@cursor.com>

fix(history): align Rust path normalization with Swift

9de4361

Trim leading and trailing slashes so //private matches across platforms. Co-authored-by: Cursor <cursoragent@cursor.com>

fix(server): inject Computer History context for all providers

81b26d8

Codex already received recent history on sendTurn; load the same block for Claude, Cursor, and Grok so enabled History works regardless of harness. Co-authored-by: Cursor <cursoragent@cursor.com>

fix(web): clean agent cursor settings icon

7d434e9

Use a transparent circular badge (no black square crop) and rounded-full clipping so the Computer Use settings row matches the pointer favicon. Co-authored-by: Cursor <cursoragent@cursor.com>

fix(web): use real agent-cursor overlay art in settings

476bf37

Point the Computer Use settings icon at the same soft-glow cursor-112 artwork as the desktop overlay, on a transparent canvas with no rounded crop. Co-authored-by: Cursor <cursoragent@cursor.com>

fix(web): soften agent-cursor glow padding in settings icons

385b92e

Match Computer Use tip so the lavender radial glow does not square-crop at settings row size. Co-authored-by: Cursor <cursoragent@cursor.com>

fix(web): soft-extend agent-cursor badge glow into purple frame

aa6cde7

Keep the original pointer art and fade the glow into the badge rim so Settings and Chrome tab favicons no longer show a nested square crop. Co-authored-by: Cursor <cursoragent@cursor.com>

sheehanmunim force-pushed the feat/computer-history-v3 branch from 92c688d to 2978695 Compare August 16, 2026 17:09

macroscopeapp Bot reviewed Aug 16, 2026

View reviewed changes

cursor Bot reviewed Aug 16, 2026

View reviewed changes

Comment thread apps/desktop/src/computerHistory/ComputerHistoryManager.ts

macroscopeapp Bot reviewed Aug 16, 2026

View reviewed changes

Comment thread apps/desktop/src/computerHistory/ComputerHistoryManager.ts

fix(desktop): wait for recorder exit after SIGKILL on stop timeout

c248595

Only clear the daemon child after confirmed exit (or failed signal delivery), so a timed-out stop cannot report success while capture continues. Co-authored-by: Cursor <cursoragent@cursor.com>

cursor Bot reviewed Aug 16, 2026

View reviewed changes

Comment thread native/t3-desktop-mcp-rs/src/history.rs

Comment thread native/t3-desktop-mcp-rs/src/platform/linux.rs

macroscopeapp Bot reviewed Aug 16, 2026

View reviewed changes

fix(desktop-mcp): parse escaped app ids and keep ambiguous activate f…

d9b242d

…ail-closed Close app-list ids at the first unescaped bracket, and only fall back from AT-SPI to X11 when there is no match — not when the query is ambiguous. Co-authored-by: Cursor <cursoragent@cursor.com>

cursor Bot reviewed Aug 16, 2026

View reviewed changes

matsvarn mentioned this pull request Aug 16, 2026

fix(desktop): --version segfaults without a display #7256

Closed

2 of 4 tasks

sheehanmunim closed this Aug 18, 2026

sheehanmunim deleted the feat/computer-history-v3 branch August 18, 2026 19:29

Conversation

sheehanmunim commented Aug 15, 2026 • edited by macroscopeapp Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Demo

Test plan

Add Computer History recording and Desktop MCP server for macOS, Windows, and Linux

Note

Note

Uh oh!

coderabbitai Bot commented Aug 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review skipped

Uh oh!

macroscopeapp Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

macroscopeapp Bot commented Aug 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Approvability

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sheehanmunim commented Aug 15, 2026 •

edited by macroscopeapp Bot

Loading

coderabbitai Bot commented Aug 15, 2026 •

edited

Loading

macroscopeapp Bot commented Aug 15, 2026 •

edited

Loading