Skip to content

bug(remote): bound outgoing queues when paired clients stop draining #804

Closed

Description

@Juliusolsson05

Confirmed bounded experiment at 5d641845 (Node v24.14.1, macOS arm64, installed ws). Priority: medium; relevant when Remote is enabled and a paired phone stops draining output. Remote is off by default. This is not a claim about an observed production OOM.

Expected and actual behavior

A slow/disconnected-in-practice remote consumer must have a byte-bounded send queue. It must not retain an unlimited backlog of live agent output in the Electron main process.

RemoteServer.broadcast and send check only WebSocket readyState before send. There is no buffered-byte threshold, coalescing/overflow policy, or liveness deadline on these paths. OPEN does not imply the consumer is draining. Screen, semantic, and committed-entry events keep entering the same queue.

Reproduction and measurements

An isolated harness transpiled the unchanged RemoteServer class in memory, stubbing unused app-history imports to avoid initializing Electron/performance singletons. It connected a real ws client/server pair on an ephemeral loopback port, registered only the synthetic server socket in RemoteServer, then paused the client via client.pause().

Feed the actual broadcast method 512 synthetic screen frames with a 64 KiB recent field, yielding to the event loop every 16 frames. Record the real server socket's bufferedAmount; terminate only those test sockets afterwards. No real app connection, pairing secret, private output, or running session was used.

Frames sent Encoded bytes/frame bufferedAmount Server socket OPEN
128 65,627 7,810,803 yes
256 65,627 16,212,339 yes
512 65,627 33,015,411 (~31.5 MiB) yes

The experiment intentionally stopped at 512 frames; it demonstrates continuing queue growth, not a crash. Frame size is a synthetic stress fixture, not a measured typical frame. bufferedAmount is queued transport bytes, not a heap/RSS measurement.

Impact, tradeoffs, and acceptance

A paired phone on a stalled network can increase main-process memory pressure while agents continue working. More consumers can add separate queues; no live incidence or memory growth rate is inferred here.

  • Add a per-socket byte budget used by both broadcast and direct replies/bootstrap.
  • Coalesce replaceable screen/process snapshots. Preserve ordering for committed/semantic events; do not silently drop transcript deltas and leave an apparently complete feed.
  • Choose an explicit overflow/resync behavior (e.g. close the lagging connection and reconnect/backfill), with bounded heartbeat/liveness handling where needed.
  • Verify reconnect restores durable history and current state; current reconnect alone should not be assumed to backfill every missed live event for a session already marked historyLoaded.
  • Test a paused consumer alongside a draining consumer: queue bytes stay bounded, the healthy client continues receiving ordered data, and the slow client recovers without blocking desktop sessions.

Tradeoff: disconnect/resync costs a reconnect and history request; snapshot coalescing alone cannot solve an unbounded semantic/JSONL backlog. Screen deduplication from #746/#765 reduces traffic but does not establish a memory bound.

Refs #103, #365, #420. Separate from desktop IPC screen-interest work in #762 and the external operator toolkit in #795.

Related client-side retention issue: #805. Bounding socket queues does not bound the phone transcript store.

Activity

  1. Juliusolsson05 commented on Sep 5, 2026

    @Juliusolsson05
    OwnerAuthor

    Implementation: #813 (base main).

    A paired client that stops reading currently retains unbounded queued output in main (#804). Every server writer now shares a 4 MiB per-socket budget; overflow terminates that socket, while healthy clients continue receiving ordered frames. A reconnect resets the phone's disconnected transcript window and backfills durable history so gaps larger than a page cannot silently disappear.

    Fixes #804

    Acceptance criteria and decisions

    • Broadcasts, replies and bootstrap use the same byte budget. Bootstrap awaits each write so a healthy replay larger than the budget can complete.
    • History replies preserve a contiguous suffix and matching offsets under 3 MiB; excluded older records remain pageable. An individually oversized record returns a visible error without a retry loop.
    • Disconnect invalidates stale history replies and resets/reloads viewed windows. Semantic suffixes wait for a fresh turn boundary; durable completion remains authoritative.
    • Pending prompt outcomes remain uncertain; reconnect never resends mutations. Old socket events and disposed reconnect timers cannot revive the client.
    • Real paused/draining sockets, reconnect across multiple pages and prompt uncertainty have regression coverage.

    Validation and evidence

    • 105 remote tests / 12 files pass, including actual socket backpressure, 80 cached screens totaling over 4 MiB, and reconnecting across 310 durable entries with exact older-page reconstruction.
    • Typecheck, test contract, remote production build and diff check pass. Both full CI gates pass at 6182392, including coverage and distributable verification.
    • Before: audit synthetic paused receiver queued 31.5 MiB and remained open. After: regression verifies queued bytes never exceed 4 MiB and the healthy receiver gets all 160 test frames in order. This establishes a bound, not production memory savings.

    Tradeoffs and scope

    Disconnect/reset is the recovery contract rather than silently dropping ordered deltas or maintaining a second queue. An interrupted uncommitted semantic turn can temporarily disappear until durable completion; an individual history record above 3 MiB must be viewed on desktop. Existing remote bundle size/mixed import warnings remain. No provider, worktree, desktop rendering or toolkit changes. The #805 retention PR (#814) depends on this branch because it shares transcript lifecycle code. No merge is authorized.

    Current validation: rebased onto main f7507980 (toolkit #812 and MCP repair #818), head 205eb9dd. Source changes from the earlier implementation are preserved; both required CI gates pass on this head (CI run). No merge is authorized.

    2026-09-07 merge status: PR #813 merged as bb84772a with explicit user authorization.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions