[Bug]: a stalled command pipeline gives an endless spinner, and the message typed while it stalls is dropped #23661

Use this form for broken behavior, regressions, crashes, or reliability problems.
Feature requests belong in Discussions.
Search existing issues first and keep the report focused on one problem.

• I searched existing issues and did not find a duplicate.
• I included enough detail to reproduce or investigate the problem.

Area

Not sure

(The change is in packages/client-runtime, which the Area dropdown does not list, and in apps/web.)

Steps to reproduce

1. Put the host under sustained load, so that a command it accepts never comes back. On the machine this was
   observed on, the host was serving a 25.5 GB SQLite state database, the load average was 68-116 on 10 cores,
   and the server's event loop was stalling for 2-24 s at a time.
2. Open a thread and press Enter to send a message.
3. While that send is still in flight, type a second message and press Enter.
4. The second message is discarded. The composer shows a spinner that never resolves, and no error appears.

The same stall, observed from another machine over Tailscale, showed a command that got no answer for 163
seconds while small requests to the same host still answered in 6-126 ms.

Expected behavior

• A send that the host does not answer should fail with an error the user can see, rather than waiting forever.
• A message typed while a send is in flight should be queued and sent when the composer is free, not dropped.
• A connection whose command pipeline has stalled should be reported as stalled, even though the socket itself
  is still healthy.

Actual behavior

1. A request waits forever. packages/client-runtime/src/rpc/client.ts has no timeout, deadline, or
Effect.timeout anywhere in its 446 lines, so requestGuarded waits for a reply that never comes. The only
bound in the transport is the socket-level ping timeout in packages/client-runtime/src/rpc/protocol.ts:

// Pings go out every 5 seconds, and a server busy for a few seconds stops
// answering them. Allow 15 seconds without any frame before dropping the socket.
export const PING_TIMEOUT = "15 seconds";

That is a no-traffic check. A host under load keeps answering small requests, so the socket stays alive and
the one command that is stuck is invisible to it. This is why the host above answered small requests in 6-126 ms
while a command got no answer for 163 s.

2. The composer drops Enter, silently. apps/web/src/components/ChatView.tsx:4665-4679 returns early when
the send guard fires:

if (
  !readEnvironmentScope(environmentId, AuthOrchestrationOperateScope) ||
  !activeThread ||
  (isSendBusy && !sendQueuesBehindSetup) ||
  isConnecting ||
  isRevertingCheckpoint ||
  !clientSettingsHydrated ||
  threadDetailLoading ||
  readPendingEnvironmentChange() !== null ||
  sendInFlightRef.current ||
  feedbackUploadsInFlightRef.current.has(routeThreadKey)
) {
  notifyDirectAnnotationAttached();
  return;
}

sendQueuesBehindSetup only covers the worktree-setup case, not a general in-flight send. The text the user
typed is never sent, never queued, and never reported: the only sign is the spinner.

3. Liveness cannot see a stalled command pipeline. The 15-second heartbeat answers "is the connection still
carrying frames?", not "is the host still doing the work we ask of it?".

Impact

Blocks work completely. The user cannot send a message and cannot queue one, and nothing tells them the
message was lost.

Version or commit

0.1.1

Environment

macOS 27.0.1, arm64. Observed against a host serving a 25.5 GB statev2.sqlite under load 68-116 on 10 cores.
The client-side behaviour is load-independent: any request the host does not answer is affected.

Logs or stack traces

Measured on the affected host, from the server's own trace log over a 10-minute window:

MCP tool calls:            1,857 in ~10 minutes (peak 506 in one minute)
DB lock waits > 100 ms:    1,608   (518 seconds of lock wait in that window)
correlation calls/min vs
  total lock wait/min:     r = 0.751

The reply to the command was stuck behind a database write lock that waited up to 6,622 ms. The command
itself was accepted quickly - ws.rpc.orchestration.dispatchCommand p50 5 ms, and every command that
started also ended (43/43). The reply path is what stalled.

Attachments

A prepared fix is in TBD: a request deadline with a named EnvironmentRpcTimeoutError, a local composer
queue, and a stall tracker that reports the longest-outstanding command.

Related issues

• [Bug]: Thread snapshot HTTP handlers have no server-side timeout; cold-open herd times out at 6s and floods relay logs with context-canceled warnings #17765 - server-side HTTP handler timeouts. The client side of the same gap; that issue is about the server
  bounding its handlers, this one is about the client bounding its requests.
• fix(client-runtime): queries stay pending while a connection hands off #18155 - keeps a query pending across a session handoff. The opposite of a timeout.
• [Bug]: Submit button in Input status does nothing — enabled pending-user-input Submit is swallowed by normal-send guards #9318 - the same onSend guards swallow a pending-user-input answer. Closest existing report of the guard
  behaviour, but for a different path.
• feat(web): queue messages sent client-side while the agent is working #11673 - a client-side queue for messages sent while a turn is running on the server. That queue holds a
  message the server has already accepted. This issue is about a message the client could not send at all,
  because the client itself is the thing that is busy.
• Local Windows desktop disconnects during 36–44 second server stalls with a 12.5 GB database #21539 - a Windows host that disconnects during 36-44 s server stalls, with a 12.5 GB database. Cites the
  same 15-second no-frame heartbeat. Closed as not reproducible; the diagnosis there was about the stall
  source, not about the client being unable to see it.
• fix(client): a frozen host no longer shows a heartbeat as slow or blames your DNS #22212, feat(server,web,mobile): a host that drops because it froze now says why #22213 - reword the ping-timeout disconnect reason and report the server's own event-loop stall on
  reconnect. Both are about the socket-level heartbeat; neither adds command-level liveness.
• [Bug]: 26.5 GB V2 state database: thread-shell and recovery queries take 2–22 s; six indexes cut startup from ~137 s to ~17 s #22659, [Bug]: Thread-shell and recovery queries take 2–22 s on a 26.5 GB V2 database; six proposed indexes (follow-up to #22659) #23000 - the 26.5 GB state database and slow thread-shell/recovery queries. The load that produces
  the stall this issue is about.