bug(gateway): sandbox log/event buffers are freed on only one replica #5401

User Story

I run shared OpenShell gateways where agents create short-lived sandboxes all day (create, run a few commands, delete), and the gateways are expected to stay up for weeks. In HA runs, each gateway held 0.7–1.3 GiB with 300 sandboxes on a 5-node Kubernetes cluster, and 2.7 GiB with 1,080 sandboxes on a larger HA run: about 2–3.6 MiB per sandbox on every gateway. We run several gateway replicas behind a load balancer. Each keeps memory for sandboxes it once served, so every replica grows with the whole fleet and a delete frees memory on only one.

Problem Statement

A sandbox's log and event buffers (TracingLogBus::remove) are freed only from cleanup_removed_sandbox_state in crates/openshell-server/src/compute/mod.rs. In multi-replica mode that runs only on the replica that served DeleteSandbox, or on the reconciler-lease holder (the only replica running the driver watch).

Any other replica that created buffers for the sandbox, through a supervisor log stream, a WatchSandbox subscriber, or gateway logs tagged with the sandbox, keeps them for the life of the process.

Impact / Why This Matters

As sessions reconnect and rebalance, every replica accumulates buffers for every sandbox. Deleting sandboxes frees memory on at most one replica.

Acceptance Criteria

• After a sandbox is deleted, its buffers are gone on every replica within a few minutes (about 3 with the default sweep settings), including while a log watch on another replica is still connected.
• Each replica's buffer count stays bounded by the number of live sandboxes.

Reproduction Steps

1. Deploy two or more gateway replicas with external Postgres.
2. Create sandboxes whose supervisors or watchers connect to different replicas.
3. Delete the sandboxes through one replica.
4. RSS on the other replicas does not drop.

Environment

• OpenShell: 0.1.8-dev.51+gaf020160f
• Kubernetes driver, Helm chart, multi-replica gateway

Notes

Likely fix: a periodic sweep on every replica that drops buffers for sandboxes absent from the store, or idle beyond a TTL.

Related

Other causes of the same gateway memory growth: #5399, #5400, #5402. Allocator follow-up: #5404