Skip to content

resource_digest children_rss misses host-service's own process #6700

Open

Description

@Kitenite

Context

The resource_digest telemetry (#6697) reports child-process memory by summing terminal-session subtrees from the resource-metrics collector. The host-service daemon itself (and pty-daemon) belong to no workspace session, so a host-service memory leak is invisible to the one metric that reads as "everything outside Electron". host-service is our top crasher (7.7k SIGABRT events in the last 14 days on Sentry), so this is the process we most want a memory series for.

References

Source Who Link Date
PR follow-up @Kitenite #6697 2026-08-19

Implementation notes

Files

  • apps/desktop/src/main/lib/resource-digest/resource-digest.ts — SAMPLE_KEYS and collectDefaultSample
  • apps/desktop/src/main/lib/host-service-coordinator.ts — knows the spawned host-service child PIDs
  • apps/desktop/src/main/lib/resource-metrics/process-tree.ts — existing single-ps-snapshot helpers to read RSS for a PID set

Approach

Add a host_service_rss_bytes sample key: ask the coordinator for its live child PIDs and sum their RSS (macOS phys_footprint where available, matching the collector). Consider a matching host_service_rss_high alert threshold.

Gotchas

  • Sum across orgs (the coordinator can run one host-service per org).
  • A dead/restarting host-service should contribute 0, not fail the sample.

Activity

  1. github-actions commented on Aug 20, 2026

    @github-actions
    Contributor

    Summary

    This is a follow-up task from #6697 rather than a user-visible bug: resource_digest's child-process memory sample only ever walks subtrees rooted at terminal session PIDs, so the host-service daemon (and the pty-daemon it supervises) are invisible to the one number that reads as "everything outside Electron" — which is unfortunate given host-service is the top SIGABRT source. I confirmed the gap against the collector code, and the approach in the issue is sound, but there are several traps in the current coordinator/process-tree APIs that will bite a naive implementation. Note up front: apps/desktop/src/main/lib/resource-digest/ does not exist at main (b104fd0) — no file in the repo mentions resource_digest, SAMPLE_KEYS, collectDefaultSample, or children_rss, so #6697 hasn't landed here yet and this work has to be based on that branch.

    The gap is real

    The only PID roots the collector knows about come from terminal sessions:

    • apps/desktop/src/main/lib/resource-metrics/session-sources.ts:30 (v1, from the runtime registry's listSessions) and :70 (v2, from host-service's /terminal/resource-sessions endpoint).
    • apps/desktop/src/main/lib/resource-metrics/index.ts:275-278 expands exactly those roots via getSubtreePids, and :326 sums each session subtree.

    Nothing ever adds the host-service PID as a root. collectV2WorkspaceSessionMap talks to host-service over HTTP (session-sources.ts:91) but only asks for its sessions; the serving process's own memory never enters the snapshot.

    Implementation traps in the existing code

    The coordinator exposes no PID accessor. instances is private (host-service-coordinator.ts:171) and the public surface is getConnection/getConnections/getProcessStatus/getActiveOrganizationIds (:396-426) — port, secret, machineId, status, org ids, no PIDs. You'll need something like getChildPids(): { organizationId: string; pid: number; owned: boolean }[].

    A starting instance has pid: 0, and PID 0 is a machine-wide footgun. The instance is registered with pid: 0 at host-service-coordinator.ts:699-715 and only assigned the real PID at :780. If a 0 reaches getSubtreePids, childrenOf.get(0) is not empty — on both macOS and Linux PID 1 reports ppid 0 (process-tree.ts:60-67 builds the map straight from ps -eo ppid=), so the walk descends into the entire process table and the sample reports the whole machine's RSS as host-service. Guard with pid > 1, mirroring isSignalablePid in host-service-manifest.ts:126.

    Adopted instances aren't ours. tryAdopt registers entries with owned: false (host-service-coordinator.ts:663-675) pointing at a host-service another live app instance spawned. If you include them, two running desktop instances both report the same RSS under the same key; if you exclude them, an adopting instance reports 0 for an org that is genuinely running. Either is defensible — pick one deliberately and tag the sample (owned/adopted) so the series is interpretable.

    Own PID only, not the subtree. In v2, terminal PTYs live under the pty-daemon that host-service spawns (packages/host-service/src/daemon/DaemonSupervisor.ts:1193), so a subtree sum from the host-service PID double-counts everything already in children_rss. Sum the host-service process alone.

    The pty-daemon is not reliably a descendant. It's spawned detached: !isDev (DaemonSupervisor.ts:1194) and is explicitly designed to outlive host-service restarts (packages/host-service/src/daemon/manifest.ts:1-3, plus the adopt path around DaemonSupervisor.ts:916), so after any host-service crash the surviving daemon is reparented to PID 1 and drops out of the tree — i.e. it would vanish from the metric exactly during the crash loop you're trying to measure. Read its PID from readPtyDaemonManifest/listPtyDaemonManifests (manifest.ts:95, :139) plus isProcessAlive (:164) instead, and emit it as a separate pty_daemon_rss_bytes key — folding it into host_service_rss_bytes makes a leak un-attributable. (v1's daemon is spawned by Electron main instead: apps/desktop/src/main/lib/terminal-host/client.ts:1260-1266.)

    phys_footprint parity is opt-in per PID. enrichWithPhysFootprint (process-tree.ts:132-149) only rewrites the PIDs you hand it and no-ops off macOS. If the host-service PIDs aren't in that list, the new key is RSS while children_rss is phys_footprint, and the two aren't comparable on the platform most of your users are on.

    Cost. captureProcessSnapshot shells out to ps -eo with a 5s timeout (process-tree.ts:156). Reuse the digest's existing snapshot if it has one rather than adding a second ps per sample.

    "Dead host-service contributes 0"

    That mostly falls out for free — snapshot.byPid.get(pid) returns undefined after exit and contributes nothing — but be aware handleChildExit deletes the instance (host-service-coordinator.ts:957) before the respawn timer fires, so during a crash loop there is no entry at all. Emit an explicit 0 rather than omitting the key, and consider pairing it with host_service_uptime_ms (the coordinator already tracks spawnedAt, :703): without it, a respawn resets RSS to baseline and a leak reads as a fix.

    Reproduction / verification

    Manual, on top of the #6697 branch: start the app, jq .pid ~/.superset/host/<org>/manifest.json, then ps -o pid,ppid,rss -p <pid> and compare against the emitted sample; leave it running and confirm the series tracks Activity Monitor's "Memory" column for that process rather than RSS on macOS. Then kill -9 the PID and confirm the sample reads 0 through the respawn window instead of throwing or dropping the key.

    For a bun:test, keep the arithmetic in a pure helper — sumHostServiceRss(snapshot: ProcessSnapshot, pids: number[]) — and drive it with a hand-built ProcessSnapshot (the pattern in resource-metrics/process-tree.test.ts), covering: PID 0/starting (must be 0, not the whole tree), a PID absent from the snapshot (dead → 0), two orgs (summed), and an adopted entry (whatever you decide). If any test reaches listPtyDaemonManifests, note assertIsolatedDaemonNamespaceInTests (manifest.ts:52) throws under NODE_ENV=test unless SUPERSET_HOME_DIR points at a temp dir.

    Suggested shape

    getChildPids() on the coordinator (filtering status === "running" and pid > 1, exposing owned) → digest collects those plus daemon-manifest PIDs → both PID sets are appended to the list passed to enrichWithPhysFootprint → emit host_service_rss_bytes (own PID only, summed across orgs, 0 when nothing is running) and pty_daemon_rss_bytes separately. I'd hold off on host_service_rss_high until there's a week of the raw series to pick a threshold from — a static byte threshold on a process whose baseline varies with workspace count will mostly alert on large-but-healthy installs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions