Skip to content

fix(core): exempt live-child locations from idle eviction - #51827

Open
KamitobiHaru wants to merge 1 commit into
anomalyco:devfrom
KamitobiHaru:shell-eviction-guard
Open

KamitobiHaru wants to merge 1 commit into
anomalyco:devfrom
KamitobiHaru:shell-eviction-guard

Conversation

@KamitobiHaru

@KamitobiHaru KamitobiHaru commented Sep 28, 2026 •

Copy link
Copy Markdown

Issue for this PR

Closes #51828

Type of change

  • Bug fix
  • New feature
  • Refactor / code improvement
  • Documentation

What does this PR do?

A background shell running a long job gets SIGTERM'd when LocationActivity
evicts its location after 60 minutes without session events. The service
process itself stays alive (same PID throughout); only the location-scoped
Shell/Pty graph is torn down, because locations.invalidate() closes the
scope-bound child-process handles.

Seen on 2026-09-27 (service 2.0.10): location services evicted directory=/data3/zsy/asr2report_pipeline at 16:24:48.527Z, torchrun
Received 15 death signal at 16:24:48.563Z, payload's own handler wrote
rc=143, signal=15. No service restart near the window, so this is the
inactivity sweeper, not the ensure() replacement path.

Why it happens: the sweep only exempts locations that own active
SessionExecution sessions. A backgrounded shell/pty emits no session
events and holds no execution claim, so its location looks idle and gets
invalidated out from under the live child.

Changes:

  • New LocationRetention registry (packages/core/src/location-retention.ts,
    global node) tracking locations with live child processes, keyed by the
    same identity the sweep uses (LocationServiceMap.canonical +
    workspaceID via a shared keyOf, so pins can't silently miss).
  • shell.ts / pty.ts: retain on create, release on settle/remove/teardown
    (per-command retained flag, no double-release). Teardown still kills
    running children and now logs them.
  • location-activity.ts: expired entries with live children are touched
    (deadline deferred, no reference taken) and skipped; rest unchanged.
    Explicit invalidate still tears down immediately (registry is only read
    in the sweep). No task persistence.

How did you verify your code works?

  • bun typecheck clean in packages/core.
  • New tests: location-retention.test.ts (5 unit) and
    location-activity-retention.test.ts (2 TestClock sweep tests: a pinned
    location survives past TTL and evicts after release; explicit invalidate
    still tears down a pinned location without leaking the pin).
  • Existing suites still pass: pty-session (7), location-activity (4),
    browser-idle (1, only needed the new node dep), session-shell /
    tool-shell / shell-retention (97 pass, 19 platform skips).
  • Live server with production 60min TTL, idle for over an hour, real
    background shell (POST /api/shell, sleep 4000, DEBUG, isolated
    XDG_DATA_HOME): pinned 08:48:43Z, eviction deferred 09:49:22Z (first
    sweep after the 09:48:43Z deadline; the touch pushes the deadline an hour
    out so a single line is the whole story), running + process-alive polls
    at 09:48:58Z and 09:53:58Z, natural exit=0 with unpinned, idle eviction resumed at 09:55:23Z. Zero location services evicted, zero SIGTERM.
    Full logs attached: v2-server-monitor.log, v2-server-serve.log,
    v2-server-opencode.log, v2-local-verify.log.

Screenshots / recordings

N/A (no UI change).

Checklist

  • I have tested my changes locally
  • I have not included unrelated changes in this PR

LocationActivity tore down location-scoped Shell/Pty services after 60 minutes without session events, SIGTERMing background children while the service stayed alive.

Add LocationRetention registry. Shell and Pty retain on create, release on settle/remove/teardown. The sweep touches pinned locations instead of invalidating them. Explicit invalidate still tears down immediately. No task persistence.
@github-actions github-actions Bot added needs:compliance This means the issue will auto-close after 2 hours. needs:issue labels Sep 28, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Thanks for your contribution!

This PR doesn't have a linked issue. All PRs must reference an existing issue.

Please:

  1. Open an issue describing the bug/feature (if one doesn't exist)
  2. Add Fixes #<number> or Closes #<number> to this PR description

See CONTRIBUTING.md for details.

@github-actions

Copy link
Copy Markdown
Contributor

The following comment was made by an LLM, it may be inaccurate:

@github-actions github-actions Bot removed the needs:compliance This means the issue will auto-close after 2 hours. label Sep 28, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Thanks for updating your PR! It now meets our contributing guidelines. 👍

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Location inactivity eviction SIGTERMs live background shells after 60 minutes

1 participant