Repository navigation
Actor stays RUNNING after its sandbox dies #2211
Description
If the sandbox dies after the actor is up, nobody notices. The actor stays RUNNING until someone suspends it or the worker pod restarts.
I tried it on kind (gVisor):
- kill -9 the sentry: still RUNNING after 6 min. Requests get a 502. The only log line is an atunnel WARN with no actor labels.
- OOM kill the sandbox via memory.max: memory.events shows oom_kill 3. Nothing is logged anywhere.
- kill the app inside the sandbox: still RUNNING, 502s, no exit code.
The stats keep reporting all three, so they look like idle actors.
ateom gets SIGCHLD for the sentry, but the child reaper throws the exit status away (internal/childreap/childreap.go:153). Nothing waits on the app process after start, and the ateom API has no way to tell atelet something died. From the code, microVM looks the same (I didn't test it).
Hung sandboxes are a separate problem (#141).
Activity
git286 commentedon Oct 5, 2026
cc Benjamin Elder (@BenTheElder) Julian Gutierrez Oschmann (@juli4n) Dmitry Berkovich (@dberkov) not sure if this is a duplicate issue of any existing one. Do we know if anyone is working on the general sandbox/uVM death detection problem?
git286 commentedon Oct 5, 2026
This is a big troubleshooting gap. When it happens there is almost nothing to go on: the actor says RUNNING, stats look like an idle actor, and the only trace is one unlabeled WARN after a request fails.
Some of this we can improve on the o11y side without touching core:
Those would make it visible, but the actor would still stay RUNNING. The router keeps sending it traffic, resume skips it, and the worker slot stays taken. The real fix needs core detection: ateom watches the sandbox and app processes, reports exits up, and ateapi moves the actor to CRASHED.
git286 commentedon Oct 5, 2026
cc Tim Bai (@baizhenyu) Jeff Luo (@JeffLuoo)
git286 commentedon Oct 5, 2026
Filed #2212 to track the o11y side of things we can improve before the core detection implementation.
baizhenyu commentedon Oct 5, 2026
We considered this in #1640 ("Deferred: dead-guest visibility"). The plan there was for atelet to record when each actor first reports pending and, past a threshold, emit a warning event or a long-pending gauge per template. It was deferred while each worker hosted one actor, because readyz caught a dead lone guest.
Your test shows that plan is not enough. A dead gVisor sandbox's cgroup still reads, so its samples look idle, not pending. Only a micro-VM guest whose agent stops answering turns pending. The atelet plan would catch only that case, and only after the threshold.
The ateom has the direct signals: the exit status of its runsc or cloud-hypervisor child,
oom_killinmemory.events, and an emptycgroup.procs. With #2194 and #2195 each ateom already samples its actors on a timer, so the sweep can check these. Detection belongs there; what is missing is the way up to atelet and ateapi, as you say.ateom-gvisor: warn when a hosted actor's sandbox has no processes left
ateom-gvisor: warn when a hosted actor's sandbox has no processes left
ateom-gvisor logs a warning when a running actor's sandbox dies (agen…
Sync upstream sandbox, snapshot, and HTTPS egress updates (#69)