You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Make sandbox crashes visible in logs and metrics #2212
Sub-issue of #2211. #2211 needs core changes to move a dead actor to CRASHED. This one covers the o11y fixes we can do now, so at least there is something to look at when a sandbox dies.
Today we get almost nothing. In my kind tests (sentry kill, OOM kill, app kill), the only trace was one unlabeled atunnel WARN after a request failed. When a runtime operation fails, the error just says exit status 128.
Flag dead sandboxes in the stats sweep. ateom-gvisor already reads the actor cgroup every minute (cmd/ateom-gvisor/internal/cgroupstats). Also read oom_kill from memory.events and check whether the cgroup still has any processes. If either changes, log a WARN with actor labels and bump a counter. After my OOM test the cgroup still showed oom_kill 3 with no processes left, so the data is there.
Put the runsc stderr tail in the error. Right now runsc.go returns only the exit status. With the last ~2 KB of stderr, the crash message would say cannot checkpoint container "_pause" in state stopped instead of exit status 128.
Label runsc logs with the actor. Pause-container runsc lines have no actor labels. App-container runsc lines show up in kubectl ate logs as if the app wrote them.
Better crash event. Add the crash message to ate.actor.crashed. When ateom restarts, add the kubelet reason and exit code to the message. On kind, a node OOM only showed up as "ateom restarted while hosting the actor".
Small ones. In atelet, log stats sweep failures as WARN instead of DEBUG (cmd/atelet/statspoller.go:276). For microVM, log the tail of console.log when something fails, before cleanup deletes it.
Out of scope here: changing actor state, and reason labels on the metric. Both belong in #2211.
Items 2–5: runsc's stderr tail in errors, actor labels on runsc logs, sentry panic logs, and the crash message (plus the kubelet reason and exit code on an ateom restart) in ate.actor.crashed.
Item 6: atelet sweep failures at WARN; the micro-VM console.log tail when a lifecycle step fails.
micro-VM guest OOM (a container over its limit, or the guest out of RAM): nothing reports it. The kata agent's GetOOMEvent RPC, already in our vendored agentpb, returns the container ID on each OOM kill; a watcher per guest could log it with the actor's keys.
micro-VM VMM killed on the host: the exit status reaches the child reaper and is discarded; the usage records only turn pending.
Activity
ateom-gvisor: warn when a hosted actor's sandbox has no processes left
ateom-gvisor: warn when a hosted actor's sandbox has no processes left
ateom-gvisor logs a warning when a running actor's sandbox dies (agen…
baizhenyu commentedon Oct 9, 2026
Where sandbox-death observability stands, after #2258:
Covered
ate.sandbox.oom_kills(ateom-gvisor logs a warning when a running actor's sandbox dies #2258).ate.actor.crashed, but the message gives no reason.Planned (this issue and #2211)
ate.actor.crashed.console.logtail when a lifecycle step fails.Not covered yet
GetOOMEventRPC, already in our vendoredagentpb, returns the container ID on each OOM kill; a watcher per guest could log it with the actor's keys.All of these fit the "detect in the ateom" direction in #2211.
Sync upstream sandbox, snapshot, and HTTPS egress updates (#69)