Skip to content

Make sandbox crashes visible in logs and metrics #2212

Open

Description

@git286

Sub-issue of #2211. #2211 needs core changes to move a dead actor to CRASHED. This one covers the o11y fixes we can do now, so at least there is something to look at when a sandbox dies.

Today we get almost nothing. In my kind tests (sentry kill, OOM kill, app kill), the only trace was one unlabeled atunnel WARN after a request failed. When a runtime operation fails, the error just says exit status 128.

  1. Flag dead sandboxes in the stats sweep. ateom-gvisor already reads the actor cgroup every minute (cmd/ateom-gvisor/internal/cgroupstats). Also read oom_kill from memory.events and check whether the cgroup still has any processes. If either changes, log a WARN with actor labels and bump a counter. After my OOM test the cgroup still showed oom_kill 3 with no processes left, so the data is there.
  2. Put the runsc stderr tail in the error. Right now runsc.go returns only the exit status. With the last ~2 KB of stderr, the crash message would say cannot checkpoint container "_pause" in state stopped instead of exit status 128.
  3. Label runsc logs with the actor. Pause-container runsc lines have no actor labels. App-container runsc lines show up in kubectl ate logs as if the app wrote them.
  4. Keep sentry panics. Pass --panic-log and log it with actor labels when it isn't empty. Today nothing is kept (see gVisor sandbox dies on restore for python:3.12-slim workloads (arm64) #1769).
  5. Better crash event. Add the crash message to ate.actor.crashed. When ateom restarts, add the kubelet reason and exit code to the message. On kind, a node OOM only showed up as "ateom restarted while hosting the actor".
  6. Small ones. In atelet, log stats sweep failures as WARN instead of DEBUG (cmd/atelet/statspoller.go:276). For microVM, log the tail of console.log when something fails, before cleanup deletes it.

Out of scope here: changing actor state, and reason labels on the metric. Both belong in #2211.

Activity

  1. baizhenyu commented on Oct 9, 2026

    @baizhenyu
    Collaborator

    Where sandbox-death observability stands, after #2258:

    Covered

    • gVisor sandbox dies (sentry crash, kill, or OOM kill): the ateom's usage sweep logs one WARN per activation, with the actor's keys and ate.sandbox.oom_kills (ateom-gvisor logs a warning when a running actor's sandbox dies #2258).
    • Worker pod dies, or the ateom restarts: ateapi crashes its actors and emits ate.actor.crashed, but the message gives no reason.

    Planned (this issue and #2211)

    Not covered yet

    • micro-VM guest OOM (a container over its limit, or the guest out of RAM): nothing reports it. The kata agent's GetOOMEvent RPC, already in our vendored agentpb, returns the container ID on each OOM kill; a watcher per guest could log it with the actor's keys.
    • micro-VM VMM killed on the host: the exit status reaches the child reaper and is discarded; the usage records only turn pending.
    • gVisor app exits while the sentry stays up: ateom-gvisor logs a warning when a running actor's sandbox dies #2258 doesn't catch it, because the leaf stays populated. It needs the container's exit status, for example from runsc.
    • gVisor usage records after a death keep reporting the dead sandbox as measured (noted on ateom-gvisor logs a warning when a running actor's sandbox dies #2258).

    All of these fit the "detect in the ateom" direction in #2211.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions