Skip to content

stop dying when a container disappears mid-stats - #17

Merged
mrq1911 merged 1 commit into
masterfrom
fix/stats-nil-deref
Aug 21, 2026
Merged

mrq1911 merged 1 commit into
masterfrom
fix/stats-nil-deref

Conversation

@mrq1911

@mrq1911 mrq1911 commented Aug 21, 2026

Copy link
Copy Markdown
Member

Fixes swarmpit/swarmpit#736.

Root cause

ContainerUsage declares var v *types.StatsJSON and then:

dec := json.NewDecoder(resp.Body)
if err := dec.Decode(&v); err != nil {
    dec = json.NewDecoder(io.MultiReader(dec.Buffered(), resp.Body))
}          // rebuilds the decoder, never retries, never returns
...
previousCPU = v.PreCPUStats.CPUUsage.TotalUsage    // stats.go:157 - v is still nil

The error branch is a no-op left over from Docker CLI's stats loop, which retries — here there is no loop, so v stays nil and the very next line dereferences it. stats.go:157 is exactly the frame in the reported trace.

null is the nastier case: it decodes successfully and leaves v nil, so an error check alone isn't enough.

Verified against the pre-fix logic — all four bodies panic with the reported error:

body=""                   -> invalid memory address or nil pointer dereference
body="null"               -> invalid memory address or nil pointer dereference
body="{\"cpu_stats\": "   -> invalid memory address or nil pointer dereference
body="<html>error</html>" -> invalid memory address or nil pointer dereference

This happens routinely: a container that exits while its stats are being read returns an empty or truncated body.

Why it takes the whole agent down

ContainerUsage runs in a goroutine (ContainersUsage, stats.go:122) and there is no recover() anywhere in the agent, so one dying container panics the process. That makes it a second, independent cause of "Statistics not ready" in the UI — the agent logs Stats collector started, then goes silent.

Changes

  • return early when the body doesn't decode, or decodes to nil
  • ContainerUsage now returns (status, ok) so failed reads aren't reported as zero-valued containers
  • recover() per collection goroutine, so a future panic in one container degrades that container instead of killing the agent
  • drop the now-unused io import

Tests

New swarmpit/task/stats_test.go drives ContainerUsage/ContainersUsage against an httptest fake of the daemon stats endpoint: empty, null, truncated, non-object and HTML bodies all return ok=false instead of panicking; a well-formed body still parses; and a mixed run reports the healthy container while skipping the bad one.

go build ./..., go vet ./... and go test ./swarmpit/task/... pass on golang:1.12 (matching the Dockerfile). Only stats.go and the new test are touched — go.mod is unchanged, and I left the repo's pre-existing gofmt drift in other files alone.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Agent / invalid memory address or nil pointer dereference

1 participant