Repository navigation
Bound compression and upload memory across concurrent snapshots #2553
Description
What happened?
Atelet was OOMKilled five times during concurrent microVM snapshot uploads with a 1 GiB memory limit. Raising the limit to 2 GiB stopped further restarts during validation, but leaves the allocation pressure unresolved.
Two problems compound each other:
newParZstdeagerly allocates two 8 MiB chunk buffers per worker.writeSparseZstdrequests at least one worker, so even a 36-byte metadata file gets 16 MiB of chunk capacity before encoder state and output allocations.- Snapshot uploads start one goroutine per file, and multiple snapshots can upload concurrently. There is no shared memory budget across requests. Current upstream runs this work in
NodePlugin.UploadSnapshot; the compression allocation behavior is unchanged.
Observed version: v0.5.0-alpha2, 9c4b1fb. Sandbox runtime: microVM. Source paths above checked against upstream be3aa280; the live OOM reproduction has not been repeated on that commit.
Steps to reproduce
The compression allocations reproduce without Kubernetes or an object store: create 36-byte regular files and call writeContent(io.Discard, file) concurrently from a test in pkg/objectstorage. Run GC before each sample and measure the change in runtime.MemStats.TotalAlloc.
| Concurrent files | Allocated during compression |
|---|---|
| 1 | 16.6 MiB |
| 6 | 99.5 MiB |
| 24 | 397.8 MiB |
These are cumulative allocations, not peak live memory. The allocation profile attributed 94.75% of allocations across the samples to newParZstd.
Expected behavior
Small inputs should use proportionate compression memory. Total in-flight upload memory should be bounded across snapshot requests, accounting for compressor buffers/state and object-store multipart buffers together. A per-snapshot concurrency limit alone would still let simultaneous snapshots multiply memory use.
Validate the fix with concurrent multi-file snapshots, large sparse files, and slow multipart uploads, measuring peak memory and throughput. Waiting uploads must cancel promptly and release capacity after failures. Preserve sparse round trips, reader compatibility, and the manifest's role as the commit marker after all files succeed.
Activity
EItanya commentedon Oct 13, 2026
Generally I think we should be:
dberkov commentedon Oct 13, 2026
Walter Fender (@cheftako) - could you PTAL on it.
dberkov commentedon Oct 13, 2026
Aditya Shantanu (@aditya-shantanu) - FYI
abhisheksharma2411 commentedon Oct 14, 2026
Could I take the first half of this, the proportionate allocation for small inputs?
I had a look and your numbers line up.
newParZstdfillsworkers * parZstdQueuebuffers of 8 MiB before anything is compressed, andwriteSparseZstdalways asks for at least one worker, so a 36-byte file really does reserve 16 MiB.Filling the free list lazily, and sizing chunks to the input when it's smaller than 8 MiB, should sort that part out without touching the on-disk format or the reader.
I'd leave the shared budget across concurrent snapshots out of it though - that one seems like its own design discussion rather than something to bolt on. Happy to follow up with it separately if you'd like.
mastersingh24 commentedon Oct 15, 2026
We're seeing the same thing on the v0.5.0 release with gVisor, where this code now runs in atelet's
objectstore-pluginsidecar. Ten agents on one worker node idle-suspended within the same minute. Each snapshot uploads about 8 files, so that's roughly 80 concurrent compressed uploads. The sidecar went from about 30 MiB to 1.08 GiB during the burst, then dropped back to 33 MiB once the uploads finished, so it looks like allocation pressure rather than a leak.Because the sidecar only requests 128 MiB and sets no memory limit, it didn't get OOM-killed like in your microVM case. Instead the spike landed on the node. Our worker node (c4-highcpu-2, about 2.7 GiB allocatable) went to 88% memory, with memory requests already at 97%, so a bigger burst would have started evicting pods. That's probably worth keeping in mind for the fix: until total upload memory is bounded, the sidecar's request is far below what it actually uses, and with no limit the cost lands on whatever else runs on the node.
+1 to a global budget across snapshot requests. A per-snapshot limit wouldn't have helped here, since this was many snapshots at once. A synchronized suspend of many idle agents is a common pattern when they share the same idle timeout.
clusters/agent-substrate: give worker nodes at least 7 GB
Limit memory used by concurrent snapshot uploads (agent-substrate#2708)
BenTheElder commentedon Oct 16, 2026
#2708 reviewed and merged.
msau42 commentedon Oct 16, 2026
As part of Actor Lifecycle v2, we are adding support to asynchronously upload snapshots to object storage. The async upload goes through a workqueue where you can configure the # workers. That can help provide the global budget, although it does not address synchronous uploads: #2571
EItanya commentedon Oct 16, 2026
#2708 should already handle a basic queue in the meantime, I can re-review #2633 today
roycaihw commentedon Oct 17, 2026
Some live throughput data on the tradeoff from #2708, in case it helps size the default.
Setup: one
c3-highmem-192-metalnode (v0.5, microVM, GCS backend), snapshot staging on a dedicated Hyperdisk Balanced disk. Three probe actors suspend (synchronousSuspendActor) and resume every 20–30 s while N SWE-bench replay actors run on the same node. p95 over 12 minutes (two reps at 512Mi, one at 16Gi), with the ratio to the probes-only baseline in parentheses:512Mi(default)16Gi512Mi/16GiWith GCS, one upload reserves at least 384Mi, so the default admits one upload at a time per node and synchronous suspends queue behind each other: the plugin sent about 50 MB/s at 512Mi and 240 MB/s at 16Gi. Resume and the local snapshot write stayed under 1 s at p95 in every run. At 16Gi the plugin's working set peaked around 2.3 GiB, well under the budget, which matches the README's note that the GCS accounting is conservative.
Some options:
The async upload workqueue in #2571 would help asynchronous suspends, but synchronous suspends still wait on this budget. Happy to test changes on the same setup.
EItanya commentedon Oct 18, 2026
Agreed that this absolutely needs further optimization, I just wanted to get rid of the OOM kills and move from there
roycaihw commentedon Oct 18, 2026
Follow-up with measurements and a proposed fix: #2847. On a large node, each upload's reservation grows with the budget and
GOMAXPROCS, so a 16Gi budget fits only 5–6 uploads at once. Reserving for the path an upload actually takes gets 64Gi's result at 16Gi.objectstorage: write snapshot parts as sized zstd frames and pool upl…
objectstorage: size small snapshot upload buffers to actual data (age…