atelet: serve shared snapshot restores from the node-local file cache - #1551
Draft
Dmitry Berkovich (dberkov) wants to merge 4 commits into
Draft
Dmitry Berkovich (dberkov) wants to merge 4 commits into
Dmitry Berkovich (dberkov) wants to merge 4 commits into
Conversation
Dmitry Berkovich (dberkov)
force-pushed
the
filecache-m2
branch
from
September 9, 2026 01:25
d0bdb37 to
240c6ce
Compare
Dmitry Berkovich (dberkov)
marked this pull request as draft
September 10, 2026 03:59
Dmitry Berkovich (dberkov)
force-pushed
the
filecache-m2
branch
from
September 10, 2026 04:04
240c6ce to
667cd76
Compare
Dmitry Berkovich (dberkov)
force-pushed
the
filecache-m2
branch
3 times, most recently
from
September 14, 2026 00:14
03ae2e4 to
251378a
Compare
Dmitry Berkovich (dberkov)
changed the title
Dmitry Berkovich (dberkov)
force-pushed
the
filecache-m2
branch
from
September 17, 2026 04:03
251378a to
076951e
Compare
Bowei Du (bowei)
added
kind/feature
Dmitry Berkovich (dberkov)
force-pushed
the
filecache-m2
branch
5 times, most recently
from
September 26, 2026 05:41
b691d42 to
232e494
Compare
a4-a4s1 Bot
pushed a commit
to AlexBulankou/substrate
that referenced
this pull request
Sep 26, 2026
… golden_snapshot_uri (agent-substrate#1602) Refactors the atelet Restore API so the DATA_ON_GOLDEN base snapshot travels as a typed restore source instead of a bare string, preparing for the node-local snapshot cache (agent-substrate#690, agent-substrate#1551): `RestoreRequest` gets its own external snapshot *source* message, and read-side attributes of a restore source have a home — today the URI, next (in the M2 cache work) a sharing property set by the control plane. **This change is not backward compatible.** `golden_snapshot_uri` is removed from `RestoreRequest` with no transition: an ateapi from before this change sends a DATA_ON_GOLDEN restore an updated atelet rejects (no `base_config`), and an updated ateapi sends one an old atelet rejects (no `golden_snapshot_uri`). ateapi and atelet must be rolled together. Fresh starts, FULL/DATA restores of an actor's own snapshot, and local pause restores without a golden base are unaffected. Three commits, each buildable and tested: 1. **`ateletpb: give Restore its own external snapshot source message`** — `ExternalRestoreConfiguration` replaces `ExternalCheckpointConfiguration` in `RestoreRequest`'s config oneof (same wire shape; the checkpoint-side write-destination message is untouched), and `base_config` is added for the DATA_ON_GOLDEN base. No behavior change: nothing sets or reads the new field yet. 2. **`ateapi, atelet: carry the restore base snapshot in base_config`** — the cutover: ateapi sets `base_config` on both golden-data resume paths (external data snapshot, and local pause checkpoint combining with the golden); atelet reads and validates only `base_config`. The characteristic test of the ateapi→atelet request and the golden-data functional test pin the new field. 3. **`ateletpb: remove golden_snapshot_uri from RestoreRequest`** — the old field is deleted and `base_config` takes its field number, keeping the numbering dense. Tested: full ateapi + atelet suites including the controlapi functional tests; gofmt, golangci-lint, boilerplate clean. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
Whether a snapshot's files may be cached node-locally is the control plane's knowledge, not the node's: atelet should not infer it from URI layout or atespace conventions, which change as tags supersede golden snapshots. Add SnapshotSharing to the restore source message — deliberately the property (PRIVATE or SHARED) rather than the snapshot's kind, so new shared snapshot kinds need no atelet change — and set it in ateapi's resume workflow: SHARED when the restore reads the template's golden snapshot, PRIVATE for the actor's own snapshot. Nothing consumes the field yet.
Shared snapshot files (a template's golden snapshot today, tag snapshots later) are immutable once published, yet every restore that needs them re-downloads them into its own per-actor dir. Introduce the node-level cache that will hold them: a filecache.Store rooted under the ateom base path (cache hits will be served as hard links into per-actor restore dirs, which requires one mounted filesystem), opened at startup behind --snapshot-cache-dir with crash debris swept before atelet serves. No restore behavior changes yet: the store is wired into the herder but nothing reads through it. The cached download path and the eviction loop follow separately.
Snapshot files were downloaded into every actor's restore dir, even when a dozen actors resume from one shared snapshot on the same node. Route the cacheable legs of the restore download paths through the snapshot cache: the DATA_ON_GOLDEN base half (shared by construction), the local-pause combine leg, and an external restore whose source the control plane declared SHARED (a fresh-from-golden start). Private snapshots and requests from callers that predate the sharing field download fresh; atelet never infers cacheability from the URI. Cache hits are hard links to a read-only shared copy, so serving is gated per sandbox class: ateom-gvisor consumes restore-state strictly read-only and gets links; ateom-microvm rewrites config.json in place at restore and merges checkpoint deltas into memory-ranges' inode at suspend, either of which would corrupt a shared inode, so its class downloads fresh until a private-copy mode serves it.
The snapshot cache served only gVisor restores: its hits are read-only hard links, and ateom-microvm mutates staged files in place (config.json is rewritten at restore, checkpoint deltas are merged into memory-ranges' inode at suspend), which would corrupt a shared inode. Add cacheModeCopy: hits are hole-preserving private copies (see filecache.GetFileCopyTo), so the micro-VM class keeps its in-place mutations — no ateom-microvm changes — while concurrent restores of one snapshot still share a single download and later restores read the node cache instead of object storage. A link-mode fetch that fails with EXDEV (cache configured on a different mount than the restore dirs) now degrades to a copy too, keeping the download savings where it previously fell back to a fresh download per restore.
Dmitry Berkovich (dberkov)
force-pushed
the
filecache-m2
branch
from
September 26, 2026 16:27
232e494 to
05ca2ad
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements milestone M2 of the node-local artifact cache proposed in #690, on top of the Restore source refactor from #1602: atelet restores now serve shared snapshot files (a template's golden snapshot today, tag snapshots later) from a node-local cache backed by the
filecachestore from #1517, so concurrent restores of one snapshot share a single download and later restores read from disk instead of object storage.ateletpb, ateapi: declare how a restore's external snapshot is shared— cacheability is the control plane's knowledge, not the node's: atelet must not infer it from URI layout or atespace conventions, which change as tags supersede golden snapshots.SnapshotSharing(PRIVATE / SHARED) is deliberately the property rather than the snapshot's kind, so new shared snapshot kinds need no atelet change; ateapi's resume workflow sets SHARED when the restore reads the template's golden and PRIVATE for the actor's own snapshot. Nothing consumes the field yet.atelet: open a snapshot file cache at startup— lifecycle only:--snapshot-cache-dir(default<base-path>/snapshot-cacheviaateletpath.SnapshotCacheDir, empty disables; must share the actor dirs' mount because hits are hard links — the sharednodepath.UnderBasePathcheck, now also used by the image cache, warns otherwise) and--snapshot-cache-min-age; the store opens at startup andSweepDebrisclears crash debris before atelet serves. No behavior change yet.atelet: serve shared snapshot downloads from the file cache— routes the cacheable legs through the cache: the DATA_ON_GOLDEN base half (shared by construction), the local-pause combine leg, and an external restore whose source is declared SHARED (a fresh-from-golden start). Private snapshots — and requests from callers predating the field — download fresh. Serving is per sandbox class: gVisor consumes restore-state read-only and gets zero-cost hard links.atelet: serve micro-VM snapshot downloads as private copies— micro-VM cannot share inodes (ateom-microvm rewritesconfig.jsonin place at restore and merges checkpoint deltas intomemory-ranges' inode at suspend), so its hits are hole-preserving private copies (GetFileCopyTo): one download per snapshot per node, mutations stay private, no ateom-microvm changes. A link-modeEXDEV(cache on a different mount) degrades to a copy too, keeping the download savings.The M2 exit criterion — N concurrent restores of one shared snapshot = one download — is pinned at the unit level by
TestDownloadExternalCheckpointSharesOneGoldenDownload(4 concurrent restores, 1 GCS get, shared inode) and its copy-mode counterpart; the sharing gate has its own matrix test, and a functional test drives the golden-data resume end to end asserting the control plane declares SHARED on the fresh resume and PRIVATE on the actor's own snapshot.Still to come for M2 (will follow once this lands): the pressure-driven GC loop for the snapshot cache (mirroring
imagegc's watermark formula) — until then the cache grows unbounded, which matches today's behavior forstatic-filesand is bounded in practice by golden churn.Tested:
go test -race ./cmd/atelet/...plus the controlapi functional suite; every commit builds and passes tests individually; the cross-mount EXDEV fallback has a dedicated test that exercises two real filesystems where available (/dev/shmon Linux CI) and skips elsewhere;golangci-lint, gofmt, and boilerplate checks clean.🤖 Generated with Claude Code