Fetch private sandbox assets through the object-store plugin by aramase · Pull Request #2439 · agent-substrate/substrate

Anish Ramasekar (aramase) · 2026-10-09T23:08:51Z

atelet held its own storage credential to download sandbox assets, and private micro-VM assets only worked on GCS or S3. Assets now go through the node's object-store plugin, which already has the credentials.

Add AssetProvider.FetchAsset to the plugin API.
The plugin refuses snapshot locations, stages downloads where atelet can't read them, and replaces the target only once the sha256 matches; pkg/assetfetch shares these checks with other plugins.
atelet asks the plugin first for every asset. Only for a gs:// URL, and only when the plugin returns NotFound, InvalidArgument, Unimplemented or Unavailable, does it fall back to an anonymous read (which serves public buckets such as gs://gvisor). It never falls back when the plugin refuses a snapshot location or the size or digest is wrong.
atelet re-checks the size and sha256 of whatever it caches, since the plugin is a separate, possibly third-party process and atelet executes these bytes.

Tested with the kind gVisor and micro-VM lanes before the plugin-first change; only the NFS external-volume tests failed, same as on main. Unit tests cover each fallback and non-fallback code.

Related #2063: assets in the cluster's own store no longer wait on an anonymous GCS lookup first.

Breaking change

Plugins must implement AssetProvider to serve sandbox assets; public gs:// assets still load anonymously if the plugin can't serve them. atelet exits if ATE_STORAGE_BACKEND is set on it; set storage settings on the snapshot-plugin sidecar. Please add the breaking-change label.

Fixes #2385

Tests pass
Appropriate changes to documentation are included in the PR

Anish Ramasekar (aramase) · 2026-10-09T23:13:26Z

cc ILLUM1N0X

Anish Ramasekar (aramase) · 2026-10-12T18:01:49Z

ILLUM1N0X thanks, both addressed in 8467c5c and 625a6b5.

On the anonymous read for private gs:// assets: agreed. atelet no longer has the storage config, so it can't tell which bucket is the snapshot bucket, and pinning gs://gvisor would break public mirrors. I flipped the order instead. atelet asks the plugin first for every asset, and falls back to an anonymous read only for gs:// when the plugin returns NotFound, InvalidArgument, Unimplemented or Unavailable. A snapshot-location refusal or a size/digest failure never falls back. This is also the order #2063 proposes.

On verifying again in atelet: the plugin is a separate process and may be third-party. pkg/assetfetch is a helper plugins can use, not something atelet can rely on, and atelet executes these bytes, so it checks them itself. The cost is one local re-hash. I added a line to the verifyAssetFile comment saying so.

Anish Ramasekar (aramase) · 2026-10-12T18:16:02Z

The e2e-test failure is unrelated: only TestActorEgressCredentialInjection fails, because httpbin.org returned 503s. The same test fails on main in this run. Everything else in the gVisor lane passed. I don't have permission to rerun jobs here; could a maintainer rerun the failed job once httpbin recovers?

ILLUM1N0X · 2026-10-12T18:20:04Z

The e2e-test failure is unrelated: only TestActorEgressCredentialInjection fails, because httpbin.org returned 503s. The same test fails on main in this run. Everything else in the gVisor lane passed. I don't have permission to rerun jobs here; could a maintainer rerun the failed job once httpbin recovers?

I would wait for #2518 to merge and rebase

Anish Ramasekar (aramase) · 2026-10-12T18:26:33Z

Thanks, #2518 merged and I rebased onto main (448e31f, no changes to the PR commits), so CI is running again with that test skipped.

…ned, snapshot-refusing reads

…plugin

… T-08

… in one pass

…asset staging

…ents

Zoe Zhao (zoez7) · 2026-10-12T20:23:58Z

+// longer reads it: object storage settings belong on the snapshot-plugin
+// sidecar. Only this variable is a signal: EKS pod identity sets AWS_* in
+// every container.
+func rejectStorageEnv() error {


Nit: do we need this check? If atelet never used the ATE_STORAGE_BACKEND env var, we wouldn't add this check right?

Zoe Zhao (zoez7) · 2026-10-12T20:27:27Z

+		return "", wrapFileSystemErr("while closing temp file", err)
+	}
+
+	anonymous := u.Scheme == "gs"


anonymous := u.Scheme == "gs"

Could you explain why "gs" scheme automatically means it's annoymous?

Zoe Zhao (zoez7) · 2026-10-12T20:33:53Z

+	if err != nil {
+		return "", "", err
+	}
+	if u.Scheme != "gs" && u.Scheme != "s3" {


🤖 blocking 🔴 – On an S3 backend this accepts a gs:// URI and reads its bucket name as an S3 bucket. For gs://gvisor/..., AWS answers 403 (someone else owns the gvisor bucket name) or 301, not 404. toStatus maps that to Internal, which is not in atelet's anonymousFallbackCodes, so atelet never falls back to the anonymous read and gVisor actors fail to start on EKS. kind only works because rustfs returns 404.

Reject a scheme that does not match the configured backend with InvalidArgument, so atelet falls back for gs:// on S3.

Zoe Zhao (zoez7) · 2026-10-12T20:37:48Z

+
+	// An object larger than the capacity cannot be staged whatever max_bytes
+	// allows, so a request never reserves more than all of it.
+	reserve := min(req.GetMaxBytes(), s.capacity)


🤖 should-fix 🟡 – atelet always sends max_bytes = 8 GiB and the manifest sets --asset-staging-capacity=8Gi, so every request reserves the whole capacity. All asset fetches on a node therefore run one at a time, even a small gVisor tarball. A request for a missing object also waits here, because open() only runs after Acquire. On kind, a gs://gvisor fetch that will end in NotFound can wait for an unrelated multi-GiB micro-VM download before atelet falls back to the anonymous read.

Suggested change: call open() before Acquire, so NotFound and other open errors return without waiting. Then reserve the object's actual size, capped at max_bytes, instead of max_bytes itself. ObjectStorage.GetObject does not expose a size today, so this needs a small interface addition. The GCS reader has Attrs.Size and S3 has ContentLength.

…ytes

Anish Ramasekar (aramase) · 2026-10-12T22:39:03Z

Zoe Zhao (@zoez7) thanks, all four addressed:

S3 accepting gs:// (blocking): fixed in 1fd4dca. The plugin serves only its backend's scheme and returns InvalidArgument otherwise, so gs://gvisor on EKS falls back to the anonymous read.
Staging reservation: fixed in 4d3ca6b. The object is opened before reserving, so NotFound returns immediately, and it reserves the object's reported size instead of max_bytes.
Why gs means anonymous: it doesn't; the plugin is always tried first, and gs only enables the anonymous fallback, since atelet's own client is anonymous GCS. Renamed to anonFallback with a comment in 7507304.
ATE_STORAGE_BACKEND check: it's a migration guard, so a stale atelet manifest fails at startup instead of the sidecar silently defaulting to GCS. Happy to drop it after a release or two.

…rns Internal Fixes an issue introduced in agent-substrate#2439 where running on GKE with a GCS backend causes gs://gvisor asset downloads to fail. When querying gs://gvisor using the cluster's ambient GCP credentials, GCS returns 403 Forbidden, which status.go maps to codes.Internal. Because codes.Internal was omitted from anonymousFallbackCodes, atelet aborted rather than falling back to an anonymous GCS read. This adds codes.Internal to anonymousFallbackCodes and adds unit tests covering the GCS Internal fallback path. - **Unit tests**: `go test ./cmd/atelet/...` passes cleanly. - **E2E validation on GKE**: Verified on Rapid GKE 1.37 (`DATAPLANE=envoy`) with Counter demo: - Actor creation, memory persistence across requests 1 and 2, suspension to snapshot, and resumption from snapshot on request 3 all pass 100%. Fixes agent-substrate#2613 - [ ] Tests pass - [ ] Appropriate changes to documentation are included in the PR

…Internal (agent-substrate#2614) Fixes agent-substrate#2613 On GKE with a GCS backend, the object-store plugin's Workload Identity token gets `412 PreconditionFailed` ("The operation requires that Uniform Bucket Level Access be enabled.") from the public `gs://gvisor`. `toStatus` maps that, like any error it doesn't recognize, to `Internal`, and atelet didn't fall back on `Internal`, so sandbox creation failed after agent-substrate#2439. **Changes** - atelet falls back to an anonymous read on `Internal`. In practice that's any unexpected plugin failure. That's acceptable because anonymous reads only reach public objects, and atelet still checks the sha256. - So a broken plugin isn't hidden: the plugin logs a scrubbed warning when it returns `Internal`, and atelet warns when it falls back. - Updated the `AssetProvider` contract, `api-guide.md` and threat model T-08. **Testing** - Unit tests: `go test ./cmd/atelet/... ./internal/objectstoreplugin/... ./pkg/objectstorage/...` - GKE 1.37 (Envoy dataplane): reproduced the 412 on `main`. With this PR, gVisor prewarm falls back and succeeds, and the counter demo passes create, persistence across requests, suspend, and resume from snapshot. A follow-up issue will discuss the long-term design for plugin fallback. --------- Co-authored-by: Anna Pendleton <13793293+annapendleton@users.noreply.github.com>

Anish Ramasekar (aramase) marked this pull request as ready for review October 9, 2026 23:38

ILLUM1N0X reviewed Oct 12, 2026

View reviewed changes

Comment thread cmd/atelet/sandbox_assets.go Resolved

Comment thread cmd/atelet/sandbox_assets.go Resolved

Eitan Yarmush (EItanya) mentioned this pull request Oct 12, 2026

Support plugin-backed artifact volumes for S3 and other sources #2514

Open

Anish Ramasekar (aramase) force-pushed the aramase/f/sandbox_asset_provider_v2 branch from 78e1bd0 to 625a6b5 Compare October 12, 2026 18:01

ILLUM1N0X approved these changes Oct 12, 2026

View reviewed changes

Anish Ramasekar (aramase) force-pushed the aramase/f/sandbox_asset_provider_v2 branch from 625a6b5 to 448e31f Compare October 12, 2026 18:25

Anish Ramasekar (aramase) added 9 commits October 12, 2026 12:17

feat(snapshot-plugin): serve AssetProvider.FetchAsset with sha256-pin…

1902fb1

…ned, snapshot-refusing reads

feat(atelet)!: fetch private sandbox assets through the object-store …

89b2e53

…plugin

fix(atelet): mask sandbox asset URLs in RPC request logs

6f1a738

docs(threat-model): describe what the asset plugin checks cover under…

24fcb8f

… T-08

fix(snapshot-plugin): cap asset URI length and find snapshot prefixes…

ba3c60c

… in one pass

refactor(snapshot-plugin): move AssetProvider checks to pkg/assetfetch

65296b9

fix(snapshot-plugin): replace asset write paths atomically and bound …

4b88ba3

…asset staging

fix(atelet): fetch sandbox assets through the object-store plugin first

a99160b

docs(atelet): describe plugin-first asset order in flag help and comm…

21a1647

…ents

Anish Ramasekar (aramase) force-pushed the aramase/f/sandbox_asset_provider_v2 branch from 448e31f to 21a1647 Compare October 12, 2026 19:18

Zoe Zhao (zoez7) reviewed Oct 12, 2026

View reviewed changes

Anish Ramasekar (aramase) added 3 commits October 12, 2026 14:46

fix(snapshot-plugin): reserve asset staging by object size, not max_b…

4d3ca6b

…ytes

fix(snapshot-plugin)!: serve only the storage backend's asset URI scheme

1fd4dca

refactor(atelet): name the gs:// anonymous fallback condition

7507304

ILLUM1N0X requested a review from Zoe Zhao (zoez7) October 12, 2026 22:59

Zoe Zhao (zoez7) enabled auto-merge October 12, 2026 23:03

Zoe Zhao (zoez7) approved these changes Oct 12, 2026

View reviewed changes

Zoe Zhao (zoez7) added this pull request to the merge queue Oct 12, 2026

Merged via the queue into agent-substrate:main with commit 16ccb13 Oct 12, 2026
14 checks passed

Anish Ramasekar (aramase) deleted the aramase/f/sandbox_asset_provider_v2 branch October 13, 2026 01:39

ILLUM1N0X mentioned this pull request Oct 13, 2026

[Bug]: atelet asks anonymous public GCS before the cluster's object store for every uncached sandbox asset #2063

Closed

2 tasks done

Anna Pendleton (annapendleton) mentioned this pull request Oct 13, 2026

[Bug]: GCS backend fails to fall back to anonymous read for gs://gvisor sandbox assets #2613

Closed

2 tasks done

This was referenced Oct 13, 2026

fix(atelet): fall back to anonymous GCS read with objectstore-plugin Internal #2614

Merged

Design: how should object-store plugins signal that atelet may read a public sandbox asset anonymously? #2630

Open

skamboj mentioned this pull request Oct 14, 2026

ate-setup: allow overriding the bundled gVisor asset location #2649

Open

skamboj mentioned this pull request Oct 16, 2026

Share snapshot storage settings through an ate-storage-config ConfigMap/Secret #2378

Open

Grant McCloskey (MushuEE) mentioned this pull request Oct 28, 2026

[EPIC] Make repository Provider Neutral #3161

Open

Steven Shriver (stshrive) mentioned this pull request Nov 5, 2026

[FEATURE] atelet fetches public sandbox assets over HTTPS #3672

Open

Navigation Menu

Repository navigation

Fetch private sandbox assets through the object-store plugin - #2439

Fetch private sandbox assets through the object-store plugin#2439
Zoe Zhao (zoez7) merged 12 commits into
agent-substrate:mainfrom
aramase:aramase/f/sandbox_asset_provider_v2

Anish Ramasekar (aramase) commented Oct 9, 2026 •

edited

Loading

Uh oh!

Anish Ramasekar (aramase) commented Oct 9, 2026

Uh oh!

Uh oh!

Uh oh!

Anish Ramasekar (aramase) commented Oct 12, 2026

Uh oh!

Anish Ramasekar (aramase) commented Oct 12, 2026

Uh oh!

ILLUM1N0X commented Oct 12, 2026

Uh oh!

Anish Ramasekar (aramase) commented Oct 12, 2026

Uh oh!

Zoe Zhao (zoez7) Oct 12, 2026

Uh oh!

Zoe Zhao (zoez7) Oct 12, 2026

Uh oh!

Zoe Zhao (zoez7) Oct 12, 2026

Uh oh!

Zoe Zhao (zoez7) Oct 12, 2026

Uh oh!

Anish Ramasekar (aramase) commented Oct 12, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

Anish Ramasekar (aramase) commented Oct 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Breaking change

Uh oh!

Anish Ramasekar (aramase) commented Oct 9, 2026

Uh oh!

Uh oh!

Uh oh!

Anish Ramasekar (aramase) commented Oct 12, 2026

Uh oh!

Anish Ramasekar (aramase) commented Oct 12, 2026

Uh oh!

ILLUM1N0X commented Oct 12, 2026

Uh oh!

Anish Ramasekar (aramase) commented Oct 12, 2026

Uh oh!

Zoe Zhao (zoez7) Oct 12, 2026

Choose a reason for hiding this comment

Uh oh!

Zoe Zhao (zoez7) Oct 12, 2026

Choose a reason for hiding this comment

Uh oh!

Zoe Zhao (zoez7) Oct 12, 2026

Choose a reason for hiding this comment

Uh oh!

Zoe Zhao (zoez7) Oct 12, 2026

Choose a reason for hiding this comment

Uh oh!

Anish Ramasekar (aramase) commented Oct 12, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Anish Ramasekar (aramase) commented Oct 9, 2026 •

edited

Loading