[Anna Pendleton (annapendleton)]

Fixes #2613

On GKE with a GCS backend, the object-store plugin's Workload Identity token gets 412 PreconditionFailed ("The operation requires that Uniform Bucket Level Access be enabled.") from the public gs://gvisor. toStatus maps that, like any error it doesn't recognize, to Internal, and atelet didn't fall back on Internal, so sandbox creation failed after #2439.

Changes

• atelet falls back to an anonymous read on Internal. In practice that's any unexpected plugin failure. That's acceptable because anonymous reads only reach public objects, and atelet still checks the sha256.
• So a broken plugin isn't hidden: the plugin logs a scrubbed warning when it returns Internal, and atelet warns when it falls back.
• Updated the AssetProvider contract, api-guide.md and threat model T-08.

Testing

• Unit tests: go test ./cmd/atelet/... ./internal/objectstoreplugin/... ./pkg/objectstorage/...
• GKE 1.37 (Envoy dataplane): reproduced the 412 on main. With this PR, gVisor prewarm falls back and succeeds, and the counter demo passes create, persistence across requests, suspend, and resume from snapshot.

A follow-up issue will discuss the long-term design for plugin fallback.

[ILLUM1N0X]

One thing the description should say: toStatus maps every unrecognized plugin error to Internal. So in practice this change means "fall back on any unexpected plugin failure," not just "fall back on GCS 403." I think that's fine. The main cost is that a broken plugin is now easier to miss.

Could you also paste the body of the 403 from the plugin logs? It'd be good to record the actual cause in the PR.

[Anna Pendleton (annapendleton)]

Neither component logged the actual error: the plugin doesn't log failures, and atelet deliberately drops the plugin's status text so credentials can't leak into crash messages, so all we have is code=Internal. I'm updating this PR so the plugin logs any error it maps to Internal and atelet warns whenever it falls back after a plugin failure, and I'll paste the error from a fresh GKE run once it's redeployed.

[Benjamin Elder (BenTheElder)]

thanks for working on a fix, this is blocking all manual testing on GKE

note: you can workaround it in the short term by running your own authenticated mirror ... but it's not great.

[Anna Pendleton (annapendleton)]

After getting local logging into the plugin and atelet for the issue, it looks like I'm seeing a 412 error instead (still an unrecognized error which is why it gives Internal)

objectstore-plugin:
{"time":"2026-10-14T03:37:22.496348953Z","level":"WARN","msg":"Object-store plugin failed with an unrecognized error; returning Internal","http_status":412,"reasons":[],"service_code":"PreconditionFailed","message":"The operation requires that Uniform Bucket Level Access be enabled."}

atelet:
{"time":"2026-10-14T03:37:22.496702505Z","level":"WARN","msg":"Object-store plugin failed to fetch sandbox asset","url":"gs://gvisor/releases/nightly/2026-10-10/x86_64/gvisor.tar.zstd","code":"Internal"} {"time":"2026-10-14T03:37:22.496790374Z","level":"WARN","msg":"Sandbox asset prewarm failed; will retry","config":"gvisor-default","retries":0,"err":"while fetching sandbox asset \"gvisor\": sandbox asset gs://gvisor/releases/nightly/2026-10-10/x86_64/gvisor.tar.zstd: the object-store plugin failed (Internal)"}

Jetski analysis of this failure on GKE is that the objectstore-plugin authenticates with a Workload Identity federation token, which GCS rejects on buckets that use object ACLs instead of uniform access, like the public gs://gvisor. Anonymous reads of the same object succeed.

Thanks for all the feedback, reading through the comments rn

[Anna Pendleton (annapendleton)]

Thanks all for the thorough review! I've pushed an update addressing the feedback:

• Root cause: GCS returns 412 PreconditionFailed ("The operation requires that Uniform Bucket Level Access be enabled.") to the plugin's Workload Identity token for gs://gvisor, not a 403. toStatus maps it to Internal, like any error it doesn't recognize.
• Visibility: the plugin now logs a scrubbed warning when it returns Internal, and atelet warns when it falls back after Internal/Unavailable.
• Docs: comments, the AssetProvider contract, api-guide.md and T-08 now reflect the new fallback behavior.

I reran validation of this version on GKE 1.37 (Envoy dataplane): gVisor prewarm falls back and succeeds, and the counter demo passes create, persistence across requests, suspend, and resume from snapshot. I'll also open an issue to discuss the longer-term design for plugin fallback.

[Anna Pendleton (annapendleton)]

ILLUM1N0X Updated the description to say exactly that. The real error (posted above) is a 412 PreconditionFailed ("Uniform Bucket Level Access be enabled"), not a 403, and the plugin now logs it so a broken plugin isn't silent.

[Anna Pendleton (annapendleton)]

Benjamin Elder (@BenTheElder) Thanks! Aiming to land this quickly to unblock GKE testing, and I'll follow up on the long-term fallback design separately.

[Anna Pendleton (annapendleton)]

Opened #2630 to discuss the longer-term design for plugin fallback to anonymous reads of public assets, including Unknown from third-party plugins. Please add thoughts there so this PR can stay focused on unblocking GKE.

[Anna Pendleton (annapendleton)]

Something I don't have an immediately good idea on how to handle - the error message proliferation - error messages need to be parsed in a different way across our existing in-tree providers, and I figured each provider would want control over what info is surfaced per error (in case there is sensitive info in an error message).

For now I have the init adding optional per provider processing in but it's not my favorite long-term option as it won't play nicely with 3rd party plugins and it loops through each option regardless which in-tree provider is being used - I think long-term I'd prefer something more extensible.

Curious if anyone has any thoughts here and if we need to solve this cleanly in this PR or if it could be a followup cleanup issue.

[Anish Ramasekar (aramase)]

Anna Pendleton (@annapendleton) I wouldn't solve it here. Since every backend lives in pkg/objectstorage, DescribeServiceError could just call describeGCSError before the HTTPStatusCode fallback and skip the init registry, and third-party plugins log their own errors anyway, so the extensible version can be a follow-up.

[Lior Lieberman (LiorLieberman)]

We are cutting 0.5 today and would like this to be included. Can we make sure this is approved and merged in the next few hours?

[Anna Pendleton (annapendleton)]

I'm working on quick fix to address Anish Ramasekar (@aramase) 's feedback to remove the init registry, then I'd say this is ready for submission if there's no other outstanding blockers. I'm creating an issue for the extensible error messaging to track that for future PR.

[Benjamin Elder (BenTheElder)]

one nit for follow-up.

[Benjamin Elder (BenTheElder)]

🤖 nit 🟢 – "The plugin logs why it failed" only holds for Internal. toStatus logs its catch-all, but the plugin's own Unavailable statuses aren't logged, and an unreachable plugin can't log anything. Maybe scope that sentence to Internal?