Share snapshot storage settings through an ate-storage-config ConfigMap/Secret

Problem

Since #2224 the object-store settings (ATE_STORAGE_BACKEND, AWS_REGION, AWS_ENDPOINT_URL, AWS_S3_USE_PATH_STYLE, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY) are consumed by three containers that must agree:

the snapshot-plugin sidecar in ate-api-server (cleanup, copy)
the snapshot-plugin sidecar in atelet (checkpoint upload, restore download)
the atelet container itself (sandbox asset downloads via pkg/objectstorage; stays until that path moves off object storage)

Each is configured with its own inline env: block. The GCS default is repeated three times in manifests/ate-install/{ate-api-server,atelet}.yaml, and the rustfs settings are repeated three times across manifests/ate-install/kind/kustomization.yaml and manifests/ate-install/kind/atelet/kustomization.yaml. Nothing enforces they match. If they drift, ate-api-server deletes or copies snapshots in one bucket while atelet writes to another, and the failure surfaces only as a runtime NotFound on cleanup or restore.

That drift already happens on one supported path. ate-setup --deploy-ate-apiserver renders the bare ate-api-server.yaml with no kind branch (DeployAteAPIServer in cmd/ate-setup/internal/steps/deploy.go), while DeployAtelet switches to the kind/atelet overlay when --kind is set. On a kind cluster a targeted API-server redeploy therefore gets a snapshot-plugin sidecar pointed at GCS with no credentials, while atelet keeps writing to rustfs. The ATE_STORAGE_BACKEND refusal added to ate-api-server in #2224 does not catch this: the variable is on the sidecar, where it belongs; it is just the wrong value.

Proposal

Do what the install already does for OTLP (ate-otel-config) and for PostgreSQL (CreateAPIServerEnvVars: ConfigMap + Secret via optional envFrom, ate.dev/env-hash annotation so a change rolls the pods):

Add ate-storage-config (ConfigMap: backend, region, endpoint, path style) and ate-storage-credentials (Secret: access key/secret, present only where static keys are used, i.e. kind/rustfs). Base ships the GCS ConfigMap; the kind overlay ships the rustfs ConfigMap and Secret.
Replace the inline storage env: blocks on the three consumers with envFrom referencing both, optional: true on the Secret. Attach them only to the containers listed above; the ate-api-server container must not get them or rejectStorageEnv will exit the process.
Have ate-setup apply the ConfigMap/Secret in DeployAteSystem, DeployAteAPIServer, and DeployAtelet (next to applyOtelConfig), and give DeployAteAPIServer the same kind branch DeployAtelet has, or make it unnecessary by sourcing everything from the ConfigMap.
Delete the duplicated env: copies in the kind overlays and the explanatory comments that point at them.

Notes

env: entries override envFrom, so any remaining inline value silently wins. Step 4 is what makes the ConfigMap authoritative; verify with kubectl kustomize manifests/ate-install/kind --load-restrictor LoadRestrictionsNone.
Production GKE/EKS installs use workload/pod identity; the Secret is a kind convenience and the docs should say so.
When sandbox asset downloads leave pkg/objectstorage (separate issue), the atelet container drops out of the consumer list; the ConfigMap shape does not change.
Not GA-blocking: the shipped overlays are internally consistent today, and the known bad path is a dev-loop redeploy on kind.

Follow-up to #2224.

+1, and here's a reason for this beyond kind. We run an on-prem install against an S3-compatible store: static access keys, path-style addressing, its own endpoint. Today that takes an overlay that sets the same S3 settings on both objectstore-plugin sidecars. That's the drift this issue describes, outside kind.

Two updates to the consumer list since the issue was written:

Only the two objectstore-plugin sidecars read these settings now (Rename the snapshot-plugin sidecar, image and flag to objectstore-plugin #2540 renamed them from snapshot-plugin). atelet's own container stopped reading them in Fetch private sandbox assets through the object-store plugin #2439.
atelet's container now exits if ATE_STORAGE_BACKEND is set on it, as ate-api-server's does. So step 2's warning covers both main containers: the envFrom goes on the sidecars only.

(The kind redeploy in the Problem section was fixed by #2682.)

The one change that would let us drop the overlay is for ate-setup to render ate-storage-config from settings, the way apienv.go renders the PostgreSQL ConfigMap and Secret, instead of applying a static copy. As proposed, base ships the GCS values and only the kind overlay ships S3 values, so a non-kind S3 install would still need an overlay. And ate-setup applies with Force: true, so a ConfigMap edited after install goes back to GCS on the next run. Settings next to the existing storage.bucketName would do it:

storage.backend: gcs or s3
storage.s3.endpoint
storage.s3.region
storage.s3.pathStyle

For credentials, either of these works for us:

secret settings that render ate-storage-credentials, like ateapi.postgres.readWrite.connectionString does for PostgreSQL;
a pre-created ate-storage-credentials that ate-setup leaves alone outside kind, so it can come from an external secret manager.

One wording request for the Notes: on-prem S3-compatible stores (Ceph RGW, MinIO and the like) usually have no workload identity, so static keys are their production path. Could the docs call static keys supported for those stores, rather than "a kind convenience"?

skamboj

I'd like to pick this up. Since the issue was filed, only the two objectstore-plugin sidecars read these settings (#2439), and the kind redeploy case is fixed (#2682), so here's the scope I have in mind for a first PR:

An ate-storage-config ConfigMap (base: ATE_STORAGE_BACKEND=gcs; kind: the rustfs S3 settings) and an optional ate-storage-credentials Secret (kind only), consumed via envFrom by the two sidecars only. The inline env: copies in the base manifests and both kind overlays go away.
ate-setup applies them the way it applies ate-otel-config (deploy ate-system, deploy apiserver, deploy atelet), and stamps a digest of them on ate-api-server and atelet so a config change rolls both.
The fake-data-plane component reads the same envFrom instead of copying the sidecar's env.

One thing I ran into: kind's rustfs bucket-init Job can't be pointed at the shared Secret, because a Job's pod template is immutable and the change breaks deploy ate-system on existing kind clusters. So rustfs keeps its values inline, and a unit test checks they match the shared config.

I have this working locally. Effective container env is unchanged on every overlay, and it passes the unit tests, hack/verify-all.sh and a kind run: install main, upgrade, single-component redeploys, roll-on-change, and the demo e2e suites.

Rendering the ConfigMap/Secret from installer settings (storage.backend, storage.s3.*) for non-kind S3 installs, as skamboj suggested, would be a follow-up PR.

Is it OK to do this as a first PR and handle the installer settings in a follow-up?

SamPriyadarshi

Thanks for suggesting to take this SamPriyadarshi

I think it makes sense to split this into multiple PRs

Navigation Menu

Repository navigation

Share snapshot storage settings through an ate-storage-config ConfigMap/Secret #2378

Problem

Proposal

Notes

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Share snapshot storage settings through an ate-storage-config ConfigMap/Secret #2378

Description

Problem

Proposal

Notes

Activity

skamboj commented on Oct 16, 2026

SamPriyadarshi commented on Oct 19, 2026

ILLUM1N0X commented on Oct 23, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions