Skip to content

Support plugin-backed artifact volumes for S3 and other sources #2514

Open

Description

Could Substrate support mounting immutable content from sources like S3, similar to the existing OCI image volume support?

The use case is making skills, plugins, model files, or other assets available before a container starts, including during golden initialization. Fetching these outside the guest would let workloads initialize without requiring guest egress.

I’m thinking of one generic artifact volume source, with backend-specific fetching handled by a plugin. Substrate would own the mount and lifecycle behavior, while the plugin handles retrieving the content and accessing credentials. Adding support for another backend wouldn’t require adding another volume type to the core API.

#2439 seems related: AssetProvider.FetchAsset already accepts a URI, expected SHA-256, destination, and size limit. Could that contract also support workload volumes, perhaps by returning a standard archive that Substrate extracts and mounts read-only?

The important properties would be:

  • Contents are available before container startup and golden initialization.
  • Resume and cloning use the same pinned contents.
  • Credentials remain outside the guest, with authorization for the requested content.
  • Downloads and extraction complete before the mount becomes available.

We could package these assets as OCI images today, but that adds a publishing step. CSI is another possibility, though its current external-volume lifecycle and cloning restrictions seem like a less direct fit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions