[Brendan Burns (brendandburns)]

Summary

• move snapshot storage operations behind a provider-neutral objectstorage.Store interface
• isolate GCS and S3 implementations behind a registrable backend factory
• add an Azure Blob Storage backend using the official Azure Go SDK
• support Azure connection strings or account URLs with DefaultAzureCredential
• make Locust result uploads extensible by URI scheme
• update architecture documentation, vendored dependencies, and generated licenses

Validation

• go test ./cmd/atelet/...
• python3 -m unittest runner_test.py from benchmarking/locust
• hack/verify/go-modules.sh
• hack/verify/licenses.sh
• hack/verify/gofmt.sh
• hack/verify/boilerplate.sh

[Tim Hockin (thockin)]

We need to think hard about how we want to do this. I do not want to go through this cloud-provider-extraction exercise again. Whether I like it or not S3 is the de facto standard API. We should probably excise the GCS=specific code and fall back to the S3 compatible code if we can.

But that doesn't solve the problem right? AIUI Azure doesn't have an S3 compatible API? That's a choice.

Options:

1. Import a gaggle of cloud SDKs and suffer the inevitable churn and other issues. By what principle do we say "yes" to some and "no" to others?

2. Require S3 compat. If you don't have S3 compat, someone needs to put a translation layer into the datapath and manage it, secure it, scale it, etc. I can't imagine that will be fun. It could be a "sidecar" to the components which need it, I suppose. Like Hashi's idea of plugins.

3. Allow Go plugins, with all their deficiencies. The core system can support it without owning all the plugins. It's on the installer to get the exact right plugin for the build they are using (we can do a DKMS sort of thing, but it's gross).

4. Something else?

[Brendan Burns (brendandburns)]

Tim Hockin (@thockin) I think that having a pluggable interface here is the right approach. Taking a dependency on a proprietary API that is supported by a subset of the cloud ecosystem (and even less of the on-premise ecosystem) feels like a bad idea for an open source project that wants to appeal to the entire industry. Additionally, it's not clear to me why linking in the Google and AWS SDKs are ok, and linking in the Azure SDK is not.

I'll admit that Go's plugin landscape is less than awesome, but if we're going down the side-car approach I would prefer a generic gRPC interface for storage all up and have everyone run a side-car vs. telling people who don't have access to S3 infrastructure to do something different, it seems like it is a setup for having two classes of users.

If you want me to refactor this in terms of a gRPC storage interface with appropriate implementations, let me know.

[Benjamin Elder (BenTheElder)]

So this was discussed previously in the community meeting, mailinglist, and slack ... I know multiple people from AKS interacted in those discussions.

We have loosely agreed to decouple away from GCP only but not towards "link in every other provider".

Tim Hockin (@thockin) I think that having a pluggable interface here is the right approach. Taking a dependency on a proprietary API that is supported by a subset of the cloud ecosystem (and even less of the on-premise ecosystem) feels like a bad idea for an open source project that wants to appeal to the entire industry. Additionally, it's not clear to me why linking in the Google and AWS SDKs are ok, and linking in the Azure SDK is not.

The GCP SDK is linked in because it was for bootstrapping the project, early prototypes ran on GKE, it's not indicative of the desired final state, we can use S3 on GCP as well.

We use AWS's SDK to talk to S3 outside of AWS, but it could well be any other mature client, happy to consider options.

It's not "linking azure is not ok" it's "we don't want to open the door to link in 50 SDKs".
The S3 SDK was actually added for the kind local clusters, not EKS.

The "subset of vendors" here is ... virtually all providers, and most self-hosted options:
https://blober.io/kb/articles/s3-compatible-storage-providers-complete-list/

I'm not actually aware of another cloud that completely lacks S3 support and there are plentiful on-prem options, we use one currently for CI.

If you want me to refactor this in terms of a gRPC storage interface with appropriate implementations, let me know.

How is that more reasonable than just using an S3 <> Azure Storage shim?
In slack and in the doc in the mailinglist we discussed Azure folks testing an S3<>Azure proxy implementation.

[Brendan Burns (brendandburns)]

In my mind the difference is that if everyone goes through the same interface (gRPC) then everyone is in the same boat wrt CI/CD etc.

By pushing Azure and Azure users (to use some sort of S3 shim) you are forcing a large subset of your potential userbase to go down a less reliable, less well tested path.

As an example, suppose we implement a proxy that only proxies the specific calls that agent-substrate uses today, if tomorrow you invoke a new S3 API or capability, there's no CI/CD or process to ensure that the Azure shim gets updated.

With gRPC everyone's in the same boat wrt to changing API surface areas and things like breaking changes are handled much more gracefully.

[Tim Hockin (thockin)]

Taking a dependency on a proprietary API that is supported by a subset of the cloud ecosystem (and even less of the on-premise ecosystem) feels like a bad idea for an open source project that wants to appeal to the entire industry.

FWIW, to date we have had zero candidate users balk at "s3 compatible".

if we're going down the side-car approach I would prefer a generic gRPC interface for storage all up and have everyone run a side-car vs. telling people who don't have access to S3 infrastructure to do something different, it seems like it is a setup for having two classes of users

I don't think that makes the argument you want it to. We should not penalize the majority of the (potential) user base just because.

Play it forward a bit - how do we unwind this at some point in the future? We can't host hundreds of thousands of LOC of unreviewable deps for all the same reasons kube couldn't. If this was a temporary "we'll do it for 6 months" story, I could hold my nose - so what does now + 6 months look like?

The cloud-provider extraction in kube was EXCEEDINGLY painful and expensive for providers and users.

Help us design an exit plan for this and I will be happy to endorse it.

[Brendan Burns (brendandburns)]

Tim Hockin (@thockin) I said this above, I think, but I'm 100% fine with a generic storage interface which extracts all the storage providers behind an agent substrate specific API for storage providers (e.g. the CSI model). And it doesn't have to be gRPC if you want to use some sort of Go plugin or other structure for something similar that's cool too.

If you do that you don't have to link any cloud SDKs.

[Tim Hockin (thockin)]

Talking to people who know better, Go plugins seems like a very bad idea. Will try to substantiate that opinion more.

I'm 100% fine with a generic storage interface which extracts all the storage providers behind an agent substrate specific API for storage providers (e.g. the CSI model)

Not a great analogy - CSI is not on the data path. I am not keen to impose an extra hop on the majority of the user base for no good reason, nor am I excited about adding the deployment complexity for the "normal" path.

Here's an idea, and I say this with no sarcasm - did we finally find a purpose for WASM?

[David Justice (devigned)]

did we finally find a purpose for WASM?

WASM is a possible route, but might be easier to keep the S3 path and offer a second externalized interface for other implementations. The easy path still works without infra complexity while also providing extensibility.

If the project was in Rust or other options (not Go), I would say WASM 100% (imho).

[Maya Wang (mayawang)]

Michelle Au (@msau42) this pluggable object storage PR from Brendan Burns (@brendandburns) looks like it answers #1827 (supporting non-S3 object storage through plugins). Could you say whether this is the direction you want? It's had no maintainer review since it opened in August.

[Benjamin Elder (BenTheElder)]

Maya Wang (@mayawang) we're actively discussing with Microsoft folks in other channels. There's a design doc under review.

[Benjamin Elder (BenTheElder)]

We've been working with Anish Ramasekar (@aramase) ILLUM1N0X to land a pluggable storage backend.

#777 (comment)