Need a way to support non-S3 object storage (plugins) #1827

xref #1095
xref #1282
xref https://docs.google.com/document/d/1I2SlYey0Y2mcGQBZNpfYufWhwLbQDmaqI5T56sz33_4/edit?tab=t.0#heading=h.3gp3s0dosu0o
xref https://cloud-native.slack.com/archives/C0BT98L2WFQ

I am 8unsure that "blob storage" is the right layer for a plugin, but it's what we need right now. Why not? I was posed in interesting design question which I don't know how to answer yet -- if we wanted to use a per-actor block device (obviously a struggle at scale, but not every user is at scale) to store all the data for that actor -- how would we model it? Naively (warning: handwaving ahead), we could use some sort of filesystem snapshotting within the block device. We would not need cloud storage at all until we talk about fork/clone/tag operations. If I wanted to do that, what parts of the lifecycle of storage would need to be plugin-able beyond just GET/PUT blobs?

I don't yet have an answer and in service of trying to get to a v1 that does ONE USEFUL THING, we probably won't have time to fully explore it.

I am opening this issue to track the blob-storage plugin model.

Michelle Au (@msau42) Anish Ramasekar (@aramase) Benjamin Elder (@BenTheElder)