[Feature] Investigate an erasure-coded Driver pool that stores one file across multiple limited accounts #3150

Please confirm the following

• I have read and agree to AGPL-3.0 Section 15. The program is provided "as is" without any warranties; you bear all risks of using it.
• I have read and agree to AGPL-3.0 Section 16. The copyright holders and distributors are not liable for any damages resulting from the use or inability to use the program.
• I confirm my description is clear, polite, helps developers quickly locate the issue, and complies with community rules.
• I have read the OpenList documentation.
• I confirm there are no duplicate issues or discussions.
• I believe this issue must be handled by OpenList and not by a third party.
• I confirm this feature has not been implemented yet.
• I confirm this feature is reasonable and has general demand, not just my personal need.
• I have not read these checkboxes and therefore I just ticked them all, Please close this issue.

Feature Description

The use case is one file larger than the available capacity of any individual
free-tier Driver/account, but smaller than the usable combined capacity of
several eligible accounts. Ordinary whole-file balancing cannot satisfy it.
The current .balance mechanism rotates a storage choice by request, not by a
durable object placement record. Alias quota placement chooses one destination;
Chunk splits into parts under one remote path. None is a recoverable,
cross-account, single-file erasure pool.

Please investigate a RustFS-inspired logical pool -> stable placement set ->
k data plus m parity shards -> committed object generation -> verified
read/heal architecture, adapted to heterogeneous cloud Drivers rather than
copied as a literal RustFS disk cluster. Aggregate capacity is finite, not an
unlimited-storage claim; parity, quota headroom, rate limits and egress reduce
usable capacity. The pool must report these limits honestly.

Suggested Solution

1. Define an eligible Driver capability and failure-domain matrix: stable
   object keys, streaming write/read, Range, consistency, limits, rename/delete,
   quota observability, retries, provider terms, cost and account independence.
   Reject or degrade unsupported Drivers explicitly.
2. Define versioned pool/set membership and a durable manifest per committed
   object generation: logical length, encoding k,m, stripe/block layout,
   ordered shard IDs, account/Driver references, checksums and commit state.
   Never infer completeness from a directory listing; a lost final shard must
   be detectable. Keep provider credentials outside manifests.
3. Stage shards under stable idempotent identities, verify receipts, and
   publish one manifest generation only after the chosen write/metadata quorum
   is met. Readers use only that generation, verify data, reconstruct from
   k independent valid shards, and return an explicit error below threshold.
   Fence overwrite/delete/rename from old readers; reconcile ambiguous writes
   and orphaned staging objects. Define background/on-read/manual repair and
   what happens when pool membership or credentials change.
4. Run a disposable prototype against local and S3-like accounts with per-
   account quotas. Test a file larger than each target but within effective pool
   capacity; full and cross-stripe Range reads; missing first/middle/last shard;
   corruption, outage, full account, timeout, restart, overwrite, cleanup and
   manifest backup/restore. Measure throughput, memory/temp, provider calls,
   egress and repair cost. Gate production format and UI on this evidence.
5. Audit every ingress/egress path—browser direct upload, multipart, WebDAV,
   S3, copy, archive extraction, signed links and previews. A pooled object may
   refuse an unsupported path; it must never report success for a partial
   upload or expose raw shard objects as user files.

Additional Information

RustFS architecture documents
one object in one erasure set; its erasure-coding contract
separates layout, data/parity shards, quorum, checksums and healing. These are
reference semantics, not a proposal to import RustFS's on-disk format,
fixed set geometry or peer RPC into OpenList. Relevant OpenList source:
pkg/utils/balance.go, internal/op/path.go, drivers/alias, drivers/chunk,
internal/driver/driver.go. Explicit product decisions still needed: default
k,m/loss tolerance, single versus multiple OpenList writers, metadata
durability, migration, acceptable egress and restore time.
#3141 concerns multi-connection copy between storages, not one logical file
distributed across accounts; avoid conflating the requests.

AI Generated Content

• I used AI tools to generate this content
• I did not use AI tools to generate this content

AI model used

OpenAI Codex (exact model identifier not exposed to the agent)