Feat/v8 bucket migration - #14
Merged
Merged
Conversation
…cy cache (flag-off)
Foundation for the client-side v8 fresh-bucket migration that sidesteps the
gc-damaged legacy bucket forests (which block writes). New content uploads route
to a fresh <base>-v8 sibling; reads merge legacy+v8; the legacy listing is cached
once (it is frozen post-migration). All behind a master flag default-OFF, so this
is a zero-behavior-change landing until the read-merge is wired into the UI
(Phase 2b) and the flag flips.
- BucketVersionResolver: writeBucket / readBuckets / isForbiddenWriteTarget,
managedBaseBuckets={images,videos,audio,documents}, suffix v8, enabled flag
(default false).
- Write routing in SyncService.queueUpload (the one chokepoint -> SyncTask +
SyncState.bucket) so sync-status / share / download see the real bucket.
- Read-only-legacy guard on FulaApiService write methods (not deleteObject):
refuses writes to a managed legacy bucket while enabled.
- FulaObject.sourceBucket (+ withSourceBucket, toJson/fromJson); listObjects
tags it.
- category_listing.dart: listCategoryMerged / listCategoryMergedCached /
listCategoryCached: merge legacy+v8 deduped prefer-v8, tag sourceBucket;
legacy error propagates, v8 error tolerated.
- LegacyListingCache (encrypted Hive legacy_listing_v1): freeze legacy once on a
FRESH load (incl. fresh-empty for new users), never on stale/failed; clear()
is the manual refresh.
- 33 device-free unit tests; full suite 384 pass / 24 skip / 0 fail.
- Phase 0 on-device premise harness + the migration plan doc.
Verified live in Phase 0: a fresh bucket round-trips on prod and its forest
nodes are cluster-pinned (replicated to 6 peers): durable, not a band-aid.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ting (Phase 2b, flag-off) Wire the content-category browser to the v8 read-merge so a category shows legacy + v8 uploads as one list, with the slow legacy listing cached once. Still behind BucketVersionResolver.enabled (default false) -> zero behavior change until the flag flips. - file_browser_screen: category view loads via listCategoryCached (frozen legacy U live v8), falling back to listObjectsCached when signed out; init LegacyListingCache before use. - Per-item routing: download + cloud-delete use cloudFile.sourceBucket (the bucket the object actually lives in) so legacy items still resolve. - Pull-to-refresh (_refreshCategory) clears the frozen legacy cache before reloading -- the escape hatch if a frozen listing was ever incomplete. - sync_service: restore-from-persistence re-routes UPLOAD tasks through writeBucket so a task queued to a legacy managed bucket before flag-flip doesn't strand at the read-only-legacy write guard; downloads keep their original bucket (the file may genuinely live in legacy). - user_id.dart: deriveUserId() (sha256(publicKey)[:16]) as the per-user legacy-cache key. - category_listing: drop an unnecessary non-null assertion. Flag-off proof: readBuckets collapses to [base], listCategoryCached takes the single-bucket path (no freeze, no cache), writeBucket + refresh-clear are no-ops. Full unit suite: 358 pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…footgun Advisor review (built-in + Gemini) of Phase 2b found the category-list path opened the encrypted legacy-cache Hive box on every signed-in open even with the flag OFF (init() was gated on userId, not on the resolver) -- undercutting the zero-behavior-change guarantee the commit rests on. And listCategoryCached accepted a `prefix` that, if ever passed, would freeze a filtered listing as the full-bucket cache (silent corruption). - file_browser_screen: gate the cache init()/merge path AND _refreshCategory's cache clear on `BucketVersionResolver.enabled && isManagedBase(bucket)`, so flag-off falls through to the plain listObjectsCached with no box opened. - category_listing: drop the `prefix` param from listCategoryCached (a category is a flat whole-bucket view; the frozen cache must represent the whole bucket). - import BucketVersionResolver into the browser screen. Unit suite: 358 pass. Flag-off verified a true no-op (no box opened, writeBucket + refresh-clear are no-ops, readBuckets collapses to [base]). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…n safe) Both Gemini and Cursor flagged that init() set _initialized only AFTER the await Hive.openBox, so two concurrent callers (an initial category load racing a pull-to-refresh, or two category screens opening at once) both passed the `if (_initialized) return` guard and opened the same encrypted box twice -- a concurrent double-open can throw. Memoize the in-flight Future so the box opens exactly once; _init swallows all errors so the memoized future never rejects. Flag-on path only (init is flag-gated as of the previous commit). Unit suite: 358 pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Premortem for each v8 migration phase (failure story -> root cause -> early signal -> gate), framed for a backup app where the worst case is silent data invisibility, not a crash. Augmented by Gemini + Cursor adversarial review (Cursor read the committed code and surfaced file:line gaps). Key gates surfaced (all flag-off-safe today; must-fix-before-flip): - B1: freeze only on a proven-COMPLETE legacy listing, else silently hide files - symmetry gap: v8-failure and legacy-throw must NOT collapse to silent-empty - legacy reads must be read-only (loadForest can attempt a v7->v8 write-back that both fails and mutates the supposedly-frozen bucket) - the client freeze is only coherent if the SERVER seals legacy vs writes (an old-app device can still write legacy -> silent divergence) - SyncState / linked-key / share-match / delete must thread sourceBucket - kill-switch: make the flag remote-controllable before a fleet flip - consolidated pre-flip checklist at the end (§11) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…d on-device (10.4 partial)
On-device E2E (flag-on diagnostic, real account) surfaced three v8-threading
bugs in the cloud browser, all now fixed + live-verified:
1. Cloud explorer showed v8 files as "cloud only" though on disk. The matcher
_findLocalFileForCloudObject was v8-blind: it required an exact
SyncState.bucket match, and _categoryFromBucket('images-v8') returned null
(dead directory fallback). Fix: match the bucket FAMILY (baseOf) + either
remoteKey OR remotePath (queueUpload only sets remotePath); _categoryFromBucket
strips the -v8 suffix.
2. A re-downloaded file stayed "cloud only" because _downloadCloudFile recorded
the SyncState under the base bucket. Fix: record cloudFile.sourceBucket.
3. Pull-to-refresh was slow: it cleared the frozen legacy cache and re-fetched
all 854 legacy objects. Legacy is immutable, so refresh now keeps the cache
and reloads only the live v8 part. (Escape-hatch via a freeze-TTL is a
follow-up -- see premortem B1.)
- BucketVersionResolver.baseOf(bucket) (+ unit tests).
- _categoryFromBucket v8-aware; explorer matcher family + (remoteKey||remotePath).
- _downloadCloudFile records the real (sourceBucket) bucket.
- pull-to-refresh -> _loadCategoryFiles (no legacy clear); removed _refreshCategory.
Reviewed by Gemini + Cursor. Live-verified on a moto g85: explorer status
correct, re-download links correctly, refresh fast. Flag stays OFF; 359 unit
tests pass. Does NOT close the full 10.4 gate -- reconciliation->sourceBucket,
queueUpload remoteKey, and readBuckets-aware linked-key queries remain (tracked
in docs/v8-bucket-migration-premortem.md). Unrelated pre-existing issues found
during the run: FxFiles#12 (lost-block 410), FxFiles#13 (users-index startup
stall).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…al, family links, freeze-TTL
Completes the content-category sync-state threading so links resolve across the
legacy<->v8 split (Cursor-reviewed; built-in advisor drove the caller-grep +
getSyncStateByRemoteKey/TTL safety calls). Flag stays OFF.
- queueUpload now sets SyncState.remoteKey (was only remotePath) so linked-key
lookups + the cloud-explorer matcher find fresh uploads (the root fix).
- category reconciliation records bucket = cloudFile.sourceBucket ?? base, and
HEALS a stale same-family bucket on a synced state (a file recorded under
'images' that now lives in 'images-v8') so share/delete -- which read the
EXACT SyncState.bucket -- target the right bucket.
- BucketVersionResolver.sameFamily(a,b) (+ tests). The two USED linked/mapping
lookups (getLinkedRemoteKeysWithPaths, getMappedRemoteKeysWithPaths -- both
cloud-only detection) are family-aware; getSyncStateByRemoteKey kept EXACT
(returns one authoritative state a caller may route on); removeMapping exact.
- LegacyListingCache freeze-TTL (1 day): getFrozen expires a stale freeze so a
one-off incomplete freeze (premortem B1) self-heals within a day; box v1->v2
({frozenAt,objects}); orphan v1 box dropped on init. (Replaces the
pull-to-refresh cache-clear that re-fetched all 854 legacy objects.)
Flag-off safe: no -v8 buckets exist so sameFamily==exact, reconciliation==base,
TTL path unused; only queueUpload remoteKey runs flag-off (self-correcting via
the deferred existence check). 361 unit tests pass; analyze clean.
STILL DEFERRED (premortem, NOT closed here): folder/tag share auto-update is
bucket-bound (sharing_service) -> P8; managed-legacy delete guard -> P4 (next).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… v8 deletes normally Per the simplified P4 design: a managed legacy content bucket (images/videos/ audio/documents) does NOT support deletion -- its objects are preserved so existing file/folder share links keep working (the HARD INVARIANT). Files in the fresh `-v8` bucket delete from cloud as normal. No tombstones needed. - fula_api_service: _guardLegacyDelete backstop on deleteObject (throws on a managed-legacy target when v8 enabled). Service deletes on metadata/app buckets (shares / playlists / whatsapp / shelf) are unaffected -- none are managed content bases. - file_browser: both delete handlers (_deleteFromCloud, _deleteCloudOnlyFile) resolve the item's REAL bucket (SyncState.bucket / cloudFile.sourceBucket) and, if it is a managed legacy bucket, show a friendly "legacy files can't be deleted" SnackBar and abort BEFORE the confirm dialog. v8 items proceed. - Flag-off safe: isForbiddenWriteTarget is false when disabled -> deletes work exactly as before. - 3 device-free guard tests (blocks legacy / allows v8 / inert when disabled). 364 unit tests pass; analyze clean (no errors). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
First metadata-bucket migration (Codex + Cursor + built-in reviewed). The metadata buckets are gc-damaged like content, so shelf cloud-sync was failing with 410. Route the shelf manifest write to dump-metadata-v8; read v8-then- legacy. Local Hive stays authoritative + restore is ADDITIVE, so a failed sync never loses data. - BucketVersionResolver: new managedMetadataBuckets set (incremental -- add a bucket only when ALL its writers are routed); writeBucket + isForbiddenWrite- Target now route/guard content OR metadata buckets; readBuckets/isManagedBase stay content-only (no list-merge for single-manifest metadata). - ShelfStorageService: write (encryptAndUpload), createBucket, listObjects use writeBucket(dump-metadata)=dump-metadata-v8; restore reads v8-then-legacy (prefer the v8 full snapshot, fall back to legacy if v8 absent/unreadable). - Flag-off safe: writeBucket==dump-metadata -> unchanged. - Tests: dump-metadata routes to -v8 + guarded but not list-merged; un-migrated metadata (tag/fula) still passes through. 366 unit tests pass. Per the reviews: the SHELF is additive-restore (safe with this simple approach); DESTRUCTIVE-restore services (e.g. TagStorageService clears local Hive on a bad cloud read) MUST have restore hardened before they migrate, and fula-metadata is 5 independent manifests (shares/collab/sync-mapping/folderSync/collab-group) -- both tracked for the next metadata sessions. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…+ clear on sign-out Corrects the shelf metadata model per owner review: - restore now MERGES both buckets (reads legacy AND v8, additive) instead of the wrong prefer-v8-else-legacy. v8 (the current snapshot) is applied first so it wins a conflicting id + owns the order; legacy then fills rows that exist ONLY in the pre-migration bucket (e.g. another device's items never re-synced to v8). The old fallback would silently lose a legacy-only / multi-device item. - sign-out now clears the shelf: ShelfStorageService.clearLocal() (wipes the dump_items + order + pending-deletes boxes) wired into AuthService.signOut alongside NFT/tags/shares/collab. Sign-out = wipe local FxFiles data; the next sign-in rebuilds the shelf from legacy + v8. - extracted _applyManifest (additive, per-blob) so the merge reuses one parser; removed the wrong _downloadManifestV8ThenLegacy helper. This makes the read model match the content buckets (always read legacy, combine v8 on top) and removes the false-empty risk: after sign-out clears local, login re-reads BOTH buckets, so an empty / un-migrated v8 can't leave the shelf blank. 366 unit tests pass (shelf order round-trip + v1-payload tests still green). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…e-merge restore Second metadata service (after shelf), using the corrected pattern. Tags' restore previously did clear-then-load (_tagsBox.clear() then load cloud) which, combined with sign-out-clears-local, meant a false-empty / un-migrated v8 read could WIPE local tags. Replaced with an additive MERGE. - BucketVersionResolver: tag-metadata added to managedMetadataBuckets. - TagStorageService: write (encryptAndUpload) + createBucket + listObjects use writeBucket(tag-metadata)=tag-metadata-v8; restoreFromCloud now MERGES legacy + v8 additively -- reads BOTH, v8 first wins a conflicting id, legacy fills gaps, never clobbers a tag already present locally. NO clear-on-restore. Sign-out still clears local (clearAll), so login rebuilds from both buckets. - Behavior change: restore is now local-wins / additive (was cloud-wins) -- this matches the shelf and removes the data-loss window. - Tests: dump + tag route to -v8 + guarded; un-migrated (fula/nft) pass through; fixed two pre-existing resolver tests that hardcoded tag-metadata as unmanaged. 366 unit tests pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…e-read helper - FulaApiService.downloadMetadataMerged(base,key,encKey): reads a per-user manifest from BOTH the -v8 sibling and legacy, returns the decrypted blobs in [v8, legacy] order (never throws; missing/erroring bucket skipped). Callers apply additively. Shared primitive for the remaining metadata services. - NftService: write/createBucket -> writeBucket(nft-metadata); restore gathers collections from both buckets (v8 wins a duplicate id) before the existing clear+rebuild+preserve-local-mints merge -- so a failed/empty v8 read can no longer drop collections (legacy is always read too). - resolver: nft-metadata added to managedMetadataBuckets; tests cover dump/tag/ nft routing + stable asset-bucket passthrough. 366 unit tests pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…er) to v8 website-metadata is SHARED by 2 writers (different keys); both routed before the bucket is added to managedMetadataBuckets: - WebsiteService: write/createBucket/listObjects -> writeBucket(website-metadata); restore gathers generations from [v8, legacy] (v8 wins a dup id) before the existing clear+rebuild+preserve-in-progress merge -- a failed/empty v8 read can no longer drop generations. - IpnsPointerService: write -> v8; the read-modify-write sync now reads BOTH buckets so a v8 write never drops legacy/other-device pointer entries; restore merges pointers from [v8, legacy] (already additive -- never clobbers a local pointer). 366 unit tests pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- AppStoreService: write/createBucket -> writeBucket(app-metadata); restore gathers activated apps from [v8, legacy] (v8 wins a dup appId) before the existing restore-on-empty load. Already additive/safe (no clear-on-restore). - resolver: app-metadata added. 366 unit tests pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…m.sourceBucket
The gc-damaged legacy dump/dump-thumbs forests rejected writes (410 Gone),
leaving shelf shares stuck "Syncing 0/1". Route shelf CONTENT writes to the
-v8 siblings (the metadata manifest already moved in P6).
- resolver: new managedContentKeyedBuckets={dump,dump-thumbs} -- route+guard,
NO list-merge (the shelf addresses each blob by the explicit manifest key; it
never lists these buckets).
- ShelfItem.sourceBucket (Hive field 19, hand-edited adapter, round-trip-tested):
records the body bucket at upload; drives delete routing (body
sourceBucket ?? writeBucket('dump'); thumb derived from the body's v8-ness so a
flag rollback cannot orphan v8 thumbs) and a future cloud body-download. The
crash-retry tombstone carries it too.
- shelf_service: the body rides SyncService.queueUpload's existing v8 chokepoint
(fixed by set-membership alone); the direct thumbnail PUT wraps writeBucket;
thumb re-fetch merges [v8, legacy]; bucket-ensure routes + uses a
bucket-NAME-specific init flag, so a stale "dump-thumbs created" flag cannot
skip creating dump-thumbs-v8 (which would NoSuchBucket silently).
- enabled stays true on this branch; the pre-flip gates still gate the prod merge.
Live-verified on moto g + server: a fresh share routed the body to dump-v8
(chunked content + forest + object, all 200) and the thumb to dump-thumbs-v8
(200 + pinned, P0 gc-safety, cluster_deferred=0). 367 unit tests pass, analyze
clean. Built-in + Cursor + Codex reviewed the design (Gemini quota-capped).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…d CID path, no forest, v8 inapplicable; nft-assets same
…rvices) fula-metadata is shared by 5 Flutter services (+ the portal); the gc-damaged legacy forest blocks writes (410). Route all 5 services' writes to fula-metadata-v8 and MERGE reads [v8, legacy] (legacy is never deleted), then activate by adding fula-metadata to managedMetadataBuckets -- one switch flips all five live at once. - Step 0: FulaApiService.downloadObjectMerged(base,key) -- the unencrypted sibling of downloadMetadataMerged; dedupes when unmanaged, skips 404, and RETHROWS hard errors so clear-then-load callers keep their cache (H1). - folder_watch: encrypted merge via downloadMetadataMerged (fresh-install + additive restore preserved). - cloud_collaboration_storage / cloud_share_storage: route writes / createBucket / delete; reads merge + combine the per-user manifests (revoked-id list = union). deleteShares routes to writeBucket (legacy preserved, P4). - cloud_sync_mapping: downloadMappings merges [v8, legacy]; a hard error is rethrown so ensureLoaded / relinkMappings never wipe the in-mem cache to a partial (v8-only) set. New cloud_sync_mapping_h1_test.dart (seam-based, 2 tests). - collaboration_service: the per-group manifest reads (accept + refresh) merge BOTH S3 copies via _downloadMergedManifest + CollaborationGroup.mergeWith (fork-proof against the portal's writes); the manifest write-unit (upload + storage-key lookup + manifest share-token) routes as a UNIT at the call sites, never inside the dual-use _getStorageKeyForPath; link 'b' -> writeBucket. - collaboration_group.dart mergeWith P2 fix: isRevoked is a sticky union and expiresAt takes the earlier value -- version-churn can no longer un-revoke a group or extend a shortened expiry. New collaboration_group_test.dart (6 tests). - Activate: 'fula-metadata' added to managedMetadataBuckets; resolver test now covers all 6 metadata buckets. Verified the 5 services are the ONLY fula-metadata writers (nothing strands at the legacy-write guard). 375 unit tests pass; all changed files analyze-clean. Plan reviewed by built-in + Cursor + Codex + Gemini over two rounds. The pinning-service portal (Part B) and a cross-repo E2E follow. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Introduce Type-B v8 routing and playlist tombstone merge logic. - Add face-metadata and playlists to the BucketVersionResolver managed write targets. - FaceDetectionService: add _inFlightPaths guard and a forceRescan flag; return existing faces when already scanned to avoid duplicate detections. - FaceStorageService: route creates/uploads/listing to the writeBucket and use downloadMetadataMerged for safe v8/legacy reads. - PlaylistService: route bucket operations to the writeBucket, implement per-id tombstones (playlist-deleted/...) to prevent legacy resurrection, merge legacy+v8 listings with v8-wins semantics and tombstone subtraction, and add helpers for tombstone parsing and not-found error detection. Adjust delete/create/upload flows to write tombstones before local removal and tolerate missing v8 bucket. - Add pure unit tests for playlist merge and tombstone parsing, and expand existing resolver/guard tests. - Bump package version and update fula_client dependency in pubspec.yaml. These changes enable safe migration to v8-managed buckets for Type-B data (face-metadata, playlists), prevent duplicate face records from racing scans, and ensure deleted playlists cannot be resurrected from legacy copies.
Introduce shareV8Bucket (wraps BucketVersionResolver.writeBucket) and route folder/tag/category shares to their -v8 sibling bucket when creating shares and public/password-protected links. Update SharingService to compute an effective bucket (effBucket) for enumeration-based shares and use it for storage key lookups, Fula token creation, listing objects, manifests and posted payloads. Normalize tag cloud candidates to the v8 sibling and make discovery/display lookups match bucket families via BucketVersionResolver.sameFamily (also applied in the SharesState provider). Add device-free unit tests for shareV8Bucket behavior (idempotence, managed/unmanaged buckets and flag-off passthrough). Import the resolver where required.
Add reporting for legacy (pre-v8) files that are not included in v8 folder/tag shares. Introduces a notIncludedCount on GeneratedShareLink and a _folderShareEmptyCheckAndNotIncluded helper in SharingService that refuses purely-pre-v8 folder shares with a clear re-upload message and returns a best-effort count of legacy files left out. Populate notIncludedCount across sharing/public-link creation paths and show a SnackBar in the create-share dialog when older files were excluded. Also improve user-facing error text to advise re-uploading older files when relevant. Legacy-listing errors are treated non-fatal so sharing is not blocked by counting failures.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.