Auto-recover corrupt packfiles in packfile maintenance#2052
Open
tyrielv wants to merge 1 commit into
Open
Conversation
When a packfile in the shared object cache is corrupt or truncated (e.g. from a
past disk-full event), 'git multi-pack-index write/verify' fails with "could not
load pack N". The existing self-heal only deletes and rewrites the
multi-pack-index (MIDX), which does not fix the underlying pack: the rewrite
re-scans the same bad pack and keeps failing. The corruption then recurs
indefinitely.
PackfileMaintenanceStep now routes write/verify failures through a recovery path
that, when git reports a pack-load failure:
- Detection (always runs, even with recovery disabled): verifies each pack in
the object cache with 'git verify-pack' and reports every unreadable pack via
telemetry (Operation=FoundCorruptPack). The "could not load pack N" ordinal is
an internal MIDX position, not a filename, so per-pack verification is how we
find the actual bad file.
- Removal (gated, see kill switch below): deletes each corrupt pack's files
(.pack/.idx/.keep/.rev; Operation=DeletedCorruptPack), then deletes and
regenerates the MIDX from the packs that remain (fast path, no full repack).
Missing objects are re-fetched on demand.
- Corrupt prefetch pack (special case): prefetch packs are incremental and
ordered by timestamp, so a corrupt one invalidates every later prefetch pack
too - leaving a hole would let the newest surviving timestamp advance past it
so a later prefetch never backfills the gap. Recovery removes the corrupt
prefetch pack and every later prefetch pack
(Operation=DeletedHealthyPrefetchPack for the healthy ones removed purely due
to ordering), then requests a prefetch (via a callback GitMaintenanceScheduler
wires to a PrefetchStep, only when using a cache server) to re-download them
and rebuild the commit-graph.
Kill switch: the destructive pack removal is gated by a new git config,
gvfs.enable-packfile-recovery (default true). When false, GVFS still detects and
reports corrupt packs (Operation=FoundCorruptPack, then
CorruptPackRecoverySkipped) but deletes nothing and does not request a prefetch;
the non-destructive MIDX rewrite still runs, so behavior degrades to today's.
This gives a field kill switch without a redeploy if the destructive path ever
misbehaves.
This is stacked on the git-output bounding change: recovery runs additional git
commands (verify-pack, MIDX rewrites) against the corrupt repo, so it relies on
that change to keep a noisy stderr from OOM-ing the mount mid-recovery.
Tests:
- A verify failure reporting a pack-load error removes the corrupt pack and
rewrites the MIDX from the remaining good packs (recovery enabled).
- With recovery disabled, the same failure still verifies each pack and reports
the corrupt one but deletes nothing.
- A corrupt prefetch pack removes it and every later prefetch pack, keeps the
earlier healthy one, and requests a prefetch.
Assisted-by: Claude Opus 4.8
Signed-off-by: Tyrie Vella <tyrielv@gmail.com>
cacf77d to
284fdb5
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
When a packfile in the shared object cache is corrupt/truncated (e.g. from a past disk-full event),
git multi-pack-index write/verifyfails withcould not load pack N. The existing self-heal only deletes+rewrites the multi-pack-index (MIDX), which does not fix the underlying pack — the rewrite re-scans the same bad pack and keeps failing, so the corruption recurs indefinitely.Change
PackfileMaintenanceSteproutes write/verify failures through a recovery path. When git reports a pack-load failure:git verify-packand reports every unreadable pack via telemetry (Operation=FoundCorruptPack). TheNincould not load pack Nis an internal MIDX position, not a filename, so per-pack verification is how we find the actual bad file..pack/.idx/.keep/.rev;Operation=DeletedCorruptPack), then deletes+regenerates the MIDX from the packs that remain (fast path, no full repack). Missing objects are re-fetched on demand.Operation=DeletedHealthyPrefetchPackfor the healthy ones removed purely due to ordering), then requests a prefetch (via a callbackGitMaintenanceSchedulerwires to aPrefetchStep, only when using a cache server) to re-download them and rebuild the commit-graph.Kill switch
The destructive pack removal is gated by a new git config
gvfs.enable-packfile-recovery(defaulttrue). Whenfalse, GVFS still detects and reports corrupt packs (FoundCorruptPack→CorruptPackRecoverySkipped) but deletes nothing and requests no prefetch; the non-destructive MIDX rewrite still runs, so behavior degrades to today's. This is a field kill switch — no redeploy needed if the destructive path ever misbehaves.Telemetry
Distinct, queryable
Operationvalues so rollout can be watched:FoundCorruptPack(even when disabled),CorruptPackRecoverySkipped,DeletedCorruptPack,DeletedHealthyPrefetchPack,PrefetchRestoreUnavailable.Tests
Full unit suite: 885 passed, 0 failed (11 pre-existing native-hook skips). The pre-existing
PackfileMaintenanceRewriteOnBadVerify(empty-stderr verify failure → MIDX rewrite only, no pack detection) is unchanged.Follow-up
A functional test that corrupts a real pack and asserts the mount self-heals (rather than looping) is tracked for the streaming follow-up PR.