Repository navigation
Flush pending own-node saves on hub dispose (acked-write loss; TwoSiloRecycleConvergence flake) - #627
Merged
2 commits merged intoJul 24, 2026
Merged
2 commits merged into
2 commits merged into
Conversation
…istence sample on dispose Root cause of the TwoSiloRecycleConvergenceTest flake (main run 30068597014, shard 2): a PatchDataRequest that lands in the owner hub's Quiescing window (after Dispose() posted the DisposeRequest, before RunLevel reaches DisposeHostedHubs - where the #609 ShuttingDown NACK gate starts) is processed normally: the patch commits in-RAM and the owner ACKs with PatchDataResponse Success. The own-node persistence sampler (MeshDataSource.SubscribeToOwnDeletion) debounces updates with Sample(200 ms), and the ShutDown phase's DisposeImpl disposed that subscription with the save still pending - and its RunLevel > Started branch silently DISCARDED a sampled value during teardown. The acknowledged write was durably lost; the reactivated grain loaded the stale persisted version, so the test's WaitForPersistedBeyond timed out at the pre-recycle version (trx: resilient write "succeeded" instantly, store never advanced past v12). Creates/deletes were already covered by MeshNodeTypeSource's FlushPendingWrites reactive dispose action; updates were not. Fix (update-path twin of that existing flush): - PendingOwnSave tracker: stamps every save-worthy own-node state ahead of the Sample buffer (Do), and marks a state dispatched only when its SaveMeshNodeRequest was ACCEPTED by the hub (Post did not return the teardown-guard Failed delivery). Holds only MeshNode references - nothing in the timer chain can pin the hub (preserves the abandoned-hub collectability the static-local-function sampler shape exists for). - Sampler teardown branch (RunLevel > Started): stop posting but keep tracking instead of self-disposing-and-discarding, so quiesce-window writes reach the tracker; DisposeImpl bounds the chain's lifetime via RegisterForDisposal. - FlushPendingOwnSave: reactive dispose action that persists the latest undispatched state straight through IStorageAdapter.Write (same guards as HandleSaveMeshNode: recently-deleted tombstone, PersistedSnapshot identity, Version >= 1). The write leaf runs on the mesh IO pool, which outlives the hub; faults surface via DisposeImpl's [DISPOSE-ACTION] logging. Deterministic repro: DisposalPendingSaveFlushTest forces the exact interleaving without Orleans - own-node stream.Update, observe the own stream carry the write, dispose the per-node hub inside the 200 ms sample window, assert the write reaches storage. RED on unfixed code (same TimeoutException signature as CI), GREEN with the flush. TwoSiloRecycleConvergenceTest green; full-solution Release -p:CIRun=true -warnaserror build (CI-exact restore/build split) clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012wP3c7DLHutUczcBWgs4h8
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Root cause (product data-loss bug — the flaky test was RIGHT)
CI run 30068597014 shard 2:
TwoSiloRecycleConvergenceTestfailed atWaitForPersistedBeyond, not at convergence — the post-recyclePatchDataRequestwas ACKed Success and then durably lost. A patch landing in the owner hub's Quiescing window (afterDisposeRequest, before the RunLevel where #609's ShuttingDown NACK gates) commits in-RAM and ACKs, but the per-node hub's update-persistence path is aSample(200ms)debounce that teardown destroyed two ways:DisposeImpldisposed the sampler with the save pending, and the sampler'sRunLevel > Startedbranch silently discarded sampled values. Creates/deletes already had a final-flush dispose action (FlushPendingWrites); updates had none — that asymmetry is the defect. In prod this loses the last ≤200ms of acknowledged writes on any recycle/shutdown.Fix (
MeshDataSource.cs)PendingOwnSavetracker: stamps every save-worthy state ahead of theSamplebuffer; a state counts as dispatched only when itsSaveMeshNodeRequestpost was accepted (not the teardown-guardFailedreturn). Holds onlyMeshNoderefs — nothing in the timer chain pins the hub (preserves abandoned-hub collectability).DisposeImplbounds its lifetime.FlushPendingOwnSave: reactive dispose action persisting the latest undispatched state viaIStorageAdapter.Write(same guards asHandleSaveMeshNode: delete-wins tombstone,PersistedSnapshotidentity, Version ≥ 1); the write leaf runs on the mesh IO pool (outlives the hub); faults surface via[DISPOSE-ACTION]logging.Verification
DisposalPendingSaveFlushTest: own-nodestream.Update, dispose the per-node hub inside the 200ms window, assert durability — RED pre-fix (same TimeoutException signature as CI), GREEN post-fix.TwoSiloRecycleConvergenceTestclass: 1/1 passed.--no-restore -c Release -p:CIRun=true -warnaserror): 0 errors.No What's New entry — internal durability fix (though "no more lost writes on recycle" arguably deserves one; say the word).
🤖 Generated with Claude Code