HDDS-16081. TestOMRatisSnapshots fails due to leader applied index taken before flush - #10959
Open
ss77892 wants to merge 1 commit into
Open
HDDS-16081. TestOMRatisSnapshots fails due to leader applied index taken before flush#10959ss77892 wants to merge 1 commit into
ss77892 wants to merge 1 commit into
Conversation
…rite: follower may not have all keys when the leader's last applied index is reached
adoroszlai
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Flaky TestOMRatisSnapshots#testInstallSnapshotWithClientWrite: follower may not have all keys when the leader's last applied index is reached
The test starts a stopped follower OM, writes 200 keys while it catches up, waits for the follower to reach the leader, then reads the follower's RocksDB directly to confirm every key arrived. Sometimes the last few keys aren't there yet. That happens because there is a small gap in which a write has already been acked to the client (and committed to the Ratis log) but not yet flushed to RocksDB. The test relies on the leader's lastAppliedTermIndex, which only advances on flush, so the target it hands the follower can sit behind writes that already returned. Previously, a 5-second sleep made the flush all but certain by the time that index was read. HDDS-10310 replaced it with an index-based wait and dropped the per-key sleeps from the write loop, so the writes now outrun the flush and nothing closes the gap, leaving the test to check the follower's DB for keys that are still queued.
The fix adds a single leaderOM.awaitDoubleBufferFlush() call after the writes complete, so the leader's lastAppliedTermIndex covers them before the test uses it as the follower's target.
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-16081
How was this patch tested?
To reproduce it for sure, a small delay (1 ms) has been added to OzoneManagerDoubleBuffer#flush. The test fails all the time in this case. With the fix, it no longer fails.