HDDS-15962. Remove leader readiness check on the bootstrap flow. - #10862
Conversation
|
Thanks for the patch @sadanand48 |
There was a problem hiding this comment.
Pull request overview
Warning
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
This PR relaxes OM DB checkpoint bootstrap gatekeeping by rejecting checkpoint requests only when the OM is not the leader (instead of requiring the leader to also be “ready”), preventing deadlock in degraded clusters where a lagging follower cannot catch up.
Changes:
- Add
OzoneManager#isLeader()to reflect Raft leader status independent of readiness. - Update checkpoint servlets to gate
/dbCheckpointrequests onisLeader()rather thanisLeaderReady(). - Extend unit tests to validate leader vs non-leader behavior and
isLeader()semantics.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 4 comments.
| File | Description |
|---|---|
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java |
Introduces isLeader() based on Raft server status to decouple leadership from readiness. |
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OMDBCheckpointServletInodeBasedXfer.java |
Switches checkpoint request rejection condition from isLeaderReady() to isLeader(). |
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OMDBCheckpointServlet.java |
Same leadership-only gate update for the non-inode-based servlet. |
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/TestOMDBCheckpointServletInodeBasedXferNonLeader.java |
Updates existing non-leader tests and adds new coverage for leader behavior + isLeader() mapping. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
smengcl
left a comment
There was a problem hiding this comment.
lgtm. just one more comment. also pls check the remaining copilot comment
|
Thanks @sadanand48 for the patch. Thanks @jojochuang for reviewing this. |
What changes were proposed in this pull request?
Current logic rejects checkpoint requests with 503 unless isLeaderReady() is true.
In a degraded cluster (e.g., one OM down, one follower far behind), this creates a loop:
Lagging follower needs checkpoint to catch up.
Follower requests /v2/dbCheckpoint from leader.
Leader is LEADER_AND_NOT_READY and returns 503 due to isLeaderReady() check.
Follower cannot catch up, so leader readiness does not progress.
System remains stuck until another OM is brought back.
Checkpoint creation already has consistency barriers:
awaitDoubleBufferFlush()
bootstrap write lock (BOOTSTRAP_LOCK)
So requiring LEADER_AND_READY here is stricter than necessary and can block recovery. instead only reject if NOT_LEADER
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-15962
How was this patch tested?
Unit tests