Skip to content

HDDS-15962. Remove leader readiness check on the bootstrap flow. - #10862

Merged
smengcl merged 5 commits into
apache:masterfrom
sadanand48:HDDS-15962
Aug 8, 2026
Merged

HDDS-15962. Remove leader readiness check on the bootstrap flow.#10862
smengcl merged 5 commits into
apache:masterfrom
sadanand48:HDDS-15962

Conversation

@sadanand48

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

Current logic rejects checkpoint requests with 503 unless isLeaderReady() is true.
In a degraded cluster (e.g., one OM down, one follower far behind), this creates a loop:

Lagging follower needs checkpoint to catch up.
Follower requests /v2/dbCheckpoint from leader.
Leader is LEADER_AND_NOT_READY and returns 503 due to isLeaderReady() check.
Follower cannot catch up, so leader readiness does not progress.
System remains stuck until another OM is brought back.
Checkpoint creation already has consistency barriers:

awaitDoubleBufferFlush()
bootstrap write lock (BOOTSTRAP_LOCK)
So requiring LEADER_AND_READY here is stricter than necessary and can block recovery. instead only reject if NOT_LEADER

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-15962

How was this patch tested?

Unit tests

@sadanand48
sadanand48 requested a review from jojochuang July 24, 2026 17:46
@jojochuang jojochuang added the snapshot https://issues.apache.org/jira/browse/HDDS-6517 label Jul 27, 2026
@smengcl

smengcl commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Thanks for the patch @sadanand48

@jojochuang
jojochuang marked this pull request as ready for review August 4, 2026 16:33
Copilot AI lite review requested due to automatic review settings August 4, 2026 16:33

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

This PR relaxes OM DB checkpoint bootstrap gatekeeping by rejecting checkpoint requests only when the OM is not the leader (instead of requiring the leader to also be “ready”), preventing deadlock in degraded clusters where a lagging follower cannot catch up.

Changes:

  • Add OzoneManager#isLeader() to reflect Raft leader status independent of readiness.
  • Update checkpoint servlets to gate /dbCheckpoint requests on isLeader() rather than isLeaderReady().
  • Extend unit tests to validate leader vs non-leader behavior and isLeader() semantics.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 4 comments.

File Description
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java Introduces isLeader() based on Raft server status to decouple leadership from readiness.
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OMDBCheckpointServletInodeBasedXfer.java Switches checkpoint request rejection condition from isLeaderReady() to isLeader().
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OMDBCheckpointServlet.java Same leadership-only gate update for the non-inode-based servlet.
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/TestOMDBCheckpointServletInodeBasedXferNonLeader.java Updates existing non-leader tests and adds new coverage for leader behavior + isLeader() mapping.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@smengcl smengcl left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm. just one more comment. also pls check the remaining copilot comment

@smengcl
smengcl merged commit 036745a into apache:master Aug 8, 2026
88 of 89 checks passed
@smengcl

smengcl commented Aug 8, 2026

Copy link
Copy Markdown
Contributor

Thanks @sadanand48 for the patch. Thanks @jojochuang for reviewing this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

snapshot https://issues.apache.org/jira/browse/HDDS-6517

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants