HDDS-16666. Flush SCM transaction in memory during apply transaction by sarvekshayr · Pull Request #11409 · apache/ozone

sarvekshayr · 2026-10-05T08:58:51Z

What changes were proposed in this pull request?

When an SCM follower restarts after being down for a long time, Ratis syncs the log to disk quickly, but applyTransaction lags far behind, so commitIndex can exceed appliedIndex by millions of entries. All the DB updates produced while applying that backlog accumulate in the single in-memory BatchOperation held by SCMHADBTransactionBufferImpl, causing heavy memory pressure.

This change adds a count-based flush on the apply path:

New config ozone.scm.ha.dbtransactionbuffer.flush.pending.limit (default 10000): the number of buffered DB operations after which the buffer is flushed. Values <= 0 are invalid and fall back to the default with a warning.
New SCMHADBTransactionBuffer#flushIfPendingLimitReached()
SCMStateMachine#applyTransaction calls it right after updateLatestTrxInfo(...) and updateLastAppliedTermIndex(...), still inside the beginApplyingTransaction()/endApplyingTransaction() window. Flushing after the index is recorded keeps the persisted TransactionInfo consistent with the committed batch. The existing interval-based flush is unchanged.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-16666

How was this patch tested?

Added unit tests in TestSCMHADBTransactionBufferImpl and TestSCMStateMachine.

Manually tested on a SCM HA docker cluster with the flush pending limit set to 5:

Flush lines were observed on all three SCMs, including followers, whenever the buffered operation count reached the limit.

(Note: Changed log level to INFO for testing purpose)

With one follower stopped, load was generated, and the follower restarted: it logged repeated flushes while replaying the backlog and then Follower caught up with leader.

scm3-1  | 2026-10-05 06:43:38,126 [19f565e2-432e-4235-aa33-4575539ddfd7@group-F5C4EAAC6B4A-StateMachineUpdater] INFO ha.SCMHADBTransactionBufferImpl: txFlushPending=5 reached max 5, flushing
scm3-1  | 2026-10-05 06:43:38,130 [19f565e2-432e-4235-aa33-4575539ddfd7@group-F5C4EAAC6B4A-StateMachineUpdater] INFO ha.SCMHADBTransactionBufferImpl: txFlushPending=5 reached max 5, flushing
scm3-1  | 2026-10-05 06:43:38,133 [19f565e2-432e-4235-aa33-4575539ddfd7@group-F5C4EAAC6B4A-StateMachineUpdater] INFO ha.SCMHADBTransactionBufferImpl: txFlushPending=5 reached max 5, flushing
scm3-1  | 2026-10-05 06:43:38,138 [19f565e2-432e-4235-aa33-4575539ddfd7@group-F5C4EAAC6B4A-StateMachineUpdater] INFO ha.SCMHADBTransactionBufferImpl: txFlushPending=5 reached max 5, flushing
scm3-1  | 2026-10-05 06:43:38,141 [19f565e2-432e-4235-aa33-4575539ddfd7@group-F5C4EAAC6B4A-StateMachineUpdater] INFO ha.SCMHADBTransactionBufferImpl: txFlushPending=5 reached max 5, flushing

sumitagrawl

LGTM

Copilot

Copilot review overview

🟡 Changes recommended

The state-machine test helper repeatedly registers global SCM metrics without unregistering them, potentially causing duplicate-source test failures.

Review effort: Balanced
Findings: 1

Open (1)

Prevent duplicate SCMMetrics source registration in tests · New

What changed in this PR

Adds count-based flushing to bound SCM HA transaction-buffer memory during Ratis backlog replay.

Changes:

Adds a configurable pending-operation flush limit with validation.
Flushes during transaction application after recording transaction metadata.
Adds coverage for flushing, failures, concurrency, and ordering.

File	Description
`hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/ScmConfigKeys.java`	Defines the new configuration key and default.
`hadoop-hdds/common/src/main/resources/ozone-default.xml`	Documents the configuration property.
`hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/ha/SCMHADBTransactionBuffer.java`	Adds the conditional-flush API.
`hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/ha/SCMHADBTransactionBufferImpl.java`	Implements validated count-based flushing.
`hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/ha/SCMHADBTransactionBufferStub.java`	Implements the new interface method for the stub.
`hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/ha/SCMStateMachine.java`	Checks the flush threshold after applying transactions.
`hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/ha/TestSCMHADBTransactionBufferImpl.java`	Tests threshold, persistence, failure, and concurrency behavior.
`hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/ha/TestSCMHATransactionBufferMonitorTask.java`	Supplies configuration required by the updated constructor.
`hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/ha/TestSCMStateMachine.java`	Tests state-machine flush ordering and failure handling.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

yandrey321

lgtm

sarvekshayr · 2026-10-06T08:33:13Z

Thanks @sumitagrawl and @yandrey321 for the review.

* master: (64 commits) HDDS-16716. Add description for ozone.scm.ec.pipeline.per.volume.factor (#11415) HDDS-16008. PutBlocks from Flushes also go without Raft (#11356) HDDS-16666. Flush SCM transaction in memory during apply transaction (#11409) HDDS-15749. Run specific JUnit tests if possible (#10671) HDDS-16362. GetObjectAttributes ObjectParts should return Part entries for FSO buckets (#11242). HDDS-16643. Remove CleanupTableInfo mechanism (#11365) HDDS-16721. StreamBlockInputStream.read() returns a negative value for bytes 0x80 to 0xFF (#11411) HDDS-16241. gRPC deadline kills long-lived block streams after 30 seconds and the client never recovers (#11080) HDDS-15991. Speed up deleted table scans in quota repair (#11386) HDDS-16674. Bump awssdk to 2.55.6 (#11407) HDDS-16673. Avoid redundant ListBuckets RPCs when S3 bucket listing reaches the end (#11397) HDDS-16708. Let dependabot ignore iceberg minor version upgrades (#11398) HDDS-16713. Bump develocity-maven-extension to 2.6.0 (#11405) HDDS-16300. Allow OM to dynamically reconfigure its SCM node list without a restart (#11218) HDDS-15089. Support S3 per request read consistency (#11252) HDDS-16704. ReadBlock fails with IllegalStateException when a response is shorter than responseDataSize (#11402) HDDS-16631. Fix chooseRandom for rack names with common prefixes (#11401) HDDS-16654. Replace usage of deprecated finalize() in OM (#11376) HDDS-16658. Reuse source key details when opening input stream in S3 CopyObject (#11396) HDDS-16711. Bump moment to 2.31.0 (#11373) ... Conflicts: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/HealthyReadOnlyNodeHandler.java hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/NodeStateManager.java hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/SCMNodeManager.java hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/ha/TestSCMStateMachine.java hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/node/TestDeadNodeHandler.java hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/node/TestNodeStateManager.java hadoop-ozone/client/src/test/java/org/apache/hadoop/ozone/client/rpc/TestRpcClient.java hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/OmUtils.java hadoop-ozone/common/src/test/java/org/apache/hadoop/ozone/om/ha/TestHadoopRpcOMFollowerReadFailoverProxyProvider.java hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/shell/TestOzoneShellHA.java hadoop-ozone/interface-client/src/main/proto/OmClientProtocol.proto hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerStateMachine.java hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/response/upgrade/OMCancelPrepareResponse.java hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/response/upgrade/OMCompleteFinalizeUpgradeResponse.java hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/response/upgrade/OMPrepareResponse.java hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/protocolPB/OzoneManagerRequestHandler.java hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/ratis/TestOzoneManagerStateMachine.java

HDDS-16666. Flush SCM transaction in memory during apply transaction

8e23577

github-actions Bot added the scm label Oct 5, 2026

Redundant check removed

5e42dcb

sarvekshayr requested a review from sumitagrawl October 5, 2026 10:12

sumitagrawl approved these changes Oct 5, 2026

View reviewed changes

sumitagrawl marked this pull request as ready for review October 5, 2026 12:57

Copilot AI balanced review requested due to automatic review settings October 5, 2026 12:57

Copilot started reviewing on behalf of sumitagrawl October 5, 2026 12:58 View session

Copilot AI reviewed Oct 5, 2026

View reviewed changes

Comment thread hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/ha/TestSCMStateMachine.java Outdated

Prevent duplicate SCMMetrics source registration in tests

108e329

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

yandrey321 approved these changes Oct 5, 2026

View reviewed changes

Russole self-requested a review October 5, 2026 23:37

sumitagrawl merged commit fb0adda into apache:master Oct 6, 2026
44 checks passed

Navigation Menu

Repository navigation

HDDS-16666. Flush SCM transaction in memory during apply transaction - #11409

HDDS-16666. Flush SCM transaction in memory during apply transaction#11409
sumitagrawl merged 3 commits into
apache:masterfrom
sarvekshayr:scm-flush

sarvekshayr commented Oct 5, 2026

Uh oh!

sumitagrawl left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

yandrey321 left a comment

Uh oh!

Uh oh!

sarvekshayr commented Oct 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

sarvekshayr commented Oct 5, 2026

What changes were proposed in this pull request?

What is the link to the Apache JIRA

How was this patch tested?

Uh oh!

sumitagrawl left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

Uh oh!

yandrey321 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sarvekshayr commented Oct 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants