HDDS-15991. Speed up deleted table scans in quota repair by chihsuan · Pull Request #11386 · apache/ozone

chihsuan · 2026-10-02T15:39:42Z

What changes were proposed?

Speed up quota repair after large deletes by:

Decoding active deletedTable values in parallel using the existing batch workers. Snapshot deletedTable scans remain unchanged.
Reading only keys from deletedDirectoryTable, avoiding unused value decoding. Corrupt values in this table no longer cause repair to fail.

JIRA

https://issues.apache.org/jira/browse/HDDS-15991

How was this patch tested?

Added tests in TestQuotaRepairTask.

Local benchmark with 1M synthetic rows on a MacBook with a warm cache:

deletedTable 1.9s → 1.0s.
deletedDirectoryTable 0.63s → 0.21s.

Generated-by: Claude Code (Claude Opus 5.5)

…a repair

Copilot

Copilot review overview

🟡 Changes recommended

The corrupt deleted-directory value behavior needs direct regression coverage.

Review effort: Balanced
Findings: 1

Open (1)

Test does not cover corrupt deletedDirectoryTable value tolerance · New

What changed in this PR

Speeds up OM quota repair scans after large deletions.

Changes:

Parallelizes active deletedTable value decoding.
Uses key-only deleted-directory scans.
Adds quota-repair regression tests.

File	Description
`QuotaRepairTask.java`	Optimizes deleted-table scans.
`TestQuotaRepairTask.java`	Tests batching, accounting, and decode failures.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

…epair test

Copilot

Copilot review overview

🟢 Approval recommended

The optimization preserves quota semantics and includes focused regression coverage.

Review effort: Balanced
Findings: None

Resolved since last review (1)

Test does not cover corrupt deletedDirectoryTable value tolerance

chihsuan · 2026-10-02T16:20:22Z

cc @chungen0126 🙏

rich7420 · 2026-10-03T10:53:53Z

+    // a typed iterator decodes on this thread, so keep values raw and let the workers decode
+    try (Table.KeyValueIterator<String, byte[]> keyIter = metadataManager.getStore()
+        .getTable(OMDBDefinition.DELETED_TABLE, StringCodec.get(), ByteArrayCodec.get(), CacheType.NO_CACHE)
+        .iterator()) {


nit: please use .iterator(null, VALUE_ONLY) here, with the corresponding static import. The consumer only reads kv.getValue(), so we can avoid copying and decoding unused keys.

Thanks, good point! Switched to VALUE_ONLY in d95fcef.

rich7420 · 2026-10-03T10:54:18Z

overall LGTM

sumitagrawl

LGTM

rich7420

LGTM

adoroszlai · 2026-10-05T15:26:11Z

Thanks @chihsuan for the patch, @rich7420, @sumitagrawl for the review.

* master: (64 commits) HDDS-16716. Add description for ozone.scm.ec.pipeline.per.volume.factor (#11415) HDDS-16008. PutBlocks from Flushes also go without Raft (#11356) HDDS-16666. Flush SCM transaction in memory during apply transaction (#11409) HDDS-15749. Run specific JUnit tests if possible (#10671) HDDS-16362. GetObjectAttributes ObjectParts should return Part entries for FSO buckets (#11242). HDDS-16643. Remove CleanupTableInfo mechanism (#11365) HDDS-16721. StreamBlockInputStream.read() returns a negative value for bytes 0x80 to 0xFF (#11411) HDDS-16241. gRPC deadline kills long-lived block streams after 30 seconds and the client never recovers (#11080) HDDS-15991. Speed up deleted table scans in quota repair (#11386) HDDS-16674. Bump awssdk to 2.55.6 (#11407) HDDS-16673. Avoid redundant ListBuckets RPCs when S3 bucket listing reaches the end (#11397) HDDS-16708. Let dependabot ignore iceberg minor version upgrades (#11398) HDDS-16713. Bump develocity-maven-extension to 2.6.0 (#11405) HDDS-16300. Allow OM to dynamically reconfigure its SCM node list without a restart (#11218) HDDS-15089. Support S3 per request read consistency (#11252) HDDS-16704. ReadBlock fails with IllegalStateException when a response is shorter than responseDataSize (#11402) HDDS-16631. Fix chooseRandom for rack names with common prefixes (#11401) HDDS-16654. Replace usage of deprecated finalize() in OM (#11376) HDDS-16658. Reuse source key details when opening input stream in S3 CopyObject (#11396) HDDS-16711. Bump moment to 2.31.0 (#11373) ... Conflicts: hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/HealthyReadOnlyNodeHandler.java hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/NodeStateManager.java hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/SCMNodeManager.java hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/ha/TestSCMStateMachine.java hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/node/TestDeadNodeHandler.java hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/node/TestNodeStateManager.java hadoop-ozone/client/src/test/java/org/apache/hadoop/ozone/client/rpc/TestRpcClient.java hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/OmUtils.java hadoop-ozone/common/src/test/java/org/apache/hadoop/ozone/om/ha/TestHadoopRpcOMFollowerReadFailoverProxyProvider.java hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/shell/TestOzoneShellHA.java hadoop-ozone/interface-client/src/main/proto/OmClientProtocol.proto hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerStateMachine.java hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/response/upgrade/OMCancelPrepareResponse.java hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/response/upgrade/OMCompleteFinalizeUpgradeResponse.java hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/response/upgrade/OMPrepareResponse.java hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/protocolPB/OzoneManagerRequestHandler.java hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/ratis/TestOzoneManagerStateMachine.java

chihsuan added 3 commits October 2, 2026 22:47

HDDS-15991. Scan deletedDirectoryTable keys only in quota repair

b17810c

HDDS-15991. Decode active deletedTable rows on worker threads in quot…

779f414

…a repair

HDDS-15991. Assert the decode failure cause in quota repair test

33ef342

Copilot AI balanced review requested due to automatic review settings October 2, 2026 15:39

github-actions Bot added the om label Oct 2, 2026

chihsuan changed the title ~~HDDS-15991. Decode deletedTable rows on worker threads in quota repair~~ HDDS-15991. Speed up deleted table scans in quota repair Oct 2, 2026

chihsuan marked this pull request as ready for review October 2, 2026 15:47

Copilot started reviewing on behalf of chihsuan October 2, 2026 15:53 View session

Copilot AI reviewed Oct 2, 2026

View reviewed changes

Comment thread ...zone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/service/TestQuotaRepairTask.java

chihsuan added 2 commits October 3, 2026 00:04

HDDS-15991. Use production key helpers in undecodable deletedTable test

43fc6bc

HDDS-15991. Cover undecodable deletedDirectoryTable values in quota r…

1d29e6f

…epair test

chihsuan requested a balanced review from Copilot October 2, 2026 16:04

Copilot started reviewing on behalf of chihsuan October 2, 2026 16:10 View session

Copilot AI reviewed Oct 2, 2026

View reviewed changes

rich7420 reviewed Oct 3, 2026

View reviewed changes

HDDS-15991. Read only values from active deletedTable in quota repair

d95fcef

sumitagrawl approved these changes Oct 5, 2026

View reviewed changes

rich7420 approved these changes Oct 5, 2026

View reviewed changes

adoroszlai merged commit 4066b0e into apache:master Oct 5, 2026
42 of 44 checks passed

Navigation Menu

Repository navigation

HDDS-15991. Speed up deleted table scans in quota repair - #11386

HDDS-15991. Speed up deleted table scans in quota repair#11386
adoroszlai merged 6 commits into
apache:masterfrom
chihsuan:HDDS-15991

chihsuan commented Oct 2, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

chihsuan commented Oct 2, 2026

Uh oh!

rich7420 Oct 3, 2026

Uh oh!

chihsuan Oct 3, 2026

Uh oh!

rich7420 commented Oct 3, 2026

Uh oh!

sumitagrawl left a comment

Uh oh!

rich7420 left a comment

Uh oh!

Uh oh!

adoroszlai commented Oct 5, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

chihsuan commented Oct 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed?

JIRA

How was this patch tested?

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟢 Approval recommended

Uh oh!

chihsuan commented Oct 2, 2026

Uh oh!

rich7420 Oct 3, 2026

Choose a reason for hiding this comment

Uh oh!

chihsuan Oct 3, 2026

Choose a reason for hiding this comment

Uh oh!

rich7420 commented Oct 3, 2026

Uh oh!

sumitagrawl left a comment

Choose a reason for hiding this comment

Uh oh!

rich7420 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

adoroszlai commented Oct 5, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

chihsuan commented Oct 2, 2026 •

edited

Loading