HDDS-13217. Test that snapshot checkpoint content is preserved after defrag by arunsarin85 · Pull Request #10951 · apache/ozone

arunsarin85 · 2026-08-04T19:52:36Z

What changes were proposed in this pull request?

This PR adds an integration test, TestOmSnapshotCheckpointDbContent, to verify that snapshot checkpoint DB metadata for a bucket prefix is preserved across snapshot defrag iterations (HDDS-13217).

Please describe your PR in detail:
This PR adds a new integration test, TestOmSnapshotCheckpointDbContent, to verify that snapshot checkpoint DB metadata for a bucket prefix is preserved across snapshot defrag iterations (HDDS-13217).

The test uses a dedicated MiniOzone cluster with snapshot defrag enabled and covers two scenarios:

Create snapshots S1, S2, and S3 on a bucket, trigger defrag, and verify each defragged checkpoint still matches its version-0 baseline captured at snapshot creation time.
After an initial defrag pass, delete the middle snapshot (S2), purge it, trigger defrag again, and verify the remaining snapshot (S3) still matches its baseline.

Because version-0 checkpoint directories are removed after defrag, the test captures baselines before the first defrag and compares against the active snapshot view after defrag completes.

Each test method starts a fresh cluster to avoid cross-test interference on the snapshot defrag chain.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-13217

How was this patch tested?

https://github.com/arunsarin85/ozone/actions/runs/30941614952

…adata to live OM for a bucket

…rag. Address review feedback on PR apache#10235: compare each snapshot checkpoint against its version-0 baseline before and after defrag iterations, including after deleting a middle snapshot. Rebased onto upstream master.

Use Java 8 compatible Collections.singletonMap, correct OmSnapshot and UncheckedAutoCloseableSupplier imports.

…oss-test interference on the snapshot defrag chain.

…hot purge.

Copilot

Pull request overview

This PR adds a new OM integration test to ensure snapshot defragmentation preserves the checkpoint RocksDB metadata for a bucket prefix across defrag iterations, including a follow-up defrag after deleting/purging a middle snapshot in the snapshot chain (HDDS-13217).

Changes:

Introduces TestOmSnapshotCheckpointDbContent integration test that snapshots a bucket (S1/S2/S3), captures version-0 checkpoint baselines, triggers defrag, and verifies post-defrag metadata matches baselines.
Adds a second scenario that deletes/purges S2 after an initial defrag and verifies S3 remains consistent after a subsequent defrag pass.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+    conf = new OzoneConfiguration();
+    conf.setBoolean(OZONE_FILESYSTEM_SNAPSHOT_ENABLED_KEY, true);
+    conf.setInt(OZONE_SNAPSHOT_DEFRAG_SERVICE_INTERVAL, 7200);
+    conf.setInt(SNAPSHOT_DEFRAG_LIMIT_PER_TASK, 10);
+    conf.setTimeDuration(OZONE_SNAPSHOT_DELETING_SERVICE_INTERVAL, 1, TimeUnit.SECONDS);


+  private void triggerDefragUntilDone(List<SnapshotInfo> snapshots)
+      throws TimeoutException, InterruptedException {
+    OzoneManager om = cluster.getOzoneManager();
+    for (SnapshotInfo snapshotInfo : snapshots) {
+      GenericTestUtils.waitFor(() -> {
+        if (isSnapshotDefragComplete(snapshotInfo)) {
+          return true;
+        }
+        try {
+          om.triggerSnapshotDefrag(false);
+        } catch (IOException e) {
+          return false;
+        }
+        return isSnapshotDefragComplete(snapshotInfo);
+      }, 2000, DEFRAG_WAIT_MS);
+    }
+  }


jojochuang · 2026-08-04T20:51:22Z

+    assertEquals(expected, readPrefix(current, prefix), tableName);
+  }
+
+  private static <V> SortedMap<String, V> readPrefix(


The helper method name doesn't convery its intent.

Filter tables whose key starts with prefix prefix, and add them to a sorted map.

Sure ! Renamed the method .

jojochuang

just a few cosmetic comments

jojochuang · 2026-08-04T21:01:46Z

+    return OmMetadataManagerImpl.createCheckpointMetadataManager(conf, checkpoint);
+  }
+
+  private static Map<String, SortedMap<String, ?>> readAllBucketPrefixTables(


collects the snapshot’s OM metadata for just the tables that matter to a bucket an

added Javadoc explaining it collects snapshot OM metadata for bucket-relevant tables and returns sorted maps keyed by table name.

…llBucketPrefixTables.

adoroszlai

Thanks @arunsarin85 for the patch.

There is a checkstyle failure. Please wait for clean CI run in fork before opening PR.

adoroszlai · 2026-08-05T05:27:44Z

+ * <p>Each test method starts a fresh MiniOzone cluster to avoid cross-test interference on the
+ * global snapshot defrag chain.
+ */
+@Timeout(value = 15, unit = TimeUnit.MINUTES)


Please don't add timeout, see HDDS-12575.

Suggested change

@Timeout(value = 15, unit = TimeUnit.MINUTES)

Added a patch !

adoroszlai · 2026-08-05T05:28:56Z

+    IOUtils.closeQuietly(client);
+    IOUtils.closeQuietly(cluster);


nit: merge the calls, closeQuietly accepts multiple objects

adoroszlai · 2026-08-05T05:30:18Z

+  /**
+   * Test#1 from HDDS-13217: create S1, S2, S3 on one bucket, run defrag, and verify each
+   * defragged checkpoint still matches its version-0 baseline.
+   */
+  @Test
+  public void testDefragPreservesSnapshotCheckpointContent()
+      throws Exception {
+    ThreeSnapshotSetup setup = createThreeSnapshotsOnNewBucket();
+    triggerDefragUntilDone(setup.snapshots);
+    assertCheckpointMatchesBaseline(setup.baselines, setup.snapshots);
+  }
+
+  /**
+   * Test#2 from HDDS-13217: after an initial defrag pass, delete the middle snapshot, run defrag
+   * again, and verify the remaining youngest snapshot checkpoint still matches its baseline.
+   */
+  @Test
+  public void testDefragAfterSnapshotDeletePreservesRemainingSnapshot()
+      throws Exception {
+    ThreeSnapshotSetup setup = createThreeSnapshotsOnNewBucket();
+    triggerDefragUntilDone(setup.snapshots);


It looks like test2's initial steps match test1's steps. Please merge them to reduce time.

adoroszlai · 2026-08-05T05:38:07Z

How was this patch tested?

https://github.com/arunsarin85/ozone/actions/runs/30941614952

Thanks for running flaky-test-check. There was one timeout, please check:

[ERROR] Tests run: 2, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 176.5 s <<< FAILURE! -- in org.apache.hadoop.ozone.om.snapshot.TestOmSnapshotCheckpointDbContent
[ERROR] org.apache.hadoop.ozone.om.snapshot.TestOmSnapshotCheckpointDbContent.testDefragPreservesSnapshotCheckpointContent -- Time elapsed: 146.8 s <<< ERROR!
...
        at app//org.apache.hadoop.ozone.om.snapshot.TestOmSnapshotCheckpointDbContent.waitForCheckpointReady(TestOmSnapshotCheckpointDbContent.java:211)
        at app//org.apache.hadoop.ozone.om.snapshot.TestOmSnapshotCheckpointDbContent.createThreeSnapshotsOnNewBucket(TestOmSnapshotCheckpointDbContent.java:185)
        at app//org.apache.hadoop.ozone.om.snapshot.TestOmSnapshotCheckpointDbContent.testDefragPreservesSnapshotCheckpointContent(TestOmSnapshotCheckpointDbContent.java:131)

smengcl · 2026-08-05T06:49:39Z

Thanks @arunsarin85 for the test additions.

smengcl · 2026-08-05T06:49:39Z

+  private ThreeSnapshotSetup createThreeSnapshotsOnNewBucket()
+      throws IOException, InterruptedException, TimeoutException {
+    OzoneBucket bucket =
+        DataTestUtil.createVolumeAndBucket(client, BucketLayout.OBJECT_STORE);


Please cover the FSO defrag path too. This setup hard-codes OBJECT_STORE, so the test validates keyTable only. Snapshot defrag reconstructs keyTable, directoryTable, and fileTable; the current integration test never checks two of those three tables. Add an FSO bucket to the merged scenario and compare its directory and file tables with their version-0 baselines.

Thanks for the review .

The test now runs the flow for both OBS and FSO buckets.

https://github.com/arunsarin85/ozone/actions/runs/31029322914

smengcl · 2026-08-05T06:49:39Z

+    DataTestUtil.createKey(bucket, "key-s1", TEST_KEY_CONTENT);
+    store.createSnapshot(volumeName, bucketName, "snap-s1");
+
+    DataTestUtil.createKey(bucket, "key-s2", TEST_KEY_CONTENT);


Please include an overwrite and a delete between the snapshots. The current sequence only adds keys, so this end-to-end test exercises insert deltas but cannot detect a regression in update or tombstone reconstruction. For example, overwrite one S1 key before S2 and delete another key before S3, then compare each defragged checkpoint with its baseline.

Thanks for the review .

The snapshot sequence now includes insert, update, and tombstone cases: S1 creates key-a and key-b, we overwrite key-a before S2 and add key-s2, then delete key-b before S3 and add key-s3. After each defrag pass, we still verify the checkpoint matches its version-0 baseline.

https://github.com/arunsarin85/ozone/actions/runs/31029322914

…ls, and combine both defrag scenarios into a single test

adoroszlai · 2026-08-05T15:35:53Z

Thanks @arunsarin85 for updating the patch.

…t tests

arunsarin85 added 8 commits August 4, 2026 15:49

HDDS-13217. Add integration test comparing OM snapshot checkpoint met…

32d7368

…adata to live OM for a bucket

HDDS-13217. Moved the test onto the shared cluster

803426e

Use DataTestUtil Fix TestOmSnapshotCheckpointDbContent

d784ad2

HDDS-13217. Increase defrag wait timeout

4d05c37

HDDS-13217. Fix TestOmSnapshotCheckpointDbContent compile errors.

8b5f805

Use Java 8 compatible Collections.singletonMap, correct OmSnapshot and UncheckedAutoCloseableSupplier imports.

HDDS-13217. Use a fresh MiniOzone cluster per test method to avoid cr…

1c54679

…oss-test interference on the snapshot defrag chain.

HDDS-13217. Fix TestOmSnapshotCheckpointDbContent timeout after snaps…

f91c524

…hot purge.

jojochuang requested review from smengcl and a lite review from Copilot August 4, 2026 20:36

Copilot started reviewing on behalf of jojochuang August 4, 2026 20:46 View session

Copilot AI reviewed Aug 4, 2026

View reviewed changes

jojochuang reviewed Aug 4, 2026

View reviewed changes

Rename readPrefix to filterTableEntriesByKeyPrefix and document readA…

b90bee0

…llBucketPrefixTables.

adoroszlai reviewed Aug 5, 2026

View reviewed changes

adoroszlai changed the title ~~HDDS-13217. Add integration test to verify snapshot checkpoint content is preserved after defrag~~ HDDS-13217. Test that snapshot checkpoint content is preserved after defrag Aug 5, 2026

adoroszlai added test snapshot https://issues.apache.org/jira/browse/HDDS-6517 labels Aug 5, 2026

smengcl reviewed Aug 5, 2026

View reviewed changes

HDDS-13217. Fix import order, remove @timeout, merge closeQuietly cal…

f33f749

…ls, and combine both defrag scenarios into a single test

arunsarin85 added 2 commits August 5, 2026 22:06

HDDS-13217. Cover FSO defrag path and update/delete deltas in snapsho…

8594823

…t tests

HDDS-13217. Stabilize snapshot defrag test under OBS+FSO load.

854a72d

Uh oh!

Conversation

arunsarin85 commented Aug 4, 2026

What changes were proposed in this pull request?

What is the link to the Apache JIRA

How was this patch tested?

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

jojochuang Aug 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jojochuang left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

adoroszlai left a comment

Choose a reason for hiding this comment

Uh oh!

adoroszlai Aug 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

adoroszlai commented Aug 5, 2026

How was this patch tested?

Uh oh!

smengcl commented Aug 5, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

adoroszlai commented Aug 5, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

jojochuang Aug 4, 2026 •

edited

Loading

adoroszlai Aug 5, 2026 •

edited

Loading