HDDS-12688. Set test-friendly defaults in mini-cluster - #10926
Open
rich7420 wants to merge 7 commits into
Open
Conversation
Move the test-friendly space/memory settings that only lived in integration-test's ozone-site.xml into MiniOzoneCluster.Builder so every mini-cluster consumer (integration-test, -s3, -recon, and external users of the ozone-mini-cluster artifact) gets them. setIfUnset could not be reused directly: after HDDS-12777 a key present in ozone-default.xml always returns non-null from get(), so the stock setIfUnset never applied. OzoneConfiguration now overrides setIfUnset to treat only non *-default.xml sources (programmatic, command line, *-site.xml, custom resources) as explicitly set, so Builder defaults can override default-resource values while preserving anything the caller set. ClientConfigForTesting moves into ozone-mini-cluster main and gains an applyTo(conf, onlyIfUnset) overload backed by MutableConfigurationSource.ifUnsetWrapper. Redundant settings are dropped from the three integration-test ozone-site.xml files; module-specific ones (MockSpaceUsage, transport class, 128MB container size, 5GB min free space) are kept.
ClientConfigForTesting (moved into mini-cluster main) uses org.apache.hadoop.hdds.scm.OzoneClientConfig from hdds-client, which was previously pulled in only transitively. Declare it explicitly so maven-dependency-plugin:analyze-only no longer fails the build with 'Used undeclared dependencies found: hdds-client'.
rich7420
changed the title
Contributor
There was a problem hiding this comment.
Pull request overview
This PR moves several “test-friendly” configuration overrides (previously duplicated across integration-test module ozone-site.xml files) into MiniOzoneCluster defaults, and introduces a revised setIfUnset behavior for OzoneConfiguration so MiniOzoneCluster can override shipped *-default.xml values while still preserving caller / *-site.xml overrides.
Changes:
- Apply MiniOzoneCluster-wide, test-friendly defaults (smaller buffers/sizes, shorter intervals, lower handler counts) while preserving explicitly configured values.
- Extend
ClientConfigForTestingto support container size and an “only-if-unset” application mode via a wrapper configuration target. - Update integration-test
ozone-site.xmlresources to remove now-redundant settings and keep only module-specific overrides (eg. larger container size for S3/Recon).
Reviewed changes
Copilot reviewed 9 out of 9 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| hadoop-ozone/mini-cluster/src/main/java/org/apache/hadoop/ozone/MiniOzoneCluster.java | Adds MiniOzoneCluster default config application during builder construction. |
| hadoop-ozone/mini-cluster/src/main/java/org/apache/hadoop/ozone/ClientConfigForTesting.java | Adds container size support and “apply only if unset” behavior. |
| hadoop-ozone/mini-cluster/pom.xml | Adds an explicit dependency needed by mini-cluster configuration helpers. |
| hadoop-ozone/integration-test/src/test/resources/ozone-site.xml | Removes test defaults now provided by MiniOzoneCluster. |
| hadoop-ozone/integration-test-s3/src/test/resources/ozone-site.xml | Keeps only S3-specific override (larger container size). |
| hadoop-ozone/integration-test-recon/src/test/resources/ozone-site.xml | Keeps only Recon-specific override (larger container size). |
| hadoop-hdds/config/src/main/java/org/apache/hadoop/hdds/conf/MutableConfigurationSource.java | Adds setIfUnset default method and an “if-unset” wrapper implementation. |
| hadoop-hdds/common/src/test/java/org/apache/hadoop/hdds/conf/TestOzoneConfiguration.java | Adds tests for new OzoneConfiguration#setIfUnset semantics. |
| hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/conf/OzoneConfiguration.java | Overrides setIfUnset to override built-in defaults while preserving explicit sources. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+464
to
+477
| private boolean isExplicitlySet(String name) { | ||
| String[] sources = getPropertySources(name); | ||
| if (sources == null) { | ||
| return false; | ||
| } | ||
| for (String source : sources) { | ||
| // Any source other than a *-default.xml (programmatically, command line, | ||
| // a *-site.xml, or a custom resource) counts as explicitly set. | ||
| if (source != null && !source.endsWith("-default.xml")) { | ||
| return true; | ||
| } | ||
| } | ||
| return false; | ||
| } |
Comment on lines
+77
to
+81
| @Override | ||
| public void set(String key, String value) { | ||
| wrapped.setIfUnset(key, value); | ||
| } | ||
| } |
… wrapper - isExplicitlySet() now checks the property source against Ozone's registered built-in default resources (getConfigurationResourceFiles()) instead of a "*-default.xml" suffix. A user-provided resource is preserved even when named *-default.xml. - IfUnsetWrapper now overrides setIfUnset() to delegate to wrapped.setIfUnset(), so its set() and setIfUnset() behave consistently. - Strengthen the custom-resource test to use a *-default.xml name, guarding the edge case above.
adoroszlai
marked this pull request as draft
August 5, 2026 13:27
Contributor
|
Thanks @rich7420 for the patch. Please wait for clean CI run in fork before opening PR. |
Contributor
Author
|
@adoroszlai thanks for the reminder , I'm sorry about that |
rich7420
marked this pull request as ready for review
August 6, 2026 09:32
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
hadoop-ozone/integration-test/src/test/resources/ozone-site.xml defines some settings that are useful for test environment to reduce space/memory needs. These are not applied to other modules (e.g. integration-test-s3, etc.) or external components that use mini-cluster. The goal of this task is to add (some of) the settings in mini-cluster as defaults.
Configuration#setIfUnset() does not work, because get(String) (without a default value) returns the value from ozone-default. It looks like setIfUnset only works with property names not defined in any config XML, or properties without default value ().
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-12688
How was this patch tested?
https://github.com/rich7420/ozone/actions/runs/31081372022