Repository navigation
HADOOP-20021. hadoop-bos: fix hierarchy bucket rename/create and slim shaded jar. - #8823
Merged
slfan1989 merged 2 commits intoOct 18, 2026
Merged
Conversation
… shaded jar. - Upgrade bce-java-sdk to 0.10.332 to fix the NullPointerException when listing a bucket root (missing "owner" in the list response). - Hierarchy buckets: honour the rename contract (rename into an existing directory, rename into own descendant) and throw ParentNotDirectoryException when creating a file under a file. - Map 400 InvalidArgument to BosException instead of SessionTokenExpireException so the retry policy does not retry it. - Stop bundling guava, httpclient, httpcore and commons-codec in the shaded jar and rely on the ones shipped by Hadoop. Keep bundling httpcore-nio and httpasyncclient, which the SDK needs at class load time and Hadoop does not ship. - Update LICENSE-binary accordingly. Co-Authored-By: Claude Code <noreply@anthropic.com>
|
💔 -1 overall
This message was automatically generated. |
slfan1989
requested changes
Oct 16, 2026
slfan1989
left a comment
slfan1989
left a comment
Contributor
There was a problem hiding this comment.
Thanks for the fixes and the validation against both bucket types.
Could you add focused unit tests for the new hierarchy-bucket rename/create behavior and the HTTP 400 exception mapping? The BOS contract tests were all skipped in Jenkins, so these paths currently have no automated CI coverage.
Could you also validate the packaging changes using a built Hadoop distribution with the normal BOS classpath setup? The Maven-resolved classpath check is useful, but it does not verify the distribution runtime dependencies. Please also investigate the two failing GitHub Actions build jobs before we proceed with merging.
cc: @LuciferYang
…ials. Mock the BOS SDK client and the native file system store to cover the hierarchy-bucket rename/create behaviour and the HTTP error to exception mapping in BosClientProxyImpl, so these paths are exercised in CI where the contract tests are skipped. Co-Authored-By: Claude Code <noreply@anthropic.com>
|
💔 -1 overall
This message was automatically generated. |
slfan1989
approved these changes
Oct 18, 2026
slfan1989
left a comment
slfan1989
left a comment
Contributor
There was a problem hiding this comment.
LGTM. Thanks for adding the tests and addressing the feedback. We can follow up on the built-distribution validation separately.
Contributor
|
@yangdong2398 Thanks for the contribution! Merged into trunk. cc: @LuciferYang |
|
Thank you @slfan1989 and @yangdong2398 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of PR
JIRA: https://issues.apache.org/jira/browse/HADOOP-20021
Follow-up to HDFS-11161 (hadoop-bos), found while verifying the module with the
hadoop fsCLI and the contract suite on JDK 17.bce-java-sdk0.10.330 -> 0.10.332. 0.10.330 throwsNullPointerExceptiononhadoop fs -ls bos://bucket/because the list response has noowner; 0.10.332 guards it.rename: follow the FileSystem contract when the destination is an existing directory, and refuse to rename into its own descendant.create: throwParentNotDirectoryExceptionwhen an ancestor is a file.InvalidSessionToken(e.g.InvalidArgument) now map toBosExceptioninstead ofSessionTokenExpireException; the latter was retried by the retry policy as an expired session token and caused ~150s timeouts.httpcore-nioandhttpasyncclientare still bundled: the SDK'sBceHttpClientneeds them at class load time even with async put disabled, and Hadoop does not ship them.LICENSE-binary(SDK version,httpcore-nio,httpasyncclient).How was this patch tested?
JDK 17, endpoint
http://bj.bcebos.com, run against real BOS with credentials from environment variables (EnvironmentVariableCredentialsProvider):hadoop fsCLI on both buckets:mkdir -p,put,cat,mv,ls(including bucket root),rm -r,-checksum.TestBosClientProxyImplExceptions(HTTP error to exception mapping) andTestBaiduBosFileSystemHierarchy(hierarchy rename/create). 24 cases pass; with the pre-change sources 10 of them fail, so they do exercise the fix. The module now runs 164 tests (139 contract tests are skipped without credentials).mvn checkstyle:check(0 violations),javadoc:javadocandspotbugs:check(0 bugs) onhadoop-bos.Limitations: the CLI check used a Maven-resolved classpath, not a built distribution tarball. Since guava/httpclient now come from Hadoop, hadoop-bos needs
hadoop-common(not onlyhadoop-client-api) on the classpath.For code changes:
LICENSE,LICENSE-binary,NOTICE-binaryfiles?AI Tooling
where Claude Code is the name of the AI tool used.
https://www.apache.org/legal/generative-tooling.html
Contains content generated by Claude Code.