Skip to content

HDDS-16362. GetObjectAttributes ObjectParts should return Part entries for FSO buckets - #11242

Merged
Gargi-jais11 merged 10 commits into
apache:masterfrom
Gargi-jais11:HDDS-16362
Oct 6, 2026
Merged

Gargi-jais11 merged 10 commits into
apache:masterfrom
Gargi-jais11:HDDS-16362

Conversation

@Gargi-jais11

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

Extend GetObjectAttributes ObjectParts handling so FSO layout buckets (Ozone’s equivalent of AWS directory buckets) return per-part elements (PartNumber, Size) even when no additional checksum was stored at upload time.

This follows AWS S3 behavior documented in https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObjectAttributesParts.html:

General-purpose buckets: Part elements are omitted unless an additional checksum (x-amz-checksum-crc32, crc32c, sha1, sha256) was applied.
Directory buckets: Part elements are returned regardless of checksum.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-16362

How was this patch tested?

Added IT and UT.

// FSO bucket getObjectAttributes returns parts info irrespective of checksum present or not

bash-5.1$ aws s3api get-object-attributes --endpoint-url "$ENDPOINT" \
  --bucket "$BUCKET" --key "$KEY" \
  --object-attributes ObjectParts ObjectSize \
  --output json | jq '.ObjectParts | {TotalPartsCount, IsTruncated, PartCount: (.Parts | length), Parts}'
{
  "TotalPartsCount": 2,
  "IsTruncated": false,
  "PartCount": 2,
  "Parts": [
    {
      "PartNumber": 1,
      "Size": 5242880
    },
    {
      "PartNumber": 2,
      "Size": 8
    }
  ]
}
bash-5.1$ aws s3api get-object-attributes --endpoint-url "$ENDPOINT" \
  --bucket "$BUCKET" --key "$KEY" \
  --object-attributes ObjectParts --max-parts 1 --part-number-marker 0 \
  --output json | jq '.ObjectParts | {IsTruncated, Parts}'
{
  "IsTruncated": true,
  "Parts": [
    {
      "PartNumber": 1,
      "Size": 5242880
    }
  ]
}

// OBS bucket doesnot return part info if checksum not present
bash-5.1$ aws s3api get-object-attributes --endpoint-url "$ENDPOINT" \
  --bucket "$BUCKET" --key "$KEY" \
  --object-attributes ObjectParts ObjectSize \
  --output json | jq '.ObjectParts | {TotalPartsCount, IsTruncated, PartCount: (.Parts | length), Parts}'
{
  "TotalPartsCount": 2,
  "IsTruncated": false,
  "PartCount": 0,
  "Parts": null
}

@Gargi-jais11
Gargi-jais11 marked this pull request as ready for review September 15, 2026 08:22

@ArafatKhan2198 ArafatKhan2198 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some comments for you @Gargi-jais11

@github-actions github-actions Bot added the s3 S3 Gateway label Sep 23, 2026

@chungen0126 chungen0126 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 LGTM

@Gargi-jais11

Copy link
Copy Markdown
Contributor Author

@ArafatKhan2198 and @rich7420 please re-review whenever u are free.

@rich7420 rich7420 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it's clean. LGTM

@Gargi-jais11
Gargi-jais11 merged commit 96c43b3 into apache:master Oct 6, 2026
44 checks passed
@Gargi-jais11

Copy link
Copy Markdown
Contributor Author

Thanks @chungen0126 , @ArafatKhan2198 and @rich7420 for reviewing the PR.

errose28 added a commit that referenced this pull request Oct 6, 2026
* master: (64 commits)
  HDDS-16716. Add description for ozone.scm.ec.pipeline.per.volume.factor (#11415)
  HDDS-16008. PutBlocks from Flushes also go without Raft (#11356)
  HDDS-16666. Flush SCM transaction in memory during apply transaction (#11409)
  HDDS-15749. Run specific JUnit tests if possible (#10671)
  HDDS-16362. GetObjectAttributes ObjectParts should return Part entries for FSO buckets (#11242).
  HDDS-16643. Remove CleanupTableInfo mechanism (#11365)
  HDDS-16721. StreamBlockInputStream.read() returns a negative value for bytes 0x80 to 0xFF (#11411)
  HDDS-16241. gRPC deadline kills long-lived block streams after 30 seconds and the client never recovers (#11080)
  HDDS-15991. Speed up deleted table scans in quota repair (#11386)
  HDDS-16674. Bump awssdk to 2.55.6 (#11407)
  HDDS-16673. Avoid redundant ListBuckets RPCs when S3 bucket listing reaches the end (#11397)
  HDDS-16708. Let dependabot ignore iceberg minor version upgrades (#11398)
  HDDS-16713. Bump develocity-maven-extension to 2.6.0 (#11405)
  HDDS-16300. Allow OM to dynamically reconfigure its SCM node list without a restart (#11218)
  HDDS-15089. Support S3 per request read consistency (#11252)
  HDDS-16704. ReadBlock fails with IllegalStateException when a response is shorter than responseDataSize (#11402)
  HDDS-16631. Fix chooseRandom for rack names with common prefixes (#11401)
  HDDS-16654. Replace usage of deprecated finalize() in OM (#11376)
  HDDS-16658. Reuse source key details when opening input stream in S3 CopyObject (#11396)
  HDDS-16711. Bump moment to 2.31.0 (#11373)
  ...

Conflicts:
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/HealthyReadOnlyNodeHandler.java
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/NodeStateManager.java
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/SCMNodeManager.java
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/ha/TestSCMStateMachine.java
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/node/TestDeadNodeHandler.java
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/node/TestNodeStateManager.java
hadoop-ozone/client/src/test/java/org/apache/hadoop/ozone/client/rpc/TestRpcClient.java
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/OmUtils.java
hadoop-ozone/common/src/test/java/org/apache/hadoop/ozone/om/ha/TestHadoopRpcOMFollowerReadFailoverProxyProvider.java
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/shell/TestOzoneShellHA.java
hadoop-ozone/interface-client/src/main/proto/OmClientProtocol.proto
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerStateMachine.java
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/response/upgrade/OMCancelPrepareResponse.java
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/response/upgrade/OMCompleteFinalizeUpgradeResponse.java
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/response/upgrade/OMPrepareResponse.java
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/protocolPB/OzoneManagerRequestHandler.java
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/ratis/TestOzoneManagerStateMachine.java
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants