Skip to content

Unable to use Java AWS SDK v2 to get object using SHA256 #4694

Closed
Closed
Unable to use Java AWS SDK v2 to get object using SHA256#4694
Assignees
debora-ito
Labels
bugThis issue is a bug.closed-for-stalenessp2This is a standard priority issue

Description

@gorell
gorell
opened on Nov 13, 2023

Describe the bug

hi team,
after putting the object using the checksum algorithm SHA256 as following,

        PutObjectRequest putObjectRequest = PutObjectRequest.builder()
            .bucket(bucketName)
            .key(key)
            .contentType(contentType)
            .checksumAlgorithm(ChecksumAlgorithm.SHA256)
            .build();
        awsClient.putObject(putObjectRequest, RequestBody.fromString(content));

we're trying to get this object:

          GetObjectResponse getObjectResponse = awsClient.getObject(
                  b -> b
                      .bucket(bucketName)
                      .key(key)
                      .checksumMode(ChecksumMode.ENABLED))
              .response();

but MD5 checksum (and not SHA256) is applied instead, via this hard-coded reference this
and we're getting the exception: java.lang.IllegalStateException: Unexpected error creating MD5 checksum
note: headObject works fine
what are we missing?
thanks in advance

Expected Behavior

GetObject should work

Current Behavior

Getting an exception (see above)

Reproduction Steps

Please see the code in the bug description

Possible Solution

No response

Additional Information/Context

No response

AWS Java SDK version used

2.20.42

JDK version used

openjdk 11.0.14.1

Operating System and version

Linux service-5b7c8c4788-c62fp 5.4.238-148.347.amzn2.x86_64 #1 SMP Thu Apr 6 19:42:57 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Activity

  1. gorell
    added
    bugThis issue is a bug.
    needs-triageThis issue or PR still needs to be triaged.
    on Nov 13, 2023

  2. gorell commented on Nov 13, 2023

    @gorell
    gorell
    on Nov 13, 2023
    Author
    Screenshot 2023-11-13 at 21 25 29 Screenshot 2023-11-13 at 21 25 48 Screenshot 2023-11-13 at 21 32 09 Screenshot 2023-11-13 at 21 35 49
  3. debora-ito
    self-assigned this
    on Nov 13, 2023

  4. debora-ito commented on Nov 13, 2023

    @debora-ito
    debora-ito
    on Nov 13, 2023
    Member

    @gorell can you share the full stacktrace of the error?

  5. gorell commented on Nov 14, 2023

    @gorell
    gorell
    on Nov 14, 2023
    Author

    @debora-ito here it is:

    Caused by: java.lang.IllegalStateException: Unexpected error creating MD5 checksum
	at software.amazon.awssdk.core.checksums.Md5Checksum.getDigest(Md5Checksum.java:63)
	at software.amazon.awssdk.core.checksums.Md5Checksum.<init>(Md5Checksum.java:32)
	at software.amazon.awssdk.services.s3.internal.handlers.SyncChecksumValidationInterceptor.modifyHttpResponseContent(SyncChecksumValidationInterceptor.java:74)
	at software.amazon.awssdk.core.interceptor.ExecutionInterceptorChain.modifyHttpResponse(ExecutionInterceptorChain.java:148)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.AfterTransmissionExecutionInterceptorsStage.execute(AfterTransmissionExecutionInterceptorsStage.java:46)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.AfterTransmissionExecutionInterceptorsStage.execute(AfterTransmissionExecutionInterceptorsStage.java:28)
	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:73)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:42)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:78)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:40)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:50)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:36)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:81)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:36)
	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
	at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:56)
	at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:36)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:80)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:60)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:42)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:48)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:31)
	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26)
	at software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:193)
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:103)
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute(BaseSyncClientHandler.java:171)
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$0(BaseSyncClientHandler.java:68)
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:179)
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:62)
	at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:52)
	at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:63)
	at software.amazon.awssdk.services.s3.DefaultS3Client.getObject(DefaultS3Client.java:4535)
	at software.amazon.awssdk.services.s3.S3Client.getObjectAsBytes(S3Client.java:8465)
	at .......doGetObject(....java:322)
	... 65 more
Caused by: java.security.NoSuchAlgorithmException: MD5 MessageDigest not available
	at java.base/sun.security.jca.GetInstance.getInstance(GetInstance.java:159)
	at java.base/java.security.MessageDigest.getInstance(MessageDigest.java:185)
	at software.amazon.awssdk.core.checksums.Md5Checksum.getDigest(Md5Checksum.java:61)

  6. gorell commented on Nov 14, 2023

    @gorell
    gorell
    on Nov 14, 2023
    Author

    @debora-ito just to clarify - we've customized our JDK to use SHA256 (and not MD5 which we don't support) for checksum

  7. debora-ito commented on Nov 15, 2023

    @debora-ito
    debora-ito
    on Nov 15, 2023
    Member

    The Java SDK will validate the MD5 checksum by default on GetObject. This is a client-side configuration, separate from the GetObject checksumMode option, and this is what the code in SyncChecksumValidationInterceptor is doing.

    You can disable this default MD5 checksum validation in the S3 Client by setting a custom S3Configuration:

    S3Client s3Client = S3Client.builder()
        .serviceConfiguration(S3Configuration.builder().checksumValidationEnabled(Boolean.FALSE).build())
        .region(Region.US_WEST_2)
        .build();
  8. debora-ito
    added
    closing-soonThis issue will close in 4 days unless further comments are made.
    guidanceQuestion that needs advice or information.
    and removed
    bugThis issue is a bug.
    needs-triageThis issue or PR still needs to be triaged.
    on Nov 15, 2023

  9. gorell commented on Nov 15, 2023

    @gorell
    gorell
    on Nov 15, 2023
    Author

    Got it @debora-ito , thank you. So is there a way to configure the GetObject validation to be SHA256? Given how the PutObject is configured.

  10. github-actions
    removed
    closing-soonThis issue will close in 4 days unless further comments are made.
    on Nov 15, 2023

  11. debora-ito commented on Nov 16, 2023

    @debora-ito
    debora-ito
    on Nov 16, 2023
    Member

    In this use-case, the SDK will automatically validate the SHA256 checksum on GetObject, after finishing reading the object. The SDK validates any of the 4 checksum types supported by the ChecksumAlgorithm in PutObject.

  12. debora-ito
    added
    closing-soonThis issue will close in 4 days unless further comments are made.
    on Nov 16, 2023

  13. gorell commented on Nov 16, 2023

    @gorell
    gorell
    on Nov 16, 2023
    Author

    But it looks like it does it with MD5, that's the reason I've opened this issue.

  14. github-actions
    removed
    closing-soonThis issue will close in 4 days unless further comments are made.
    on Nov 16, 2023

  15. debora-ito commented on Nov 16, 2023

    @debora-ito
    debora-ito
    on Nov 16, 2023
    Member

    Ok, I took a step back and went through the question again. I also talked to the rest of the team about this issue.

    1. Firstly, there is a known bug that I think is directly impacting you - we are performing both the default md5 checksum validation AND the checksumAlgorithm validation (SHA256 in your case) upon calling GetObject. We are going to change this to: if checksumMode is enabled on GetObject, we'll only perform the checksumAlgorithm validation, and not perform the md5 validation - the double validation is not necessary in this case.

    2. While the fix for (1) is not released, as a workaround you can disable the md5 checksum in the s3 client by using the S3Configuration I showed in my previous comment -

    S3Client s3Client = S3Client.builder()
        .serviceConfiguration(S3Configuration.builder().checksumValidationEnabled(Boolean.FALSE).build())
        .region(Region.US_WEST_2)
        .build();

    To clarify: setting checksumValidationEnabled() to false will only disable the md5 checksum validation, not the SHA256 validation. The SHA256 checksum validation will still be performed if checksumMode is enabled in GetObject.

    @gorell Will the fix of (1) address the issue you are reporting here? Is there anything else I'm missing?

  16. gorell commented on Nov 16, 2023

    @gorell
    gorell
    on Nov 16, 2023
    Author

    @debora-ito The answer to you question is 'yes', the fix of (1) will do the job, thank you very much

  17. debora-ito commented on Nov 16, 2023

    @debora-ito
    debora-ito
    on Nov 16, 2023
    Member

    Great, thank you for confirming.

    I'll comment here once we have more updates on the fix.

  18. debora-ito
    added
    bugThis issue is a bug.
    p2This is a standard priority issue
    and removed
    guidanceQuestion that needs advice or information.
    on Nov 16, 2023

  19. debora-ito commented on Dec 4, 2023

    @debora-ito
    debora-ito
    on Dec 4, 2023
    Member

    The fix is available in version 2.21.38.

    @gorell

  20. debora-ito
    added
    closing-soonThis issue will close in 4 days unless further comments are made.
    on Dec 5, 2023
  21. github-actions
    added
    closed-for-staleness
    and removed
    closing-soonThis issue will close in 4 days unless further comments are made.
    on Dec 7, 2023
  22. github-actions
    closed this as completedon Dec 7, 2023

  23. kishd commented on Aug 13, 2024

    @kishd
    kishd
    on Aug 13, 2024

    @debora-ito - This appears to be an issue with the delete api. "There is no way to disable MD5 for S3Client.deleteObjects because the service API requires the MD5 checksum". https://github.com/aws/aws-sdk-java-v2/blob/master/core/sdk-core/src/main/java/software/amazon/awssdk/core/internal/http/pipeline/stages/HttpChecksumStage.java

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugThis issue is a bug.closed-for-stalenessp2This is a standard priority issue

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions