Skip to content

NPE in RetryOnErrorCodeCondition.shouldRetry() #4600

Open
Open
NPE in RetryOnErrorCodeCondition.shouldRetry()#4600
Assignees
debora-ito
Labels
bugThis issue is a bug.p2This is a standard priority issue

Description

@steveloughran
steveloughran
opened on Oct 16, 2023

Describe the bug

NPE in error handling code of RetryOnErrorCodeCondition.shouldRetry(); in bundle-2.20.128.jar


Caused by: java.lang.NullPointerException
	at software.amazon.awssdk.awscore.retry.conditions.RetryOnErrorCodeCondition.shouldRetry(RetryOnErrorCodeCondition.java:45) ~[bundle-2.20.128.jar:?]
	at software.amazon.awssdk.core.retry.conditions.OrRetryCondition.lambda$shouldRetry$0(OrRetryCondition.java:46) ~[bundle-2.20.128.jar:?]
	at java.util.stream.MatchOps$1MatchSink.accept(MatchOps.java:90) ~[?:1.8.0_382]

looking at source its caught AwsServiceException but the error details are (somehow) null and there's no handling of that

        if (ex instanceof AwsServiceException) {
            AwsServiceException exception = (AwsServiceException) ex;

            return retryableErrorCodes.contains(exception.awsErrorDetails().errorCode()); /** here*/
        }

Expected Behavior

A call of S3AFileSystem.isDirectory() to return true if there's a directory at the path, that is: does LIST path or HEAD path+"/" return something.

Current Behavior

Caused by: java.lang.NullPointerException
	at software.amazon.awssdk.awscore.retry.conditions.RetryOnErrorCodeCondition.shouldRetry(RetryOnErrorCodeCondition.java:45)
	at software.amazon.awssdk.core.retry.conditions.OrRetryCondition.lambda$shouldRetry$0(OrRetryCondition.java:46)
	at java.util.stream.MatchOps$1MatchSink.accept(MatchOps.java:90)
	at java.util.Spliterators$IteratorSpliterator.tryAdvance(Spliterators.java:1812)
	at java.util.stream.ReferencePipeline.forEachWithCancel(ReferencePipeline.java:126)
	at java.util.stream.AbstractPipeline.copyIntoWithCancel(AbstractPipeline.java:499)
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:486)
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
	at java.util.stream.MatchOps$MatchOp.evaluateSequential(MatchOps.java:230)
	at java.util.stream.MatchOps$MatchOp.evaluateSequential(MatchOps.java:196)
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.util.stream.ReferencePipeline.anyMatch(ReferencePipeline.java:516)
	at software.amazon.awssdk.core.retry.conditions.OrRetryCondition.shouldRetry(OrRetryCondition.java:46)
	at software.amazon.awssdk.core.retry.conditions.AndRetryCondition.lambda$shouldRetry$0(AndRetryCondition.java:47)
	at java.util.stream.MatchOps$1MatchSink.accept(MatchOps.java:90)
	at java.util.Spliterators$IteratorSpliterator.tryAdvance(Spliterators.java:1812)
	at java.util.stream.ReferencePipeline.forEachWithCancel(ReferencePipeline.java:126)
	at java.util.stream.AbstractPipeline.copyIntoWithCancel(AbstractPipeline.java:499)
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:486)
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
	at java.util.stream.MatchOps$MatchOp.evaluateSequential(MatchOps.java:230)
	at java.util.stream.MatchOps$MatchOp.evaluateSequential(MatchOps.java:196)
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.util.stream.ReferencePipeline.allMatch(ReferencePipeline.java:521)
	at software.amazon.awssdk.core.retry.conditions.AndRetryCondition.shouldRetry(AndRetryCondition.java:47)
	at software.amazon.awssdk.core.retry.conditions.AndRetryCondition.lambda$shouldRetry$0(AndRetryCondition.java:47)
	at java.util.stream.MatchOps$1MatchSink.accept(MatchOps.java:90)
	at java.util.Spliterators$IteratorSpliterator.tryAdvance(Spliterators.java:1812)
	at java.util.stream.ReferencePipeline.forEachWithCancel(ReferencePipeline.java:126)
	at java.util.stream.AbstractPipeline.copyIntoWithCancel(AbstractPipeline.java:499)
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:486)
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
	at java.util.stream.MatchOps$MatchOp.evaluateSequential(MatchOps.java:230)
	at java.util.stream.MatchOps$MatchOp.evaluateSequential(MatchOps.java:196)
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.util.stream.ReferencePipeline.allMatch(ReferencePipeline.java:521)
	at software.amazon.awssdk.core.retry.conditions.AndRetryCondition.shouldRetry(AndRetryCondition.java:47)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper.retryPolicyAllowsRetry(RetryableStageHelper.java:109)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:66)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:36)
	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
	at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:56)
	at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:36)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:80)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:60)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:42)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:50)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:32)
	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26)
	at software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:196)
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:103)
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute(BaseSyncClientHandler.java:171)
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:82)
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:179)
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:76)
	at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45)
	at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:56)
	at software.amazon.awssdk.services.s3.DefaultS3Client.headObject(DefaultS3Client.java:5445)
	at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$getObjectMetadata$8(S3AFileSystem.java:2758)
	at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:468)
	at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:431)
	at org.apache.hadoop.fs.s3a.S3AFileSystem.getObjectMetadata(S3AFileSystem.java:2746)
	at org.apache.hadoop.fs.s3a.S3AFileSystem.getObjectMetadata(S3AFileSystem.java:2726)
	at org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:3857)
	at org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:3785)
	at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$getFileStatus$24(S3AFileSystem.java:3762)
	at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.invokeTrackingDuration(IOStatisticsBinding.java:543)
	at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.lambda$trackDurationOfOperation$5(IOStatisticsBinding.java:524)
	at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.trackDuration(IOStatisticsBinding.java:445)
	at org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2571)
	at org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2590)
	at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:3760)
	at org.apache.hadoop.hive.metastore.Warehouse.isDir(Warehouse.java:853)
	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler$8.run(HiveMetaStore.java:1449)
	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler$8.run(HiveMetaStore.java:1447)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899)
	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_database_core(HiveMetaStore.java:1447)
	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_database(HiveMetaStore.java:1575)
	... 41 more

Reproduction Steps

see stack trace; this is hadoop trunk up to HADOOP-18908. Improve S3A region handling; but with some internals on the execution chain (s3a auditing)

Possible Solution

look for a null errorDetails() field before calling errorCode() in it; if null it is non-retryable.

 return retryableErrorCodes.contains(exception.awsErrorDetails().errorCode());

Additional Information/Context

there is some underlying deployment quirk triggering this. fix the NPE and we'll be able to find out what

AWS Java SDK version used

2.20.128

JDK version used

OpenJDK Runtime Environment (build 1.8.0_382-b05)

Operating System and version

linux

Activity

  1. steveloughran
    added
    bugThis issue is a bug.
    needs-triageThis issue or PR still needs to be triaged.
    on Oct 16, 2023
  2. steveloughran
    added a commit that references this issue on Oct 17, 2023

    HADOOP-18939. NPE in AWS v2 SDK RetryOnErrorCodeCondition.shouldRetry…

    e5eb404
  3. debora-ito
    self-assigned this
    on Oct 20, 2023

  4. debora-ito commented on Oct 20, 2023

    @debora-ito
    debora-ito
    on Oct 20, 2023
    Member

    Is it possible that retryableErrorCodes is null, and not errorDetails?

  5. debora-ito
    added
    p2This is a standard priority issue
    on Nov 2, 2023

  6. steveloughran commented on Nov 13, 2023

    @steveloughran
    steveloughran
    on Nov 13, 2023
    Author

    maybe both; all we saw was the failure from the missing error code

  7. ahmarsuhail
    added a commit that references this issue on Nov 27, 2023

    HADOOP-18939. NPE in AWS v2 SDK RetryOnErrorCodeCondition.shouldRetry…

    ba252ac
  8. ahmarsuhail
    added 2 commits that reference this issue on Dec 5, 2023

    HADOOP-18939. NPE in AWS v2 SDK RetryOnErrorCodeCondition.shouldRetry…

    e1cb1b9

    HADOOP-18939. NPE in AWS v2 SDK RetryOnErrorCodeCondition.shouldRetry…

    b7fee11
  9. jiajunmao
    added a commit that references this issue on Feb 6, 2024

    HADOOP-18939. NPE in AWS v2 SDK RetryOnErrorCodeCondition.shouldRetry…

    97cfbd6

  10. bhoradc commented on Jan 6, 2025

    @bhoradc
    bhoradc
    on Jan 6, 2025
    Contributor

    Hi @steveloughran,

    If this issue still persists, can you please provide a self-contained code sample with the steps that we can use to reproduce this issue at our end?

    Regards,
    Chaitanya

  11. bhoradc
    added
    response-requestedWaiting on additional info and feedback. Will move to "closing-soon" in 10 days.
    and removed
    needs-triageThis issue or PR still needs to be triaged.
    on Jan 6, 2025

  12. steveloughran commented on Jan 7, 2025

    @steveloughran
    steveloughran
    on Jan 7, 2025
    Author

    @bhoradc this was actually our own sdk exception being built up incompletely -the retry logic just cannot handle that

    apache/hadoop#6193

    I think you should consider what is the appropriate behaviour here, I'd consider any exception without a status code as "do not retry"

  13. github-actions
    removed
    response-requestedWaiting on additional info and feedback. Will move to "closing-soon" in 10 days.
    on Jan 7, 2025
  14. aws-sdk-java-automation
    added a commit that references this issue on Dec 18, 2025

    Merge pull request #4600 from aws/staging/cef3dd20-c116-40be-a2f6-180…

    74eda4d
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugThis issue is a bug.p2This is a standard priority issue

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions