Skip to content

SdkClientException: Unable to execute HTTP request: Only one connection receive subscriber allowed #2444

Closed
Closed
SdkClientException: Unable to execute HTTP request: Only one connection receive subscriber allowed#2444
Assignees
debora-ito
Labels
closed-for-stalenessguidanceQuestion that needs advice or information.

Description

@nikitsenka
nikitsenka
opened on Apr 29, 2021

SdkClientException: Unable to execute HTTP request: Only one connection receive subscriber allowed

We are using java AWS SDK v2 for sending data to s3 using Java Http POST endpoint inside Reactive Spring Boot Webflux application.

Describe the issue

This error occurs 1-3 times in a week and only in production where the load is approx 10-30 RPS. It causes the dataloss as 1-3 HTTP json messages are not delivered. The SDK retry looks like applied for this case.

Steps to Reproduce

The reason of the issue is unknown
Unfortunately it happens only in production and not connected with the user data. Any recommendation how to simulate the issue in local environment will be really helpful.

Your Environment

  • AWS Java SDK version used: 2.16.32
  • JDK version used: OpenJDK 64-Bit Server VM AdoptOpenJDK (build 14.0.2+12, mixed mode, sharing)
  • Operating System and version: Linux ingress-traffic-c5ddf5dd7-qm4vt 4.14.181-142.260.amzn2.x86_64 Add Travis config #1 SMP Wed Jun 24 19:07:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Full log log.txt
Snippet from the log:

426		... 55 common frames omitted
427	Caused by: java.lang.IllegalStateException: Only one connection receive subscriber allowed.
428		... 89 common frames omitted
429		at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.maybeRetryExecute(AsyncRetryableStage.java:143)
430		at software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper.setLastException(RetryableStageHelper.java:194)
431		at software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper.setLastException(RetryableStageHelper.java:198)
432		at software.amazon.awssdk.core.exception.SdkClientException.create(SdkClientException.java:43)
433		at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:98)
434	Caused by: software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request: Only one connection receive subscriber allowed.
435		at java.base/java.lang.Thread.run(Thread.java:832)
436		at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
437		at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
438		at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
439		at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
440		at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
441		at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)

Metrics details

10-04-2021 17:19:25.752 trace= span= �[35m[aws-java-sdk-NettyEventLoop-1-7]�[0;39m �[39mDEBUG�[0;39m c.n.e.o.t.c.a.LoggingMetricPublisher.? - aws sdk metrics: MetricCollection(name=ApiCall, metrics=[MetricRecord(metric=MarshallingDuration, value=PT0.000085333S), MetricRecord(metric=RetryCount, value=3), MetricRecord(metric=ApiCallSuccessful, value=false), MetricRecord(metric=OperationName, value=PutObject), MetricRecord(metric=ApiCallDuration, value=PT0.867155059S), MetricRecord(metric=CredentialsFetchDuration, value=PT0.000000895S), MetricRecord(metric=ServiceId, value=S3)], children=[MetricCollection(name=ApiCallAttempt, metrics=[MetricRecord(metric=SigningDuration, value=PT0.000059589S), MetricRecord(metric=ServiceCallDuration, value=PT0.481107161S), MetricRecord(metric=AwsExtendedRequestId, value=Qdpz3ZCYk6RFbSYc0SRQu8cjhikHwH78oekWHw65vc8aFR1moZvuwEYP3re31+YohV2H7PubDqc=), MetricRecord(metric=HttpStatusCode, value=500), MetricRecord(metric=BackoffDelayDuration, value=PT0S), MetricRecord(metric=AwsRequestId, value=D41X7XHXFWAMT2KX)], children=[MetricCollection(name=HttpClient, metrics=[MetricRecord(metric=AvailableConcurrency, value=2), MetricRecord(metric=MaxConcurrency, value=50), MetricRecord(metric=LeasedConcurrency, value=0), MetricRecord(metric=PendingConcurrencyAcquires, value=0), MetricRecord(metric=HttpClientName, value=NettyNio)], children=[])]), MetricCollection(name=ApiCallAttempt, metrics=[MetricRecord(metric=BackoffDelayDuration, value=PT0.083S), MetricRecord(metric=SigningDuration, value=PT0.000098252S)], children=[MetricCollection(name=HttpClient, metrics=[MetricRecord(metric=AvailableConcurrency, value=1), MetricRecord(metric=MaxConcurrency, value=50), MetricRecord(metric=LeasedConcurrency, value=0), MetricRecord(metric=PendingConcurrencyAcquires, value=0), MetricRecord(metric=HttpClientName, value=NettyNio)], children=[])]), MetricCollection(name=ApiCallAttempt, metrics=[MetricRecord(metric=BackoffDelayDuration, value=PT0.194S), MetricRecord(metric=SigningDuration, value=PT0.000170456S)], children=[MetricCollection(name=HttpClient, metrics=[MetricRecord(metric=AvailableConcurrency, value=1), MetricRecord(metric=MaxConcurrency, value=50), MetricRecord(metric=LeasedConcurrency, value=0), MetricRecord(metric=PendingConcurrencyAcquires, value=0), MetricRecord(metric=HttpClientName, value=NettyNio)], children=[])]), MetricCollection(name=ApiCallAttempt, metrics=[MetricRecord(metric=BackoffDelayDuration, value=PT0.051S), MetricRecord(metric=SigningDuration, value=PT0.000090042S)], children=[MetricCollection(name=HttpClient, metrics=[MetricRecord(metric=AvailableConcurrency, value=0), MetricRecord(metric=MaxConcurrency, value=50), MetricRecord(metric=LeasedConcurrency, value=0), MetricRecord(metric=PendingConcurrencyAcquires, value=0), MetricRecord(metric=HttpClientName, value=NettyNio)], children=[])])])

Activity

  1. nikitsenka
    added
    guidanceQuestion that needs advice or information.
    needs-triageThis issue or PR still needs to be triaged.
    on Apr 29, 2021

  2. debora-ito commented on Apr 30, 2021

    @debora-ito
    debora-ito
    on Apr 30, 2021
    Member

    @nikitsenka a quick search for java.lang.IllegalStateException: Only one connection receive subscriber allowed and the results point to an issue with Spring Webflux:

    Looking at the logs the error is returned with a http status code 500, which the SDK will retry, as you mentioned. This doesn't look like an issue with the SDK itself, but if you'd like us to investigate further please provide a self-contained sample code we can use to reproduce.

  3. debora-ito
    added
    response-requestedWaiting on additional info and feedback. Will move to "closing-soon" in 10 days.
    and removed
    needs-triageThis issue or PR still needs to be triaged.
    on Apr 30, 2021

  4. nikitsenka commented on Apr 30, 2021

    @nikitsenka
    nikitsenka
    on Apr 30, 2021
    Author

    @debora-ito thanks for the links. The only helpful is the first one but unfortunately it is a WebClient case and we are using AWS SDK NettyNio client. Please see the code snippet below. It is not compilable but just to show the usage of SDK.

    package example;

import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
import lombok.AllArgsConstructor;
import lombok.Getter;
import lombok.NoArgsConstructor;
import lombok.ToString;
import lombok.extern.slf4j.Slf4j;
import org.apache.commons.lang3.StringUtils;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.config.ConfigurableBeanFactory;
import org.springframework.cloud.sleuth.instrument.async.TraceableExecutorService;
import org.springframework.context.annotation.Scope;
import org.springframework.core.io.buffer.DataBuffer;
import org.springframework.http.HttpStatus;
import org.springframework.stereotype.Component;
import org.springframework.util.Assert;
import reactor.core.publisher.Mono;
import software.amazon.awssdk.auth.credentials.AwsCredentialsProvider;
import software.amazon.awssdk.auth.credentials.DefaultCredentialsProvider;
import software.amazon.awssdk.core.async.AsyncRequestBody;
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.s3.S3AsyncClient;
import software.amazon.awssdk.services.s3.S3AsyncClientBuilder;
import software.amazon.awssdk.services.s3.model.PutObjectRequest;
import software.amazon.awssdk.services.s3.model.S3Exception;
import software.amazon.awssdk.services.sts.StsClient;
import software.amazon.awssdk.services.sts.StsClientBuilder;
import software.amazon.awssdk.services.sts.auth.StsAssumeRoleCredentialsProvider;
import software.amazon.awssdk.services.sts.model.AssumeRoleRequest;

import java.net.URI;
import java.time.Clock;
import java.time.LocalDateTime;
import java.time.format.DateTimeFormatter;
import java.util.HashMap;
import java.util.Map;
import java.util.Objects;
import java.util.UUID;
import java.util.concurrent.locks.Lock;
import java.util.concurrent.locks.ReentrantLock;

import static java.lang.String.format;
import static org.apache.commons.lang3.StringUtils.defaultString;
import static org.apache.commons.lang3.StringUtils.isNotBlank;
import static software.amazon.awssdk.core.client.config.SdkAdvancedAsyncClientOption.FUTURE_COMPLETION_EXECUTOR;


/**
 * S3 implementation for sink adapter.
 */
@Component
@Getter
@Slf4j
@NoArgsConstructor
@AllArgsConstructor
@Scope(ConfigurableBeanFactory.SCOPE_PROTOTYPE)
@ToString(onlyExplicitlyIncluded = true, includeFieldNames = false)
public class S3SinkAdapter implements SinkAdapter<S3SinkConfiguration> {

    @ToString.Include
    private S3SinkConfiguration s3Sink;
    private Lock lock = new ReentrantLock(true);
    @Autowired
    private ObjectMapper objectMapper;
    private Map<String, S3AsyncClient> s3AsyncClientMap = new HashMap<>();
    @Autowired
    private TraceableExecutorService traceableExecutorService;
    @Autowired
    private TraceHelper traceHelper;
    @Autowired
    private Clock clock;


    /**
     * {@inheritDoc}
     */
    @Override
    public SinkType getType() {
        return SinkType.S3;
    }

    /**
     * {@inheritDoc}
     */
    @Override
    public SinkAdapter<S3SinkConfiguration> withSink(JsonNode sinkConfiguration) {
        Assert.notNull(sinkConfiguration, "Null S3 sink configuration was provided");
        this.s3Sink = objectMapper.convertValue(sinkConfiguration, S3SinkConfiguration.class);
        return this;
    }

    /**
     * {@inheritDoc}
     */
    @Override
    public S3SinkConfiguration getSink() {
        return this.s3Sink;
    }

    @Override
    public Mono<String> storeData(Payload payload) {
        if (Objects.nonNull(payload.getBytes())) {
            log.info("S3SinkAdapter start json data storing");
            // Aws sdk will throw socket connection timeout error if number of bytes set in the "Content-Length" header
            // is more than the actual file size
            // https://aws.amazon.com/premiumsupport/knowledge-center/s3-socket-connection-timeout-error/
            payload.setContentLength(payload.getBytes().length);
            return storeBody(payload, AsyncRequestBody.fromBytes(payload.getBytes()));
        } else {
            log.info("S3SinkAdapter start raw data storing");
            AsyncRequestBody requestBody = AsyncRequestBody.fromPublisher(payload.getDataBufferFlux().map(DataBuffer::asByteBuffer));
            return storeBody(payload, requestBody);
        }
    }

    private Mono<String> storeBody(Payload payload, AsyncRequestBody requestBody) {
        // We reuse connections for different request so we do not need to close clients
        @SuppressWarnings("PMD.CloseResource")
        S3AsyncClient s3AsyncClient = asyncClient(s3Sink.getRoleArn());
        PutObjectRequest build = PutObjectRequest.builder()
                .bucket(s3Sink.getS3BucketName())
                .contentLength(payload.getContentLength())
                .key(getKey(s3Sink.getFolderPath(),
                        payload.getEventId(),
                        s3Sink.getNamingTimeFormat(),
                        payload.getExtension()))
                .acl(s3Sink.getAcl())
                .build();
        return Mono
                .fromFuture(s3AsyncClient
                        .putObject(build,
                                requestBody
                        )
                )
                .transformDeferred(traceHelper::propagateTraceFromContextToMono)
                .onErrorMap(throwable -> {
                    if (throwable instanceof S3Exception
                            && ((S3Exception) throwable).statusCode() < HttpStatus.BAD_REQUEST.value()) {
                        return new IngestionValidationException("Cannot process request to S3", throwable);
                    }
                    log.debug("s3 client executor details: " + ((ExecutorAwareService) traceableExecutorService).getExecutor());
                    return new SinkIngestionException("S3 ingestion error. EventId - " + payload.getEventId(), throwable);
                })
                .doOnSuccess(s -> log.info("S3SinkAdapter stored data successfully"))
                .thenReturn(payload.getEventId())
                .name("s3sinkadapter_storedata")
                .metrics();
    }

    private S3AsyncClient asyncClient(String roleArn) {
        String key = StringUtils.isBlank(roleArn) ? "default" : roleArn;
        if (s3AsyncClientMap.get(key) == null) {
            lock.lock();
            try {
                if (s3AsyncClientMap.get(key) == null) {
                    log.info("s3 async client configuration start");
                    S3AsyncClientBuilder builder = S3AsyncClient.builder()
                            .credentialsProvider(getAwsCredentialsProvider(roleArn))
                            .asyncConfiguration(configuration -> configuration
                                    .advancedOption(FUTURE_COMPLETION_EXECUTOR, traceableExecutorService)
                                    .build()
                            )
                            .overrideConfiguration(c -> c.addMetricPublisher(LoggingMetricPublisher.builder().build()))
                            .region(getRegion());

                    addEndpointUrl(builder);
                    s3AsyncClientMap.put(key, builder.build());
                }
            } finally {
                lock.unlock();
            }
        }
        log.info("using s3 client with role: " + key);
        return s3AsyncClientMap.get(key);
    }

    private AwsCredentialsProvider getAwsCredentialsProvider(String roleArn) {
        if (StringUtils.isBlank(roleArn)) {
            return DefaultCredentialsProvider.create();
        }
        AssumeRoleRequest assumeRoleRequest = AssumeRoleRequest
                .builder()
                .roleArn(roleArn)
                .roleSessionName(UUID.randomUUID().toString())
                .build();
        StsClientBuilder stsClientBuilder = StsClient.builder()
                .region(getRegion());
        addEndpointUrl(stsClientBuilder);
        return StsAssumeRoleCredentialsProvider.builder()
                .stsClient(stsClientBuilder.build())
                .refreshRequest(assumeRoleRequest)
                .build();
    }

    private void addEndpointUrl(S3AsyncClientBuilder builder) {
        if (isNotBlank(s3Sink.getEndpointUrl())) {
            builder.endpointOverride(URI.create(s3Sink.getEndpointUrl()));
        }
    }

    private void addEndpointUrl(StsClientBuilder builder) {
        if (isNotBlank(s3Sink.getEndpointUrl())) {
            builder.endpointOverride(URI.create(s3Sink.getEndpointUrl()));
        }
    }

    private Region getRegion() {
        return s3Sink.getS3RegionName() != null ? Region.of(s3Sink.getS3RegionName()) : Region.US_EAST_2;
    }

    private String getKey(String folderPath, String eventId, String dateTimeFormat, String extension) {
        return format("%s/%s/%s.%s", defaultString(folderPath), getCurrentTimeFormatted(dateTimeFormat), eventId, extension);
    }

    private String getCurrentTimeFormatted(String dateTimeFormat) {
        return DateTimeFormatter.ofPattern(StringUtils.isEmpty(dateTimeFormat) ? "yyyyMMdd" : dateTimeFormat)
                .format(LocalDateTime.now(clock));
    }

}

    Any recommendations will be helpful

  5. github-actions
    removed
    response-requestedWaiting on additional info and feedback. Will move to "closing-soon" in 10 days.
    on Apr 30, 2021

  6. nikitsenka commented on Apr 30, 2021

    @nikitsenka
    nikitsenka
    on Apr 30, 2021
    Author

    We are using s3AsyncClientMap to store client object in memory and reuse them for subsequent requests from the same client. The idea is to not create client every time but reuse existing one to improve performance. Not sure is it really make sense. Maybe we can just create new client for each request. I did't found any example or recommendation regarding it on AWS SDK User guide. Will be great if you can suggest better approach or provide some example for it.

  7. nikitsenka commented on May 10, 2021

    @nikitsenka
    nikitsenka
    on May 10, 2021
    Author

    @debora-ito any ideas?

  8. pkgonan commented on Sep 24, 2021

    @pkgonan
    pkgonan
    on Sep 24, 2021

    I have same issue, any solution?

    software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request: Connection has been closed
	at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:98) ~[sdk-core-2.17.36.jar!/:na]
	at software.amazon.awssdk.core.exception.SdkClientException.create(SdkClientException.java:43) ~[sdk-core-2.17.36.jar!/:na]
	at software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper.setLastException(RetryableStageHelper.java:204) ~[sdk-core-2.17.36.jar!/:na]
	at software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper.setLastException(RetryableStageHelper.java:200) ~[sdk-core-2.17.36.jar!/:na]
	at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.maybeRetryExecute(AsyncRetryableStage.java:179) ~[sdk-core-2.17.36.jar!/:na]
	at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.lambda$attemptExecute$1(AsyncRetryableStage.java:159) ~[sdk-core-2.17.36.jar!/:na]
	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[na:na]
	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[na:na]
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[na:na]
	at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088) ~[na:na]
	at software.amazon.awssdk.utils.CompletableFutureUtils.lambda$forwardExceptionTo$0(CompletableFutureUtils.java:76) ~[utils-2.17.36.jar!/:na]
	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[na:na]
	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[na:na]
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[na:na]
	at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088) ~[na:na]
	at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.lambda$null$0(MakeAsyncHttpRequestStage.java:104) ~[sdk-core-2.17.36.jar!/:na]
	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[na:na]
	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[na:na]
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[na:na]
	at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088) ~[na:na]
	at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage$WrappedErrorForwardingResponseHandler.onError(MakeAsyncHttpRequestStage.java:158) ~[sdk-core-2.17.36.jar!/:na]
	at software.amazon.awssdk.http.nio.netty.internal.NettyRequestExecutor.handleFailure(NettyRequestExecutor.java:300) ~[netty-nio-client-2.17.36.jar!/:na]
	at software.amazon.awssdk.http.nio.netty.internal.NettyRequestExecutor.lambda$writeRequest$8(NettyRequestExecutor.java:254) ~[netty-nio-client-2.17.36.jar!/:na]
	at io.opentelemetry.javaagent.instrumentation.netty.common.FutureListenerWrappers$WrappedFutureListener.operationComplete(FutureListenerWrappers.java:67) ~[na:na]
	at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:578) ~[netty-common-4.1.67.Final.jar!/:4.1.67.Final]
	at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:552) ~[netty-common-4.1.67.Final.jar!/:4.1.67.Final]
	at io.netty.util.concurrent.DefaultPromise.access$200(DefaultPromise.java:35) ~[netty-common-4.1.67.Final.jar!/:4.1.67.Final]
	at io.netty.util.concurrent.DefaultPromise$1.run(DefaultPromise.java:502) ~[netty-common-4.1.67.Final.jar!/:4.1.67.Final]
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) ~[netty-common-4.1.67.Final.jar!/:4.1.67.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469) ~[netty-common-4.1.67.Final.jar!/:4.1.67.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500) ~[netty-transport-4.1.67.Final.jar!/:4.1.67.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) ~[netty-common-4.1.67.Final.jar!/:4.1.67.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.67.Final.jar!/:4.1.67.Final]
	at java.base/java.lang.Thread.run(Thread.java:829) ~[na:na]
Caused by: reactor.netty.channel.AbortedException: Connection has been closed
	at reactor.netty.http.server.HttpServerOperations.onInboundClose(HttpServerOperations.java:559) ~[reactor-netty-http-1.0.10.jar!/:1.0.10]
	at reactor.netty.channel.ChannelOperationsHandler.channelInactive(ChannelOperationsHandler.java:73) ~[reactor-netty-core-1.0.10.jar!/:1.0.10]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) ~[netty-transport-4.1.67.Final.jar!/:4.1.67.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248) ~[netty-transport-4.1.67.Final.jar!/:4.1.67.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241) ~[netty-transport-4.1.67.Final.jar!/:4.1.67.Final]
	at io.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:81) ~[netty-transport-4.1.67.Final.jar!/:4.1.67.Final]
	at io.netty.handler.codec.MessageAggregator.channelInactive(MessageAggregator.java:438) ~[netty-codec-4.1.67.Final.jar!/:4.1.67.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) ~[netty-transport-4.1.67.Final.jar!/:4.1.67.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248) ~[netty-transport-4.1.67.Final.jar!/:4.1.67.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241) ~[netty-transport-4.1.67.Final.jar!/:4.1.67.Final]
	at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelInactive(CombinedChannelDuplexHandler.java:418) ~[netty-transport-4.1.67.Final.jar!/:4.1.67.Final]
	at io.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:81) ~[netty-transport-4.1.67.Final.jar!/:4.1.67.Final]
	at io.netty.channel.CombinedChannelDuplexHandler.channelInactive(CombinedChannelDuplexHandler.java:221) ~[netty-transport-4.1.67.Final.jar!/:4.1.67.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) ~[netty-transport-4.1.67.Final.jar!/:4.1.67.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248) ~[netty-transport-4.1.67.Final.jar!/:4.1.67.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241) ~[netty-transport-4.1.67.Final.jar!/:4.1.67.Final]
	at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelInactive(CombinedChannelDuplexHandler.java:418) ~[netty-transport-4.1.67.Final.jar!/:4.1.67.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:389) ~[netty-codec-4.1.67.Final.jar!/:4.1.67.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:354) ~[netty-codec-4.1.67.Final.jar!/:4.1.67.Final]
	at io.netty.channel.CombinedChannelDuplexHandler.channelInactive(CombinedChannelDuplexHandler.java:221) ~[netty-transport-4.1.67.Final.jar!/:4.1.67.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) ~[netty-transport-4.1.67.Final.jar!/:4.1.67.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248) ~[netty-transport-4.1.67.Final.jar!/:4.1.67.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241) ~[netty-transport-4.1.67.Final.jar!/:4.1.67.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1405) ~[netty-transport-4.1.67.Final.jar!/:4.1.67.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) ~[netty-transport-4.1.67.Final.jar!/:4.1.67.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248) ~[netty-transport-4.1.67.Final.jar!/:4.1.67.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:901) ~[netty-transport-4.1.67.Final.jar!/:4.1.67.Final]
	at io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:831) ~[netty-transport-4.1.67.Final.jar!/:4.1.67.Final]
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) ~[netty-common-4.1.67.Final.jar!/:4.1.67.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469) ~[netty-common-4.1.67.Final.jar!/:4.1.67.Final]
	at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) ~[netty-transport-native-epoll-4.1.67.Final-linux-x86_64.jar!/:4.1.67.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) ~[netty-common-4.1.67.Final.jar!/:4.1.67.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.67.Final.jar!/:4.1.67.Final]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.67.Final.jar!/:4.1.67.Final]
	... 1 common frames omitted```
  9. debora-ito
    self-assigned this
    on Mar 19, 2022

  10. debora-ito commented on May 20, 2022

    @debora-ito
    debora-ito
    on May 20, 2022
    Member

    @nikitsenka I'm sorry, getting back to this issue after a long time. Are you still seeing the error?

    It is recommended to reuse the same client whenever possible, but if you need different client configurations than you need to create different clients. Creating a new client for every request is not recommended. I don't see any obvious problems with your implementation of s3AsyncClientMap. We list some best practices in our Dev Guide and on Github.

    Marking this to auto-close soon, but let us know if you're still getting this error.

  11. debora-ito
    added
    closing-soonThis issue will close in 4 days unless further comments are made.
    on May 20, 2022
  12. github-actions
    added
    closed-for-staleness
    and removed
    closing-soonThis issue will close in 4 days unless further comments are made.
    on May 22, 2022
  13. github-actions
    closed this as completedon May 22, 2022
  14. aws-sdk-java-automation
    added a commit that references this issue on Apr 5, 2023

    Merge pull request #2444 from aws/staging/045a76ff-d694-4e30-8e43-f0b…

    accb975
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

closed-for-stalenessguidanceQuestion that needs advice or information.

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions