Repository navigation
Asynchronous DynamoDB Client V2 is slower than V1 #3422
Description
Describe the bug
I tried the DynamoDB asynchronous client v. 2.17.271 and noticed than requests runs slower than on version 1.11.698 (synchronous one)
I also wrote a simple tests for both client versions which calls GetItem request and Scan request in a loop and measure the time.
I used the same HTTP settings, tried on both DynamoDBLocal emulator and actual cloud DynamoDB.
The test for async v2 client runs slower than the test for sync v1 both on DynamoDBLocal and cloud one.
The bigger request (Scan with limit 100) time difference is higher than the smalle one (GetItem). But they both runs slower.
I tried different HTTP netty client configuration (OpenSSL for SSL, completion on thread pool and on event loop thread). Results are the same
What else should I look for (maybe turn on some metrics)? to find where the bottleneck is
Expected Behavior
At least the same performance. And for multiple concurrent requests should be better due to nature of asynchronous requests.
Current Behavior
Current measurements (on DynamoDBLocal), smaller number is better
`
version 2.17.271
scan sequential 200 total: 933ms consumed 0 rcu/0 wcu
scan sequential percentiles: 90th 7ms, 50th 5ms
scan parallel 200 total: 226ms consumed 0 rcu/0 wcu
scan parallel percentiles: 90th 61ms, 50th 29ms
getitem sequential 200 total: 357ms consumed 0 rcu/0 wcu
getitem sequential percentiles: 90th 3ms, 50th 3ms
getitem parallel 200 total: 103ms consumed 0 rcu/0 wcu
getitem parallel percentiles: 90th 9ms, 50th 4ms
version 1.11.698
scan sequential 200 total: 540ms consumed 0.0 rcu/0.0 wcu
scan sequential percentiles: 90th 4ms, 50th 4ms
scan parallel 200 total: 215ms consumed 0.0 rcu/0.0 wcu
scan parallel percentiles: 90th 45ms, 50th 38ms
getitem sequential 200 total: 225ms consumed 0.0 rcu/0.0 wcu
getitem sequential percentiles: 90th 6ms, 50th 2ms
getitem parallel 200 total: 51ms consumed 0.0 rcu/0.0 wcu
getitem parallel percentiles: 90th 20ms, 50th 6ms
`
Reproduction Steps
Basically it measures the time for running a loop
for (int i = 0; i < nrequests; i++) {
stat.measure(() -> {
ScanResult rs = test.client.scan(new ScanRequest(TABLE)
.withConsistentRead(FALSE)
.withLimit(limit)
.withFilterExpression("#nid > :n")
.withExpressionAttributeNames(Collections.singletonMap("#nid", "nid"))
.withExpressionAttributeValues(Collections.singletonMap(":n", new AttributeValue().withN(Long.toString(test.rnd.nextInt(8000)))))
);
if (rs.getCount() == 0 || rs.getCount() > limit) throw new RuntimeException("invalid number of rows returned " + rs.getCount());
});
}or
for (int i = 0; i < nrequests; i++) {
ScanResponse rs = stat.measure(
() -> test.client.scan(ScanRequest.builder()
.tableName(TABLE)
.limit(limit)
.filterExpression("#nid > :n")
.expressionAttributeNames(Collections.singletonMap("#nid", "nid"))
.expressionAttributeValues(Collections.singletonMap(":n", AttributeValue.fromN(Long.toString(test.rnd.nextInt(8000)))))
.consistentRead(FALSE)
.returnConsumedCapacity(ReturnConsumedCapacity.TOTAL)
.build())
.join()
);
stat.addConsumed(rs.consumedCapacity());
if (rs.count() == 0 || rs.count() > limit) throw new RuntimeException("wrong number of rows returned " + rs.count());
}Client configuration besides authentication
builder
.withClientConfiguration(new ClientConfiguration()
.withMaxConnections(200)
.withConnectionMaxIdleMillis(60_000)
.withConnectionTimeout(30_000)
.withClientExecutionTimeout(30_000)
.withConnectionTTL(60_000)
.withSocketTimeout(30_000)
.withTcpKeepAlive(true)
.withThrottledRetries(true)
)
.build()builder
.httpClientBuilder(NettyNioAsyncHttpClient.builder()
.maxConcurrency(200)
.maxPendingConnectionAcquires(10_000)
.connectionMaxIdleTime(Duration.ofSeconds(60))
.connectionTimeout(Duration.ofSeconds(30))
.connectionAcquisitionTimeout(Duration.ofSeconds(30))
.readTimeout(Duration.ofSeconds(30)))
.asyncConfiguration(b -> b.advancedOption(SdkAdvancedAsyncClientOption.FUTURE_COMPLETION_EXECUTOR, Runnable::run))
.build()Possible Solution
No response
Additional Information/Context
No response
AWS Java SDK version used
2.17.271
JDK version used
openjdk version "1.8.0_342", openjdk version "11.0.16" 2022-07-19
Operating System and version
Ubuntu Linux 22.04 x86_64
Activity
Merge pull request #3422 from aws/hdavidh/dsql-utilities-update
debora-ito commentedon Jan 23, 2026
Quick (and overdue) update: we are looking into this and other reports of performance issues in v2.
For anyone that is impacted by this, please add a 👍 to the top comment, and share your use case here if you'd like.
batrived commentedon Apr 8, 2026
Production profiling data: AwsExecutionContextBuilder overhead on DynamoDB-heavy workload
We've been profiling a JanusGraph server that uses the DynamoDB storage backend heavily (Query-dominated workload, ~thousands of DynamoDB calls/sec) on SDK version 2.42.28. Sharing concrete flame graph numbers in case they're useful for this investigation.
What we measured
Using async-profiler on a multi-core JVM (total samples = 7185s wall-clock CPU across all threads over a 43-minute window):
AwsExecutionContextBuilder.invokeInterceptorsAndCreateExecutionContextDynamoDbResolveEndpointInterceptor.modifyRequest(within above)DynamoDbAuthSchemeInterceptor(within above)ExecutionInterceptorChain$$Lambda_.acceptExecutionAttributes.getAttributeSo ~6-7% of all JVM CPU is spent constructing per-request execution context — for a workload where DynamoDB calls are the dominant operation.
Root cause analysis
DynamoDbResolveEndpointInterceptorcallsDefaultDynamoDbEndpointProvider.resolveEndpoint()on every single request, which re-evaluates the endpoint rules engine and re-parses the URI, even whenendpointOverrideis statically configured at client construction time.DynamoDbAuthSchemeInterceptorre-resolves auth scheme per request.ExecutionAttributesis constructed asnew HashMap<>()(default capacity 16) and then has ~30–40 attributes added during the pipeline, causing multiple resize/rehash cycles per request.Workaround we found for #1
Bypassing
endpointOverridein favor of a customendpointProvidereliminates the per-request endpoint resolution cost:This saves ~0.5% CPU on our workload by short-circuiting
DynamoDbResolveEndpointInterceptorentirely.Note on SDK version
All the improvements from PRs #5643 (unmarshalling), #6383 (SigV4 25% speedup), #4867 are already included in 2.42.28 — the numbers above reflect the current state after those fixes.
Hope this data point helps the investigation. Happy to share more profiling details if useful.