Repository navigation
potential memory leak in aws sdk s3 code #3924
Description
Describe the bug
Hi Team. We run our code (that reads-from/writes-to s3) over time. We use v2.20.42. When taking heap dumps we observe consistent memory growth, like this:
-rw-r--r-- 1 user staff 394378864 Apr 18 13:54 /Users/user/heap1.bin
-rw-r--r-- 1 user staff 402753779 Apr 18 14:05 /Users/user/heap2.bin
-rw-r--r-- 1 user staff 409832891 Apr 18 14:11 /Users/user/heap3.bin
-rw-r--r-- 1 user staff 415905366 Apr 18 14:23 /Users/user/heap4.bin
When comparing heap dumps we can see that the majority of the new objects on the heap seem to be related to AWS SDK S3 library, almost solely reflecting the above memory growth. Screenshots attached. Code attached. Please your advice on the above. Thanks in advance.
aws-dal.zip
Expected Behavior
The memory dumps should remain consistent over time, no leaks should occur.
Current Behavior
The memory grows on a regular basis due to AWS SDK S3 library objects being created on the heap and not released.
Reproduction Steps
Run the attached code over time.
Possible Solution
No response
Additional Information/Context
No response
AWS Java SDK version used
2.20.42
JDK version used
openjdk 11.0.14.1
Operating System and version
Linux service-5b7c8c4788-c62fp 5.4.238-148.347.amzn2.x86_64 #1 SMP Thu Apr 6 19:42:57 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Activity
[-]memory leak in aws sdk s3 code[/-][+]potential memory leak in aws sdk s3 code[/+]debora-ito commentedon Apr 21, 2023
@gorell memory leaks are usually related to some bad practices with resource management while using the SDK.
lamis31 commentedon Apr 24, 2023
@debora-ito
Yes, we create a new client for every request, is this can cause a memory leak?
Can you provide the api or code example we should use to close the input stream?
Thanks
debora-ito commentedon Apr 24, 2023
Yes, this is most likely the cause of the memory leak. Try changing to a shared instance of the SDK client, and remember to close the client when it is not needed anymore.
I meant the input stream (or any data type) used to read the data from a streaming operation. For example, if you use ResponseInputStream to retrieve data from
S3Client#getObject, callResponseInputStream#closeas soon as you're done reading the stream.We have a list of best practices in the Developer Guide -
https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/best-practices.html
lamis31 commentedon Apr 24, 2023
ok thanks, will try the suggested changes