Repository navigation
Support parallel uploads for multiparts while using client side encryption #2248
Description
We are creating AmazonS3Encryption object using AmazonS3EncryptionClientBuilder. We are trying to apply AES 256 client side encryption and use multipart upload in parallel using ThreadPoolExecutor and Callables. The uploads fails as parallel upload is not supported by aws s3 Java SDK. Is there a plan to support this feature? We have a large single file (maximum size of 1 TB) in S3 which is unencrypted. We need to read this file, apply client side encryption using our client managed key and upload encrypted bytes back to S3. We want parallel upload as sequential upload would take long time to complete. The sequential upload is working fine but takes long time to complete.
Is this something that can be supported in near future? If not, are there any workarounds that we could look at?
Caused by: com.amazonaws.SdkClientException: Parts are required to be uploaded in series (partNumber=0, nextPartNumber=9)
at com.amazonaws.services.s3.internal.crypto.MultipartUploadCryptoContext.beginPartUpload(MultipartUploadCryptoContext.java:82)
at com.amazonaws.services.s3.internal.crypto.S3CryptoModuleBase.uploadPartSecurely(S3CryptoModuleBase.java:304)
at com.amazonaws.services.s3.internal.crypto.CryptoModuleDispatcher.uploadPartSecurely(CryptoModuleDispatcher.java:168)
Activity
debora-ito commentedon Mar 6, 2020
Hi @palak123i I'm marking this as a feature request. Will discuss plans for support this with the team.
debora-ito commentedon Mar 12, 2020
So there are no plans to add support for this on V1. The challenge right now is to change the requirement of the encryption process that uses the data from one part for the initialization vector of the next part.
This might be considered for V2. Closing, feel free to reach out if you have further questions.
palak123i commentedon Mar 12, 2020
V2 does not even support client side encryption at the moment. So I am not sure how long it would take for this feature to be available in V2. Is there any ETA you can provide before closing this issue?
debora-ito commentedon Mar 12, 2020
We don't comment on ETAs generally, but you can track which feature is being worked on in V2's Public Features dashboard: https://github.com/aws/aws-sdk-java-v2/projects/1
tomekit commentedon Jan 14, 2023
Hi @palak123i,
Did you manage to solve your issue or workaround it?
I am actually trying to understand what are the benefits of multipart uploads when combined with client side encryption compared tosimple "putObject". Can you easily retry uploading failed part?