Skip to content

Cross-Region S3 Client #52

Closed
Closed
Cross-Region S3 Client#52
Labels
1.x Parityfeature-requestA feature should be added or improved.no-auto-closurep1This is a high priority issue

Description

@millems
millems
opened on Jul 6, 2017

The 1.11.x S3 client automatically routed requests to the correct bucket region in a complicated manner. The 2.0.x client does not perform this automatic re-routing.

We should create a wrapper to pool S3 clients for different regions and automatically route requests to the region in which the bucket was created.

Activity

  1. millems
    added
    1.x Parity
    Feature Request
    on Jul 6, 2017
  2. justnance
    added
    feature-requestA feature should be added or improved.
    and removed
    Feature Request
    on Apr 19, 2019
  3. millems
    added
    no-auto-closure
    on Mar 31, 2020

  4. manasshukla commented on Jun 20, 2020

    @manasshukla
    manasshukla
    on Jun 20, 2020

    Is there an update on when this feature would be shipped out ?

  5. millems commented on Sep 25, 2020

    @millems
    millems
    on Sep 25, 2020
    ContributorAuthor

    Sorry, no update.

    It's not our #1 priority, because with access points and other arn-based ways of accessing buckets/objects plus the use-arn-region flag, the SDK will automatically route the request to the correct region without the overhead of an extra call to AWS to find out where the bucket actually is.

    I understand there's still value in this feature request, since the caller may not have a full bucket ARN.

  6. byF commented on Jul 22, 2021

    @byF
    byF
    on Jul 22, 2021

    If anyone thinks using ARNs like arn:aws:s3:::mybucket will work, they won't #2244

  7. pkgonan commented on Nov 28, 2021

    @pkgonan
    pkgonan
    on Nov 28, 2021

    @millems Hi. Any plan of supporting this feature ? A lot of people think they need this feature from the perspective of customers using AWS S3 products.

  8. millems commented on Nov 30, 2021

    @millems
    millems
    on Nov 30, 2021
    ContributorAuthor

    At some point, probably, but in the meantime you could try using access points which automatically support cross-region request routing.

  9. pkgonan commented on Dec 1, 2021

    @pkgonan
    pkgonan
    on Dec 1, 2021

    @millems
    Oh. That's good news. When are you expecting?

  10. millems commented on Dec 1, 2021

    @millems
    millems
    on Dec 1, 2021
    ContributorAuthor

    @pkgonan Unfortunately we have no expected date to report at this time.

  11. pkgonan commented on Dec 1, 2021

    @pkgonan
    pkgonan
    on Dec 1, 2021

    @millems Thanks for your reply. :)

  12. maciejwalkowiak commented on Mar 24, 2022

    @maciejwalkowiak
    maciejwalkowiak
    on Mar 24, 2022

    FYI we are working on adding cross-region S3 client to Spring Cloud AWS. It's early stage but looks promising so far: https://github.com/awspring/spring-cloud-aws/blob/3.0.x-s3/spring-cloud-aws-s3-parent/spring-cloud-aws-s3/src/main/java/io/awspring/cloud/s3/CrossRegionS3Client.java

    Basic idea is that every region-specific method is wrapped in a try catch that on http status 301 uses client created for the region where the bucket got created. Clients are cached both by region and also by bucket name (to speed up subsequent calls for the same bucket). Implementation is a bit Spring specific as we use LRU cache from spring-core, but if needed this can be replaced with something else or abstracted out.

    Repetitive chunk of this class is generated with JavaParser. If you're interested this is a very sketchy implementation:

  13. yasminetalby
    added
    p1This is a high priority issue
    on Nov 12, 2022

  14. zoewangg commented on Jul 14, 2023

    @zoewangg
    zoewangg
    on Jul 14, 2023
    Contributor

    This feature has been released in 2.20.99. You can enable it by setting crossRegionAccessEnabled to true on the S3 client builder.
    Example:

    S3AsyncClient client = S3AsyncClient.builder()
                                    .crossRegionAccessEnabled(true)
                                    .build();

    I'll go ahead and close this issue. As always, feedback is welcome! :)

  15. zoewangg
    closed this as completedon Jul 14, 2023

  16. github-actions commented on Jul 14, 2023

    @github-actions
    github-actionsBot
    on Jul 14, 2023

    ⚠️COMMENT VISIBILITY WARNING⚠️

    Comments on closed issues are hard for our team to see.
    If you need more assistance, please open a new issue that references this one.
    If you wish to keep having a conversation with other community members under this issue feel free to do so.

  17. dasmowenator commented on Mar 25, 2024

    @dasmowenator
    dasmowenator
    on Mar 25, 2024

    So the addition of .crossRegionAccessEnabled(true) is nice... but it still doesn't work with opt-in regions, even if your account is opted into the given region. For example, if I create an S3 client in the us-east-1 region and then try to head an object in an af-south-1 bucket that I own, it will result in a 400 error. Or if I create an S3 client in the af-south-1 region and then try to head an object in an us-east-1 bucket that I own, it will also result in a 400 error. The SDKv1 client didn't have this problem...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    1.x Parityfeature-requestA feature should be added or improved.no-auto-closurep1This is a high priority issue

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions