Repository navigation
Adaptive retry for Glue Catalog calls to fix Glue throttling #1294
Description
Feature Request / Improvement
I have experienced throttling exceptions when multiple nodes are reading and writing an Iceberg table with a Glue catalog. I have wrapped my library calls in a retry wrapper, but this functionality should be implemented in the library instead.
I would add an option to the Glue catalog constructor to define the maximum number of retries and set the boto retry policy to standard instead of the default legacy. Standard is adding jitter to the retry interval which would ensure independent processes won't block each other (Thundering herd problem)
Metadata
Metadata
Assignees
Labels
No labels
Activity
Fokko commentedon Nov 5, 2024
Hey @mark-major thanks for jumping in here. I see that we only added retries to the REST catalog so far. Having this for the Glue catalog would be a great addition 👍
mark-major commentedon Nov 5, 2024
I'm happy to work on this if I have some time.
Fokko commentedon Nov 5, 2024
That would be great, happy to review! 🙌
kevinjqliu commentedon Nov 6, 2024
we use boto3 to create the glue client
looks like it has some built-in retry mechanism
https://boto3.amazonaws.com/v1/documentation/api/latest/guide/retries.html#configuring-a-retry-mode
mark-major commentedon Nov 6, 2024
@kevinjqliu Yes, we'll need the standard retry mode and the ability to configure the max attempts.
mark-major commentedon Nov 8, 2024
PR #1307 is opened