Repository navigation
Amazon emr serverless get session endpoint - #70766
Merged
vincbeck merged 1 commit intoOct 14, 2026
Merged
Conversation
boring-cyborg
Bot
added
area:providers
kind:documentation
provider:amazon
This was referenced Jul 30, 2026
Contributor
|
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions. |
github-actions
Bot
added
the
stale
vgkowski
force-pushed
the
amazon-emr-serverless-get-session-endpoint
branch
from
September 26, 2026 10:52
2a21b55 to
03f5cca
Compare
github-actions
Bot
removed
the
stale
vgkowski
marked this pull request as ready for review
September 29, 2026 18:32
vgkowski
force-pushed
the
amazon-emr-serverless-get-session-endpoint
branch
from
September 29, 2026 19:16
03f5cca to
f303fe9
Compare
Contributor
Author
|
@vincbeck @o-nikolas this is the next step for EMR-S session integration |
vincbeck
reviewed
Oct 12, 2026
Spark Connect clients need a session's endpoint and a short-lived auth token to attach to an EMR Serverless interactive session. Fetching them in their own task, right before the connecting task, keeps the token fresh. The token is a credential, so it is masked in task logs. Airflow 3 redacts task logs in the supervisor, so common.compat resolves mask_secret to the variant that forwards the mask there, and falls back to the older locations on earlier Airflow versions. The token remains readable in XCom, which the docs call out.
vgkowski
force-pushed
the
amazon-emr-serverless-get-session-endpoint
branch
from
October 13, 2026 14:26
f303fe9 to
1b77352
Compare
vincbeck
approved these changes
Oct 13, 2026
This was referenced Oct 14, 2026
1 task done
1 task done
10 of 93 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add
EmrServerlessGetSessionEndpointOperator, which resolves the Spark Connectendpoint URL and a short-lived authentication token for a running EMR Serverless
interactive session so a client can attach to it.
Why:
so the endpoint must be fetched fresh right before connecting rather than cached.
secrets masker before returning it via XCom, keeping it out of task logs and the
rendered XCom value in the UI.
Changes:
EmrServerlessHook.get_session_endpointreturns the rawGetSessionEndpointboto3response (thin wrapper, per the provider convention); the operator does the field
transformation before pushing to XCom.
mask_secretto thecommon.compatSDK shim so the masker import resolves acrosssupported Airflow versions.
The feature is gated on
botocore >= 1.43.0at runtime (introduced in the base PR), sothe provider's dependency floor is unchanged.
Note
This PR is part of a stack and depends on
#70763 (adds
EmrServerlessStartSessionOperatorand the shared session plumbing). Please review/mergethat one first — this PR's base branch is
vgkowski/amazon-emr-serverless-start-sessionuntil itlands, then it retargets to
main.Was generative AI tooling used to co-author this PR?
Generated-by: Kiro, following the Gen-AI guidelines in contributing-docs/05_pull_requests.rst