fix(providers/amazon): handle multi-node job log streams in BatchClientHook - #63476
Merged
potiuk merged 2 commits intoMar 14, 2026
Merged
Conversation
…ntHook Multi-node AWS Batch jobs return log stream names under `attempts[].taskProperties[].containers[].logStreamName` instead of the single-node path `attempts[].container.logStreamName`. The existing code only checked the single-node path, producing `[None]` for multi-node jobs which bypassed the emptiness guard and caused a TypeError in urllib.parse.quote_plus() downstream. Now extracts stream names from both schemas and filters out None values. Closes apache#54254 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
boring-cyborg
Bot
added
area:providers
provider:amazon
o-nikolas
reviewed
Mar 12, 2026
Skip multi-node iteration when single-node stream is already found, as suggested in review. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Member
|
@o-nikolas — Could you please check whether your review feedback on this PR has been addressed? @antonio-mello-ai appears to have responded to your comments. @antonio-mello-ai, do you believe the reviewer's concerns have been resolved? If the concerns are resolved, please resolve the conversation threads. Thank you! |
Contributor
Author
|
@potiuk Yes, the reviewer's concern has been addressed — added |
potiuk
approved these changes
Mar 14, 2026
Contributor
|
Just getting back to this monday, looks all good to me |
abhijeets25012-tech
pushed a commit
to abhijeets25012-tech/airflow
that referenced
this pull request
Apr 9, 2026
…ntHook (apache#63476) * fix(providers/amazon): handle multi-node job log streams in BatchClientHook Multi-node AWS Batch jobs return log stream names under `attempts[].taskProperties[].containers[].logStreamName` instead of the single-node path `attempts[].container.logStreamName`. The existing code only checked the single-node path, producing `[None]` for multi-node jobs which bypassed the emptiness guard and caused a TypeError in urllib.parse.quote_plus() downstream. Now extracts stream names from both schemas and filters out None values. Closes apache#54254 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: add continue after single-node stream match Skip multi-node iteration when single-node stream is already found, as suggested in review. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
get_job_all_awslogs_info()to extract log stream names from both single-node (attempts[].container.logStreamName) and multi-node (attempts[].taskProperties[].containers[].logStreamName) AWS Batch job schemasNonevalues that previously bypassed theif not stream_names:guard, causingTypeErrorinurllib.parse.quote_plus()downstreamRoot Cause
Multi-node AWS Batch jobs return log stream names under
attempts[].taskProperties[].containers[].logStreamName, not the single-node pathattempts[].container.logStreamName. The existing code only checked the single-node path, producingstream_names = [None]for multi-node jobs. Since[None]is truthy, it passed the emptiness guard and propagatedNonedownstream.Test Plan
taskPropertiesschema extracts stream names correctlylogStreamNameincontainerfalls through to warning pathprek)Closes #54254
🤖 Generated with Claude Code
Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com