Fix KubernetesPodOperator XCom sidecar hang on Alpine by YahelyUshpiz · Pull Request #58931 · apache/airflow

YahelyUshpiz · 2025-12-02T10:05:41Z

The [extract_xcom_kill] method previously used to identify and kill the sidecar process. However, the implementation in some Alpine/BusyBox versions does not support the flag, causing the command to fail and the sidecar to hang indefinitely. This commit replaces the command with a portable shell loop that iterates over to identify processes owned by the current user. This ensures compatibility with all Alpine versions and removes the dependency on .

Bug Report: KubernetesPodOperator XCom Sidecar Hangs due to pgrep missing -u option
Title
KubernetesPodOperator XCom sidecar hangs indefinitely because "pgrep -u" is not supported in some Alpine versions

Description
The KubernetesPodOperator injects a sidecar container to capture XCom values. Once the main container completes, Airflow attempts to terminate this sidecar to allow the Pod to complete.

The termination logic in PodManager.extract_xcom_kill executes the following command inside the sidecar:

kill -2 $(pgrep -u $(id -u) -f 'sh')

This command relies on pgrep supporting the -u (user) flag to filter processes by the current user. However, many lightweight container images used for sidecars (like alpine) use BusyBox's implementation of pgrep. Some versions of BusyBox pgrep do not support the -u flag, causing the command to fail with an invalid option error.

As a result:
The kill command fails.
The sidecar container continues running (executing its infinite sleep loop).
The Pod remains in the Running phase.
The Airflow task eventually times out despite the main work completing successfully.
Impact:
Task Hangs: Tasks that successfully finish their work will hang until the operator times out and causing task to fail.
False Failures: Successful tasks are marked as failed due to timeout.

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

boring-cyborg · 2025-12-02T10:05:47Z

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide (https://github.com/apache/airflow/blob/main/contributing-docs/README.rst)
Here are some useful points:

Pay attention to the quality of your code (ruff, mypy and type annotations). Our prek-hooks will help you with that.
In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
Be sure to read the Airflow Coding style.
Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
Apache Airflow is a community-driven project and together we are making it better 🚀.
In case of doubts contact the developers at:
Mailing List: dev@airflow.apache.org
Slack: https://s.apache.org/airflow-slack

potiuk · 2025-12-07T20:08:08Z

I think this script should generally follow the original script with -u and fallback to alpine way if it fails, or detect busybox/alpine and use alpine-specific way. going through all processes in /proc in bash might take a long time potentially

potiuk

Please make the alpine approach conditional - not default

Nataneljpwd

Looks good, maybe it is a good idea to convert to exception and try catch instead of changing the method definition, as I feel it is more idiomatic, other than that, great find!

Nataneljpwd · 2026-01-03T14:03:27Z

Maybe it is also worth to add a test so that we can avoid regression in the future

YahelyUshpiz · 2026-01-03T14:35:29Z

I agree, I'll convert to it into exception and try/catch.

potiuk · 2026-01-07T14:58:32Z

Still failing

YahelyUshpiz · 2026-01-18T14:26:46Z

Still failing

Fixed

Nataneljpwd

Looks good to me!

jscheffl · 2026-01-18T21:26:29Z

One general question: You say "some Alpine versions" - do you know which? Is it then in general rather advisable to prevent usage of these versions?
On the other hand if you know this is happening in your environment, ist this then not for all Pods? Should the kill command for this case maybe made configurable that you can override it in your environment and it is in general configurable (e.g. on the default kubernetes connection like the Alpine image?

YahelyUshpiz · 2026-01-19T09:12:31Z

One general question: You say "some Alpine versions" - do you know which? Is it then in general rather advisable to prevent usage of these versions? On the other hand if you know this is happening in your environment, ist this then not for all Pods? Should the kill command for this case maybe made configurable that you can override it in your environment and it is in general configurable (e.g. on the default kubernetes connection like the Alpine image?

On Alpine versions older than 3.14 the "-u" flag in "pgrep" doesn't exist, and it happened because the xcom sidecar image was specifically set to this version.
It is an old version, and I agree that the solution may be just to upgrade the xcom sidecar image, but im my opinion if Airflow gives the user the option to set the xcom sidecar image - a better user experience would be to handle this fallback inside Airflow instead of handing it to the client.
Also, this solution gives the most generic fallback (deleting the actual proc files), and is suitable if more cases similar to this will happen in the future.

jscheffl · 2026-01-25T21:49:49Z

I am not sure.

Have you considered the option to update the Alpine image? Why should we support an older version if there is a new out? This is already ~3 years old...

YahelyUshpiz · 2026-01-26T08:28:41Z

I am not sure.

Have you considered the option to update the Alpine image? Why should we support an older version if there is a new out? This is already ~3 years old...

Upgrading the image does solve the issue in this case, but since there isn’t a compatibility matrix for the XCom sidecar and users can specify any version, it might be better to implement a generic fallback. Having the user manually configure the kill command could add unnecessary friction to the user experience.

The [extract_xcom_kill] method previously used to identify and kill the sidecar process. However, the implementation in some Alpine/BusyBox versions does not support the flag, causing the command to fail and the sidecar to hang indefinitely. This commit replaces the command with a portable shell loop that iterates over to identify processes owned by the current user. This ensures compatibility with all Alpine versions and removes the dependency on .

This reverts commit 8c4b5f5.

jscheffl · 2026-02-05T19:35:50Z

I am not sure.
Have you considered the option to update the Alpine image? Why should we support an older version if there is a new out? This is already ~3 years old...

Upgrading the image does solve the issue in this case, but since there isn’t a compatibility matrix for the XCom sidecar and users can specify any version, it might be better to implement a generic fallback. Having the user manually configure the kill command could add unnecessary friction to the user experience.

For me this is not fully convincing. If you assume somebody is updating providers to have this convenience why not expecting to also update Alpine image? Is there a reason to use the 3y old image?

Nataneljpwd · 2026-02-08T08:17:29Z

Maybe a good idea might be to add a compatibility matrix in the docs?
I think it can also aid in solving the issue

potiuk · 2026-03-11T19:49:30Z

I am not sure.
Have you considered the option to update the Alpine image? Why should we support an older version if there is a new out? This is already ~3 years old...

Upgrading the image does solve the issue in this case, but since there isn’t a compatibility matrix for the XCom sidecar and users can specify any version, it might be better to implement a generic fallback. Having the user manually configure the kill command could add unnecessary friction to the user experience.

For me this is not fully convincing. If you assume somebody is updating providers to have this convenience why not expecting to also update Alpine image? Is there a reason to use the 3y old image?

I am fine with it - as long as it is fallback

boring-cyborg · 2026-03-11T19:49:37Z

Awesome work, congrats on your first merged pull request! You are invited to check our Issue Tracker for additional contributions.

github-actions · 2026-03-11T19:50:24Z

Hi maintainer, this PR was merged without a milestone set.
We've automatically set the milestone to Airflow 3.1.9 based on: backport label targeting v3-1-test
If this milestone is not correct, please update it to the appropriate milestone.

This comment was generated by Milestone Tag Assistant.

…8931) * Fix KubernetesPodOperator XCom sidecar hang on Alpine The [extract_xcom_kill] method previously used to identify and kill the sidecar process. However, the implementation in some Alpine/BusyBox versions does not support the flag, causing the command to fail and the sidecar to hang indefinitely. This commit replaces the command with a portable shell loop that iterates over to identify processes owned by the current user. This ensures compatibility with all Alpine versions and removes the dependency on . * changed alpine fallback command * added conditional fallback xcom kill command when the primary one fails * Revert "changed alpine fallback command" This reverts commit 8c4b5f5. * Convert the xcom kill command conditioning to exception and try/catch * Fixed command string format (cherry picked from commit ed78d6c) Co-authored-by: Yahely Ushpiz <102915448+YahelyUshpiz@users.noreply.github.com>

github-actions · 2026-03-11T19:50:50Z

Backport successfully created: v3-1-test

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status	Branch	Result
✅	v3-1-test

…ache#58931) * Fix KubernetesPodOperator XCom sidecar hang on Alpine The [extract_xcom_kill] method previously used to identify and kill the sidecar process. However, the implementation in some Alpine/BusyBox versions does not support the flag, causing the command to fail and the sidecar to hang indefinitely. This commit replaces the command with a portable shell loop that iterates over to identify processes owned by the current user. This ensures compatibility with all Alpine versions and removes the dependency on . * changed alpine fallback command * added conditional fallback xcom kill command when the primary one fails * Revert "changed alpine fallback command" This reverts commit 8c4b5f5. * Convert the xcom kill command conditioning to exception and try/catch * Fixed command string format (cherry picked from commit ed78d6c) Co-authored-by: Yahely Ushpiz <102915448+YahelyUshpiz@users.noreply.github.com>

eladkal · 2026-03-11T20:06:12Z

This is provider only.
No backport needed

* Fix KubernetesPodOperator XCom sidecar hang on Alpine The [extract_xcom_kill] method previously used to identify and kill the sidecar process. However, the implementation in some Alpine/BusyBox versions does not support the flag, causing the command to fail and the sidecar to hang indefinitely. This commit replaces the command with a portable shell loop that iterates over to identify processes owned by the current user. This ensures compatibility with all Alpine versions and removes the dependency on . * changed alpine fallback command * added conditional fallback xcom kill command when the primary one fails * Revert "changed alpine fallback command" This reverts commit 8c4b5f5. * Convert the xcom kill command conditioning to exception and try/catch * Fixed command string format

YahelyUshpiz requested review from hussein-awala and jedcunningham as code owners December 2, 2025 10:05

boring-cyborg Bot added area:providers provider:cncf-kubernetes Kubernetes (k8s) provider related issues labels Dec 2, 2025

potiuk requested changes Dec 7, 2025

View reviewed changes

YahelyUshpiz force-pushed the fix/kubernetes-pod-operator-xcom-hang branch from d7016d4 to 8c4b5f5 Compare December 31, 2025 07:33

Nataneljpwd reviewed Jan 3, 2026

View reviewed changes

YahelyUshpiz requested a review from potiuk January 4, 2026 18:16

Nataneljpwd approved these changes Jan 18, 2026

View reviewed changes

eladkal requested a review from jscheffl January 18, 2026 19:43

YahelyUshpiz added 6 commits February 5, 2026 09:17

changed alpine fallback command

aba3763

added conditional fallback xcom kill command when the primary one fails

ccd3bb0

Revert "changed alpine fallback command"

8b3ca12

This reverts commit 8c4b5f5.

Convert the xcom kill command conditioning to exception and try/catch

cefcad6

Fixed command string format

bdba85a

eladkal force-pushed the fix/kubernetes-pod-operator-xcom-hang branch from a55d78f to bdba85a Compare February 5, 2026 08:17

potiuk approved these changes Mar 11, 2026

View reviewed changes

potiuk merged commit ed78d6c into apache:main Mar 11, 2026
199 of 200 checks passed

potiuk added the backport-to-v3-1-test label Mar 11, 2026

github-actions Bot added this to the Airflow 3.1.9 milestone Mar 11, 2026

eladkal removed this from the Airflow 3.1.9 milestone Mar 11, 2026

eladkal removed the backport-to-v3-1-test label Mar 11, 2026

jscheffl mentioned this pull request Mar 24, 2026

Status of testing Providers that were prepared on March 24, 2026 #64195

Closed

Uh oh!

Conversation

YahelyUshpiz commented Dec 2, 2025

Uh oh!

boring-cyborg Bot commented Dec 2, 2025

Uh oh!

potiuk commented Dec 7, 2025

Uh oh!

potiuk left a comment

Choose a reason for hiding this comment

Uh oh!

Nataneljpwd left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Nataneljpwd commented Jan 3, 2026

Uh oh!

YahelyUshpiz commented Jan 3, 2026

Uh oh!

potiuk commented Jan 7, 2026

Uh oh!

YahelyUshpiz commented Jan 18, 2026

Uh oh!

Nataneljpwd left a comment

Choose a reason for hiding this comment

Uh oh!

jscheffl commented Jan 18, 2026

Uh oh!

YahelyUshpiz commented Jan 19, 2026

Uh oh!

jscheffl commented Jan 25, 2026

Uh oh!

YahelyUshpiz commented Jan 26, 2026

Uh oh!

jscheffl commented Feb 5, 2026

Uh oh!

Nataneljpwd commented Feb 8, 2026

Uh oh!

potiuk commented Mar 11, 2026

Uh oh!

Uh oh!

boring-cyborg Bot commented Mar 11, 2026

Uh oh!

github-actions Bot commented Mar 11, 2026

Uh oh!

github-actions Bot commented Mar 11, 2026

Backport successfully created: v3-1-test

Uh oh!

eladkal commented Mar 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants