KubernetesPodOperator task is incorrectly marked as SUCCESS instead of FAILED when both the task pod and its child pod are interrupted in close succession

[RayVilaca]

Under which category would you file this issue?

Providers

Apache Airflow version

3.3.0

What happened and how to reproduce it?

Architecture context

This happens when using KubernetesExecutor together with KubernetesPodOperator, which involves two separate pods:

1. The per-task pod created by KubernetesExecutor to execute the task instance.
2. The child pod created by KubernetesPodOperator.execute() to run the user's workload.

Issue Description

When both pods are interrupted within roughly one second of each other, the task instance is sometimes finalized as success, even though the child pod never completed its work.

Steps to reproduce

1. Trigger a DAG task using KubernetesPodOperator whose child pod runs long enough to still be running when interrupted. Example:

from datetime import datetime
from airflow import DAG
from airflow.models import Variable
from airflow.providers.cncf.kubernetes.operators.pod import KubernetesPodOperator

default_args = {
    "owner": "data",
    "depends_on_past": False,
    "start_date": datetime.strptime("2026-08-05", "%Y-%m-%d"),
    "params": {
        "severity": 0,
    },
    "retries": 1,
}

dag = DAG(
    dag_id="air-test-sigterm-repro",
    default_args=default_args,
    catchup=False,
    schedule=None,
    max_active_runs=1
)

sleep_worker = KubernetesPodOperator(
    dag=dag,
    task_id="sleep-worker",
    name="sleep-worker",
    namespace=Variable.get("var-k8s-namespace"),
    image="python:3.9",
    cmds=["python", "-c"],
    do_xcom_push=False,
    get_logs=True,
    is_delete_operator_pod=True,
    arguments=[
        "import time\n"
        "for i in range(120):\n"
        "    print('tick', i, 'still running, still streaming logs', flush=True)\n"
        "    time.sleep(5)\n"
        "print('finished without interruption', flush=True)\n"
    ],
)

2. While the task is running, delete both the child pod and the per-task pod within the same short time window (rather than waiting for one deletion to complete). The most reliable sequence we found is to delete the child pod first, then the per-task pod about one second later.

   Terminal 1 — child pod:

   kubectl delete pod sleep-worker-xxxxxxx -n airflow --grace-period=30

   Terminal 2 (~1 second later) — per-task pod:

   kubectl delete pod air-test-sigterm-repro-sleep-worker-xxxxxxxxx -n airflow --grace-period=30

3. Observe the task's final state in the Airflow UI or API.

logs sleep-worker-xxxxxxx

tick 0 still running, still streaming logs
tick 1 still running, still streaming logs
tick 2 still running, still streaming logs
tick 3 still running, still streaming logs
tick 4 still running, still streaming logs
tick 5 still running, still streaming logs
tick 6 still running, still streaming logs
tick 7 still running, still streaming logs
tick 8 still running, still streaming logs
tick 9 still running, still streaming logs
tick 10 still running, still streaming logs
tick 11 still running, still streaming logs
tick 12 still running, still streaming logs
tick 13 still running, still streaming logs
tick 14 still running, still streaming logs
tick 15 still running, still streaming logs
tick 16 still running, still streaming logs
tick 17 still running, still streaming logs

logs air-test-sigterm-repro-sleep-worker-xxxxxxxxx

{"timestamp":"2026-08-05T18:49:28.622020Z","level":"info","event":"::endgroup::","task_id":"sleep-worker","map_index":-1,"try_number":1,"run_id":"manual__2026-08-05T18:49:11.709471+00:00","dag_id":"air-test-sigterm-repro","ti_id":"019fd342-172e-7f4b-8c58-eebe3cb6bc91","logger":"airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager","filename":"pod_manager.py","lineno":186}
{"timestamp":"2026-08-05T18:49:29.219279Z","level":"info","event":"[base] tick 0 still running, still streaming logs","task_id":"sleep-worker","map_index":-1,"try_number":1,"run_id":"manual__2026-08-05T18:49:11.709471+00:00","dag_id":"air-test-sigterm-repro","ti_id":"019fd342-172e-7f4b-8c58-eebe3cb6bc91","logger":"airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager","filename":"pod_manager.py","lineno":500}
{"timestamp":"2026-08-05T18:49:34.224101Z","level":"info","event":"[base] tick 1 still running, still streaming logs","task_id":"sleep-worker","map_index":-1,"try_number":1,"run_id":"manual__2026-08-05T18:49:11.709471+00:00","dag_id":"air-test-sigterm-repro","ti_id":"019fd342-172e-7f4b-8c58-eebe3cb6bc91","logger":"airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager","filename":"pod_manager.py","lineno":500}
{"timestamp":"2026-08-05T18:49:39.229335Z","level":"info","event":"[base] tick 2 still running, still streaming logs","task_id":"sleep-worker","map_index":-1,"try_number":1,"run_id":"manual__2026-08-05T18:49:11.709471+00:00","dag_id":"air-test-sigterm-repro","ti_id":"019fd342-172e-7f4b-8c58-eebe3cb6bc91","logger":"airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager","filename":"pod_manager.py","lineno":500}
{"timestamp":"2026-08-05T18:49:44.230741Z","level":"info","event":"[base] tick 3 still running, still streaming logs","task_id":"sleep-worker","map_index":-1,"try_number":1,"run_id":"manual__2026-08-05T18:49:11.709471+00:00","dag_id":"air-test-sigterm-repro","ti_id":"019fd342-172e-7f4b-8c58-eebe3cb6bc91","logger":"airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager","filename":"pod_manager.py","lineno":500}
{"timestamp":"2026-08-05T18:49:49.235802Z","level":"info","event":"[base] tick 4 still running, still streaming logs","task_id":"sleep-worker","map_index":-1,"try_number":1,"run_id":"manual__2026-08-05T18:49:11.709471+00:00","dag_id":"air-test-sigterm-repro","ti_id":"019fd342-172e-7f4b-8c58-eebe3cb6bc91","logger":"airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager","filename":"pod_manager.py","lineno":500}
{"timestamp":"2026-08-05T18:49:54.240729Z","level":"info","event":"[base] tick 5 still running, still streaming logs","task_id":"sleep-worker","map_index":-1,"try_number":1,"run_id":"manual__2026-08-05T18:49:11.709471+00:00","dag_id":"air-test-sigterm-repro","ti_id":"019fd342-172e-7f4b-8c58-eebe3cb6bc91","logger":"airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager","filename":"pod_manager.py","lineno":500}
{"timestamp":"2026-08-05T18:49:59.245828Z","level":"info","event":"[base] tick 6 still running, still streaming logs","task_id":"sleep-worker","map_index":-1,"try_number":1,"run_id":"manual__2026-08-05T18:49:11.709471+00:00","dag_id":"air-test-sigterm-repro","ti_id":"019fd342-172e-7f4b-8c58-eebe3cb6bc91","logger":"airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager","filename":"pod_manager.py","lineno":500}
{"timestamp":"2026-08-05T18:50:04.247038Z","level":"info","event":"[base] tick 7 still running, still streaming logs","task_id":"sleep-worker","map_index":-1,"try_number":1,"run_id":"manual__2026-08-05T18:49:11.709471+00:00","dag_id":"air-test-sigterm-repro","ti_id":"019fd342-172e-7f4b-8c58-eebe3cb6bc91","logger":"airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager","filename":"pod_manager.py","lineno":500}
{"timestamp":"2026-08-05T18:50:09.247528Z","level":"info","event":"[base] tick 8 still running, still streaming logs","task_id":"sleep-worker","map_index":-1,"try_number":1,"run_id":"manual__2026-08-05T18:49:11.709471+00:00","dag_id":"air-test-sigterm-repro","ti_id":"019fd342-172e-7f4b-8c58-eebe3cb6bc91","logger":"airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager","filename":"pod_manager.py","lineno":500}
{"timestamp":"2026-08-05T18:50:14.250719Z","level":"info","event":"[base] tick 9 still running, still streaming logs","task_id":"sleep-worker","map_index":-1,"try_number":1,"run_id":"manual__2026-08-05T18:49:11.709471+00:00","dag_id":"air-test-sigterm-repro","ti_id":"019fd342-172e-7f4b-8c58-eebe3cb6bc91","logger":"airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager","filename":"pod_manager.py","lineno":500}
{"timestamp":"2026-08-05T18:50:19.255868Z","level":"info","event":"[base] tick 10 still running, still streaming logs","task_id":"sleep-worker","map_index":-1,"try_number":1,"run_id":"manual__2026-08-05T18:49:11.709471+00:00","dag_id":"air-test-sigterm-repro","ti_id":"019fd342-172e-7f4b-8c58-eebe3cb6bc91","logger":"airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager","filename":"pod_manager.py","lineno":500}
{"timestamp":"2026-08-05T18:50:24.256810Z","level":"info","event":"[base] tick 11 still running, still streaming logs","task_id":"sleep-worker","map_index":-1,"try_number":1,"run_id":"manual__2026-08-05T18:49:11.709471+00:00","dag_id":"air-test-sigterm-repro","ti_id":"019fd342-172e-7f4b-8c58-eebe3cb6bc91","logger":"airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager","filename":"pod_manager.py","lineno":500}
{"timestamp":"2026-08-05T18:50:29.261832Z","level":"info","event":"[base] tick 12 still running, still streaming logs","task_id":"sleep-worker","map_index":-1,"try_number":1,"run_id":"manual__2026-08-05T18:49:11.709471+00:00","dag_id":"air-test-sigterm-repro","ti_id":"019fd342-172e-7f4b-8c58-eebe3cb6bc91","logger":"airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager","filename":"pod_manager.py","lineno":500}
{"timestamp":"2026-08-05T18:50:34.266976Z","level":"info","event":"[base] tick 13 still running, still streaming logs","task_id":"sleep-worker","map_index":-1,"try_number":1,"run_id":"manual__2026-08-05T18:49:11.709471+00:00","dag_id":"air-test-sigterm-repro","ti_id":"019fd342-172e-7f4b-8c58-eebe3cb6bc91","logger":"airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager","filename":"pod_manager.py","lineno":500}
{"timestamp":"2026-08-05T18:50:39.272003Z","level":"info","event":"[base] tick 14 still running, still streaming logs","task_id":"sleep-worker","map_index":-1,"try_number":1,"run_id":"manual__2026-08-05T18:49:11.709471+00:00","dag_id":"air-test-sigterm-repro","ti_id":"019fd342-172e-7f4b-8c58-eebe3cb6bc91","logger":"airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager","filename":"pod_manager.py","lineno":500}
{"timestamp":"2026-08-05T18:50:44.277099Z","level":"info","event":"[base] tick 15 still running, still streaming logs","task_id":"sleep-worker","map_index":-1,"try_number":1,"run_id":"manual__2026-08-05T18:49:11.709471+00:00","dag_id":"air-test-sigterm-repro","ti_id":"019fd342-172e-7f4b-8c58-eebe3cb6bc91","logger":"airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager","filename":"pod_manager.py","lineno":500}
{"timestamp":"2026-08-05T18:50:49.282109Z","level":"info","event":"[base] tick 16 still running, still streaming logs","task_id":"sleep-worker","map_index":-1,"try_number":1,"run_id":"manual__2026-08-05T18:49:11.709471+00:00","dag_id":"air-test-sigterm-repro","ti_id":"019fd342-172e-7f4b-8c58-eebe3cb6bc91","logger":"airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager","filename":"pod_manager.py","lineno":500}
{"timestamp":"2026-08-05T18:50:54.286787Z","level":"info","event":"[base] tick 17 still running, still streaming logs","task_id":"sleep-worker","map_index":-1,"try_number":1,"run_id":"manual__2026-08-05T18:49:11.709471+00:00","dag_id":"air-test-sigterm-repro","ti_id":"019fd342-172e-7f4b-8c58-eebe3cb6bc91","logger":"airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager","filename":"pod_manager.py","lineno":500}
{"timestamp":"2026-08-05T18:50:59.288915Z","level":"info","event":"[base] tick 18 still running, still streaming logs","task_id":"sleep-worker","map_index":-1,"try_number":1,"run_id":"manual__2026-08-05T18:49:11.709471+00:00","dag_id":"air-test-sigterm-repro","ti_id":"019fd342-172e-7f4b-8c58-eebe3cb6bc91","logger":"airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager","filename":"pod_manager.py","lineno":500}
{"timestamp":"2026-08-05T18:51:04.293390Z","level":"info","event":"[base] tick 19 still running, still streaming logs","task_id":"sleep-worker","map_index":-1,"try_number":1,"run_id":"manual__2026-08-05T18:49:11.709471+00:00","dag_id":"air-test-sigterm-repro","ti_id":"019fd342-172e-7f4b-8c58-eebe3cb6bc91","logger":"airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager","filename":"pod_manager.py","lineno":500}
{"timestamp":"2026-08-05T18:51:09.296800Z","level":"info","event":"[base] tick 20 still running, still streaming logs","task_id":"sleep-worker","map_index":-1,"try_number":1,"run_id":"manual__2026-08-05T18:49:11.709471+00:00","dag_id":"air-test-sigterm-repro","ti_id":"019fd342-172e-7f4b-8c58-eebe3cb6bc91","logger":"airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager","filename":"pod_manager.py","lineno":500}
{"timestamp":"2026-08-05T18:51:14.301846Z","level":"info","event":"[base] tick 21 still running, still streaming logs","task_id":"sleep-worker","map_index":-1,"try_number":1,"run_id":"manual__2026-08-05T18:49:11.709471+00:00","dag_id":"air-test-sigterm-repro","ti_id":"019fd342-172e-7f4b-8c58-eebe3cb6bc91","logger":"airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager","filename":"pod_manager.py","lineno":500}
{"timestamp":"2026-08-05T18:51:19.306809Z","level":"info","event":"[base] tick 22 still running, still streaming logs","task_id":"sleep-worker","map_index":-1,"try_number":1,"run_id":"manual__2026-08-05T18:49:11.709471+00:00","dag_id":"air-test-sigterm-repro","ti_id":"019fd342-172e-7f4b-8c58-eebe3cb6bc91","logger":"airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager","filename":"pod_manager.py","lineno":500}
{"timestamp":"2026-08-05T18:51:24.311349Z","level":"info","event":"[base] tick 23 still running, still streaming logs","task_id":"sleep-worker","map_index":-1,"try_number":1,"run_id":"manual__2026-08-05T18:49:11.709471+00:00","dag_id":"air-test-sigterm-repro","ti_id":"019fd342-172e-7f4b-8c58-eebe3cb6bc91","logger":"airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager","filename":"pod_manager.py","lineno":500}
{"timestamp":"2026-08-05T18:51:29.315332Z","level":"info","event":"[base] tick 24 still running, still streaming logs","task_id":"sleep-worker","map_index":-1,"try_number":1,"run_id":"manual__2026-08-05T18:49:11.709471+00:00","dag_id":"air-test-sigterm-repro","ti_id":"019fd342-172e-7f4b-8c58-eebe3cb6bc91","logger":"airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager","filename":"pod_manager.py","lineno":500}
{"timestamp":"2026-08-05T18:51:34.316805Z","level":"info","event":"[base] tick 25 still running, still streaming logs","task_id":"sleep-worker","map_index":-1,"try_number":1,"run_id":"manual__2026-08-05T18:49:11.709471+00:00","dag_id":"air-test-sigterm-repro","ti_id":"019fd342-172e-7f4b-8c58-eebe3cb6bc91","logger":"airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager","filename":"pod_manager.py","lineno":500}
{"timestamp":"2026-08-05T18:51:36.975521Z","level":"info","event":"Received signal, forwarding to task subprocess","signal":"SIGTERM","pid":13,"logger":"supervisor","filename":"supervisor.py","lineno":1403}
{"timestamp":"2026-08-05T18:51:39.344660Z","level":"info","event":"[base] tick 26 still running, still streaming logs","task_id":"sleep-worker","map_index":-1,"try_number":1,"run_id":"manual__2026-08-05T18:49:11.709471+00:00","dag_id":"air-test-sigterm-repro","ti_id":"019fd342-172e-7f4b-8c58-eebe3cb6bc91","logger":"airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager","filename":"pod_manager.py","lineno":500}
{"timestamp":"2026-08-05T18:51:44.322040Z","level":"info","event":"[base] tick 27 still running, still streaming logs","task_id":"sleep-worker","map_index":-1,"try_number":1,"run_id":"manual__2026-08-05T18:49:11.709471+00:00","dag_id":"air-test-sigterm-repro","ti_id":"019fd342-172e-7f4b-8c58-eebe3cb6bc91","logger":"airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager","filename":"pod_manager.py","lineno":500}

Deleting only one of the two pods did not reproduce the issue in our testing. In those cases, the task correctly transitioned to failed or up_for_retry.

What we observe when it reproduces

The last log emitted by the per-task pod is:

{"event":"Received signal, forwarding to task subprocess","signal":"SIGTERM","pid":13,"logger":"supervisor","filename":"supervisor.py","lineno":1403}

After that, we never see:

• on_kill() being invoked.
• The provider's "Deleting pod: ..." log line.
• Any exception or traceback.
• Any further output from the child pod.

Despite the child pod being interrupted before completing its work, the task is later reported by the API server as success.

Observed in production

We observed this four times across two different DAGs over four days. Every occurrence coincided with our autoscaler reclaiming a Spot/preemptible node hosting one or both pods.

When the issue reproduces:

19:12:28Z  [child pod progress log — task actively running]
19:12:42Z  <node hosting the per-task pod begins draining>
19:12:44Z  {"event":"Received signal, forwarding to task subprocess","signal":"SIGTERM","pid":13,"logger":"supervisor"}
           <nothing else logged>
           --> Task instance later shows state = success

A normal execution of the same task instead includes:

{"event":"Deleting pod: <pod-name>", ...}
{"event":"::group::Post Execute", ...}
{"event":"Workload finished", "final_state":"success", ...}

Contrast: interrupting only the child pod

If only the child pod is interrupted while the per-task pod continues running, the task correctly fails. We've observed both AirflowException (PodFailed) and NotFoundException (404 after pod deletion), but in both cases the task is never reported as success.

What you think should happen instead?

The task should end in failed (or up_for_retry), just as it does when only the child pod is interrupted. Reporting success for a task whose workload never completed is a correctness issue, downstream tasks proceed as if the work succeeded, on_failure_callback is never triggered, and no retry occurs.

We haven't identified the root cause and are only reporting the observed, reproducible behavior. This may be related to #58936 (fixed by #61627 in 3.3.0), which changed how SIGTERM is forwarded from the supervisor to the task subprocess. The last log we consistently see ("Received signal, forwarding to task subprocess") originates from that change, although we can't confirm whether this is a regression from that fix or a separate issue.

Operating System

Debian GNU/Linux 12 (bookworm)

Deployment

Official Apache Airflow Helm Chart

Apache Airflow Provider(s)

cncf-kubernetes

Versions of Apache Airflow Providers

apache-airflow-providers-cncf-kubernetes==10.19.0

Official Helm Chart version

1.18.0

Kubernetes Version

No response

Helm Chart configuration

No response

Docker Image customizations

No response

Anything else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[adrianrego]

I've observed this behavior as well on our production clusters.