KubernetesJobOperator.get_pods() fails the task instead of waiting for the Job controller to create pods

Under which category would you file this issue?

Providers

Apache Airflow version

2.11.2

What happened and how to reproduce it?

get_pods() issues discover_pods_retry_number + 1 (default 4) list_namespaced_pod calls back to back with no delay, then raises No pods running with labels ... if the pods are not visible yet. Raising discover_pods_retry_number only adds more immediate GETs — there is no way to wait.

Production trace for one occurrence:

time (UTC)	event
`00:00:23.223`	`job.py` — `Building job job-<task>-<suffix>`
`00:00:23.360`	task failed — `AirflowException: No pods running with labels ...`
`00:00:23`	Job controller — `SuccessfulCreate: Created pod job-<task>-<suffix>-<id>`
`00:01:28`	Job controller — `Job completed`

kube_job_created and kube_pod_created are both 00:00:23: the pod existed in the same second the operator gave up, 137 ms after the Job was built. The workload then ran to success unmonitored. ~20 occurrences in 5 days across 8+ unrelated DAGs.

Consequences:

The task is marked failed while its Job succeeds. Nothing waits on, monitors or cleans up the Job; on_kill() does not fire.
self.pods is never assigned, so _cleanup_monitoring_pods() in the finally block no-ops.
A retry or manual clear starts a second Job while the first is still running or has already committed its writes — silent double execution for non-idempotent tasks.

The message is also inaccurate: the pods are not "not running", they do not exist yet.

What you think should happen instead?

Wait for the pods, bounded by a timeout. No new parameters needed — KubernetesJobOperator extends KubernetesPodOperator, so schedule_timeout_seconds (deadline, default 120 via startup_timeout_seconds) and startup_check_interval_seconds (interval, default 5) are already inherited and documented. The first probe stays immediate, so tasks whose pods appear at once are unaffected.

Also: break on >= parallelism rather than ==, and deprecate discover_pods_retry_number as a no-op.

Operating System

No response

Deployment

Official Apache Airflow Helm Chart

Apache Airflow Provider(s)

No response

Versions of Apache Airflow Providers

cncf provider at 10.13.0 but related code is unchanged in current main

Official Helm Chart version

1.19.0

Kubernetes Version

1.35.6

Helm Chart configuration

Not Applicable

Docker Image customizations

Not Applicable

Anything else?

The issue is Inherited by KubernetesStartKueueJobOperator, GKEStartJobOperator, GKEStartKueueJobOperator — i.e. exactly the Kueue case where pod creation is slowest.

The issue is known and was raised in #53368 but rejected as out of scope for this PR.

Are you willing to submit PR?

Yes I am willing to submit a PR!

Code of Conduct

I agree to follow this project's Code of Conduct

@mo-rieger

@mo-rieger do you have any updates on this? I'm also experiencing the same issue in Airflow V3.

I worked around this issue by creating a subclass of KubernetesJobOperator that adds a delay to the retry loop in get_pods, e.g.:

POD_DISCOVERY_POLL_INTERVAL = 1

class CusomKubernetesJobOperator(KubernetesJobOperator):
    """Waits for the Job controller to create the pods. Drop once cncf-kubernetes ships https://github.com/apache/airflow/issues/70585"""

    def get_pods(self, pod_request_obj, context, *, exclude_checked=True):
        label_selector = self._build_find_pod_label_selector(
            context, exclude_checked=exclude_checked
        )
        pod_list = []

        for attempt in range(self.discover_pods_retry_number + 1):
            if attempt:
                time.sleep(POD_DISCOVERY_POLL_INTERVAL)
            pod_list = self.client.list_namespaced_pod(
                namespace=pod_request_obj.metadata.namespace,
                label_selector=label_selector,
            ).items
            if len(pod_list) >= (self.parallelism or 1):
                break

        if not pod_list:
            raise AirflowException(f"No pods running with labels {label_selector}")

        for pod_instance in pod_list:
            self.log_matching_pod(pod=pod_instance, context=context)

        return pod_list

But I would appreciate a review of the linked PR by a maintainer :)

Navigation Menu

Repository navigation

KubernetesJobOperator.get_pods() fails the task instead of waiting for the Job controller to create pods #70585

Under which category would you file this issue?

Apache Airflow version

What happened and how to reproduce it?

What you think should happen instead?

Operating System

Deployment

Apache Airflow Provider(s)

Versions of Apache Airflow Providers

Official Helm Chart version

Kubernetes Version

Helm Chart configuration

Docker Image customizations

Anything else?

Are you willing to submit PR?

Code of Conduct

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

KubernetesJobOperator.get_pods() fails the task instead of waiting for the Job controller to create pods #70585

Description

Under which category would you file this issue?

Apache Airflow version

What happened and how to reproduce it?

What you think should happen instead?

Operating System

Deployment

Apache Airflow Provider(s)

Versions of Apache Airflow Providers

Official Helm Chart version

Kubernetes Version

Helm Chart configuration

Docker Image customizations

Anything else?

Are you willing to submit PR?

Code of Conduct

Activity

DrTuup commented on Aug 27, 2026

mo-rieger commented on Aug 27, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions