Description

Task-level deadline alerts

Airflow 3 introduced Deadline Alerts, replacing the previous SLA mechanism. This is a useful improvement, but currently deadlines can only be defined at the DAG run level.

I would like to propose extending Deadline Alerts to support task-level deadlines as well.

Use case/motivation

Motivation

A DAG run can consist of many tasks with very different expected completion times. A single deadline for the entire DAG does not provide enough granularity to detect delays in individual parts of the pipeline.

For example:

DAG: daily_etl

extract_data       → expected by 01:00
transform_data     → expected by 02:00
load_data          → expected by 03:00
publish_report     → expected by 04:00

With a DAG-level deadline of 04:00, a problem with extract_data might only become visible much later. If the task has its own deadline, the problem can be detected and alerted immediately.

This is particularly useful for:

• long-running ETL/ELT pipelines
• data availability monitoring
• pipelines with different deadlines for individual stages
• operational monitoring where downstream tasks depend on upstream data
• detecting bottlenecks before they cause the entire DAG run to miss its deadline

Proposed API

The existing DAG-level mechanism could be extended with an optional task-level deadline configuration.

For example:

@dag(
    deadline=DeadlineAlert(
        interval=timedelta(hours=4),
        callback=...
    )
)
def my_dag():

    extract = PythonOperator(
        task_id="extract",
        ...,
        deadline=DeadlineAlert(
            interval=timedelta(hours=1),
            callback=...
        )
    )

Alternatively, the deadline could be defined directly on the task:

extract = PythonOperator(
    task_id="extract",
    ...,
    deadline=timedelta(hours=1),
    deadline_callback=...
)

The exact API is of course open for discussion.

Expected behavior

A task-level deadline should be evaluated independently for each task instance.

For example, if:

extract_data deadline = 01:00
transform_data deadline = 02:00
DAG run deadline = 04:00

and extract_data has not completed by 01:00, the configured deadline callback should be triggered, even though the overall DAG run is still within its deadline.

The DAG-level deadline should continue to work independently.

Why not use execution_timeout?

execution_timeout is useful, but it solves a different problem.

execution_timeout answers:

How long is a task allowed to run?

A deadline answers:

By what point should the task have completed?

For example, a task might normally run for 10 minutes but must be completed by 01:00 because downstream processing starts afterwards. An execution timeout does not express this requirement.

Backwards compatibility

This should be backwards compatible:

• existing DAG-level deadlines continue to work unchanged
• task-level deadlines are optional
• a DAG can use DAG-level deadlines, task-level deadlines, or both
• no deadline configuration means no deadline alert

I believe task-level deadlines would make the new Deadline Alert functionality significantly more useful for real-world production pipelines, especially for data engineering and operational workloads.

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct