Repository navigation
Conversation
|
why do you think this ? If you want to detect the case of a dag_run that did not run any task then add a very last task from airflow.providers.standard.operators.empty import EmptyOperator
from airflow.providers.standard.operators.python import PythonOperator
from airflow.sdk import TaskInstanceState, TriggerRule, get_current_context
from airflow.sdk import DAG, CronDataIntervalTimetable, timezone
with DAG(
dag_id="test_dag_run",
schedule=CronDataIntervalTimetable("0 0 1 * *", "Europe/Paris"),
start_date=timezone.parse("2026-07-05T00:00:00", "Europe/Paris"),
catchup=False,
):
task_1 = EmptyOperator(
task_id="Empty",
)
def final_status_callable():
context = get_current_context()
ti = context["ti"]
dag_run = context["dag_run"]
states_by_run = ti.get_task_states(
dag_id=ti.dag_id,
run_ids=[dag_run.run_id],
)
task_states = states_by_run.get(dag_run.run_id, {})
print(f"Status of all tasks for dag_run {dag_run.run_id!r}:")
failed_tasks = []
all_tasks = []
for task_id, state in sorted(task_states.items()):
if task_id == ti.task_id:
continue # this task itself is still "running"
print(f" {task_id:<30} {state}")
all_tasks.append(task_id)
if state in (TaskInstanceState.FAILED, TaskInstanceState.UPSTREAM_FAILED):
failed_tasks.append(task_id)
if failed_tasks:
raise RuntimeError(f"These tasks did not succeed: {failed_tasks}")
if len(all_tasks) == 0:
raise RuntimeError(f"Zero tasks run")
task_2 = final_status = PythonOperator(
task_id="final_status",
python_callable=final_status_callable,
trigger_rule=TriggerRule.ALL_DONE,
)
task_1 >> task_2
|
|
@raphaelauv |
|
what I would prefer is the option to mark skipped tasks that have out of date start_date or end_date and then if you need run a PythonOperator that check that all task where skipped |
|
You have a point that nothing actually failed there, and marking the tasks as skipped would certainly read better than today's empty grid. What tips it for me is an inconsistency that already exists between backfill and manual runs. On a Dag whose So the date window is not really a rule about whether these tasks may run at that logical date. It is a rule about what the scheduler creates, which is also how the docs describe Did you try your snippet with the |
|
Okay, it makes sense to make the behavior of 'backfill' and 'manual trigger' equivalent. In that case, could you add to the documentation that start_date and end_date on tasks are ignored in the event of a manual trigger? And also update the BaseOperator doc to state that these settings are only taken into account for scheduled runs, and not for manual or backfill runs. Yes my example do not works with tasks that are not run by the scheduler |
|
Done. Thanks for pushing on this, the semantics are better documented now than before the discussion. |
|
maybe also add the info here -> https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/dag-run.html#manual-triggering-and-data-intervals |
|
could you also rename your pr somrhing like
|
yuseok89
changed the title
|
Both done. Thanks for pushing on this, the PR ended up better than where it started. |
|
gentle ping @amoghrajesh , thanks |
potiuk
added
the
closed because of open PR limit
|
Hello @yuseok89 - thank you for your contributions to Apache Airflow! The Airflow community has introduced a limit of 5 open pull requests at a time for contributors without write access to the repository. You currently have 11 open pull requests, so - as a one-time step of introducing the limit - we closed the ones where maintainers have not engaged yet:
These pull requests stay open because maintainers are already engaged in them - they count towards your limit:
This is not a judgement of you or of your changes. We never told contributors before that opening many pull requests at once was a problem, so there is nothing to feel bad about - and nothing is lost: your branches, commits and the review history stay where they are. What we ask you to do is to make your first prioritization decision: choose which of the pull requests above matter most to you, and reopen them (up to 5 open at a time, including the ones still open) with the "Reopen pull request" button or While your pull requests are waiting for review, the most valuable thing you can do is help in other ways - reviewing other contributors' pull requests, helping with issues, and taking part in the discussions on the devlist and Slack. Why we introduced the limit, what it means for you and how to reopen or restore a pull request is explained in https://github.com/apache/airflow/blob/main/contributing-docs/32_open_pull_request_limit.rst. Drafted-by: Claude Code (Opus 5); reviewed by @potiuk before posting |
closes: #38513
A manual run whose logical date falls outside a task's
start_date/end_datewindow creates no task instance for that task. When every task is filtered out this way, the run ends up with no task instances at all and is immediately reported as success, with no error and no failed task to alert on. Manual runs are now exempt from that window, as backfill runs already are.The issue has been assigned since 2024 with no activity, happy to hand it back.
Verified in the UI
Using a Dag modelled on the one from the issue, with a
start_dateof 2050 and three chained@taskfunctions (extract→transform→load).Before
Screen.Recording.2026-08-18.at.11.19.52.PM.mov
After
Screen.Recording.2026-08-18.at.11.24.09.PM.mov
Was generative AI tooling used to co-author this PR?
{pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.