Repository navigation
Deferred AzureDataFactoryRunPipelineOperator + Airflow 3.0.3 failure #55728
Description
Apache Airflow Provider(s)
microsoft-azure
Versions of Apache Airflow Providers
apache-airflow-providers-microsoft-azure==12.6.1
Apache Airflow version
3.0.6
Operating System
Linux
Deployment
Astronomer
Deployment details
No response
What happened
Running the AzureDataFactoryRunPipelineOperator async + Airflow 3.0.3 gives this failure:
ERROR - Task failed with exception: source="task"
AirflowException: You cannot use AsyncToSync in the same thread as an async event loop - just await the async function directly.
File "/usr/local/lib/python3.12/site-packages/airflow/sdk/execution_time/task_runner.py", line 920 in run
File "/usr/local/lib/python3.12/site-packages/airflow/sdk/execution_time/task_runner.py", line 1215 in _execute_task
File "/usr/local/lib/python3.12/site-packages/airflow/sdk/bases/operator.py", line 1606 in resume_execution
File "/usr/local/lib/python3.12/site-packages/airflow/providers/microsoft/azure/operators/data_factory.py", line 256 in execute_complete
The operator runs fine on Airflow 2.
What you think should happen instead
The operator should not fail
How to reproduce
Run AzureDataFactoryRunPipelineOperator async + Airflow 3.0.3 or higher
Anything else
Same error also found in other issues/PRs:
- MSGraphAsyncOperator fails to call API #54350
- RuntimeError in trigger creating hooks, Airflow 3.0.3 #53447
- Fix sync to async issue when getting connection for SQLExecuteQueryTrigger #55094
- Avoid using extra_dejson method on connection in KiotaRequestAdapterHook to avoid AsyncToSync RuntimeError #55179
- Made get_conn method async for KiotaRequestAdapterHook to fix AsyncToSync error #54598
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct
Activity
ashb commentedon Sep 17, 2025
L256 seems to be a
raise AirflowException(event["message"])-- is that really the line it's blowing up on and the end of the stack trace?Ohh. that is the error being propogated from the trigger, so it's got the wrong stack trace.
ashb commentedon Sep 17, 2025
@BasPH Is there anything further up in the task logs from the trigger itself that might actually tell us the correct location?
Vamsi-klu commentedon Aug 27, 2026
Avoid extra_dejson in ADF and Synapse async hooks (#72130)
Avoid extra_dejson in ADF and Synapse async hooks (apache#72130)