Repository navigation
Add a check in BigQueryInsertJobOperator to verify the Job state before marking it as success in Airflow #40839
Description
Apache Airflow Provider(s)
Versions of Apache Airflow Providers
apache-airflow-providers-google==10.12.0
Apache Airflow version
2.7.3
Operating System
debian
Deployment
Astronomer
Deployment details
No response
What happened
When the BigQueryInsertJobOperator submits a job to BigQuery, there are instances where the TaskInstance in Airflow is marked as success rarely before the Job actually completes in BigQuery.
When the execute method of a TaskInstance’s Operator completes without raising an exception, the TaskInstance is marked as success. Since these TaskInstances are being marked as success, it implies that the BigQueryInsertJobOperator’s execute method is completing without any exceptions. When QueryJob.result returns something, allowing the execute method to finish.
What you think should happen instead
Need to add an additional check here to verify if the Job state is DONE in BigQuery, ensuring that Airflow only marks the TaskInstance as success when the job is indeed complete in BigQuery.
How to reproduce
I haven't been able to replicate the issue. It happens infrequently.
Anything else
There might be a bug in google-cloud-bigquery==3.13.0 that causes the QueryJob.result method to return before the job is actually DONE.
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct
Activity
boring-cyborg commentedon Jul 17, 2024
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
RNHTTR commentedon Jul 17, 2024
Worth noting that even though this is on an older version of the provider, there's no check in main to see whether the job completed in BigQuery. I think this is probably a bug in
google-cloud-bigquery==3.13.0, but it still might make sense to check whether the job is complete to avoid this situation completely.shahar1 commentedon Nov 26, 2024
@kandharvishnu @pankajastro
I'd like to have 3 requests:
* @eladkal I agree with your statement that we shouldn't work with permanent workarounds - however, would it suffice for now to implement one of the PRs while adding a link to an appropriate issue in Bigquery?
pankajastro commentedon Nov 28, 2024
Hey @shahar1, Thank you for the review feedback I appreciate it. I have added my thoughts and also proposed some modified solutions which might help us in detecting future errors. Please do let me your thought. thank you! #44279 (comment)
eladkal commentedon Dec 1, 2024
Closed by #44279