Handle race condition in DatabricksReposCreateOperator when concurrently creating the same repo path - #62422
Merged
potiuk merged 1 commit intoMar 10, 2026
Conversation
…gnore_existing_repo=True. If create_repo() fails, the operator now re-checks repository existence and proceeds if the repository was created concurrently; otherwise, the original exception is re-raised. Add unit tests covering recovery and failure propagation under concurrent create scenarios.
SameerMesiah97
force-pushed
the
DatabricksReposCreateOperator-Race-Fix
branch
from
February 24, 2026 19:19
14fbd2e to
ca03239
Compare
Contributor
Author
|
Requesting review for this. |
potiuk
approved these changes
Mar 10, 2026
dominikhei
pushed a commit
to dominikhei/airflow
that referenced
this pull request
Mar 11, 2026
…gnore_existing_repo=True. If create_repo() fails, the operator now re-checks repository existence and proceeds if the repository was created concurrently; otherwise, the original exception is re-raised. Add unit tests covering recovery and failure propagation under concurrent create scenarios. (apache#62422) Co-authored-by: Sameer Mesiah <smesiah971@gmail.com>
Pyasma
pushed a commit
to Pyasma/airflow
that referenced
this pull request
Mar 13, 2026
…gnore_existing_repo=True. If create_repo() fails, the operator now re-checks repository existence and proceeds if the repository was created concurrently; otherwise, the original exception is re-raised. Add unit tests covering recovery and failure propagation under concurrent create scenarios. (apache#62422) Co-authored-by: Sameer Mesiah <smesiah971@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This change makes
DatabricksReposCreateOperatorresilient to a race condition when multiple tasks attempt to create a repository at the samerepo_pathconcurrently.Previously, the operator performed a
get_repo_by_pathcheck followed bycreate_repo. If two tasks ran at the same time, both could observe that the repository did not exist and both attempt creation. One task would succeed, while the other would fail with a 400 error from the Databricks API indicating that the repo already exists.The operator now treats this as a recoverable condition. If
create_repofails because the repo already exists, the operator re-fetches the repo ID viaget_repo_by_path. If the repo is found, execution proceeds normally and preserves the existingignore_existing_reposemantics.Rationale
The previous implementation relied on a non-atomic existence check followed by creation. In concurrent DAG runs, this leads to a classic time-of-check/time-of-use race condition. Two tasks can both pass the existence check and attempt creation, even though only one creation can succeed.
Since repository creation is an external side-effect managed by the Databricks API, the operator cannot assume exclusivity or single-writer behavior. It must defensively handle the possibility that another task or DAG run creates the resource between the check and the create call. Handling this explicitly makes the operator more robust under concurrency without changing its single-run behavior.
Tests
Add tests that verify that:
create_reporaises an “already exists” error by re-fetching the repo ID and proceeding successfully.