Skip to content

Handle race condition in DatabricksReposCreateOperator when concurrently creating the same repo path - #62422

Merged
potiuk merged 1 commit into
apache:mainfrom
SameerMesiah97:DatabricksReposCreateOperator-Race-Fix
Mar 10, 2026
Merged

Handle race condition in DatabricksReposCreateOperator when concurrently creating the same repo path#62422
potiuk merged 1 commit into
apache:mainfrom
SameerMesiah97:DatabricksReposCreateOperator-Race-Fix

Conversation

@SameerMesiah97

Copy link
Copy Markdown
Contributor

Description

This change makes DatabricksReposCreateOperator resilient to a race condition when multiple tasks attempt to create a repository at the same repo_path concurrently.

Previously, the operator performed a get_repo_by_path check followed by create_repo. If two tasks ran at the same time, both could observe that the repository did not exist and both attempt creation. One task would succeed, while the other would fail with a 400 error from the Databricks API indicating that the repo already exists.

The operator now treats this as a recoverable condition. If create_repo fails because the repo already exists, the operator re-fetches the repo ID via get_repo_by_path. If the repo is found, execution proceeds normally and preserves the existing ignore_existing_repo semantics.

Rationale

The previous implementation relied on a non-atomic existence check followed by creation. In concurrent DAG runs, this leads to a classic time-of-check/time-of-use race condition. Two tasks can both pass the existence check and attempt creation, even though only one creation can succeed.

Since repository creation is an external side-effect managed by the Databricks API, the operator cannot assume exclusivity or single-writer behavior. It must defensively handle the possibility that another task or DAG run creates the resource between the check and the create call. Handling this explicitly makes the operator more robust under concurrency without changing its single-run behavior.

Tests

Add tests that verify that:

  • the operator recovers when create_repo raises an “already exists” error by re-fetching the repo ID and proceeding successfully.
  • a genuine creation failure (where the repo still cannot be found after the error) is propagated and not silently swallowed.

…gnore_existing_repo=True. If create_repo() fails, the operator now re-checks repository existence and proceeds if the repository was created concurrently; otherwise, the original exception is re-raised. Add unit tests covering recovery and failure propagation under concurrent create scenarios.
@SameerMesiah97
SameerMesiah97 force-pushed the DatabricksReposCreateOperator-Race-Fix branch from 14fbd2e to ca03239 Compare February 24, 2026 19:19
@SameerMesiah97

Copy link
Copy Markdown
Contributor Author

Requesting review for this.

@potiuk
potiuk merged commit 9217499 into apache:main Mar 10, 2026
90 checks passed
dominikhei pushed a commit to dominikhei/airflow that referenced this pull request Mar 11, 2026
…gnore_existing_repo=True. If create_repo() fails, the operator now re-checks repository existence and proceeds if the repository was created concurrently; otherwise, the original exception is re-raised. Add unit tests covering recovery and failure propagation under concurrent create scenarios. (apache#62422)

Co-authored-by: Sameer Mesiah <smesiah971@gmail.com>
Pyasma pushed a commit to Pyasma/airflow that referenced this pull request Mar 13, 2026
…gnore_existing_repo=True. If create_repo() fails, the operator now re-checks repository existence and proceeds if the repository was created concurrently; otherwise, the original exception is re-raised. Add unit tests covering recovery and failure propagation under concurrent create scenarios. (apache#62422)

Co-authored-by: Sameer Mesiah <smesiah971@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants