Skip to content

fix(providers/google): sanitize Dataproc batch labels to use dashes instead of underscores - #62926

Merged
potiuk merged 3 commits into
apache:mainfrom
YoannAbriel:fix/issue-59332
Mar 12, 2026
Merged

fix(providers/google): sanitize Dataproc batch labels to use dashes instead of underscores#62926
potiuk merged 3 commits into
apache:mainfrom
YoannAbriel:fix/issue-59332

Conversation

@YoannAbriel

Copy link
Copy Markdown
Contributor

Problem

DataprocCreateBatchOperator.__update_batch_labels() replaces dots and spaces with underscores when building GCP labels from DAG/task IDs. GCP labels only allow lowercase letters, numbers, and dashes — underscores are invalid. This causes InvalidArgument errors when task IDs contain underscores (common with task groups, e.g. process_data).

Root Cause

re.sub(r"[.\s]", "_", ...) only strips dots and whitespace, converting them to underscores. Any existing underscores in the ID pass through unchanged. The validation regex r"^[a-z][\w-]{0,62}$" also incorrectly allows underscores via \w.

Additionally, the dag_display_name validation was checking dag_id instead of the actual dag_display_name value.

Fix

  • Replace the substitution pattern with re.sub(r"[^a-z0-9-]", "-", ...) — strips everything except valid label characters
  • Update the validation regex to r"^[a-z0-9][a-z0-9-]{0,62}$" matching GCP's actual requirements
  • Fix the dag_display_name validation to check the right variable
  • Added test for underscore/dot sanitization

Closes: #59332


Was generative AI tooling used to co-author this PR?
  • Yes — Claude Code

Generated-by: Claude Code following the guidelines


  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.

@YoannAbriel
YoannAbriel requested a review from shahar1 as a code owner March 5, 2026 10:09
@boring-cyborg boring-cyborg Bot added area:providers provider:google Google (including GCP) related issues labels Mar 5, 2026
@YoannAbriel
YoannAbriel force-pushed the fix/issue-59332 branch 6 times, most recently from 9cd8af9 to 05571b3 Compare March 5, 2026 20:03
…nstead of underscores

GCP labels only allow lowercase letters, numbers, and dashes.
`__update_batch_labels` was replacing dots/spaces with underscores,
causing InvalidArgument errors when DAG/task IDs contained underscores
(e.g. task groups with dots in their path).

Also fixes the validation regex to match GCP's actual label requirements
and corrects the dag_display_name validation check which was incorrectly
testing dag_id instead.

Closes: apache#59332
@potiuk
potiuk merged commit 9ce4b8e into apache:main Mar 12, 2026
92 checks passed
PascalEgn pushed a commit to PascalEgn/airflow that referenced this pull request Mar 12, 2026
…nstead of underscores (apache#62926)

* fix(providers/google): sanitize Dataproc batch labels to use dashes instead of underscores

GCP labels only allow lowercase letters, numbers, and dashes.
`__update_batch_labels` was replacing dots/spaces with underscores,
causing InvalidArgument errors when DAG/task IDs contained underscores
(e.g. task groups with dots in their path).

Also fixes the validation regex to match GCP's actual label requirements
and corrects the dag_display_name validation check which was incorrectly
testing dag_id instead.

Closes: apache#59332

* fix: use valid dag_id in test (no spaces allowed)

* fix: apply ruff format
@YoannAbriel
YoannAbriel deleted the fix/issue-59332 branch March 12, 2026 10:05
Pyasma pushed a commit to Pyasma/airflow that referenced this pull request Mar 13, 2026
…nstead of underscores (apache#62926)

* fix(providers/google): sanitize Dataproc batch labels to use dashes instead of underscores

GCP labels only allow lowercase letters, numbers, and dashes.
`__update_batch_labels` was replacing dots/spaces with underscores,
causing InvalidArgument errors when DAG/task IDs contained underscores
(e.g. task groups with dots in their path).

Also fixes the validation regex to match GCP's actual label requirements
and corrects the dag_display_name validation check which was incorrectly
testing dag_id instead.

Closes: apache#59332

* fix: use valid dag_id in test (no spaces allowed)

* fix: apply ruff format
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:providers provider:google Google (including GCP) related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

DataprocCreateBatchOperator fails with InvalidArgument when DAG/task IDs contain dots or spaces

2 participants