Skip to content

Check GCSToAzureBlobStorageOperator match_glob support after template… - #70574

Merged
potiuk merged 3 commits into
apache:mainfrom
bramhanandlingala:fix/70296-6
Aug 13, 2026
Merged

Check GCSToAzureBlobStorageOperator match_glob support after template…#70574
potiuk merged 3 commits into
apache:mainfrom
bramhanandlingala:fix/70296-6

Conversation

@bramhanandlingala

@bramhanandlingala bramhanandlingala commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

Fixes the microsoft/azure provider's GCSToAzureBlobStorageOperator entry from the #70296 exemption-list burn-down.

match_glob is a template field, but __init__ validated it against the installed apache-airflow-providers-google version and raised immediately if unsupported — so this check ran on the un-rendered Jinja expression instead of the actual rendered value.

Moved the compatibility check from __init__ into execute(), right before match_glob is used to build the GCS list call. __init__ now only does a plain assignment. _is_match_glob_supported itself stays computed in __init__, since it depends on the installed provider version, not on any template field.

  • Removed the raise from __init__; self.match_glob = match_glob is now a plain assignment.
  • Added the same check in execute(), guarded by _is_match_glob_supported.
  • Updated the existing test to construct the operator successfully and assert the error on .execute() instead of construction.
  • Removed the operator's entry from validate_operators_init_exemptions.txt.

Verified locally that scripts/ci/prek/validate_operators_init.py reports zero findings for this class after the change, and that the existing unit tests in test_gcs_to_wasb.py pass with the updated test.

Related to #70296


Gen-AI disclosure: I used a generative AI tool to help identify the root
cause, write tests, and draft the PR description. I reviewed, tested, and
verified all changes locally before submitting.

Was generative AI tooling used to co-author this PR?
  • Yes - Claude

Generated-by: Claude following the guidelines

@potiuk potiuk left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks — same fix as #70723 does for the Amazon GCSToS3Operator, and correct for the same reason: match_glob is in template_fields, so checking it in __init__ validated the un-rendered Jinja string and turned a bad value into a Dag parse failure rather than a task failure. Keeping the _is_match_glob_supported probe in __init__ is right, since that depends on the installed Google provider rather than anything templated.

Using ValueError here is the better choice — I've suggested on #70723 that it match yours rather than the other way round, since Airflow is trying to reduce direct AirflowException raises. Worth the two of you syncing so the sibling changes land consistent.

One test nit inline, not blocking.


Drafted-by: Claude Code (Opus 5); reviewed by @potiuk before posting

@potiuk potiuk left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Retracting my earlier approval — my mistake, and I'm sorry for the churn.

I approved this before reading #70296, which sets out the rule for this burn-down, and by that rule this change moves the wrong kind of check.

A check that only asks whether an argument was passed … belongs in __init__ and must not be moved. Fix these by rewriting in place, not by moving. Use the is not None polarity.

The reasoning in the issue is sound and I'd missed both halves of it:

  • With render_template_as_native_obj=True, a field that was provided can render to None, so the same check in execute() reports a supplied argument as missing.
  • Raising in the constructor surfaces a static authoring mistake as a Dag import error, rather than once per task instance and per retry on a worker.

if not self._is_match_glob_supported and match_glob: asks whether match_glob was passed (combined with an environment capability that __init__ can already answer). It's a provision check, so it should be rewritten in place rather than relocated:

# in __init__ — keep it here
if not self._is_match_glob_supported and match_glob is not None:
    raise ValueError(
        "The 'match_glob' parameter requires 'apache-airflow-providers-google>=10.3.0'."
    )

Note the is not None polarity matters: if match_glob: is a truthiness test on the un-rendered Jinja string, which is a third question that matches neither intent. You already use ValueError here, which is right.

This is the twin of #70723 — worth the two of you landing the same shape.

Good news on mechanics: #70505 (which narrows the hook to allow provision checks written with is not None) merged on 28 July, so the rewrite below passes validate-operators-init and you can still remove the exemption-file entry in this PR — the burn-down goal is unaffected.

There's an active follow-up, #70503, cataloguing already-merged PRs that made exactly this move so they can be put back. Fixing it here saves this PR from joining that list.


Drafted-by: Claude Code (Opus 5); reviewed by @potiuk before posting

@bramhanandlingala

Copy link
Copy Markdown
Contributor Author

@potiuk Thanks for catching this — restored the check to init using
match_glob is not None, removed the duplicate from execute(), and
updated the test to assert the raise on construction instead of execute().

@potiuk
potiuk merged commit cc279cd into apache:main Aug 13, 2026
83 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

Backport failed to create: v3-3-test. View the failure log Run details

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status Branch Result
v3-3-test Commit Link

You can attempt to backport this manually by running:

cherry_picker cc279cd v3-3-test

This should apply the commit to the v3-3-test branch and leave the commit in conflict state marking
the files that need manual conflict resolution.

After you have resolved the conflicts, you can continue the backport process by running:

cherry_picker --continue

If you don't have cherry-picker installed, see the installation guide.

dabla pushed a commit to dabla/airflow that referenced this pull request Aug 14, 2026
apache#70574)

* Check GCSToAzureBlobStorageOperator match_glob support after template rendering

* Restore match_glob provision check in __init__ per apache#70296 rules
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:dev-tools area:providers backport-to-v3-3-test Backport to v3-3-test provider:microsoft-azure Azure-related issues ready for maintainer review Set after triaging when all criteria pass.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants