Skip to content

Deprecate Dataproc ClusterGenerator helper class - #71427

Merged
shahar1 merged 1 commit into
apache:mainfrom
VladaZakharova:deprecate_dataproc_create_cluster
Oct 3, 2026
Merged

shahar1 merged 1 commit into
apache:mainfrom
VladaZakharova:deprecate_dataproc_create_cluster

Conversation

@olegkachur-e

Copy link
Copy Markdown
Contributor
  • Becomes obsolete following the deprecation of keyword arguments in DataprocCreateClusterOperator.
  • Direct configuration using 'cluster_config'/'virtual_cluster_config' is preferred in modern workflows.
  • Reduces maintainer overhead by removing legacy helper code.

Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

@olegkachur-e
olegkachur-e requested a review from shahar1 as a code owner August 11, 2026 11:20
@boring-cyborg boring-cyborg Bot added area:providers provider:google Google (including GCP) related issues labels Aug 11, 2026
@olegkachur-e
olegkachur-e force-pushed the deprecate_dataproc_create_cluster branch 3 times, most recently from e6723d6 to ad6173f Compare August 12, 2026 12:04
@olegkachur-e
olegkachur-e marked this pull request as draft August 18, 2026 09:43
@olegkachur-e
olegkachur-e force-pushed the deprecate_dataproc_create_cluster branch from ad6173f to fcce6dd Compare August 28, 2026 12:54
@olegkachur-e
olegkachur-e marked this pull request as ready for review August 28, 2026 12:56

@shahar1 shahar1 left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI Review

Deprecating ClusterGenerator makes sense. One blocking issue: the migrated system test config is missing a property that ClusterGenerator used to add, so cluster creation will fail (see the inline comment on example_dataproc_batch_persistent.py). The other inline comments are smaller consistency fixes: the removal dates don't match, the operator warning still points at ClusterGenerator, the reason text names a class that doesn't exist, the enum deprecation never reaches users, and there is no test for the new warnings.

Smaller observations

  • Nit: without use_instead, the rendered ClusterGenerator warning reads "There is no replacement." and then gives replacement instructions. The helper-class warnings also say there is no replacement. A short use_instead would read better.
  • Nit: this PR moves the keyword-arguments path's removal date from October 5, 2026 to September 1, 2027. That's reasonable, since the old date is days away, but please mention it in the PR description so it's visible in the changelog.
  • Nit: deleting example_dataproc_cluster_generator.py removes system-test coverage for a class that stays supported until September 2027. That's fine if intended, but worth confirming.
  • CI: six checks on the head commit were cancelled, including MyPy providers checks. A rebase will give a fresh run.

This review was drafted by an AI-assisted tool and
confirmed by an Apache Airflow maintainer. After you've
addressed the points above and pushed an update, an Apache Airflow
maintainer — a real person — will take the next look
at the PR. The findings cite the project's review criteria;
if you think one of them is mis-applied, please reply on the
PR and a maintainer will weigh in.

More on how Apache Airflow handles maintainer review:
contributing-docs/05_pull_requests.rst.

Comment thread providers/google/docs/operators/cloud/dataproc.rst Outdated
Comment thread providers/google/src/airflow/providers/google/cloud/operators/dataproc.py Outdated
Comment thread providers/google/src/airflow/providers/google/cloud/operators/dataproc.py Outdated
Comment thread providers/google/tests/deprecations_ignore.yml Outdated
@olegkachur-e
olegkachur-e force-pushed the deprecate_dataproc_create_cluster branch 4 times, most recently from 0639e22 to 95ace6c Compare September 28, 2026 16:43
- Becomes obsolete following the deprecation of keyword arguments in DataprocCreateClusterOperator.
- Direct configuration using 'cluster_config'/'virtual_cluster_config' is preferred in modern workflows.
- Reduces maintainer overhead by removing legacy helper code.
@olegkachur-e
olegkachur-e force-pushed the deprecate_dataproc_create_cluster branch from 95ace6c to 5201c28 Compare September 29, 2026 15:17
@olegkachur-e
olegkachur-e requested a review from shahar1 October 1, 2026 10:41

@aaron-y-chen aaron-y-chen left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, Overall LGTM :)

@shahar1
shahar1 merged commit 232c264 into apache:main Oct 3, 2026
86 checks passed
@olegkachur-e
olegkachur-e deleted the deprecate_dataproc_create_cluster branch October 19, 2026 13:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:providers provider:google Google (including GCP) related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants