Repository navigation
Deprecate Dataproc ClusterGenerator helper class - #71427
Conversation
e6723d6 to
ad6173f
Compare
ad6173f to
fcce6dd
Compare
There was a problem hiding this comment.
AI Review
Deprecating ClusterGenerator makes sense. One blocking issue: the migrated system test config is missing a property that ClusterGenerator used to add, so cluster creation will fail (see the inline comment on example_dataproc_batch_persistent.py). The other inline comments are smaller consistency fixes: the removal dates don't match, the operator warning still points at ClusterGenerator, the reason text names a class that doesn't exist, the enum deprecation never reaches users, and there is no test for the new warnings.
Smaller observations
- Nit: without
use_instead, the renderedClusterGeneratorwarning reads "There is no replacement." and then gives replacement instructions. The helper-class warnings also say there is no replacement. A shortuse_insteadwould read better. - Nit: this PR moves the keyword-arguments path's removal date from October 5, 2026 to September 1, 2027. That's reasonable, since the old date is days away, but please mention it in the PR description so it's visible in the changelog.
- Nit: deleting
example_dataproc_cluster_generator.pyremoves system-test coverage for a class that stays supported until September 2027. That's fine if intended, but worth confirming. - CI: six checks on the head commit were cancelled, including
MyPy providers checks. A rebase will give a fresh run.
This review was drafted by an AI-assisted tool and
confirmed by an Apache Airflow maintainer. After you've
addressed the points above and pushed an update, an Apache Airflow
maintainer — a real person — will take the next look
at the PR. The findings cite the project's review criteria;
if you think one of them is mis-applied, please reply on the
PR and a maintainer will weigh in.More on how Apache Airflow handles maintainer review:
contributing-docs/05_pull_requests.rst.
0639e22 to
95ace6c
Compare
- Becomes obsolete following the deprecation of keyword arguments in DataprocCreateClusterOperator. - Direct configuration using 'cluster_config'/'virtual_cluster_config' is preferred in modern workflows. - Reduces maintainer overhead by removing legacy helper code.
95ace6c to
5201c28
Compare
aaron-y-chen
left a comment
There was a problem hiding this comment.
Thanks, Overall LGTM :)
Was generative AI tooling used to co-author this PR?
{pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.