Repository navigation
Support Table objects in BigQueryHook.create_table table_resource - #73099
Conversation
|
Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide
|
shahar1
left a comment
There was a problem hiding this comment.
Welcome to Apache Airflow!
Before letting maintainers to review your PR:
- Please add AI attribution as part of the PR body according to the PR template.
- We don't use conventional commits - please remove
fix(google):from title.
aa554db to
3ab9f00
Compare
|
Thanks @shahar1 for the review. Both points addressed:
Ready for another look. |
|
Friendly ping — both review points were addressed on Sep 17 (generative-AI disclosure added to the PR description, conventional-commit prefix removed from the title). GitHub checks are green. Happy to adjust anything else — could someone take another look when you get a chance? |
|
Hi @X33834, Thank you for your PR! The change seems GDM. In the google provider side, we want to also see the system tests results after a change. If so, please share your results as well.
Thank you in advance! |
|
Thanks @molcay! The PR's CI run includes the BigQuery unit test suite and is green ( For the system tests under For what it's worth, the change is contained to input normalization in |
|
Thanks for the pointer, @molcay. I don't have access to a GCP project with the BigQuery API enabled, so I can't run the system tests from here. If someone on the team can trigger To make the change easy to reason about while those run:
Happy to adjust the implementation if the system tests surface anything. |
|
Hi @X33834, I am trying to run the system tests for bigquery on your branch, when I have a result I will let you know. |
|
Hi @molcay, thanks a lot for running the BigQuery system tests on my branch — really appreciate it! Quick note: the Heads-up: the CI workflows on the new commit are currently in |
|
Hi @X33834, It seems like your change did not break anything. Since your change is about handling different kind of table definitions for the Can you make this change as well to make the system test future-proofed? Also, please remove the |
…tem test Per review feedback in apache#73099: add an object-based create_table example alongside the existing dict-config one, so both supported input types of BigQueryHook.create_table stay covered by the system test.
Not needed for provider changes per review feedback in apache#73099.
|
Both pushed to the branch:
CI is running; happy to add a |
…_resource The `table_resource` parameter is typed and documented to accept a dict, `Table`, `TableReference`, or `TableListItem`, but the implementation only worked with a plain dict. `Table.from_api_repr()` was called backwards on a `Table` instance (TypeError) and raw `**`-unpacking was applied to `TableReference`/`TableListItem` objects (TypeError: not a mapping). Convert any object input to a plain dict via `.to_api_repr()`, wrapping a bare `TableReference` under `"tableReference"` since its serialization is flat, before merging `schema_fields`. Add unit tests covering dict, Table, TableReference, TableListItem inputs and schema merging. Closes: apache#73091
…tem test Per review feedback in apache#73099: add an object-based create_table example alongside the existing dict-config one, so both supported input types of BigQueryHook.create_table stay covered by the system test.
Not needed for provider changes per review feedback in apache#73099.
Generated-by: Claude Opus 5
372590f to
008ff6b
Compare
potiuk
left a comment
There was a problem hiding this comment.
Approving. Passing a Table used to crash (Table.from_api_repr(Table) and then **Table), and the new path handles both Table and a bare TableReference correctly. to_api_repr() returns a copy, so the caller's object isn't modified. I pushed a fixup that only moves the google.cloud.bigquery import into the third-party import block, since ruff's import-order check flagged it.
Drafted-by: Claude Code (Opus 5.5); reviewed by @potiuk before posting
|
Awesome work, congrats on your first merged pull request! You are invited to check our Issue Tracker for additional contributions. |
Problem
Closes: #73091
BigQueryHook.create_table()'stable_resourceparameter is typed and documented to accept adict,Table,TableReference, orTableListItem, but the implementation only worked with a plaindict:Tableinstance:Table.from_api_repr()was called backwards (it expects an API-JSON dict, not aTableobject) →TypeError: argument of type 'Table' is not iterableTableReference/TableListItem: raw**-unpacking on a non-mapping →TypeError: 'TableReference' object is not a mappingThis is reachable from
BigQueryCreateTableOperator, which has the identical type hint and forwardstable_resourcestraight into the hook. A DAG author who follows the documented type hint and passes aTableobtained fromget_table()/list_tables()hits a crash.Solution
Convert any object input to a plain dict via
.to_api_repr()before mergingschema_fieldsand building the finalTable. A bareTableReferenceis wrapped under"tableReference"since itsto_api_repr()is flat, unlikeTable/TableListItem.Tests
Added unit tests in
TestTableOperationscovering all four input forms (dict,Table,TableReference,TableListItem) plus schema merging:test_create_table_with_table_objecttest_create_table_with_table_reference_objecttest_create_table_with_table_list_itemtest_create_table_with_table_object_and_schema_fieldsRan locally: conversion logic verified against real
google.cloud.bigqueryobjects;ruff checkandpy_compileclean. Full provider test suite runs in CI via breeze.Was generative AI tooling used to co-author this PR?
Generated-by: DeepSeek via TraeCode AI assistant following the guidelines