Skip to content

Support Table objects in BigQueryHook.create_table table_resource - #73099

Merged
potiuk merged 6 commits into
apache:mainfrom
X33834:fix-bigquery-create-table
Oct 5, 2026
Merged

potiuk merged 6 commits into
apache:mainfrom
X33834:fix-bigquery-create-table

Conversation

@X33834

@X33834 X33834 commented Sep 13, 2026 •

Copy link
Copy Markdown
Contributor

Problem

Closes: #73091

BigQueryHook.create_table()'s table_resource parameter is typed and documented to accept a dict, Table, TableReference, or TableListItem, but the implementation only worked with a plain dict:

  • Table instance: Table.from_api_repr() was called backwards (it expects an API-JSON dict, not a Table object) → TypeError: argument of type 'Table' is not iterable
  • TableReference / TableListItem: raw **-unpacking on a non-mapping → TypeError: 'TableReference' object is not a mapping

This is reachable from BigQueryCreateTableOperator, which has the identical type hint and forwards table_resource straight into the hook. A DAG author who follows the documented type hint and passes a Table obtained from get_table() / list_tables() hits a crash.

Solution

Convert any object input to a plain dict via .to_api_repr() before merging schema_fields and building the final Table. A bare TableReference is wrapped under "tableReference" since its to_api_repr() is flat, unlike Table / TableListItem.

Tests

Added unit tests in TestTableOperations covering all four input forms (dict, Table, TableReference, TableListItem) plus schema merging:

  • test_create_table_with_table_object
  • test_create_table_with_table_reference_object
  • test_create_table_with_table_list_item
  • test_create_table_with_table_object_and_schema_fields

Ran locally: conversion logic verified against real google.cloud.bigquery objects; ruff check and py_compile clean. Full provider test suite runs in CI via breeze.

Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

Generated-by: DeepSeek via TraeCode AI assistant following the guidelines


@X33834
X33834 requested a review from shahar1 as a code owner September 13, 2026 17:22
@boring-cyborg boring-cyborg Bot added area:providers provider:google Google (including GCP) related issues labels Sep 13, 2026
@boring-cyborg

boring-cyborg Bot commented Sep 13, 2026

Copy link
Copy Markdown

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide
Here are some useful points:

  • Pay attention to the quality of your code (ruff, mypy and type annotations). Our prek-hooks will help you with that.
  • In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example Dag that shows how users should use it.
  • Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
  • Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
  • Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
  • Be sure to read the Airflow Coding style.
  • Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
    Apache Airflow is a community-driven project and together we are making it better 🚀.
    In case of doubts contact the developers at:
    Mailing List: dev@airflow.apache.org
    Slack: https://s.apache.org/airflow-slack

@shahar1 shahar1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Welcome to Apache Airflow!
Before letting maintainers to review your PR:

  1. Please add AI attribution as part of the PR body according to the PR template.
  2. We don't use conventional commits - please remove fix(google): from title.

@X33834
X33834 force-pushed the fix-bigquery-create-table branch from aa554db to 3ab9f00 Compare September 15, 2026 15:23
@X33834 X33834 changed the title fix(google): Support Table objects in BigQueryHook.create_table table_resource Support Table objects in BigQueryHook.create_table table_resource Sep 17, 2026
@X33834

X33834 commented Sep 17, 2026

Copy link
Copy Markdown
Contributor Author

Thanks @shahar1 for the review. Both points addressed:

  1. Added the generative-AI disclosure section to the PR description per the PR template.
  2. Removed the conventional-commit prefix from the title.

Ready for another look.

@shahar1
shahar1 dismissed their stale review September 17, 2026 17:28

Concerns addressed

@X33834

X33834 commented Sep 29, 2026

Copy link
Copy Markdown
Contributor Author

Friendly ping — both review points were addressed on Sep 17 (generative-AI disclosure added to the PR description, conventional-commit prefix removed from the title). GitHub checks are green.

Happy to adjust anything else — could someone take another look when you get a chance?

@molcay

molcay commented Sep 29, 2026

Copy link
Copy Markdown
Contributor

Hi @X33834,

Thank you for your PR! The change seems GDM.

In the google provider side, we want to also see the system tests results after a change.
Did you also run the system tests which are related to the BigQuery service?

If so, please share your results as well.

Since you change the BigQueryHook, it is good to run all BigQuery related system tests

Thank you in advance!

@X33834

X33834 commented Sep 29, 2026

Copy link
Copy Markdown
Contributor Author

Thanks @molcay!

The PR's CI run includes the BigQuery unit test suite and is green (providers/google/tests/unit/google/cloud/hooks/test_bigquery.py, including the new cases for Table/TableReference/TableListItem inputs).

For the system tests under providers/google/tests/system/google/cloud/bigquery/: I don't have access to a GCP project with the BigQuery API enabled, so I couldn't run them myself. Happy to run them if credentials can be provided, or feel free to trigger them on your side if that's easier.

For what it's worth, the change is contained to input normalization in create_table: it accepts Table/TableReference/TableListItem objects and converts them via to_api_repr() (previously a Table object was passed through Table.from_api_repr, which raised). The dict path is unchanged, so existing system tests that pass dicts exercise the same code path as before.

@X33834

X33834 commented Oct 2, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for the pointer, @molcay. I don't have access to a GCP project with the BigQuery API enabled, so I can't run the system tests from here. If someone on the team can trigger providers/google/tests/system/google/cloud/bigquery/ on this branch, I'd be glad to look at any failures.

To make the change easy to reason about while those run:

  • create_table now normalizes Table / TableReference / TableListItem inputs through to_api_repr() before calling the API; the dict path is unchanged, so existing system tests that pass dicts exercise the same code path as before.
  • The BigQuery unit suite in this PR's CI is green, including the new cases covering each accepted input type.

Happy to adjust the implementation if the system tests surface anything.

@molcay

molcay commented Oct 2, 2026

Copy link
Copy Markdown
Contributor

Hi @X33834,

I am trying to run the system tests for bigquery on your branch, when I have a result I will let you know.

@X33834

X33834 commented Oct 2, 2026

Copy link
Copy Markdown
Contributor Author

Hi @molcay, thanks a lot for running the BigQuery system tests on my branch — really appreciate it!

Quick note: the Static checks failure comes from the breeze doc-image regeneration hooks (dev/breeze/doc/images/output_workflow-run*), unrelated to the actual PR changes. I've updated the branch with the latest main to clear it.

Heads-up: the CI workflows on the new commit are currently in action_required state — would you mind re-approving/triggering them when you get a chance? Looking forward to the system test results. 🙏

X33834 added a commit to X33834/airflow that referenced this pull request Oct 2, 2026
@molcay

molcay commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

Hi @X33834,

It seems like your change did not break anything.

Since your change is about handling different kind of table definitions for the BigQueryCreateTableOperator; we must also add more tasks to the bigquery_tables system test.
As you can (see)[https://github.com/X33834/airflow/blob/fix-bigquery-create-table/providers/google/tests/system/google/cloud/bigquery/example_bigquery_tables.py#L85-L97], we already have one create_table task with dict config for table_resource argument. We must add the other type example in this DAG.

Can you make this change as well to make the system test future-proofed?

Also, please remove the newsfragment file.

Comment thread newsfragments/73099.feature.rst Outdated
X33834 added a commit to X33834/airflow that referenced this pull request Oct 3, 2026
…tem test

Per review feedback in apache#73099: add an object-based create_table example
alongside the existing dict-config one, so both supported input types
of BigQueryHook.create_table stay covered by the system test.
X33834 added a commit to X33834/airflow that referenced this pull request Oct 3, 2026
Not needed for provider changes per review feedback in apache#73099.
@X33834

X33834 commented Oct 3, 2026

Copy link
Copy Markdown
Contributor Author

Both pushed to the branch:

  • Added a create_table_from_table_object task to example_bigquery_tables.py passing a google.cloud.bigquery.Table instance (same schema as the dict example), wired into the chain right after create_table, so both supported input types of BigQueryHook.create_table stay covered.
  • Removed the newsfragment.

CI is running; happy to add a TableListItem example too if you think it's worth covering.

X33834 and others added 6 commits October 5, 2026 18:32
…_resource

The `table_resource` parameter is typed and documented to accept a dict,
`Table`, `TableReference`, or `TableListItem`, but the implementation only
worked with a plain dict. `Table.from_api_repr()` was called backwards on a
`Table` instance (TypeError) and raw `**`-unpacking was applied to
`TableReference`/`TableListItem` objects (TypeError: not a mapping).

Convert any object input to a plain dict via `.to_api_repr()`, wrapping a
bare `TableReference` under `"tableReference"` since its serialization is
flat, before merging `schema_fields`.

Add unit tests covering dict, Table, TableReference, TableListItem inputs
and schema merging.

Closes: apache#73091
…tem test

Per review feedback in apache#73099: add an object-based create_table example
alongside the existing dict-config one, so both supported input types
of BigQueryHook.create_table stay covered by the system test.
Not needed for provider changes per review feedback in apache#73099.
@potiuk
potiuk force-pushed the fix-bigquery-create-table branch from 372590f to 008ff6b Compare October 5, 2026 16:32

@potiuk potiuk left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving. Passing a Table used to crash (Table.from_api_repr(Table) and then **Table), and the new path handles both Table and a bare TableReference correctly. to_api_repr() returns a copy, so the caller's object isn't modified. I pushed a fixup that only moves the google.cloud.bigquery import into the third-party import block, since ruff's import-order check flagged it.


Drafted-by: Claude Code (Opus 5.5); reviewed by @potiuk before posting

@potiuk
potiuk merged commit fbdec95 into apache:main Oct 5, 2026
87 checks passed
@boring-cyborg

boring-cyborg Bot commented Oct 5, 2026

Copy link
Copy Markdown

Awesome work, congrats on your first merged pull request! You are invited to check our Issue Tracker for additional contributions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:providers provider:google Google (including GCP) related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

BigQueryHook.create_table() crashes when table_resource is a Table, TableReference, or TableListItem instead of a dict

4 participants