Upstream Pydantic 2.12.0 has a Regression that will break PyIceberg

Apache Iceberg version

None

Please describe the bug 🐞

Pydantic recently 2.12.0 and unfortunately it contains a regression: pydantic/pydantic#12347 that ValidationInfo object will lose field_names in some cases.

I have a draft PR that experiments 2.12.0 and confirmed that the regression affect us: #2589


table_metadata = ValidationInfo(config={'title': 'TableMetadataV2', 'validate_by_alias': True, 'validate_by_name': True}, context=None, data=None, field_name=None)

    def check_schemas(table_metadata: TableMetadata) -> TableMetadata:
        """Check if the current-schema-id is actually present in schemas."""
>       current_schema_id = table_metadata.current_schema_id
E       AttributeError: 'pydantic_core._pydantic_core.ValidationInfo' object has no attribute 'current_schema_id'

More info:
pydantic 2.11.10

pydantic 2.12.0

While waiting for upstream fix, I think we may want to pin the pydantic version to <2.12.0

Willingness to contribute

I can contribute a fix for this bug independently
I would be willing to contribute a fix for this bug with guidance from the Iceberg community
I cannot contribute a fix for this bug at this time

reopened the issue. lets close when there's a resolution to the underlying issue :)

[Upstream Error] Pin pydantic version <2.12.0 to avoid regression (ap…

Is there any workaround to make pyiceberg work? I tried reinstalling pydantic to 2.11.0 and pyiceberg to 0.9.1 and still got the same error

pydantic_core._pydantic_core.ValidationInfo' object has no attribute 'current_schema_id

Is there any workaround to make pyiceberg work? I tried reinstalling pydantic to 2.11.0 and pyiceberg to 0.9.1 and still got the same error

pydantic_core._pydantic_core.ValidationInfo' object has no attribute 'current_schema_id

Hi @FranMorilloAWS,

After some investigation, it looks like both pyiceberg 0.9.x and pyiceberg 0.10.x don’t currently exclude pydantic 2.12.0 in their pyproject.toml file:
Reference line in pyproject.toml

However, the main branch already includes an explicit exclusion for this version, which seems to address the problem.

For now, the only stable workaround I’ve found is to:

Downgrade pyiceberg to 0.9.1
Pin pydantic to <2.10.0

I haven’t yet had the chance to test with pydantic 2.10.0 or 2.11.0, so they might still work — but I can only confirm that <2.10.0 has fixed the issue on my end.

This combination allows my data connectors to run without triggering the validation error.

@nickjoanis

Hi @FranMorilloAWS ,

The other CI pipline I had that initially discovered this issue now works with pyiceberg==0.10.0 and pydantic==2.11.10. Could you please check if the 2.11.10 or <2.10.0 as suggested by @nickjoanis works? If it still not work, could you please share the example that you observed this error? Thanks!

`setup.py`: pin `pydantic` version to `2.11.0`

+1

ci: Pin pydantic version in `provision.py` for iceberg tests (#5352)

`setup.py`: pin pydantic version to `2.11.0`

I can confirm pyiceberg==0.9.1 and pydantic==2.11.10 works

@arthurql

I can confirm pyiceberg==0.9.1 and pydantic==2.11.10 works

I can second @arthurql 's findings.

It doesn't necessarily have to be pyiceberg==0.9.1. Any pyiceberg version with pydantic < 2.12 will work.

Here's a simple script to check:

uv run --with 'pyiceberg[pyarrow,sql-sqlite]' --with 'pydantic<2.12' python - <<'PY'
import pyiceberg, pydantic, pyarrow as pa
print(f"pyiceberg.__version__ = {pyiceberg.__version__}")
print(f"pydantic.__version__ = {pydantic.__version__}")

from pyiceberg.catalog import load_catalog

catalog = load_catalog("catalog", type="in-memory")
catalog.create_namespace_if_not_exists("default")

data = pa.table({"x": [1, 2, 3], "y": [4, 5, 6]})
iceberg_table = catalog.create_table("default.test", schema=data.schema)
iceberg_table.append(data)

print("Table created and data appended successfully.")
print(len(iceberg_table.scan().to_arrow()))
PY

outputs:

pyiceberg.__version__ = 0.10.0
pydantic.__version__ = 2.11.10
Table created and data appended successfully.
3

removing the --with 'pydantic<2.12' will default back to the latest pydantic version, which is 2.12, and will fail the script

Thanks everyone for chiming in here! 🙌 It looks like Pydantic is preparing a patch release: pydantic/pydantic#12347

It doesn't necessarily have to be pyiceberg==0.9.1. Any pyiceberg version with pydantic < 2.12 will work.

Here's a simple script to check:

uv run --with 'pyiceberg[pyarrow,sql-sqlite]' --with 'pydantic<2.12' python - <<'PY'
import pyiceberg, pydantic, pyarrow as pa
print(f"pyiceberg.__version__ = {pyiceberg.__version__}")
print(f"pydantic.__version__ = {pydantic.__version__}")

from pyiceberg.catalog import load_catalog

catalog = load_catalog("catalog", type="in-memory")
catalog.create_namespace_if_not_exists("default")

data = pa.table({"x": [1, 2, 3], "y": [4, 5, 6]})
iceberg_table = catalog.create_table("default.test", schema=data.schema)
iceberg_table.append(data)

print("Table created and data appended successfully.")
print(len(iceberg_table.scan().to_arrow()))
PY

outputs:

pyiceberg.__version__ = 0.10.0
pydantic.__version__ = 2.11.10
Table created and data appended successfully.
3

removing the --with 'pydantic<2.12' will default back to the latest pydantic version, which is 2.12, and will fail the script

Alternatively you can also install the previous pydantic after install pyiceberg (that should override the broken pydantic version)
e.g. pip install pydantic==2.11.10

pydantic has release 2.12.3 which completely fix the regression. I've verified using the script above.

To summarize, all pyiceberg releases should work without any issues. By default, fresh installation will pull the latest pydantic dependency, pydantic==2.12.3. If you're still seeing this issue, please upgrade pydantic to 2.12.3 (or downgrade to 2.11.10).

For future reference:

pydantic 2.12.0 introduced the regression
pydantic 2.12.1 was yanked by its maintainers
pydantic 2.12.2 fixed the regression in pydantic, but did not resolve the issue for pyiceberg due to Fix after model validator signatures #2626
pydantic 2.12.3 fixed the regression in both pydantic and pyiceberg

As next steps:

merge Fix after model validator signatures #2626 since pydantic consider the previous signature to be deprecated
partially revert [Upstream Error] Pin pydantic version <2.12.0 to avoid regression #2591, we should still disallow 2.12.0 and 2.12.1. (2.12.2 is ok after Fix after model validator signatures #2626)

Thanks everyone for working on this issue!

Fix after model validator signatures (#2626)

dep: disallow pydantic 2.12.0 and 2.12.1, update poetry lock to use p…

Navigation Menu

Repository navigation

Upstream Pydantic 2.12.0 has a Regression that will break PyIceberg #2590

Apache Iceberg version

Please describe the bug 🐞

Willingness to contribute

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Upstream Pydantic 2.12.0 has a Regression that will break PyIceberg #2590

Description

Apache Iceberg version

Please describe the bug 🐞

Willingness to contribute

Activity

kevinjqliu commented on Oct 7, 2025

fmorillo7694 commented on Oct 8, 2025

nickjoanis commented on Oct 8, 2025

HonahX commented on Oct 8, 2025

sheinbergon commented on Oct 9, 2025

arthurql commented on Oct 10, 2025

ForeverAngry commented on Oct 11, 2025

kevinjqliu commented on Oct 11, 2025

Fokko commented on Oct 11, 2025

lolloz98 commented on Oct 12, 2025

kevinjqliu commented on Oct 19, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions