You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Removing then adding the same partition in update_spec fails聽#1131
When adding a field to partition spec, if the field previously existed but was deleted it will fail when not explictly defining a column name:
For example this would fail:
table.update_spec().add_identity("VendorID").commit()
table.update_spec().remove_field("VendorID").commit()
table.update_spec().add_identity("VendorID").commit()
This would fail:
table.update_spec().add_field("VendorID", IdentityTransform()).commit()
table.update_spec().remove_field("VendorID").commit()
table.update_spec().add_field("VendorID", IdentityTransform()).commit()
This would not fail when defining the partition field name:
table.update_spec().add_field("VendorID", IdentityTransform(), "VendorID").commit()
table.update_spec().remove_field("VendorID").commit()
table.update_spec().add_field("VendorID", IdentityTransform(), "VendorID").commit()
Example trace:
Traceback (most recent call last):
File "C:\Users\a\PycharmProjects\pythonProject\test.py", line 24, in
table.update_spec().add_identity("VendorID").commit()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\a\PycharmProjects\pythonProject.venv\Lib\site-packages\pyiceberg\table_init_.py", line 3682, in add_identity
return self.add_field(source_column_name, IdentityTransform(), None)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\a\PycharmProjects\pythonProject.venv\Lib\site-packages\pyiceberg\table_init_.py", line 3657, in add_field
new_field = self.partition_field((bound_ref.field.field_id, transform), partition_field_name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\a\PycharmProjects\pythonProject.venv\Lib\site-packages\pyiceberg\table_init.py", line 3833, in _partition_field
return PartitionField(source_id, field_key[1], transform, name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\a\PycharmProjects\pythonProject.venv\Lib\site-packages\pyiceberg\partitioning.py", line 111, in init
super().init(**data)
File "C:\Users\a\PycharmProjects\pythonProject.venv\Lib\site-packages\pydantic\main.py", line 193, in init
self.pydantic_validator.validate_python(data, self_instance=self)
pydantic_core._pydantic_core.ValidationError: 1 validation error for PartitionField
name
Field required [type=missing, input_value={'source-id': 1, 'field-i...m': IdentityTransform()}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.8/v/missing
Activity
sungwy commentedon Sep 11, 2024
Hi @lkindere thank you very much for reporting this issue! I've put up a PR to fix this, and I've created tests based on your examples here: #1161