Repository navigation
Design of Serialization/Deserialization #2
Description
I would like to have a discussion about how to implement the Serialization/Deserialization of the iceberg metadata. There are currently two different designs each with its pros and cons. The designs are:
- Split representation into in_memory/on_disk part (example)
- Use custom implementations of
Serialize/Deserialize(example)
I would like to get your opinion which design you prefer.
Activity
ZENOTME commentedon Jul 21, 2023
If I'm not mistaken, use the split representation also need to custom implementation for on_disk part. And we should maintain two system: in_memory::type, in_memory::value, on_disk::type, on_disk::value. And we also need to maintain the conversion between in_memory and on_disk. So I think strictly separate two part may cause redundant code. Can we only separate some part🤔?
JanKaul commentedon Jul 21, 2023
I also think that a split representation leads to redundancy which makes maintaining the code more difficult.
I personally think that the added complexity of custom implementations of
Serialize/Deserializeis worth avoiding the redundancy.JanKaul commentedon Jul 21, 2023
I realized that this discussion is closely related to #3 , because it gives additional motivation for splitting the representation into an in_memory and on_disk part.
I think @ZENOTME made a good point about splitting only a part of the spec.
We could split the representation for all structs that have user interaction (i.e.
TableMetadata) but use a single representation for the smaller building blocks (i.e. types, values, ..)liurenjie1024 commentedon Jul 21, 2023
Personally, I prefer the
Serialize/Deserializeapproach. From my experience, the effort doesn't reduce much since both way you need to maintain the conversion logic. Theon_diskapproach has some shorting comings:https://github.com/icelake-io/icelake/blob/eee3520f993837f4e1745c60538ba7ab1b421481/src/types/in_memory.rs#L64
ZENOTME commentedon Jul 21, 2023
Sounds good for me.
Xuanwo commentedon Jul 21, 2023
Ok, let's pick the
Serialize/Deserializeapproach.Nice idea!
liurenjie1024 commentedon Dec 14, 2023
Close this as we have finished discussion. Feel free to open if needed.
feat(reader): Add PartitionSpec support to FileScanTask and RecordBat…
Merge pull request apache#2 from KKould/fix/deletes_blocking
Wait for containers (apache#2)
docs: remediate review findings (testing contract + status reconcilia…