GH-50622: [Docs][Format] Align Variant `typed_value` primitive type mappings with the Parquet shredding spec by sdf-jkl · Pull Request #50810 · apache/arrow

sdf-jkl · 2026-08-04T20:46:16Z

Rationale for this change

The existing Arrow spec is misleading and allows certain Arrow types to be typed value Variant fields. The Parquet Shredding spec specifically rejects them - Unsigned integers, Null.

Decimal mapping needs clarification about precision.

What changes are included in this PR?

Change the Primitive Type Mappings table for Parquet Variant extension type.

Rendered version - https://github.com/sdf-jkl/arrow/blob/gh50622-variant-typed-value-mappings/docs/source/format/CanonicalExtensions.rst#primitive-type-mappings

Are these changes tested?

N/A

Are there any user-facing changes?

Changes to the Arrow docs

Closes #50622

GitHub Issue: [Docs][Format] Variant extension: align typed_value primitive type mappings with the Parquet shredding spec #50622

github-actions · 2026-08-04T20:46:52Z

⚠️ GitHub issue #50622 has been automatically assigned in GitHub to PR creator.

sdf-jkl · 2026-08-13T15:33:54Z

Please take a look @alamb @wgtmac @zeroshade

alamb

Thank you @sdf-jkl -- this looks very nice to me except for a few small details I left comments on

I don't think we should merge this without making sure we ping the mailing list about it (perhaps again) as I think it could be interpreted as changing the spec.

alamb · 2026-08-17T19:16:27Z

+only valid Parquet representation.
+
+----------------------------------------+--------------------------------------------------+---------------------------------------------+
+| Variant Type                           | Parquet Type                                     | Arrow ``typed_value`` Type                  |


Perhaps we could also add a reference to the location of the parquet and arrow type lists

I think the arrow ones are here:
https://arrow.apache.org/docs/format/Columnar.html#data-types

The parquet ones are here:
https://parquet.apache.org/docs/file-format/types/
https://parquet.apache.org/docs/file-format/types/logicaltypes/

I am a little confused about what (logical or physical) type is used in the Parquet Type column. Some seem to be just the physical type (INT64) , and some seem to have both logical and physical (INT32, INT(8, true) ). It would help to be spcific about what the Parquet type column represents and use it consistently

alamb · 2026-08-17T19:20:16Z

+----------------------------------------+--------------------------------------------------+---------------------------------------------+
+| double                                 | DOUBLE                                           | Float64                                     |
+----------------------------------------+--------------------------------------------------+---------------------------------------------+
+| decimal4 (1 <= P <= 9, 0 <= S <= P)    | INT32, DECIMAL(P, S)                             | Decimal32(P, S)                             |


https://parquet.apache.org/docs/file-format/types/variantencoding/ says

The Decimal type contains a scale, but no precision. The implied precision of a decimal value is floor(log_10(val)) + 1.

I don't unersatnd how that maps to 1 <= P <= 9, 0 <= S <= P 🤔

There's a small separate table for decimals that shows what precision can fit in the underlying physical type of a typed_value.

https://github.com/apache/parquet-format/blob/3ef43e00cea6dcd06da5b77541f84bef879aff5b/VariantEncoding.md?plain=1#L440-L447

alamb · 2026-08-17T19:22:20Z

+| uuid                                   | FIXED_LEN_BYTE_ARRAY[len=16], UUID               | :ref:`UUID extension type <uuid_extension>` |
+----------------------------------------+--------------------------------------------------+---------------------------------------------+
+
+The decimal precision bands follow the `Variant encoding types


I am still a little confused about how decimals are working -- is the this text talking about each variant value itself (as the scale/precision can vary row to row) or the min/max precision of all values in a column (presumably what is needed when shredding) 😕

It's talking about the constraint of physical types of the typed_value field, so it would be the latter. I'll polish it to make it more clear.

sdf-jkl · 2026-08-18T16:23:27Z

@alamb I think it's more clear now.

zeroshade · 2026-08-18T19:59:48Z

+   * A Variant null is always encoded in the ``value`` field (as ``00``),
+     never in ``typed_value``: a null ``typed_value`` signals that the row is
+     not shredded, and for shredded object fields a null ``typed_value``
+     together with a null ``value`` means the field is missing.


Should we clarify that a variant null should still be indicated in the null validity bitmap of the corresponding array (accompanying the 00 value in the value field)?

I don't think it should.

The spec explicitly says:

The value field must be annotated as required for unshredded Variant values, or optional if parts of the value are shredded as typed Parquet columns.

link

That makes Variant::Null and SQL NULL different things

I think it might be more precise to say it makes Variant::Null and a null value in the corresponding arrow array different things

alamb · 2026-08-19T11:40:35Z

+----------------------------------------+-----------------------------------+--------------------------+---------------------------------------------+
+| int16                                  | INT32                             | INT(16, true)            | Int16                                       |
+----------------------------------------+-----------------------------------+--------------------------+---------------------------------------------+
+| int32                                  | INT32                             |                          | Int32                                       |


i am confused that there is no logical type for int32, though this is the same in the parquet variant table too https://github.com/apache/parquet-format/blob/master/VariantShredding.md#shredded-value-types

alamb · 2026-08-19T11:42:46Z

+   * A Variant null is always encoded in the ``value`` field (as ``00``),
+     never in ``typed_value``: a null ``typed_value`` signals that the row is
+     not shredded, and for shredded object fields a null ``typed_value``
+     together with a null ``value`` means the field is missing.


I think it might be more precise to say it makes Variant::Null and a null value in the corresponding arrow array different things

alamb

Thank you @sdf-jkl -- I think this makes sense to me. I'll wait for more confirmation from other community members before merging this in

sdf-jkl · 2026-08-21T19:09:48Z

Thanks @alamb addressed your comments here - 84af803

sdf-jkl · 2026-08-24T21:12:40Z

@alamb I think my message to the mailing list about this went to spam too 😢

HuaHuaY · 2026-08-25T02:28:30Z

@alamb I think my message to the mailing list about this went to spam too 😢

I received the email successfully. I guess the community people may be busy lately. Perhaps you could send another email to remind.

### Rationale for this change A Null `typed_value` column is not a valid shredded Variant type. Nulls are encoded in the `value` column as Variant null (`00`). A dedicated Null `typed_value` would also make “field present and null” indistinguishable from “field missing” for shredded objects. See apache/arrow#50622 and apache/arrow#50810. Fixes #1205 ### What changes are included in this PR? - `NewVariantType` rejects a Null `typed_value` field - Nested shredded fields with a Null `typed_value` are also rejected ### Are these changes tested? - `go test ./arrow/extensions` ### Are there any user-facing changes? Yes — constructing a Variant extension type with a Null `typed_value` now returns `arrow.ErrInvalid`. --------- Signed-off-by: Digvijay <digvijay.vaghela@yahoo.com>

wgtmac · 2026-09-03T05:46:42Z

+   stores its unscaled value as a 4-, 8-, or 16-byte integer, capping the
+   precision at 9, 18, or 38 significant digits.
+
+   ``(P, S)`` refers to the Arrow data type of the ``typed_value`` field


Do we really need to re-explain what a standard Arrow Decimal Array is?

per @alamb's comment above it seems like it was not clean enough before

I am still a little confused about how decimals are working -- is the this text talking about each variant value itself (as the scale/precision can vary row to row) or the min/max precision of all values in a column (presumably what is needed when shredding) 😕

I can reword it if needed

sdf-jkl · 2026-09-15T16:54:08Z

Re-requested the reviews.

I no one has objections I'd like to merge by the end of the week

alamb · 2026-09-16T20:40:25Z

I'll plan to merge this on Friday Sep 18 2026 unless anyone objects or would like more time to review

alamb · 2026-09-21T10:23:23Z

Thanks again everyone

pin the typed value typed to corresponding Parquet types

3afdca0

sdf-jkl requested a review from pitrou as a code owner August 4, 2026 20:46

github-actions Bot added Component: Documentation awaiting review Awaiting review labels Aug 4, 2026

sdf-jkl mentioned this pull request Aug 4, 2026

[Docs][Format] Variant extension: align typed_value primitive type mappings with the Parquet shredding spec #50622

Closed

alamb reviewed Aug 17, 2026

View reviewed changes

sdf-jkl added 3 commits August 18, 2026 10:09

split parquet physical and logical type columns

683dc27

clean up decimal precision notes

cd26764

clean up pass

97fc879

zeroshade reviewed Aug 18, 2026

View reviewed changes

github-actions Bot added awaiting changes Awaiting changes and removed awaiting review Awaiting review labels Aug 18, 2026

alamb reviewed Aug 19, 2026

View reviewed changes

alamb approved these changes Aug 19, 2026

View reviewed changes

apacheGH-50622: Clarify Variant integer and null mappings

84af803

github-actions Bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Aug 21, 2026

peterxcli mentioned this pull request Aug 26, 2026

[Variant] Track and remove native Variant compatibility workarounds after upstream fixes apache/datafusion-comet#5477

Open

7 of 28 tasks

dgvj-work mentioned this pull request Aug 28, 2026

fix(arrow/extensions): reject Null Variant typed_value apache/arrow-go#1243

Merged

wgtmac reviewed Sep 3, 2026

View reviewed changes

sdf-jkl requested review from wgtmac and zeroshade September 15, 2026 16:53

wgtmac approved these changes Sep 16, 2026

View reviewed changes

HuaHuaY mentioned this pull request Sep 17, 2026

GH-45937: [C++][Parquet] support to read, write and validate variant #50252

Open

alamb merged commit 808338e into apache:main Sep 21, 2026
34 of 35 checks passed

sdf-jkl deleted the gh50622-variant-typed-value-mappings branch September 21, 2026 12:50

Conversation

sdf-jkl commented Aug 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

github-actions Bot commented Aug 4, 2026

Uh oh!

sdf-jkl commented Aug 13, 2026

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sdf-jkl Aug 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sdf-jkl commented Aug 18, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

sdf-jkl commented Aug 21, 2026

Uh oh!

sdf-jkl commented Aug 24, 2026

Uh oh!

HuaHuaY commented Aug 25, 2026

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sdf-jkl commented Sep 15, 2026

Uh oh!

alamb commented Sep 16, 2026

Uh oh!

Uh oh!

alamb commented Sep 21, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

sdf-jkl commented Aug 4, 2026 •

edited

Loading

sdf-jkl Aug 18, 2026 •

edited

Loading