[PATHFINDING] Parse json as variant

[scovich]

• Related to Variant: Read/Parse JSON value as Variant #7425

This is a pathfinding exercise, to see how easy/hard it might be to parse JSON text into parquet's new variant type, using the tape decoder. Not intended to merge, it is more of a conversation starter.

In particular:

• It would be better to leverage a general variant library for variant bit-wrangling instead of doing it all manually here.
• TBD Where/how to expose this functionality through a public API
• Still TBD how to assemble a bunch of variant metadata+value pairs inside an arrow array data that can eventually become a usable arrow array
• For comparison, the same exercise is repeated using serde_json instead. This would almost certainly not belong in an actual contribution to arrow-json.

[scovich]

Attn @alamb

[alamb]

See also the related PR for variant here:

• Variant Support for Arrow and Parquet [DRAFT] #7404

[alamb]

Thank you for this PR @scovich

It would be better to leverage a general variant library for variant bit-wrangling instead of doing it all manually here.

• 100% agree. I think we'll do this as part of [EPIC] [Parquet] Implement Variant type support in Parquet #6736. I am currently working on breaking up the work into some smaller pieces so we can get the code flowing

TBD Where/how to expose this functionality through a public API

In my mind this functionality feels like a "computation kernel" (aka similarly to the functions in https://docs.rs/arrow/latest/arrow/compute/index.html)

The signature seems like it would roughly be something like:

/// Covert text stored as JSON in an input `StringArray`, `LargeStringArray` or `StringViewArray` into
/// a single "Variant" array (`StructArray` with an extension type)
fn json_to_variant(input: &ArrayRef) -> ArrayRef {
 ...
}

Since the arrow-json crate is currently for converting JSON to arrow it is not 100% clear to me that this functionality belongs in the arrow-json crate at all, espcially as variant is not part of the "core" arrow spec it seems.

Still TBD how to assemble a bunch of variant metadata+value pairs inside an arrow array data that can eventually become a usable arrow array

I think we will sort this out as part of implementing varint in #6736. TLDR is via a StructArray annotated with an extension type I think.

[scovich]

TBD Where/how to expose this functionality through a public API

In my mind this functionality feels like a "computation kernel" (aka similarly to the functions in https://docs.rs/arrow/latest/arrow/compute/index.html)

Since the arrow-json crate is currently for converting JSON to arrow it is not 100% clear to me that this functionality belongs in the arrow-json crate at all, espcially as variant is not part of the "core" arrow spec it seems.

I agree something like arrow-compute makes a lot of sense. Unfortunately, the tape decoder machinery is private to arrow-json crate, so I had to do the initial pathfinding here. Is there a better way forward?

[alamb]

I agree something like arrow-compute makes a lot of sense. Unfortunately, the tape decoder machinery is private to arrow-json crate, so I had to do the initial pathfinding here. Is there a better way forward?

SOme other options might be (not sure which one we should go with):

1. copy/paste the code to avoid a dependency
2. refactor the tape machinery into a new crate that they can both depend on

I have been thinking a lot about how we should introduce variant. What do you think about a structure like this (crates)

• variant: Core definition of the open variant type, no dependencies
• arrow-variant: Arrow extension type for variant, including conversion to/from JSON and arrow arrays (e.g. a compute kernel, etc)

I think depending on how arrow-variant is implemented, maybe it depends directly on arrow-json and maybe expose relevant parts

[alamb]

I filed #7423 to track this item

[github-actions]

Thank you for your contribution. Unfortunately, this pull request is stale because it has been open 60 days with no activity. Please remove the stale label or comment or this will be closed in 7 days.