Repository navigation
docs: Add documentation for supported casts with complex types #2760
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
|
|
@@ -191,3 +191,57 @@ The following cast operations are not compatible with Spark for all inputs and a | |||||
|
|
||||||
| Any cast not listed in the previous tables is currently unsupported. We are working on adding more. See the | ||||||
| [tracking issue](https://github.com/apache/datafusion-comet/issues/286) for more details. | ||||||
|
|
||||||
| ### Complex Type Casting | ||||||
|
|
||||||
| Comet provides support for casting involving complex types (arrays, structs, and maps) with the following rules: | ||||||
|
|
||||||
| #### Array Type Casting | ||||||
|
|
||||||
| - **Array to String**: Arrays can be cast to string. The cast is compatible if the element type can be cast to string. | ||||||
| - Recursively checks that the array element type supports casting to string | ||||||
| - Example: `CAST(array(1, 2, 3) AS STRING)` → `"[1, 2, 3]"` | ||||||
|
|
||||||
| - **Array to Array**: Arrays can be cast to other array types if the element types are compatible. | ||||||
| - Recursively validates that the source element type can be cast to the target element type | ||||||
| - Example: `CAST(array(1, 2, 3) AS ARRAY<BIGINT>)` → `array(1L, 2L, 3L)` | ||||||
| - The compatibility level (Compatible/Incompatible/Unsupported) is determined by the element type cast | ||||||
|
|
||||||
| - **Special case**: Arrays with `NullType` elements (empty arrays) can be cast to any array type | ||||||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Is it correct to say
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Agree with @martin-g
|
||||||
|
|
||||||
| #### Struct Type Casting | ||||||
|
|
||||||
| - **Struct to Struct**: Structs can be cast to other struct types if all field types are compatible. | ||||||
| - Requires the same number of fields in source and target structs | ||||||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This is not quite correct!
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Thanks @martin-g appreciate if you can provide an example. so it would be more clear for the user and we can also track this in the issue as future work |
||||||
| - Each field is validated recursively: `source_field[i]` must be castable to `target_field[i]` | ||||||
| - Field names do not need to match; casting is done by position | ||||||
| - Example: `CAST(struct(1, 'a') AS STRUCT<x: BIGINT, y: STRING>)` → `struct(1L, 'a')` | ||||||
| - If any field cast is Incompatible or Unsupported, the entire struct cast inherits that level | ||||||
|
|
||||||
| - **Struct to String**: Structs can be cast to string if all field types can be cast to string. | ||||||
| - Recursively validates each field type can be cast to string | ||||||
| - Example: `CAST(struct(1, 'a') AS STRING)` → `"{1, a}"` | ||||||
|
|
||||||
| #### Map Type Casting | ||||||
|
|
||||||
| Currently, Comet does not support casting to or from map types. Map-related casts will fall back to Spark. | ||||||
|
|
||||||
| #### Nested Complex Types | ||||||
|
|
||||||
| Comet supports nested complex types with recursive casting validation: | ||||||
| - **Array of Structs**: `ARRAY<STRUCT<...>>` casting is supported if both the array and struct casting rules are satisfied | ||||||
| - **Struct of Arrays**: `STRUCT<..., ARRAY<...>, ...>` casting is supported if all fields (including array fields) satisfy casting rules | ||||||
| - **Deeply nested types**: Any level of nesting is supported as long as each level satisfies the casting requirements | ||||||
|
|
||||||
| Examples: | ||||||
| ```sql | ||||||
| -- Array of structs to array of structs | ||||||
| CAST(array(struct(1, 'a'), struct(2, 'b')) AS ARRAY<STRUCT<x: BIGINT, y: STRING>>) | ||||||
|
|
||||||
| -- Struct containing arrays | ||||||
| CAST(struct(1, array(1, 2)) AS STRUCT<id: BIGINT, values: ARRAY<BIGINT>>) | ||||||
| ``` | ||||||
|
|
||||||
| **Note**: The compatibility level of a complex type cast is determined by the "worst" compatibility level among all | ||||||
| its constituent element/field casts. For example, if any field in a struct cast is Incompatible, the entire struct | ||||||
| cast is Incompatible. | ||||||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There are no tests for this.
datafusion-comet/spark/src/test/scala/org/apache/comet/CometCastSuite.scala
Line 1050 in e77998a
cast ArrayType to StringTypeseems to be the onlycast ArrayType to ...test in the suite