[FLINK-37926][table] Support casting from VARIANT to ARRAY by raminqaf · Pull Request #29073 · apache/flink

raminqaf · 2026-09-02T16:14:28Z

What is the purpose of the change

Let CAST and TRY_CAST convert a VARIANT into an ARRAY<T>. This is the first of three stacked PRs for FLINK-37926 (VARIANT to constructed types); the follow-ups add ROW/STRUCTURED and MAP<STRING, V> and build on this branch. It extends FLINK-37925 (VARIANT to scalar types, already merged): a constructed cast is that same scalar cast applied to every element, plus a shape check at each level, so no new leaf semantics are introduced.

A variant array casts to ARRAY<T> by casting each element to T with the full VARIANT-to-T rule, recursively. The recursion bottoms out at the scalar cast, which keeps one mental model for the whole type and one deliberate strictness: a stored value keeps its kind, so a stored string is never parsed into a number. CAST(PARSE_JSON('["1","2"]') AS ARRAY<INT>) therefore fails; read it as ARRAY<STRING> and convert with a regular cast instead.

Brief change log

Add VariantToArrayCastRule: shape-check the variant is an array, then cast each element to the target element type, recursively reusing the existing VARIANT cast rules (including the VARIANT-to-VARIANT identity, so ARRAY<VARIANT> shreds exactly one level).
A JSON null element maps to SQL NULL for a nullable element type and fails the cast for a NOT NULL one; an empty array casts to an empty ARRAY<T>; a top-level JSON null casts to SQL NULL for a nullable target before the shape check. CAST fails on any bad element and TRY_CAST returns NULL for the whole array (all-or-nothing), matching nested ARRAY casts today.
Add a shared AbstractVariantToConstructedCastRule base (JSON-null passthrough and canFail), reused by the follow-up ROW and MAP PRs.
Castability is explicit only and recursive: VARIANT to ARRAY<T> is castable iff VARIANT to T is, so an element type with no variant counterpart (e.g. INTERVAL) is rejected at plan time.
Extend the cast-rules table and the data types docs (EN and zh).

Verifying this change

This change added tests and can be verified as follows:

LogicalTypeCastsTest: VARIANT to ARRAY castability, including nesting, ARRAY<VARIANT>, and an unsupported leaf.
CastRuleProviderTest#testResolveVariantToArray: rule resolution and recursive resolvability.
CastRulesTest: codegen behavior for ARRAY<INT> (nullable and NOT NULL), ARRAY<STRING>, ARRAY<DOUBLE>, nested ARRAY<ARRAY<INT>>, ARRAY<VARIANT>, empty and null-element arrays, a heterogeneous array, and shape mismatches.
CastFunctionITCase: end-to-end CAST/TRY_CAST over PARSE_JSON, covering the same shapes, the strict-leaf failures, and validation errors.

Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): no
The public API, i.e., is any changed class annotated with @Public(Evolving): no
The serializers: no
The runtime per-record code paths (performance sensitive): no (a new opt-in cast rule; existing paths are unchanged)
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
The S3 file system connector: no

Documentation

Does this pull request introduce a new feature? yes
If yes, how is the feature documented? docs (data-types.md, EN and zh) and JavaDocs

Was generative AI tooling used to co-author this PR?

Yes (please specify the tool below)

Generated-by: Opus 4.8

flinkbot · 2026-09-02T16:29:10Z

CI report:

9e3a72c Azure: SUCCESS

Bot commands

The @flinkbot bot supports the following commands:

@flinkbot run azure re-run the last Azure build

mateczagany · 2026-09-03T10:23:42Z

+    }
+    result$0 = new org.apache.flink.table.data.GenericArrayData(objArray$3);
+
+    A JSON null element leaves the slot null (SQL NULL); a NOT NULL element type emits a throw instead.


For ARRAY<VARIANT> this drops a value: the identity cast keeps a variant null at top level, but here it becomes SQL NULL.

I think by skipping the isNull when the elementType is VARIANT casts could keep null elements as variant null instead.

This would also match Spark, Trino and Snowflake in this regard.

Excellent catch @mateczagany! I have addressed your feedback and the codegen should be able to handle this now. I have added a test:

CAST VARIANT -> ARRAY<VARIANT> [1, null, 3]

It is green now.

twalthr

Thank you @raminqaf!

twalthr · 2026-09-03T12:25:36Z

+            return allowExplicit
+                    && supportsCasting(sourceType, ((ArrayType) targetType).getElementType(), true);


why this?

Suggested change

return allowExplicit

&& supportsCasting(sourceType, ((ArrayType) targetType).getElementType(), true);

return supportsCasting(sourceType, ((ArrayType) targetType).getElementType(), true);

We should keep the allowExplicit so the user must write CAST(v AS ARRAY<...>) the cast to an ARRAY from a variant is explicit.

twalthr · 2026-09-03T12:36:05Z

+                                ARRAY(INT()).notNull())
+                        // a JSON null element fails a NOT NULL element type
+                        .testTableApiRuntimeError(
+                                call("PARSE_JSON", "[1, null, 3]").cast(ARRAY(INT().notNull())),


is there no "Table API" way of parsing JSON? Can we avoid call()?

there's no dedicated Table API parseJson() — PARSE_JSON exists only as BuiltInFunctionDefinitions.PARSE_JSON, invoked via call(). So call() can't be avoided

Created a ticket for this:
https://issues.apache.org/jira/browse/FLINK-40545

twalthr

LGTM, thanks @raminqaf

…RAY<VARIANT> A variant array element casts to a VARIANT target by the identity cast, so a JSON null element is a valid variant null and is kept as-is rather than downgraded to SQL NULL, matching the top-level VARIANT cast and Spark, Trino and Snowflake. The null guard is skipped for a VARIANT element type. Also reuse a shared VariantBuilder across the cast test fixtures.

raminqaf force-pushed the FLINK-37926-variant-to-array branch from 230c201 to 64be049 Compare September 3, 2026 06:36

snuyanzin reviewed Sep 3, 2026

View reviewed changes

Comment thread ...r/src/main/java/org/apache/flink/table/planner/functions/casting/VariantToArrayCastRule.java

snuyanzin reviewed Sep 3, 2026

View reviewed changes

Comment thread ...table-planner/src/test/java/org/apache/flink/table/planner/functions/CastFunctionITCase.java Outdated

snuyanzin reviewed Sep 3, 2026

View reviewed changes

Comment thread ...le-planner/src/test/java/org/apache/flink/table/planner/functions/casting/CastRulesTest.java

raminqaf force-pushed the FLINK-37926-variant-to-array branch 4 times, most recently from f3432e9 to c068f50 Compare September 3, 2026 09:09

manner reviewed Sep 3, 2026

View reviewed changes

Comment thread ...table-planner/src/test/java/org/apache/flink/table/planner/functions/CastFunctionITCase.java Outdated

mateczagany reviewed Sep 3, 2026

View reviewed changes

twalthr reviewed Sep 3, 2026

View reviewed changes

twalthr approved these changes Sep 3, 2026

View reviewed changes

github-actions Bot added the community-reviewed PR has been reviewed by the community. label Sep 3, 2026

raminqaf mentioned this pull request Sep 3, 2026

[FLINK-37926][table] Support casting from VARIANT to ROW and STRUCTURED #29092

Open

1 task

raminqaf and others added 5 commits September 4, 2026 10:23

[FLINK-37926][table] Support casting from VARIANT to ARRAY

0d76c63

[FLINK-37926][table] Add tests for casting from VARIANT to ARRAY

6b076b7

[FLINK-37926][docs] Document casting from VARIANT to ARRAY

4bd5aec

[FLINK-37926] Address Feedbacks

9e3a72c

raminqaf force-pushed the FLINK-37926-variant-to-array branch from fd62734 to 9e3a72c Compare September 4, 2026 08:25

snuyanzin approved these changes Sep 4, 2026

View reviewed changes

snuyanzin merged commit 5d3597c into apache:master Sep 4, 2026
1 check passed

		return allowExplicit
		&& supportsCasting(sourceType, ((ArrayType) targetType).getElementType(), true);

	return allowExplicit
	&& supportsCasting(sourceType, ((ArrayType) targetType).getElementType(), true);
	return supportsCasting(sourceType, ((ArrayType) targetType).getElementType(), true);

Uh oh!

Conversation

raminqaf commented Sep 2, 2026

What is the purpose of the change

Brief change log

Verifying this change

Does this pull request potentially affect one of the following parts:

Documentation

Was generative AI tooling used to co-author this PR?

Uh oh!

flinkbot commented Sep 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

CI report:

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mateczagany Sep 3, 2026

Choose a reason for hiding this comment

Uh oh!

raminqaf Sep 3, 2026

Choose a reason for hiding this comment

Uh oh!

twalthr left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

twalthr Sep 3, 2026

Choose a reason for hiding this comment

Uh oh!

raminqaf Sep 3, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

twalthr Sep 3, 2026

Choose a reason for hiding this comment

Uh oh!

raminqaf Sep 3, 2026

Choose a reason for hiding this comment

Uh oh!

raminqaf Sep 3, 2026

Choose a reason for hiding this comment

Uh oh!

twalthr left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

flinkbot commented Sep 2, 2026 •

edited

Loading