Describe the bug
Comet's native IF fails at runtime when its two branches have the same Spark type but differ in nested nullability, for example a struct field or map value that can be NULL in one branch and not in the other:
org.apache.comet.CometNativeException: Invalid argument error: column types must match schema types, expected Struct("x": Int32) but found Struct("x": non-null Int32) at column index 0
Spark treats the two branch types as the same when only their nullability differs, so it adds no cast, and Comet serializes each branch with its own Arrow type. The native IfExpr reports the THEN branch's type as its output type. When every row of a batch takes the ELSE branch, though, it returns the ELSE branch's array unchanged, and that array has the other nullability.
CASE WHEN does not fail, because the planner casts each branch to a common type first.
Steps to reproduce
CREATE TABLE t(q boolean, i int) USING parquet;
INSERT INTO t VALUES (true, 1), (false, 2);
-- both fail
SELECT IF(q, named_struct('x', i), named_struct('x', 0)) FROM t;
SELECT IF(q, map('k', i), map('k', 0)) FROM t;
-- passes
SELECT CASE WHEN q THEN named_struct('x', i) ELSE named_struct('x', 0) END FROM t;Reproduced on main at 7649361 through CometSqlFileTestSuite. The failure needs a batch in which no row takes the THEN branch, which this two-row table gives in one of its tasks. IF(q, array(i), array(0)) passes.
Expected behavior
The same result as Spark.
Additional context
Found while adding tests for #3025. The planner could build IF the way it builds CASE WHEN, casting a branch whose Arrow type differs from the common type.
Describe the bug
Comet's native
IFfails at runtime when its two branches have the same Spark type but differ in nested nullability, for example a struct field or map value that can be NULL in one branch and not in the other:Spark treats the two branch types as the same when only their nullability differs, so it adds no cast, and Comet serializes each branch with its own Arrow type. The native
IfExprreports the THEN branch's type as its output type. When every row of a batch takes the ELSE branch, though, it returns the ELSE branch's array unchanged, and that array has the other nullability.CASE WHENdoes not fail, because the planner casts each branch to a common type first.Steps to reproduce
Reproduced on
mainat 7649361 throughCometSqlFileTestSuite. The failure needs a batch in which no row takes the THEN branch, which this two-row table gives in one of its tasks.IF(q, array(i), array(0))passes.Expected behavior
The same result as Spark.
Additional context
Found while adding tests for #3025. The planner could build
IFthe way it buildsCASE WHEN, casting a branch whose Arrow type differs from the common type.