Skip to content

Comparison operators on arrays and structs with floating-point leaves do not match Spark for signed zero #6157

Description

@andygrove

Describe the bug

<=>, <, <=, > and >= on arrays and structs with FLOAT/DOUBLE leaves run natively and compare the nested values with Arrow's total order, so -0.0 and 0.0 are different values. Spark compares nested values with TypeUtils.getInterpretedOrdering, which bottoms out in SQLOrderingUtil.compareDoubles/compareFloats, where signed zeros are equal.

CometExecRule.normalize only wraps scalar float operands. #6073 fixes =, <> and IN for nested operands, but leaves <=> and the ordering operators on the raw comparator. #5507 covers ORDER BY and rank on nested float keys, and this is the same gap in the comparison predicates.

Steps to reproduce

On main at cccc08b, default Spark 4.1 profile, as a CometSqlFileTestSuite fixture:

CREATE TABLE t (a ARRAY<DOUBLE>, b ARRAY<DOUBLE>, s STRUCT<v: DOUBLE>, u STRUCT<v: DOUBLE>) USING parquet;
INSERT INTO t VALUES (
  array(CAST('-0.0' AS DOUBLE)), array(CAST('0.0' AS DOUBLE)),
  named_struct('v', CAST('-0.0' AS DOUBLE)), named_struct('v', CAST('0.0' AS DOUBLE)));

SELECT a <=> b FROM t;
SELECT a < b, a >= b, s <=> u, s < u FROM t;

Both queries run fully native (CometProject over CometNativeScan) and return different answers:

Query Spark Comet
a <=> b true false
a < b, a >= b, s <=> u, s < u false, true, true, false true, false, false, true

CAST('-0.0' AS DOUBLE) is a string cast, so the sign survives. CAST(-0.0 AS DOUBLE) would produce +0.0.

Expected behavior

The same answers as Spark: signed zeros inside arrays and structs compare equal for every comparison operator.

Additional context

Found while reviewing #6073. I only verified signed zero. NaN payloads and signs probably diverge too, since total_cmp orders a negative NaN below every other value while Spark treats every NaN as the largest value. The equivalence in #6073's nested_comparison.rs could likely be extended into an ordering comparator for these operators.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area:expressionsExpression evaluationbugSomething isn't workingcorrectnesspriority:criticalData corruption, silent wrong results, security issues

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions