Skip to content

negative: ANSI overflow check reads null slots and can raise spurious overflow errors #5093

Description

@andygrove

Describe the bug

In native/spark-expr/src/math_funcs/negative.rs, the ANSI-mode overflow check (check_overflow! macro) loops over Int8/16/32/64 and interval arrays comparing every slot's value to MIN, then calls neg_wrapping. The scan reads value(i) without checking validity, so a MIN value sitting in a null slot (arbitrary bytes left behind by filter/slice/FFI) raises a spurious ARITHMETIC_OVERFLOW error even though the row is null.

It is also a two-pass re-implementation of arrow's checked neg kernel, which negates with overflow detection in a single pass and correctly skips null slots.

Steps to reproduce

Construct an Int64 array where a null slot's underlying value is i64::MIN (for example by masking a valid i64::MIN entry to null via a validity buffer) and evaluate negative() on it in ANSI mode. The expression errors; Spark returns null for the null row and succeeds.

Expected behavior

Null slots must be ignored by the overflow check. Replacing the ANSI path with arrow::compute::kernels::numeric::neg fixes this, since the kernel skips invalid slots.

Additional context

Notes for the fix:

  • The Spark error message per type must be reconstructed in a map_err; if the exact offending value is required for Int8/Int16 wording, a rescan on the (cold) error path can recover it.
  • IntervalUnit::MonthDayNano is deliberately not overflow-checked today, and arrow's checked neg would check it, so that type must keep neg_wrapping unconditionally.
  • Legacy (non-ANSI) mode keeps neg_wrapping as today.

Found during an audit of native code that replicates existing arrow-rs kernels. The corresponding null-slot behavior is already pinned correctly for add in checked_arithmetic.rs tests.

Activity

  1. Smallfu666 commented on Jul 31, 2026

    @Smallfu666
    Contributor

    take

  2. added
    priority:mediumFunctional bugs, performance regressions, broken features
    and removed on Aug 3, 2026
  3. added this to the 1.1.0 milestone on Aug 3, 2026
  4. removed their assignment
    on Aug 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area:expressionsExpression evaluationbugSomething isn't workingpriority:mediumFunctional bugs, performance regressions, broken features

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions