Repository navigation
[SPARK-56663][SQL][FOLLOWUP] Fix silent overflow in date_trunc fast path. - #56313
chenhao-db wants to merge 2 commits into
Conversation
|
@cloud-fan could you take a look? Thanks! |
cloud-fan
left a comment
There was a problem hiding this comment.
0 blocking, 1 non-blocking, 0 nits.
Clean, correct fix for the fast-path silent overflow.
Correctness (1)
- DateTimeUtils.scala:513: same
value - Math.floorMod(value, unit)pattern in the sibling MILLISECOND/SECOND branches (lines 540/542) still silently overflows atLong.MinValue— see inline
Verification
Traced the fix end to end: Math.subtractExact equals - for all non-overflowing inputs, and the only inputs it changes are those within one unitMicros of Long.MinValue, whose truncated-down instant is below Long.MinValue and so is unrepresentable by the slow path too. On underflow it throws ArithmeticException, caught by the existing catch (lines 524-525) and falls back to truncToUnit, which itself throws via instantToMicros's multiplyExact for the same input. So the fast path now fails consistently with the slow path rather than returning a wrapped-positive value — no input produces a wrong-but-representable result.
| try { | ||
| val local = Math.addExact(micros, offsetMicros) | ||
| val truncatedLocal = local - Math.floorMod(local, unitMicros) | ||
| val truncatedLocal = Math.subtractExact(local, Math.floorMod(local, unitMicros)) |
There was a problem hiding this comment.
Nicely targeted fix. Note the same value - Math.floorMod(value, unit) shape lives one match away in the sibling branches: TRUNC_TO_MILLISECOND (line 540) and TRUNC_TO_SECOND (line 542) still use a plain -, so a timestamp at Long.MinValue micros silently wraps to a positive value there (e.g. MIN - floorMod(MIN, 1000) = 9223372036854775616) instead of throwing like the fast path now does. Those branches have no offset arithmetic and no slow-path fallback, so applying Math.subtractExact directly would make them throw ArithmeticException consistently. Worth folding into this PR (with MILLISECOND/SECOND added to the new test), or noting they're intentionally out of scope. Pre-existing, so non-blocking.
cloud-fan
left a comment
There was a problem hiding this comment.
1 addressed, 0 remaining, 1 new (1 late catch — my miss from last round).
0 blocking, 1 non-blocking, 0 nits.
Prior round's MILLISECOND/SECOND finding fully addressed. Clean, complete, well-tested overflow fix.
Suggestions (1)
- PR description: the stated mechanism is inverted — "
Math.floorMod(local, unitMicros)is also negative" is wrong;floorModwith a positive divisor is always non-negative. The real cause is subtracting the non-negative remainder from a value nearLong.MinValue, which underflows. (If it were negative as written,local - negativewould move toward zero and could not overflow.) The code is correct — only the description's explanation. Worth fixing as the rationale-of-record.
Verification
Confirmed the now-fixed SECOND/MILLISECOND branches: Math.floorMod(Long.MinValue, unit) is positive (192 for ms, 224192 for sec), so Math.subtractExact underflows -> ArithmeticException, propagating directly (these branches have no slow-path fallback). The new test covers all levels YEAR..MILLISECOND — date levels via daysToMicros's multiplyExact, MINUTE/HOUR/DAY via the truncToUnitFast -> truncToUnit -> instantToMicros fallback — and correctly excludes MICROSECOND (returns micros unchanged). Both interpreted and codegen route through DateTimeUtils.truncTimestamp, so checkExceptionInExpression validates both paths. grep confirms lines 513/540/542 are the only value - floorMod sites; no silent-overflow pattern remains.
|
pyspark failure unrelated, thanks, merging to master/4.x/4.2 (bug fix) |
…ath. ### What changes were proposed in this pull request? Fixes the bug introduced by #55610. When `local` is a very large negative value, `Math.floorMod(local, unitMicros)` is positive because `unitMicros` is positive, and the subtraction can overflow to positive values. This produces a inconsistent result from the slow path. In most common cases, this optimization still takes effect. The performance loss is ignorable. ### Why are the changes needed? Fix incorrect result. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? New unit tests. It passes before #55610, or after this change. But it fails on the current master. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #56313 from chenhao-db/fix_trunc_overflow. Authored-by: chenhao-db <chenhao.li@databricks.com> Signed-off-by: Wenchen Fan <wenchen@databricks.com> (cherry picked from commit 94d23c3) Signed-off-by: Wenchen Fan <wenchen@databricks.com>
…ath. ### What changes were proposed in this pull request? Fixes the bug introduced by #55610. When `local` is a very large negative value, `Math.floorMod(local, unitMicros)` is positive because `unitMicros` is positive, and the subtraction can overflow to positive values. This produces a inconsistent result from the slow path. In most common cases, this optimization still takes effect. The performance loss is ignorable. ### Why are the changes needed? Fix incorrect result. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? New unit tests. It passes before #55610, or after this change. But it fails on the current master. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #56313 from chenhao-db/fix_trunc_overflow. Authored-by: chenhao-db <chenhao.li@databricks.com> Signed-off-by: Wenchen Fan <wenchen@databricks.com> (cherry picked from commit 94d23c3) Signed-off-by: Wenchen Fan <wenchen@databricks.com>
|
@chenhao-db, the added test is flaky on JDK 25 - likely passes when run this single tests, but likely fails when run the whole suite https://github.com/apache/spark/actions/runs/27185554437/job/80255424191
See #54514 for more details. |
What changes were proposed in this pull request?
Fixes the bug introduced by #55610. When
localis a very large negative value,Math.floorMod(local, unitMicros)is positive becauseunitMicrosis positive, and the subtraction can overflow to positive values. This produces a inconsistent result from the slow path.In most common cases, this optimization still takes effect. The performance loss is ignorable.
Why are the changes needed?
Fix incorrect result.
Does this PR introduce any user-facing change?
No.
How was this patch tested?
New unit tests. It passes before #55610, or after this change. But it fails on the current master.
Was this patch authored or co-authored using generative AI tooling?
No.