Skip to content

[SPARK-56663][SQL][FOLLOWUP] Fix silent overflow in date_trunc fast path. - #56313

Closed
chenhao-db wants to merge 2 commits into
apache:masterfrom
chenhao-db:fix_trunc_overflow
Closed

chenhao-db wants to merge 2 commits into
apache:masterfrom
chenhao-db:fix_trunc_overflow

Conversation

@chenhao-db

@chenhao-db chenhao-db commented Jun 4, 2026 •

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

Fixes the bug introduced by #55610. When local is a very large negative value, Math.floorMod(local, unitMicros) is positive because unitMicros is positive, and the subtraction can overflow to positive values. This produces a inconsistent result from the slow path.

In most common cases, this optimization still takes effect. The performance loss is ignorable.

Why are the changes needed?

Fix incorrect result.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

New unit tests. It passes before #55610, or after this change. But it fails on the current master.

Was this patch authored or co-authored using generative AI tooling?

No.

@chenhao-db

Copy link
Copy Markdown
Contributor Author

@cloud-fan could you take a look? Thanks!

@cloud-fan cloud-fan left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

0 blocking, 1 non-blocking, 0 nits.
Clean, correct fix for the fast-path silent overflow.

Correctness (1)

  • DateTimeUtils.scala:513: same value - Math.floorMod(value, unit) pattern in the sibling MILLISECOND/SECOND branches (lines 540/542) still silently overflows at Long.MinValue — see inline

Verification

Traced the fix end to end: Math.subtractExact equals - for all non-overflowing inputs, and the only inputs it changes are those within one unitMicros of Long.MinValue, whose truncated-down instant is below Long.MinValue and so is unrepresentable by the slow path too. On underflow it throws ArithmeticException, caught by the existing catch (lines 524-525) and falls back to truncToUnit, which itself throws via instantToMicros's multiplyExact for the same input. So the fast path now fails consistently with the slow path rather than returning a wrapped-positive value — no input produces a wrong-but-representable result.

try {
val local = Math.addExact(micros, offsetMicros)
val truncatedLocal = local - Math.floorMod(local, unitMicros)
val truncatedLocal = Math.subtractExact(local, Math.floorMod(local, unitMicros))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nicely targeted fix. Note the same value - Math.floorMod(value, unit) shape lives one match away in the sibling branches: TRUNC_TO_MILLISECOND (line 540) and TRUNC_TO_SECOND (line 542) still use a plain -, so a timestamp at Long.MinValue micros silently wraps to a positive value there (e.g. MIN - floorMod(MIN, 1000) = 9223372036854775616) instead of throwing like the fast path now does. Those branches have no offset arithmetic and no slow-path fallback, so applying Math.subtractExact directly would make them throw ArithmeticException consistently. Worth folding into this PR (with MILLISECOND/SECOND added to the new test), or noting they're intentionally out of scope. Pre-existing, so non-blocking.

@chenhao-db
chenhao-db requested a review from cloud-fan June 4, 2026 03:15

@cloud-fan cloud-fan left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 addressed, 0 remaining, 1 new (1 late catch — my miss from last round).
0 blocking, 1 non-blocking, 0 nits.
Prior round's MILLISECOND/SECOND finding fully addressed. Clean, complete, well-tested overflow fix.

Suggestions (1)

  • PR description: the stated mechanism is inverted — "Math.floorMod(local, unitMicros) is also negative" is wrong; floorMod with a positive divisor is always non-negative. The real cause is subtracting the non-negative remainder from a value near Long.MinValue, which underflows. (If it were negative as written, local - negative would move toward zero and could not overflow.) The code is correct — only the description's explanation. Worth fixing as the rationale-of-record.

Verification

Confirmed the now-fixed SECOND/MILLISECOND branches: Math.floorMod(Long.MinValue, unit) is positive (192 for ms, 224192 for sec), so Math.subtractExact underflows -> ArithmeticException, propagating directly (these branches have no slow-path fallback). The new test covers all levels YEAR..MILLISECOND — date levels via daysToMicros's multiplyExact, MINUTE/HOUR/DAY via the truncToUnitFast -> truncToUnit -> instantToMicros fallback — and correctly excludes MICROSECOND (returns micros unchanged). Both interpreted and codegen route through DateTimeUtils.truncTimestamp, so checkExceptionInExpression validates both paths. grep confirms lines 513/540/542 are the only value - floorMod sites; no silent-overflow pattern remains.

@cloud-fan

Copy link
Copy Markdown
Contributor

pyspark failure unrelated, thanks, merging to master/4.x/4.2 (bug fix)

@cloud-fan cloud-fan changed the title [SPARK-56663][FOLLOWUP] Fix silent overflow in date_trunc fast path. [SPARK-56663][SQL][FOLLOWUP] Fix silent overflow in date_trunc fast path. Jun 4, 2026
@cloud-fan cloud-fan closed this in 94d23c3 Jun 4, 2026
cloud-fan pushed a commit that referenced this pull request Jun 4, 2026
…ath.

### What changes were proposed in this pull request?

Fixes the bug introduced by #55610. When `local` is a very large negative value, `Math.floorMod(local, unitMicros)` is positive because `unitMicros` is positive, and the subtraction can overflow to positive values. This produces a inconsistent result from the slow path.

In most common cases, this optimization still takes effect. The performance loss is ignorable.

### Why are the changes needed?

Fix incorrect result.

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

New unit tests. It passes before #55610, or after this change. But it fails on the current master.

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes #56313 from chenhao-db/fix_trunc_overflow.

Authored-by: chenhao-db <chenhao.li@databricks.com>
Signed-off-by: Wenchen Fan <wenchen@databricks.com>
(cherry picked from commit 94d23c3)
Signed-off-by: Wenchen Fan <wenchen@databricks.com>
cloud-fan pushed a commit that referenced this pull request Jun 4, 2026
…ath.

### What changes were proposed in this pull request?

Fixes the bug introduced by #55610. When `local` is a very large negative value, `Math.floorMod(local, unitMicros)` is positive because `unitMicros` is positive, and the subtraction can overflow to positive values. This produces a inconsistent result from the slow path.

In most common cases, this optimization still takes effect. The performance loss is ignorable.

### Why are the changes needed?

Fix incorrect result.

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

New unit tests. It passes before #55610, or after this change. But it fails on the current master.

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes #56313 from chenhao-db/fix_trunc_overflow.

Authored-by: chenhao-db <chenhao.li@databricks.com>
Signed-off-by: Wenchen Fan <wenchen@databricks.com>
(cherry picked from commit 94d23c3)
Signed-off-by: Wenchen Fan <wenchen@databricks.com>
@pan3793

pan3793 commented Jun 9, 2026

Copy link
Copy Markdown
Member

@chenhao-db, the added test is flaky on JDK 25 - likely passes when run this single tests, but likely fails when run the whole suite

[info] - TruncTimestamp of Long.MinValue overflows with ArithmeticException *** FAILED *** (65 milliseconds)
[info]   (unsafe mode) Expected `` but null error message found (ExpressionEvalHelper.scala:219)
[info]   org.scalatest.exceptions.TestFailedException:
[info]   at org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:472)
...

https://github.com/apache/spark/actions/runs/27185554437/job/80255424191

This is expected behavior. For "hot throws", the JIT will produce compiled code that does not go through the interpreter via deoptimization to throw an exception but throws a pre-allocated exception object without a stack trace or message.

See #54514 for more details.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants