[SPARK-56663][SQL] Restore fast path for date_trunc MINUTE/HOUR/DAY

[Licht-T]

Closes SPARK-56663.

What changes were proposed in this pull request?

This PR re-introduces a fast path in DateTimeUtils.truncTimestamp for TRUNC_TO_MINUTE, TRUNC_TO_HOUR, and TRUNC_TO_DAY. A new private truncToUnitFast(micros, zoneId, unitMicros, fallbackUnit) helper:

1. Resolves the zone offset for the input instant once (zoneId.getRules.getOffset(...)).
2. Truncates by Math.floorMod in local time: local = micros + offsetMicros, then truncatedLocal = local - floorMod(local, unitMicros), then back: candidate = truncatedLocal - offsetMicros.
3. Re-checks the offset at candidate and falls back to the existing truncatedTo()-based slow path if it differs from the original offset (the SPARK-30766 / SPARK-30857 DST guard).
4. The DST re-check is skipped for ZoneRules.isFixedOffset zones.
5. Math.addExact / subtractExact guard the offset arithmetic; on overflow we also fall back to the slow path.

The arithmetic also handles sub-minute LMT offsets (e.g. America/Los_Angeles -07:52:58 pre-1883 - the SPARK-33404 case) and 30/45-minute zones (Asia/Kolkata +05:30, Asia/Kathmandu +05:45) without an offset-alignment guard, because the offset is applied as part of the floorMod.

Date-level units (WEEK/MONTH/QUARTER/YEAR + trunc(date, ...)) are not touched in this PR; they still go through microsToDays / daysToMicros. Those can follow up in a separate ticket.

Why are the changes needed?

SPARK-33404 (Nov 2020) fixed a correctness bug - date_trunc('minute', '1769-10-17 17:10:02') in America/Los_Angeles returned the original timestamp because the existing fast path ignored the time-zone offset and the LMT offset has second granularity. The fix routed MINUTE/HOUR/DAY through microsToInstant().atZone(zoneId).truncatedTo(unit), which allocates ZonedDateTime per row.

The author of that fix flagged the regression on the follow-up benchmark PR:

"Slow down is around 5.5 times" - MaxGekk, #30338

That regression has stood for ~5.5 years with no follow-up ticket. This PR restores most of the lost throughput without re-introducing the LMT bug, by applying the offset before/after the floorMod and verifying the offset is unchanged at the truncated candidate (handling DST transitions that motivated SPARK-30766 / SPARK-30857).

DateTimeBenchmark Truncation results, wholestage on, ns/row on a 12th Gen Intel i7-1260P:

level	baseline	this PR	speedup
MINUTE	136.0	69.5	1.96×
HOUR	136.2	69.8	1.95×
DAY	136.5	69.8	1.96×
DD	136.7	70.1	1.95×

Other levels are unchanged (within noise).

Does this PR introduce any user-facing change?

No. The output of date_trunc for MINUTE/HOUR/DAY is identical to master in all cases; only the internal implementation changes.

How was this patch tested?

• DateTimeUtilsSuite - all 63 tests pass, including:
  • SPARK-33404: test truncTimestamp when time zone offset from UTC has a granularity of seconds, which loops MINUTE/SECOND/MILLI/MICRO truncation across every available zone (covers historical LMT zones).
  • The existing truncTimestamp test, which loops HOUR/DAY/MINUTE/etc. across every available zone.
  • Five new tests added in this PR:
    • truncTimestamp with sub-hour zone offsets - Asia/Kolkata +05:30 and Asia/Kathmandu +05:45 HOUR/DAY truncation.
    • truncTimestamp across DST transitions - America/Los_Angeles spring-forward (2024-03-10) and fall-back (2024-11-03) for HOUR/DAY, exercising the offset-equality fallback.
    • SPARK-30766/30857: truncTimestamp before the epoch in HOUR/DAY - pre-1970 instant in America/Los_Angeles for HOUR and DAY.
    • truncTimestamp at America/Sao_Paulo midnight DST gap - 2018-11-04 (the last Brazilian DST start, jumping from -3 to -2 at exactly midnight local), exercising the case where the local day boundary itself falls in a gap and DAY truncation must resolve forward.
    • truncTimestamp at Pacific/Apia after the 2011 calendar shift - mid-2012 instant in a +13:00 zone whose history skipped a calendar date, verifying the fast path under a non-DST historical offset change.
• DateExpressionsSuite - all 68 tests pass.
• DateTimeBenchmark re-run locally with the numbers shown above. Per the contributor guide, the official benchmarks/DateTimeBenchmark-results.txt should be regenerated on the standard CI runner - happy to do that via the GitHub Actions benchmark workflow on request.

Was this patch authored or co-authored using generative AI tooling?

Yes, co-authored by Claude Code.

[Licht-T]

@MaxGekk @cloud-fan friendly ping when you have a moment - this is a follow-up to SPARK-33404 that you originally fixed and merged. The slow path you introduced for correctness is preserved as the fallback; the new fast path applies the offset before/after the floorMod and verifies the offset is unchanged at the truncated candidate, so the LMT minute case (and the SPARK-30766 / SPARK-30857 hour/day cases) all still go through the safe path. ~2× speedup on MINUTE/HOUR/DAY locally, which recovers most of the regression flagged in #30338. Would appreciate your eyes.

[MaxGekk]

DateTimeBenchmark Truncation results, wholestage on, ns/row on a 12th Gen Intel i7-1260P:

@Licht-T Could you regenerate benchmark results:
./sql/core/benchmarks/DateTimeBenchmark-results.txt
./sql/core/benchmarks/DateTimeBenchmark-jdk25-results.txt
./sql/core/benchmarks/DateTimeBenchmark-jdk21-results.txt

[Licht-T]

Thanks @MaxGekk. Just ran the benchmarks on CI and pushed the results.

[cloud-fan]

Summary

What: Re-introduces a fast path in DateTimeUtils.truncTimestamp for TRUNC_TO_MINUTE / HOUR / DAY via offset arithmetic + floorMod, falling back to the existing ZonedDateTime.truncatedTo-based path when the truncated candidate crosses an offset transition. WEEK/MONTH/QUARTER/YEAR + trunc(date, ...) are deliberately deferred.

Why: SPARK-33404 (Nov 2020) routed these levels through microsToInstant().atZone(zoneId).truncatedTo(unit) to fix a correctness bug where the prior fast path floor-modded micros in UTC, silently assuming the offset was a whole multiple of the unit (so any sub-minute LMT offset was rounded away). The fix carried a ~5.5x regression that has stood ~5 years. Benchmark deltas in the result files match the PR description: ~125ns/row to ~56ns/row wholestage on for MINUTE/HOUR/DAY/DD; WEEK/MONTH/QUARTER/YEAR unchanged.

Approach — the key design move: the new algorithm shifts the boundary grid into the wall-clock frame, instead of leaving it in UTC. Concretely: resolve O0 = getOffset(micros) once; compute local = micros + O0 (the wall-clock at micros re-labelled as a UTC instant — a "shadow frame"); truncatedLocal = local - floorMod(local, unitMicros) is the previous wall-clock unit boundary; candidate = truncatedLocal - O0, then re-check getOffset(candidate) == O0 (skipped for fixed-offset zones). On mismatch — or Math.addExact/subtractExact overflow — fall back to the slow path.

This is the right structural move: because epoch (1970-01-01 00:00:00) is itself a wall-clock unit boundary and wall-clock unit boundaries recur exactly every unitMicros in the wall-clock frame, floorMod(local, unitMicros) is true wall-clock alignment and is independent of how O0 shifts the shadow zero relative to UTC zero. That is precisely why sub-second LMT offsets (e.g. -07:52:58) and 30/45-minute zones (Kolkata +05:30, Kathmandu +05:45) need no offset-alignment guard — the failure mode of the old fast path is structurally absent here.

Correctness guarantee of the offset re-check (traced by hand for LMT / LA spring-forward / LA fall-back / Sao_Paulo midnight gap / Pacific/Apia 2011 calendar shift):

• Stable offset (single valid offset for the truncated wall-clock W, equal to O0): candidate = W - O0 is exactly the slow path's result; re-check passes.
• Unambiguous W with a valid offset O_W != O0: O(candidate) = O0 would imply wallClock(candidate) = W, contradicting W having sole valid offset O_W. So re-check necessarily fails -> fallback. (LA spring-forward DAY case.)
• Overlap W (fall-back DST): O0 is necessarily one of the two ambiguous offsets (since W is within unitMicros of wall(micros)), and ZonedDateTime.ofLocal(W, zone, preferredOffset=O0) picks the same instant as candidate.
• Gap W (forward DST): candidate = W - O0 always lands in a segment with offset != O0 (regardless of whether O0 is the pre- or post-offset), so re-check fails -> fallback, where the slow path's gap resolution kicks in. (Sao_Paulo midnight gap.)

So whenever the fast path returns directly, it returns the slow path's value byte-for-byte.

Implicit precondition: the algorithm is valid only for unitMicros that divide MICROS_PER_DAY (true for MINUTE/HOUR/DAY); it doesn't extend to WEEK/MONTH/QUARTER because those aren't fixed-width in micros. Worth stating in the Scaladoc — see inline comment.

LGTM modulo the inline comments (test coverage extension and a Scaladoc precondition note).

[cloud-fan]

The existing SPARK-33404 test at line 692 loops over ALL_TIMEZONES with the 1769-10-17 LMT input but only exercises TRUNC_TO_MINUTE/SECOND/MILLI/MICRO. With this PR, TRUNC_TO_HOUR and TRUNC_TO_DAY also flow through the same offset-arithmetic path, and the LMT (sub-minute offset) case is exactly the bug class that motivated the slow path. The new test here only covers LA at 1960 (no LMT). Adding two lines to the SPARK-33404 loop would extend coverage to every available zone at near-zero cost — the most direct regression guard against re-introducing the SPARK-33404 bug:

testTrunc(DateTimeUtils.TRUNC_TO_HOUR, "1769-10-17T17:00:00", inputTS.get, zid)
testTrunc(DateTimeUtils.TRUNC_TO_DAY, "1769-10-17T00:00:00", inputTS.get, zid)

[Licht-T]

Added these two lines into the SPARK-33404 test.

[cloud-fan]

The algorithm depends on unitMicros dividing MICROS_PER_DAY — that's what makes wall-clock unit boundaries align to multiples of unitMicros in the shifted-local frame (since epoch is itself a unit boundary). It's true for all three call sites today, but if someone later extends truncToUnitFast to a non-dividing unit (or a unit that isn't fixed-width in micros) it would silently misalign. Worth one sentence in the Scaladoc, e.g.:

unitMicros must evenly divide MICROS_PER_DAY; otherwise wall-clock unit boundaries do not align to multiples of unitMicros in the shifted-local frame and the floorMod truncation is unsafe.

[Licht-T]

Added the sentence into the Scaladoc.

[cloud-fan]

Minor: "2024-03-10 02:00 PDT does not exist" mislabels the gap. It's the wall-clock reading 02:00–02:59 that doesn't exist on 2024-03-10 in LA, regardless of label; 2024-03-10T02:00-07:00 is itself a well-formed UTC instant (08:00 UTC) — one hour before the transition, actually in PST.

Suggested change

	// Spring-forward in LA: 2024-03-10 02:00 PDT does not exist; 02:30 local maps to
	// 2024-03-10 03:30 PDT in wall-clock terms. Use an instant just after the transition
	// so HOUR/DAY truncation candidate falls into the pre-transition offset window.
	// Spring-forward in LA: local 02:00-02:59 doesn't exist on 2024-03-10
	// (01:59 PST jumps to 03:00 PDT). Pick 03:30 PDT just after the transition
	// so the HOUR/DAY truncation candidate falls into the pre-transition window.

[Licht-T]

Changed the comment as per your suggestion.

[Licht-T]

Thanks, @cloud-fan. Just addressed your comments!

[cloud-fan]

thanks, merging to master/4.x/4.2!