Skip to content

[SPARK-57490][SQL] Support CAST between nanosecond timestamp types of different precision#56544

Open
MaxGekk wants to merge 3 commits into
apache:masterfrom
MaxGekk:nanos-cast-nanos
Open

[SPARK-57490][SQL] Support CAST between nanosecond timestamp types of different precision#56544
MaxGekk wants to merge 3 commits into
apache:masterfrom
MaxGekk:nanos-cast-nanos

Conversation

@MaxGekk

@MaxGekk MaxGekk commented Jun 16, 2026

Copy link
Copy Markdown
Member

What changes were proposed in this pull request?

This PR adds support for CAST between same-family nanosecond-precision timestamp types of different precision:

  • TIMESTAMP_NTZ(p1) <-> TIMESTAMP_NTZ(p2)
  • TIMESTAMP_LTZ(q1) <-> TIMESTAMP_LTZ(q2)

where the precisions are in [7, 9].

Both TimestampNTZNanosType and TimestampLTZNanosType share the same physical value TimestampNanosVal(epochMicros, nanosWithinMicro) (with nanosWithinMicro in [0, 999]). A cross-precision cast only re-floors the sub-microsecond part of the value; epochMicros and the time zone are untouched:

  • Widening (target precision >= source precision): lossless; the value is unchanged.
  • Narrowing (target precision < source precision): floors nanosWithinMicro toward the past to the target precision step (drops the lowest 9 - precision sub-microsecond digits), consistent with the existing nanos -> micros narrowing rule.

The store-assignment / up-cast contract follows the established micros <-> nanos precedent (SPARK-57293):

  • Widening (lossless) is allowed as an ANSI store assignment but is not an up-cast.
  • Narrowing (lossy) is explicit-CAST only (rejected by both up-cast and store assignment).
  • Equal precision is the identity cast.

Concretely, this adds the canCast / canAnsiCast / canANSIStoreAssign rules, the interpreted and codegen eval paths in Cast, and a DateTimeUtils.truncateTimestampNanosToPrecision helper reused by both paths.

In addition, this PR adds SQLQuery coverage in cast.sql for the SQL parser/typed-literal/:: cast surface of cross-precision nanos casts, while avoiding value-semantics duplication with CastSuite*.

Why are the changes needed?

The nanosecond-capable timestamp types TIMESTAMP_NTZ(p) and TIMESTAMP_LTZ(p) (p in [7, 9]) are gated behind spark.sql.timestampNanosTypes.enabled. Casts between these types and their microsecond counterparts (TIMESTAMP_NTZ / TIMESTAMP), as well as to/from DATE and STRING, are already supported. However, casting between two nanosecond timestamps of the same family but different precision was not allowed: such a cast was absent from Cast.canCast / Cast.canAnsiCast and failed type checking.

Does this PR introduce any user-facing change?

Yes, but only behind the preview flag spark.sql.timestampNanosTypes.enabled. With the flag enabled, CAST(... AS TIMESTAMP_NTZ(p)) / CAST(... AS TIMESTAMP_LTZ(p)) from another nanosecond timestamp of the same family is now supported instead of failing type checking.

How was this patch tested?

Added unit tests in CastSuiteBase covering:

  • all NTZ/LTZ precision pairs (widening is lossless, narrowing floors the sub-microsecond digits), including pre-epoch (negative-epoch) values and nulls;
  • the store-assignment / up-cast contract for cross-precision pairs;
  • null-cast coverage for cross-precision pairs.

Added SQLQuery tests in cast.sql (and regenerated goldens) for parser + typed-literal + :: cross-precision cast paths.

Ran:

  • build/sbt 'catalyst/testOnly *CastSuite *CastWithAnsiOnSuite *CastWithAnsiOffSuite' (355 tests passed)
  • build/sbt "sql/testOnly org.apache.spark.sql.SQLQueryTestSuite -- -z cast.sql" (6 tests passed)
  • build/sbt "sql-api/scalastyle" "catalyst/scalastyle" "catalyst/Test/scalastyle"

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Cursor

MaxGekk added 3 commits June 16, 2026 19:38
… different precision

### What changes were proposed in this pull request?

This PR adds support for `CAST` between same-family nanosecond-precision timestamp
types of different precision:
- `TIMESTAMP_NTZ(p1)` <-> `TIMESTAMP_NTZ(p2)`
- `TIMESTAMP_LTZ(q1)` <-> `TIMESTAMP_LTZ(q2)`

where the precisions are in [7, 9]. A cross-precision cast only re-floors the
sub-microsecond part of the value; `epochMicros` and the time zone are untouched.
Widening (target precision >= source) is lossless; narrowing floors the
sub-microsecond digits toward the past, consistent with the existing nanos ->
micros narrowing rule.

The store-assignment / up-cast contract follows the established micros <-> nanos
precedent: widening is allowed as an ANSI store assignment but is not an up-cast,
narrowing is explicit-CAST only, and equal precision is the identity cast.

### Why are the changes needed?

Casting between two nanosecond timestamps of the same family but different
precision was previously absent from `Cast.canCast` / `Cast.canAnsiCast` and
failed type checking, even though casts to/from the microsecond counterparts,
`DATE` and `STRING` were already supported.

### Does this PR introduce _any_ user-facing change?

Yes, but only behind the preview flag `spark.sql.timestampNanosTypes.enabled`.
With the flag enabled, `CAST(... AS TIMESTAMP_NTZ(p))` / `CAST(... AS
TIMESTAMP_LTZ(p))` from another nanosecond timestamp of the same family is now
supported.

### How was this patch tested?

Added unit tests in `CastSuiteBase` covering all NTZ/LTZ precision pairs
(widening, narrowing with floor semantics, pre-epoch values, nulls) and the
store-assignment / up-cast contract.

### Was this patch authored or co-authored using generative AI tooling?

Generated-by: Cursor
Format the newly added truncateTimestampNanosToPrecision method signature in
SparkDateTimeUtils to satisfy scalafmt in the sql/api module.
Add SQLQuery test coverage in cast.sql for TIMESTAMP_NTZ(p1) <-> TIMESTAMP_NTZ(p2)
and TIMESTAMP_LTZ(q1) <-> TIMESTAMP_LTZ(q2) using typed literals and :: syntax,
while avoiding duplication with CastSuite value-semantics assertions.
@MaxGekk MaxGekk force-pushed the nanos-cast-nanos branch from cb53c9c to 30dee82 Compare June 16, 2026 17:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant