Skip to content

[fix](external) Fix Arrow and external timestamp semantics - #67784

Open
Gabriel39 wants to merge 40 commits into
apache:branch-4.1from
Gabriel39:fix/arrow-write-timestamp-semantics
Open

Gabriel39 wants to merge 40 commits into
apache:branch-4.1from
Gabriel39:fix/arrow-write-timestamp-semantics

Conversation

@Gabriel39

@Gabriel39 Gabriel39 commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Issue Number: DORIS-28352, DORIS-28353

Related PRs: #66891, #65446

Problem Summary:

This PR is based on #66891 and completes the timestamp and binary type contracts needed by Iceberg and Paimon on branch-4.1.

It fixes the Arrow writer and external timestamp semantics:

  • Allow timezone-free Arrow timestamps only for Doris DATETIMEV2; TIMESTAMPTZ continues to require instant semantics.
  • Canonicalize the UTC Z alias to UTC in Arrow timestamp metadata so Arrow clients can resolve the timezone.
  • Validate nested Array, Map, and Struct bindings recursively against their declared timestamp representation.
  • Map Doris DATETIMEV2(p) to Paimon TIMESTAMP(p) and Doris TIMESTAMPTZ(p) to Paimon TIMESTAMP_LTZ(p) without losing precision.
  • Preserve Variant's physical STRUCT when applying request-level logical timestamp types.
  • Write Parquet INT64 DATETIMEV2 as a timezone-naive logical timestamp and TIMESTAMPTZ as an adjusted-to-UTC timestamp.
  • Keep versioned Parquet timestamp plans, including Paimon native Parquet ranges hidden behind scan-level FORMAT_JNI, on scanner V2 even when enable_file_scanner_v2=false. Scanner V1 cannot honor their explicit wall-clock contract.
  • Honor an explicit Hive timezone even when an intermediate FE omits the newer version marker; keep the Hive sink timezone field separate from the existing Azure multipart field ID.
  • Retain nested timestamp semantic projection metadata when another consumer requests the whole physical parent, preventing a TIMESTAMP_LTZ child from being materialized as DATETIMEV2.
  • Use the same configured timezone for Python UDF Arrow schemas and encoded values. Preserve the explicitly declared timezone label (such as +08:00) instead of exporting cctz internal Fixed/... names, so UDF/UDTF/UDAF batches agree with Python protocol metadata. Parquet, Hive and Iceberg retain their declared schema timezone labels too.
  • Construct Arrow convertors per writer or call with explicit schema and timezone parameters. Parquet, Hive, Iceberg, Paimon, Python and Arrow Flight select their own classes; there is no global singleton or default Arrow Flight fallback.
  • Build Parquet/Hive and Iceberg Arrow schemas inside their convertors, and decode the pinned Paimon schema inside its convertor. Writers consume the same instance-owned schema used for batch conversion.

It also makes catalog binary mappings binary-safe:

  • Map Iceberg BINARY and UUID, Paimon BINARY and VARBINARY, Hive binary, and JDBC binary types directly to Doris VARBINARY.
  • Remove the catalog enable.mapping.varbinary behavior; the property no longer changes catalog type mapping.
  • Keep the file TVF enable_mapping_varbinary option as an explicit opt-in, defaulting to false, because changing an ad-hoc TVF result schema also changes CTAS type inference.
  • Preserve SQL NULL separately from text and empty binary bytes in full-static and hybrid Iceberg partition overwrites.
  • Preserve arbitrary byte values instead of exposing catalog binary columns as UTF-8 STRING, preventing Arrow Flight clients from rejecting valid binary payloads.

It ports the timestamp correctness fixes from #65446 to the direct scanner architecture used by branch-4.1:

  • Preserve Parquet INT96 wall-clock values for versioned plans and support the explicit hive.parquet.time-zone property for HMS catalogs and file TVFs.
  • Map Hudi Avro instant timestamps to TIMESTAMPTZ and local-timestamp annotations to DATETIMEV2. Prevent Iceberg/Paimon tables discovered through HMS from inheriting the Hive timezone override.
  • Preserve Paimon TIMESTAMP versus TIMESTAMP_LTZ semantics, including nested filter-only projections backed by unannotated INT96.
  • In scanner V2, round ORC timestamp nanoseconds consistently, apply TIMESTAMP_INSTANT carry before timezone conversion, reject malformed nanoseconds as data errors, and keep timestamp statistics/SARG pruning conservative.
  • Write Parquet INT64 logical timestamps by default while retaining explicit enable_int96_timestamps=true support.
  • Carry the required fixed-offset timezone detection and normalization helpers into branch-4.1.

External-to-Doris type mapping changes

The comparisons below are against the pre-PR branch-4.1 implementation, not an intermediate revision of this PR. Previously, the catalog property enable.mapping.varbinary defaulted to false. The new catalog mappings apply even when that property is omitted or explicitly set to false.

Catalog data columns

External source External type Before: catalog flag false (default) Before: catalog flag true After this PR
Hive/HMS type mapper BINARY (non-partition data column) STRING VARBINARY Always VARBINARY
Iceberg BINARY STRING VARBINARY Always VARBINARY
Iceberg FIXED(N) CHAR(N) VARBINARY(N) Always VARBINARY(N)
Iceberg UUID STRING VARBINARY(16) Always VARBINARY(16)
Paimon BINARY(N) STRING VARBINARY(N) Always VARBINARY(N)
Paimon VARBINARY(N) STRING VARBINARY(N) Always VARBINARY(N)
JDBC MySQL BINARY, VARBINARY STRING VARBINARY(N) Always VARBINARY(N)
JDBC MySQL TINYBLOB, BLOB, MEDIUMBLOB, LONGBLOB STRING VARBINARY(N) Always VARBINARY(N)
JDBC PostgreSQL BYTEA STRING VARBINARY(N) Always VARBINARY(N)
JDBC Oracle BLOB STRING VARBINARY(N) Always VARBINARY(N)
JDBC SQL Server BINARY, VARBINARY, IMAGE STRING VARBINARY(N) Always VARBINARY(N)
JDBC DB2 BLOB STRING VARBINARY(N) Always VARBINARY(N)

Here, N is a byte length: the declared external width for Iceberg/Paimon and requiredColumnSize() from JDBC metadata for the listed JDBC mappings. Unqualified VARBINARY above denotes the maximum-length binary mapping. This list names the changed JDBC branches explicitly; it does not imply that every type in every JDBC connector is remapped.

Timestamp mappings

The timestamp mapping option previously defaulted to false. Both catalog enable.mapping.timestamp_tz and file TVF enable_mapping_timestamp_tz are now compatibility inputs only: omitting them or specifying false cannot change the mapping. Precision is capped at six fractional digits.

External source External type Before this PR After this PR
Hive/HMS TIMESTAMP WITH LOCAL TIME ZONE DATETIMEV2(6) by default; TIMESTAMPTZ(6) with the option TIMESTAMPTZ(6)
Iceberg timestamp with time zone / adjust-to-UTC timestamp DATETIMEV2(6) by default; TIMESTAMPTZ(6) with the option TIMESTAMPTZ(6)
Paimon TIMESTAMP_LTZ(p) DATETIMEV2(min(p,6)) by default; TIMESTAMPTZ with the option TIMESTAMPTZ(min(p,6))
JDBC MySQL TIMESTAMP(p) DATETIMEV2(p) by default; TIMESTAMPTZ with the option TIMESTAMPTZ(p)
JDBC PostgreSQL timestamptz DATETIMEV2 by default; TIMESTAMPTZ with the option TIMESTAMPTZ, preserving supported precision
JDBC Oracle TIMESTAMP(p) WITH LOCAL TIME ZONE DATETIMEV2 by default; TIMESTAMPTZ with the option TIMESTAMPTZ(min(p,6))
JDBC Oracle TIMESTAMP(p) WITH TIME ZONE Unsupported TIMESTAMPTZ(min(p,6))
JDBC SQL Server datetimeoffset(p) STRING TIMESTAMPTZ(min(p,6))
JDBC Trino and Trino connector timestamp(p) with time zone DATETIMEV2(min(p,6)) TIMESTAMPTZ(min(p,6))
JDBC ClickHouse DateTime, DateTime64(p), including column/server timezone variants DATETIMEV2 TIMESTAMPTZ(0) / TIMESTAMPTZ(min(p,6)); these types store epoch instants
Hudi Avro timestamp-millis, timestamp-micros DATETIMEV2(3/6) TIMESTAMPTZ(3/6)
Hudi Avro local-timestamp-millis, local-timestamp-micros BIGINT DATETIMEV2(3/6)
MaxCompute TIMESTAMP DATETIMEV2(6) TIMESTAMPTZ(6)
File TVF Parquet / ORC UTC-adjusted logical timestamp / TIMESTAMP_INSTANT DATETIMEV2 by default; TIMESTAMPTZ with the option TIMESTAMPTZ at the existing supported precision
Hive, Iceberg, Paimon, JDBC, MaxCompute and file TVFs Corresponding timezone-free timestamp, TIMESTAMP_NTZ, MySQL DATETIME, Parquet timestamp with isAdjustedToUTC=false, ORC TIMESTAMP DATETIMEV2 Unchanged: DATETIMEV2, never TIMESTAMPTZ
File TVF Parquet Unannotated INT96 DATETIMEV2 by default; TIMESTAMPTZ when the option was enabled DATETIMEV2(6): bare INT96 cannot declare timezone semantics; a table-format logical schema can still identify an instant

Nested timestamp leaves follow the same rules. DATE and TIME are not remapped by this policy. SQL Server TIMESTAMP is a binary rowversion, not a timestamp; its existing mapping is unchanged.

The read/write paths are updated together with metadata: JNI instant carriers use UTC components, ClickHouse scalar and array timestamps are projected as epoch microseconds (including Query TVFs), MaxCompute instant writes retain microseconds, and Iceberg defaults preserve offsets. TIMESTAMPTZ preserves the instant, not the source's original zone identifier; its displayed offset follows the Doris session timezone.

JDBC predicates containing TIMESTAMPTZ literals, or calendar functions/casts involving TIMESTAMPTZ columns, are evaluated locally until the remote timezone semantics can be preserved. Timezone-independent direct NULL checks remain eligible for pushdown. This can increase scanned rows and also prevent limit pushdown. Trino connector domains retain UTC and fractional precision and reject non-representable short-timestamp endpoints.

Partition, nested-type and TVF boundaries

Scope Before this PR After this PR
Hive/HMS BINARY partition column With the catalog flag false: VARCHAR(MAX_VARCHAR_LENGTH); with true: VARBINARY VARCHAR(MAX_VARCHAR_LENGTH) in both cases, preserving the HMS text partition-name/literal contract
Binary leaves inside supported Hive ARRAY/MAP/STRUCT, Iceberg LIST/MAP/STRUCT, and Paimon ARRAY/MAP/ROW The nested binary leaf used the corresponding flag-dependent mapping above The nested binary leaf uses the mandatory binary mapping above; container types and unrelated leaves are unchanged
File TVF binary inference, such as Parquet/ORC through file TVFs Separate enable_mapping_varbinary option, default false; binary inference was opt-in Unchanged: the separate option remains effective and defaults to false; catalog migration does not enable it

Compatibility and migration

  • This is not merely a new default. Catalog STRING/CHAR fallback for the listed binary types can no longer be selected. New catalog creation and ALTER requests normalize the compatibility marker to true, including requests that supply false.
  • After metadata replay, a new-version FE becoming Master journals an ordinary ALTER CATALOG PROPERTIES record setting enable.mapping.varbinary=true and/or enable.mapping.timestamp_tz=true for each existing external catalog whose corresponding marker is absent or not true. The migration runs before query readiness/checkpoints; already-enabled catalogs are skipped. Followers replay the same durable change.
  • This migration changes Doris catalog metadata, not external table definitions or file contents. It does not rewrite existing native Doris STRING columns. Because the marker is persisted, rolling back FE binaries alone does not restore an earlier false setting.
  • Applications can observe different external column/result types. Existing binary scalar functions and ordinary views retain execution-layer VARBINARY. Binary transport does not imply support for binary predicates or computation; even an equality predicate can be rewritten into an unsupported hash-based path. Binary hash keys (including grouping and hash joins), column/tablet-routing hashes, IN/NOT IN, min/max, and Iceberg bucket/truncate transforms return explicit unsupported errors. Binary storage/runtime predicate factories reject the type, and binary IN does not materialize zone-map, dictionary or Bloom predicates. This PR does not add an OLAP storage type or change native-table type mappings: direct native VARBINARY columns, CTAS and MTMV materialization remain unsupported. Explicitly convert bytes to a supported native type before materializing; hexadecimal STRING encoding is reversible. Existing native tables are not migrated.
  • Binary collection kernels without VARBINARY dispatch (collect_set, array membership/position/distinct/remove/set operations and their related hash-based functions) are rejected during analysis with a clear unsupported-type error. Byte-agnostic array construction/element access and collect_list remain available; binary values are never silently coerced to text to enable a kernel.
  • The timestamp policy is semantic, not a blanket conversion of every timestamp: zoned/instant types map to TIMESTAMPTZ, while timezone-free types map to DATETIMEV2. Old false options no longer select a wall-clock fallback for instants.
  • MySQL and OceanBase MySQL-mode execution connections explicitly use a UTC session and UTC Calendar for TIMESTAMPTZ reads/writes. Remote session-sensitive expressions consequently execute in UTC; DATETIME column reads retain wall-clock fields. This prevents Connector/J connection/JVM timezone mismatches from shifting instants.
  • ClickHouse scalar and nested timestamps cross JDBC as epoch microseconds. JDBC v1 can otherwise collapse the two instants in a DST overlap, even when requesting ZonedDateTime.
  • PostgreSQL timestamptz arrays, including nested arrays and NULL elements, are converted to UTC instant carriers before JNI materialization.
  • SQL Server datetimeoffset reads use the offset-preserving getTimestamp path on both legacy and current drivers, without exception-based capability probing. Writes bind an explicit UTC ISO timestamp to avoid JVM-local datetime2 conversion and preserve microseconds.
  • Iceberg TIMESTAMPTZ identity/bucket/year/month/day/hour partitions use UTC instants. Static and dynamic writes, overwrite/delete metadata, negative epochs, and DST overlaps preserve the same partition value. Timestamp buckets hash microseconds, and temporal partition ordinals floor to the containing calendar unit.
  • Iceberg-specific Parquet schema conversion and statistics collection are isolated in VIcebergParquetWriter; generic Parquet writes retain their existing timestamp contract.
  • Newly written Iceberg ORC timestamp statistics retain sub-millisecond precision and conservative upper bounds, preventing equality filters from skipping rows that are visible to an unfiltered scan. Previously written incorrect manifest statistics are not repaired automatically.
  • The bundled ORC writer cannot losslessly encode timestamp fractions in the epoch interval [-0.999, 0) seconds (ORC-645). DATETIMEV2/TIMESTAMPTZ ORC writes now fail explicitly for that interval rather than silently producing positive timestamps; use Parquet for these values. Other negative timestamps remain supported. Existing files are not rewritten.

Release note

External catalog binary types now map directly to Doris VARBINARY. The former enable.mapping.varbinary catalog property no longer enables STRING fallback behavior. File TVFs retain the separate enable_mapping_varbinary opt-in and continue to default to STRING.

Arrow timestamp schemas now publish the UTC timezone as UTC instead of Z for client compatibility.

Parquet Export and Outfile now write INT64 logical timestamps by default: DATETIMEV2 is timezone-naive and TIMESTAMPTZ is adjusted to UTC. Set enable_int96_timestamps=true for readers that require INT96. Versioned scanner plans preserve raw INT96 wall-clock values by default, and hive.parquet.time-zone can describe Hive files normalized by a known writer timezone. External instant types now always map to TIMESTAMPTZ, including Hudi, Iceberg and Paimon; local timestamp types retain DATETIMEV2 semantics. Scanner V2 rounds ORC timestamp nanoseconds to microseconds; instant carry is applied before timezone conversion, and malformed fractional values fail the scan without terminating the BE process.

Check List (For Author)

  • Test
    • Focused BE unit tests
    • ASAN BE test executable build
    • clang-format 16
    • Full CI and SQL regression execution

Validation:

  • The converter refactor builds successfully under ASAN. Its 50 focused BE tests cover Arrow column bindings, Variant output, Parquet/Hive/Iceberg writers, internal schema initialization, malformed Paimon schema input, and instance-local timestamp bindings. The fixed-offset regression tests failed before the fix and passed afterward. clang-format 16 passed for all 16 C/C++ files affected by the timezone-label fix.

  • The earlier binary computation and review fixes passed 354 focused ASAN BE tests; one temporary-directory permission conflict was resolved by rerunning with an isolated directory.

  • Confirmed five regression tests failed before the fixes: malformed Paimon ARRAY/MAP arity and unsupported binary column hashing, hash-key selection, IN/NOT IN, and Iceberg bucket/truncate transforms.

  • Removed the new binary computation assertions from the SQL and Hive ORC regression cases. Retained coverage for byte transport, ordinary views, native materialization restrictions, nested binary values, and existing scalar functions. The long-value case verifies all 2,048 returned rows using integer-key filtering instead of a binary predicate. Both edited scripts compile with the regression framework's Groovy 4.0.19 version; end-to-end SQL reruns remain pending CI.

  • Standard CI and review are requested through separate run buildall and /review comments.

  • Behavior changed:

    • Yes
    • No
  • Does this need documentation?

    • Yes
    • No

User-facing documentation should remove catalog binary and timestamp fallback options, describe mandatory instant mappings and native materialization restrictions, and retain the TVF enable_mapping_varbinary opt-in.

@Gabriel39
Gabriel39 requested a review from yiguolei as a code owner September 10, 2026 08:10
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@Gabriel39

Copy link
Copy Markdown
Contributor Author

run buildall

@Gabriel39

Copy link
Copy Markdown
Contributor Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Found two blocking correctness and compatibility issues:

  • Request-local Parquet schema synchronization destroys the physical STRUCT contract used by native Variant readers.
  • The absent INT96 timezone field changes semantics across old and new BEs during a supported BE-first rolling upgrade.

Critical checkpoints:

  • Arrow and target-specific writers: traced recursive null/complex handling, UUID/FIXED/Variant metadata, timestamp units, converter ownership, and production call sites; no distinct writer defect survived.
  • Format-v2 readers: checked mapping, nested/filter/dictionary paths, statistics, caches, splits, and Hudi/Paimon routing; the two inline issues are the surviving defects.
  • ORC rounding and pushdown: checked half-up carry, negative time, DST, SARG envelopes, and aggregate/statistics fallbacks; no reachable additional issue survived.
  • FE, properties, and export: checked validation, catalog/TVF transport, lake-format exclusions, writer defaults, export persistence/replay, and mixed-version behavior; no distinct issue survived beyond the rolling-upgrade contract.
  • Tests and regressions: reviewed the relevant BE, FE, and regression changes plus generated output updates. No additional user focus was supplied. Tests were not run, per the review instructions.

Comment thread be/src/format_v2/parquet/reader/native_column_reader.cpp Outdated
Comment thread gensrc/thrift/PlanNodes.thrift Outdated
@hello-stephen

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 62.03% (49/79) 🎉
Increment coverage report
Complete coverage report

@Gabriel39

Copy link
Copy Markdown
Contributor Author

run buildall

@Gabriel39

Copy link
Copy Markdown
Contributor Author

/review

@github-actions

Copy link
Copy Markdown
Contributor

Codex automated review failed and did not complete.

Error: Invalid prompt: your prompt was flagged as potentially violating our usage policy. Please try again with a different prompt: https://platform.openai.com/docs/guides/reasoning#advice-on-prompting
Workflow run: https://github.com/apache/doris/actions/runs/34481885937

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

@Gabriel39

Copy link
Copy Markdown
Contributor Author

run buildall

@Gabriel39

Copy link
Copy Markdown
Contributor Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Found three blocking correctness and rolling-compatibility issues on the reviewed head:

  • Default INT64 Parquet output gives timezone-free DATETIMEV2 values instant semantics, so cross-timezone readers shift them.
  • The versioned INT96 contract is honored only by FileScannerV2; the supported V1 route returns different values for the same current-FE plan.
  • Upgraded BEs reject old-FE CHAR(N) plans for Iceberg fixed(N) writes.

Coverage: reviewed all 156 changed paths and the related Arrow writer, nested schema, Paimon history, Parquet V1/V2, ORC rounding/SARG, external binary, Export replay, and mixed-version paths through three convergence rounds. The earlier Variant physical-STRUCT and old-FE missing-marker threads are fixed on this head and were not duplicated. No additional user focus was specified, so the full PR review was used. Builds and tests were not run because this review runner explicitly disallows them.

Requesting changes until the three inline issues are addressed.

Comment thread be/src/format_v2/table_reader.cpp
Comment thread be/src/core/data_type_serde/data_type_string_serde.cpp
@Gabriel39

Copy link
Copy Markdown
Contributor Author

run buildall

@Gabriel39

Copy link
Copy Markdown
Contributor Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes for five substantiated issues on exact head 489a3c5aa8373bee9fbe9abe1708ddf72acb804b.

Findings:

  • P1: Paimon native-Parquet scans still select FileScannerV1 when enable_file_scanner_v2=false, bypassing the new per-column TIMESTAMP/TIMESTAMP_LTZ contract and returning shifted values.
  • P1: Mapping-disabled ORC TIMESTAMP_INSTANT rounding carries in local civil time across DST transitions, returning the wrong DATETIMEV2 value.
  • P1: The new plain Arrow-converter default deterministically breaks both pre-existing Iceberg Variant transformer tests; production Iceberg call sites do pass the target converter.
  • P2: An ORC value immediately below year zero that rounds into the supported boundary is rejected before its carry is applied.
  • P2: File-controlled ORC nanoseconds are checked with DORIS_CHECK, converting malformed data into a fatal-coded failure and a process-fatal path in debug or exit_on_exception configurations instead of a checked corruption status.

Critical checkpoints:

  • Architecture and lifecycle: Scanner selection, mixed native/JNI splits, request-local schemas, cache ownership, projection refresh, and teardown were traced. The Paimon scan-level/range-level format mismatch is the sole surviving scanner/lifecycle issue.
  • Schema mapping and materialization: INT96 absent/empty/named semantics, nested and dictionary readers, Paimon history annotations, Variant physical schemas, and ORC row conversion were checked. The Paimon and ORC row-value issues above remain.
  • Filtering, deletes, and pushdown: Parquet statistics do not accept INT96, row-group dictionary pruning is string-only, and the relevant non-fixed-zone ORC aggregate/SARG paths fall back. No additional false-pruning or delete-path issue survived.
  • Format and external compatibility: Iceberg/Paimon writer recursion, UUID/fixed/VARBINARY bindings, FE/BE rolling-version behavior, Parquet output defaults, EXPORT persistence/replay, Hive/Hudi/JDBC mappings, and TVF exceptions were checked. The existing old-FE Iceberg fixed thread was treated as a hard duplicate fence; no distinct production writer issue remains beyond the Paimon/ORC findings.
  • Performance and observability: Split/refinement, cache, prefetch, and profile-sensitive paths showed no material new issue.
  • Tests: The two Iceberg Variant tests fail by static control-flow proof, and the ORC tests omit the mapping-disabled DST carry, rounding-into-range lower bound, and malformed-nanoseconds cases. No build was run because the review contract explicitly prohibits builds; evidence is static review only.

User focus: no additional focus was provided. Two complete review rounds converged, including separate risk challenges, and all Round 2 reviewers returned NO_NEW_VALUABLE_FINDINGS beyond these accepted comments.

Comment thread be/src/exec/operator/file_scan_operator.cpp Outdated
Comment thread be/src/core/data_type_serde/data_type_serde.cpp Outdated
Comment thread be/src/format/transformer/vparquet_transformer.h Outdated
Comment thread be/src/core/data_type_serde/data_type_serde.cpp Outdated
Comment thread be/src/core/data_type_serde/data_type_serde.cpp Outdated
@hello-stephen

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 60.78% (62/102) 🎉
Increment coverage report
Complete coverage report

@Gabriel39

Copy link
Copy Markdown
Contributor Author

run buildall

@Gabriel39

Copy link
Copy Markdown
Contributor Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Request changes: two P1 correctness regressions remain.

  • ORC timestamp results are not scanner-independent: V2 now rounds/carries nanoseconds, while supported V1 query and load paths still truncate.
  • Explicit legacy INT96 output now uses a timezone-naive Arrow staging schema, but the reader still applies the writer timezone, shifting cross-zone round trips and contradicting the added regression oracle.

Critical checkpoints reviewed: Paimon native/JNI routing; full, partial, predicate-only, and refreshed Parquet projections; request-local schema isolation; Variant physical carriers; timestamp statistics, SARG and aggregate safety; target-specific nested Arrow conversion; the versioned absent/empty/named INT96 contract on the supported BE-first rollout; and catalog-wide binary mapping. No additional distinct issue survived those checks.

I did not repeat the plain-ORC year-zero boundary issue or old-FE Iceberg fixed-width writer compatibility issue because live threads already cover them. No additional user focus was supplied.

Tests were not run because this review environment explicitly prohibits builds/tests; conclusions are from static path tracing against the authoritative PR bundle and the changed tests/golden outputs.

Comment thread be/src/core/data_type_serde/data_type_serde.cpp Outdated
Comment thread be/src/format/transformer/vparquet_transformer.cpp Outdated
@Gabriel39

Copy link
Copy Markdown
Contributor Author

run buildall

@Gabriel39

Copy link
Copy Markdown
Contributor Author

/review

@Gabriel39

Copy link
Copy Markdown
Contributor Author

run buildall

@Gabriel39

Copy link
Copy Markdown
Contributor Author

/review

@hello-stephen

Copy link
Copy Markdown
Contributor

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 73.17% (1819/2486)
Line Coverage 60.95% (32545/53395)
Region Coverage 48.45% (30737/63438)
Branch Coverage 51.42% (9543/18560)

@github-actions

Copy link
Copy Markdown
Contributor

Codex automated review failed and did not complete.

Error: Codex completed, but no new pull request review was submitted for the current head SHA.
Workflow run: https://github.com/apache/doris/actions/runs/34581702727

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review summary for exact head f68ba57

Review opinion: no new actionable or blocking findings.

I reviewed the complete authoritative 162-file diff and deduplicated against every existing inline thread. The review converged after two rounds of full-coverage and risk-focused passes; every candidate was independently resolved, dismissed with code evidence, or fenced by existing review context.

Critical checkpoints covered:

  • ORC and Parquet timestamp row decoding, writer semantics, half-up rounding, bounds, named-zone/DST behavior, statistics, SARGs, page/footer metadata, dictionary and Bloom pruning, plus INT96 tri-state propagation and scanner routing.
  • Paimon historical TIMESTAMP/TIMESTAMP_LTZ annotations through nested and filter-only projections, native/JNI range selection, late-filter refresh, request-local schema mutation, mixed-plan marker handling, and forced Scanner V2 routing.
  • Plain, Iceberg, and Paimon Arrow adapters across nullable and nested ARRAY/MAP/STRUCT values, offsets/null maps, converter lifetimes, timestamp units/zones, UUID/fixed/VARBINARY, and Variant physical layouts.
  • FE catalog/TVF/JDBC mappings, Hive/Hudi/Iceberg/Paimon timezone contracts, OUTFILE/EXPORT INT64 versus INT96 selection, persisted export replay, irrelevant non-Parquet property handling, and old/new FE-BE compatibility paths.
  • Changed unit/regression tests and expected outputs, including explicit legacy-scope cases already discussed in existing threads.

The user focus file contained no additional focus points, so full PR scope was retained. Builds and tests were not run because the review instructions explicitly prohibited them; this conclusion is based on static code, control-flow, compatibility, and test-code review.

@Gabriel39

Copy link
Copy Markdown
Contributor Author

run buildall

@Gabriel39

Copy link
Copy Markdown
Contributor Author

/review

@Gabriel39

Copy link
Copy Markdown
Contributor Author

Review and CI follow-up:

  • Preserve the existing CDC-to-OLAP storage contract when JDBC source metadata now exposes VARBINARY. Binary values map to STRING targets, with binary primary keys following the existing VARCHAR key conversion. Added a regression test that failed before the fix.
  • Fix invalid BE test fixtures: nullable ARRAY elements, explicit predicate node kinds, Parquet predicate access paths with complete slot types, and independent named-timezone initialization. The array test also distinguishes SQL NULL from a Variant null value.
  • Align external/Python UDF regression expectations with the intended binary-safe and civil-time contracts. Binary materialization into OLAP uses explicit representations, and binary comparison fixtures explicitly choose text semantics. Retain result checks and V2 profile invariants across scanner/min-max/lazy switches.

Local validation passed: 698 related BE tests, 378 FE tests, FE Checkstyle, clang-format 16 for all 74 C/C++ files touched by the PR, and syntax parsing for all 12 updated Groovy suites. The INT96 writer test also passes in isolation.

The three unmuted FE failures from the previous CI run passed locally without relaxing their timeouts. The coverage build failed while starting an incompatible Node binary; the current TeamCity build configuration already uses the glibc-217 distribution. No shared CI configuration was changed as part of this follow-up.

Full external regression requires the new CI run; local validation does not establish that every pipeline is green.

@Gabriel39

Copy link
Copy Markdown
Contributor Author

run buildall

@Gabriel39

Copy link
Copy Markdown
Contributor Author

/review

@github-actions

Copy link
Copy Markdown
Contributor

Codex automated review failed and did not complete.

Error: Selected model is at capacity. Please try a different model.
Workflow run: https://github.com/apache/doris/actions/runs/35500433220

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

@hello-stephen

Copy link
Copy Markdown
Contributor

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 73.27% (1826/2492)
Line Coverage 61.07% (32855/53800)
Region Coverage 48.62% (31086/63943)
Branch Coverage 51.63% (9666/18720)

@Gabriel39

Copy link
Copy Markdown
Contributor Author

run buildall

@Gabriel39

Copy link
Copy Markdown
Contributor Author

/review

@github-actions

Copy link
Copy Markdown
Contributor

Codex automated review failed and did not complete.

Error: You've hit your usage limit. Visit https://chatgpt.com/codex/settings/usage to purchase more credits or try again at Sep 23rd, 2026 6:44 AM.
Workflow run: https://github.com/apache/doris/actions/runs/35501604799

The selected account is excluded until 2026-09-23T06:44:00Z. Please trigger /review again; another configured account may be available.

@github-actions

Copy link
Copy Markdown
Contributor

Codex automated review failed and did not complete.

Error: You've hit your usage limit. Visit https://chatgpt.com/codex/settings/usage to purchase more credits or try again at Sep 23rd, 2026 7:04 AM.
Workflow run: https://github.com/apache/doris/actions/runs/35500774890

The selected account is excluded until 2026-09-23T07:04:00Z. Please trigger /review again; another configured account may be available.

@Gabriel39

Copy link
Copy Markdown
Contributor Author

run buildall

@Gabriel39

Copy link
Copy Markdown
Contributor Author

/review

@github-actions

Copy link
Copy Markdown
Contributor

Codex automated review failed and did not complete.

Error: Selected model is at capacity. Please try a different model.
Workflow run: https://github.com/apache/doris/actions/runs/35507706200

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

@Gabriel39

Copy link
Copy Markdown
Contributor Author

run buildall

@Gabriel39

Copy link
Copy Markdown
Contributor Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Static review at exact head 1cb0641 (base 8d2018b).

Found one new P1 correctness/stability issue: the Paimon/Iceberg map Arrow writer dereferences a nullable key-column cast even when required map keys are represented by plain columns, which can crash a release BE. The inline comment points to the added helper and requests a guarded plain-key path plus regression coverage.

I traced the ORC timestamp SARG/zone-map paths, Parquet INT96 tri-state contract, nested/schema/cache ownership, catalog marker migration, JDBC/Hudi/MaxCompute/Trino transport, and Iceberg partition encoding. No additional changed-head defect was substantiated. The signed negative-offset boundary is already covered by the existing unresolved cast/parser thread and is not duplicated here.

This was a static-only review; no builds or tests were run.

DCHECK(nested_values_column.is_nullable());
const auto* keys_nullmap_data =
check_and_get_column<ColumnNullable>(nested_keys_column)->get_null_map_data().data();
const auto& offsets = map_column.get_offsets();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Guard required map keys before dereferencing the nullable cast

check_and_get_column<ColumnNullable>(nested_keys_column) is dereferenced unconditionally here, although map keys are not required to be nullable. DataTypeMap::create_column() and TableReader nullability alignment can produce a plain key column (for example a Paimon map whose key is non-nullable), so this returns null and crashes the BE in release builds before the row loop. Please handle plain keys without a null-map lookup (or normalize/validate the child and return a Status), and add a Paimon/Iceberg map writer test with a non-nullable key.

@hello-stephen

Copy link
Copy Markdown
Contributor

BE UT Coverage Report

Increment line coverage 86.30% (1776/2058) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 60.10% (26150/43508)
Line Coverage 44.92% (271066/603422)
Region Coverage 40.69% (214740/527784)
Branch Coverage 42.24% (99529/235621)

@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 88.04% (1811/2057) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 74.58% (31574/42333)
Line Coverage 58.93% (352767/598632)
Region Coverage 55.63% (294258/528982)
Branch Coverage 56.46% (132875/235361)

yiguolei pushed a commit that referenced this pull request Sep 22, 2026
)

### What problem does this PR solve?

This ports #68297 to `master`, preserving the first of five planned
extractions from #67784.

Binary `Field` values can retain references to released source storage,
and Hive binary text needs its own Base64 contract. TIMESTAMPTZ output
can lose historical offset seconds, format invalid NULL payloads, or
fail again while reporting a boundary cast error.

- Own long binary Field values while keeping short values inline.
Preserve execution type lengths and decoder bytes, and add Hive Base64
and hexadecimal decoding support.
- Explicitly reject unsupported binary hash keys, IN, aggregates,
predicates and computed partition transforms. Keep the existing FE
comparison/group/join restrictions and existing binary scalar functions.
Reject unsupported collection kernels in each function's legality check
before coercion.
- Preserve historical second offsets in both TIMESTAMPTZ formatting and
parsing. Skip masked NULL payloads, reject unrepresentable local years,
and preserve cast error/NULL behavior at boundaries.

Arrow convertor migration, Parquet/ORC semantics, external writer
changes and catalog mapping migration belong to the subsequent
extractions. This PR does not enable native VARBINARY storage.

### Master adaptation

- Retain the fixed-offset normalization and tests already present on
master.
- Use the current void-returning
`VInPredicate::_prepare_zonemap_min_max` interface in both the guard and
its test.
- Retain master header cleanup and existing timestamp-nanosecond tests.
- Retain the existing master binary-literal encoder and its StringView
input contract; the older std::string-based caller fix is not
applicable.

### Testing
- TIMESTAMPTZ regression follow-up: reproduced both binary-output and
stream-load failures using the master PR CI artifact, regenerated the
two snapshots through `run-regression-test.sh`, and passed both suites
in comparison mode from each branch checkout. Explicit `Asia/Shanghai`
session settings were verified with the server default session zone set
to UTC. Only historical offset seconds changed in the generated results.
- Function-local validation update: 18 FE tests passed with Checkstyle
enabled, covering direct legality checks, nested/mixed/variadic
VARBINARY arguments, both `collect_set` arities, supported ordinary
types, SQL analysis, and existing array rewrites. The new
direct-legality tests reproduced missing rejection before the change.

- BE ASAN build and **199 tests passed** across 17 suites using
`run-be-ut.sh`, including binary lifetime/SerDe/rejection, timestamp
parsing/casts, and existing Arrow/Variant serialization coverage.
- `VarBinaryUnsupportedCollectionTest`: **passed** (13 unsupported
expressions plus supported byte-preserving collection analysis). The FE
test reactor and repository Checkstyle passed after cleaning stale
branch build artifacts.
- Repository clang-format 16 check and build-header hygiene checks:
**passed**; 31 changed C++ source/header files.
- Groovy compilation of the three regression suites: **passed**. Live
SQL regression execution remains pending CI.
- clang-tidy was attempted but could not complete because master already
contains an unmatched `NOLINTEND` in `be/src/core/types.h`. A diagnostic
run with the compiler resource directory corrected reproduced that
blocker; the other reported findings in `column_varbinary.cpp` were
outside changed lines. This is not a clean clang-tidy result.

The focused BE test source list and local test/build settings were
restored before committing. No build configuration changes are included.

### Release note

Fix binary value lifetime and serialization, reject unsupported binary
computation paths, and preserve TIMESTAMPTZ historical offsets and
boundary error behavior.

### Check List (For Author)

- Test
- [x] Regression test (three self-checking suites added; execution
pending CI)
  - [x] Unit Test
- Behavior changed:
- [x] Yes. Binary rejection and timestamp boundary behavior are
described above.
- Does this need documentation?
- [x] No. This fixes existing type behavior without introducing a
configuration option.

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label


### Scoped review follow-up

This follow-up only fixes correctness/stability defects introduced by
this PR. Compatibility preservation, pre-existing limitations,
additional VARBINARY computation/validation, performance refactors, and
test-style-only rewrites are excluded.

- Separate historical TIMESTAMPTZ wire-offset parsing from session
fixed-zone limits in both parser paths.
- Use UTC diagnostics for the TIMESTAMP_NS cast/comparison failures
affected by the new local-year formatting exception.
- Validation: 31 focused ASAN BE tests passed, including ordinary
DATE/DATETIME parsing. Three targeted tests failed before the fixes.
clang-format 16 and build hygiene passed. Full clang-tidy remains
affected by pre-existing diagnostics.
- Branch-specific UTC/GMT normalization and FE folding fixes are handled
in #68297; the corresponding master behavior predates this PR or already
defers folding.

### CI test follow-up

- Keep the binary literal test's owning Field alive while reading its
StringView. Branch-4.1 now has the corresponding short/long embedded-NUL
coverage using its execution API.
- Replace the obsolete +15:00 rejection input with +24:00. Add generated
historical-offset checks in both cast modes; all prior snapshot results
are unchanged.
- Validation: 34 focused ASAN BE tests passed on each branch. The
lifetime error and the original SQL mismatch were reproduced. The
complete cast regression suite passed in comparison mode from both
branch checkouts against the reported master CI artifact. clang-format
16 passed; full clang-tidy remains blocked by pre-existing diagnostics.

This follow-up changes tests only and retains the agreed scope: no
compatibility work or additional binary computation support. Existing
muted failures are outside this fix.
Gabriel39 added a commit that referenced this pull request Sep 22, 2026
)

### What problem does this PR solve?

This is the first of five planned extractions from #67784, targeting
`branch-4.1`.

Binary `Field` values can retain references to released source storage,
and Hive binary text needs its own Base64 contract. TIMESTAMPTZ output
can lose historical offset seconds, format invalid NULL payloads, or
fail again while reporting a boundary cast error.

- Own long binary Field values while keeping short values inline.
Preserve execution type lengths and decoder bytes, fix binary literal
encoding, and add Hive Base64 and hexadecimal decoding support.
- Explicitly reject unsupported binary hash keys, IN, aggregates,
predicates and computed partition transforms. Keep the existing FE
comparison/group/join restrictions and existing binary scalar functions.
Reject unsupported collection kernels before coercion.
- Normalize fixed timezone offsets and preserve historical second
offsets in both TIMESTAMPTZ formatting and parsing. Skip masked NULL
payloads, reject unrepresentable local years, and preserve cast
error/NULL behavior at boundaries.

Arrow convertor migration, Parquet/ORC semantics, external writer
changes and catalog mapping migration belong to the subsequent
extractions. This PR does not enable native VARBINARY storage.

### Testing
- TIMESTAMPTZ regression follow-up: reproduced both binary-output and
stream-load failures using the master PR CI artifact, regenerated the
two snapshots through `run-regression-test.sh`, and passed both suites
in comparison mode from each branch checkout. Explicit `Asia/Shanghai`
session settings were verified with the server default session zone set
to UTC. Only historical offset seconds changed in the generated results.
- Function-local validation update: 17 FE tests passed after a clean
build with Checkstyle enabled. Coverage includes direct legality checks,
nested/mixed/variadic VARBINARY arguments, both `collect_set` arities,
supported ordinary types, SQL analysis, and existing array rewrites.
Collection restrictions now live in each function's legality check
before coercion; existing branch-specific argument rules are preserved.

- Rebuilt the BE ASAN test target from this extraction: **184 tests
passed**, zero failures. Coverage includes binary
lifetime/SerDe/rejection paths, timestamp parsing/casts, hash and
partition guards, and existing Arrow/Variant serialization tests.
- `VarBinaryUnsupportedCollectionTest`: **passed** (13 unsupported
collection expressions, plus existing byte-preserving array/collection
analysis).
- FE reactor `validate` with repository Checkstyle: **passed**.
- clang-format 16 check on all 34 changed C++ source/header files:
**passed**.
- Groovy compilation of the three new regression suites: **passed**.
Live SQL regression execution is pending CI.

The local BE test source list was narrowed for the focused build and
restored before committing. No build configuration changes are included.

### Release note

Fix binary value lifetime and serialization, reject unsupported binary
computation paths, and preserve TIMESTAMPTZ historical offsets and
boundary error behavior.

### Check List (For Author)

- Test
- [x] Regression test (three self-checking suites added; execution
pending CI)
  - [x] Unit Test
- Behavior changed:
- [x] Yes. Binary rejection and timestamp boundary behavior are
described above.
- Does this need documentation?
- [x] No. This fixes existing type behavior without introducing a
configuration option.

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label

### Scoped review follow-up

This follow-up only fixes correctness/stability defects introduced by
this PR. Compatibility preservation, pre-existing limitations,
additional VARBINARY computation/validation, and unrelated refactors are
excluded.

- Separate historical TIMESTAMPTZ wire-offset parsing from session
fixed-zone limits in both parser paths.
- Validate the complete UTC/GMT fixed offset and exclude rejected
endpoint values from the timezone cache.
- Decline FE string folding when the session-local year is outside the
new BE display range. Preserve the CAST for BE evaluation in both cast
modes instead of folding non-strict casts to NULL.
- Validation: 29 focused ASAN BE tests and 15 FE tests passed. Four BE
tests and the new FE boundary test failed before the fixes. clang-format
16 and FE Checkstyle passed.
- The corresponding master follow-up is in #68301. Master already has
different timezone normalization and FE folding behavior; its additional
TIMESTAMP_NS error-reporting fix does not apply to branch-4.1.

### CI test follow-up

- Keep the binary literal test's owning Field alive while reading its
StringView. Branch-4.1 now has the corresponding short/long embedded-NUL
coverage using its execution API.
- Replace the obsolete +15:00 rejection input with +24:00. Add generated
historical-offset checks in both cast modes; all prior snapshot results
are unchanged.
- Validation: 34 focused ASAN BE tests passed on each branch. The
lifetime error and the original SQL mismatch were reproduced. The
complete cast regression suite passed in comparison mode from both
branch checkouts against the reported master CI artifact. clang-format
16 passed; full clang-tidy remains blocked by pre-existing diagnostics.

This follow-up changes tests only and retains the agreed scope: no
compatibility work or additional binary computation support. Existing
muted failures are outside this fix.
yiguolei pushed a commit that referenced this pull request Sep 23, 2026
)

### What problem does this PR solve?

This is the first of five planned extractions from #67784, targeting
`branch-4.1`.

Binary `Field` values can retain references to released source storage,
and Hive binary text needs its own Base64 contract. TIMESTAMPTZ output
can lose historical offset seconds, format invalid NULL payloads, or
fail again while reporting a boundary cast error.

- Own long binary Field values while keeping short values inline.
Preserve execution type lengths and decoder bytes, fix binary literal
encoding, and add Hive Base64 and hexadecimal decoding support.
- Explicitly reject unsupported binary hash keys, IN, aggregates,
predicates and computed partition transforms. Keep the existing FE
comparison/group/join restrictions and existing binary scalar functions.
Reject unsupported collection kernels before coercion.
- Normalize fixed timezone offsets and preserve historical second
offsets in both TIMESTAMPTZ formatting and parsing. Skip masked NULL
payloads, reject unrepresentable local years, and preserve cast
error/NULL behavior at boundaries.

Arrow convertor migration, Parquet/ORC semantics, external writer
changes and catalog mapping migration belong to the subsequent
extractions. This PR does not enable native VARBINARY storage.

### Testing
- TIMESTAMPTZ regression follow-up: reproduced both binary-output and
stream-load failures using the master PR CI artifact, regenerated the
two snapshots through `run-regression-test.sh`, and passed both suites
in comparison mode from each branch checkout. Explicit `Asia/Shanghai`
session settings were verified with the server default session zone set
to UTC. Only historical offset seconds changed in the generated results.
- Function-local validation update: 17 FE tests passed after a clean
build with Checkstyle enabled. Coverage includes direct legality checks,
nested/mixed/variadic VARBINARY arguments, both `collect_set` arities,
supported ordinary types, SQL analysis, and existing array rewrites.
Collection restrictions now live in each function's legality check
before coercion; existing branch-specific argument rules are preserved.

- Rebuilt the BE ASAN test target from this extraction: **184 tests
passed**, zero failures. Coverage includes binary
lifetime/SerDe/rejection paths, timestamp parsing/casts, hash and
partition guards, and existing Arrow/Variant serialization tests.
- `VarBinaryUnsupportedCollectionTest`: **passed** (13 unsupported
collection expressions, plus existing byte-preserving array/collection
analysis).
- FE reactor `validate` with repository Checkstyle: **passed**.
- clang-format 16 check on all 34 changed C++ source/header files:
**passed**.
- Groovy compilation of the three new regression suites: **passed**.
Live SQL regression execution is pending CI.

The local BE test source list was narrowed for the focused build and
restored before committing. No build configuration changes are included.

### Release note

Fix binary value lifetime and serialization, reject unsupported binary
computation paths, and preserve TIMESTAMPTZ historical offsets and
boundary error behavior.

### Check List (For Author)

- Test
- [x] Regression test (three self-checking suites added; execution
pending CI)
  - [x] Unit Test
- Behavior changed:
- [x] Yes. Binary rejection and timestamp boundary behavior are
described above.
- Does this need documentation?
- [x] No. This fixes existing type behavior without introducing a
configuration option.

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label

### Scoped review follow-up

This follow-up only fixes correctness/stability defects introduced by
this PR. Compatibility preservation, pre-existing limitations,
additional VARBINARY computation/validation, and unrelated refactors are
excluded.

- Separate historical TIMESTAMPTZ wire-offset parsing from session
fixed-zone limits in both parser paths.
- Validate the complete UTC/GMT fixed offset and exclude rejected
endpoint values from the timezone cache.
- Decline FE string folding when the session-local year is outside the
new BE display range. Preserve the CAST for BE evaluation in both cast
modes instead of folding non-strict casts to NULL.
- Validation: 29 focused ASAN BE tests and 15 FE tests passed. Four BE
tests and the new FE boundary test failed before the fixes. clang-format
16 and FE Checkstyle passed.
- The corresponding master follow-up is in #68301. Master already has
different timezone normalization and FE folding behavior; its additional
TIMESTAMP_NS error-reporting fix does not apply to branch-4.1.

### CI test follow-up

- Keep the binary literal test's owning Field alive while reading its
StringView. Branch-4.1 now has the corresponding short/long embedded-NUL
coverage using its execution API.
- Replace the obsolete +15:00 rejection input with +24:00. Add generated
historical-offset checks in both cast modes; all prior snapshot results
are unchanged.
- Validation: 34 focused ASAN BE tests passed on each branch. The
lifetime error and the original SQL mismatch were reproduced. The
complete cast regression suite passed in comparison mode from both
branch checkouts against the reported master CI artifact. clang-format
16 passed; full clang-tidy remains blocked by pre-existing diagnostics.

This follow-up changes tests only and retains the agreed scope: no
compatibility work or additional binary computation support. Existing
muted failures are outside this fix.
yiguolei pushed a commit that referenced this pull request Sep 23, 2026
…68396)

### What problem does this PR solve?

Related PR: #68381. This is the master version of the second split from
#67784, based on the primitives merged in #68301.

Arrow batch conversion mixes protocol serialization with table-specific
UUID handling, while writers construct schemas separately. Introduce
explicit Doris, Python, Arrow Flight, Parquet, Hive, Iceberg and Paimon
convertors with instance-owned schema parameters and timezone. Move
schema construction/decoding into the convertors and route nested SerDe
writes through the selected format.

Separate Parquet, Hive and Iceberg writers and migrate existing callers.
Preserve master's tracked Arrow memory pools, Iceberg statistics and
timestamp-nanosecond support. Master does not yet contain the Paimon
write backend or physical Variant table writes present on branch-4.1;
this pick adds the converter interfaces without importing those
features. Parquet timestamp encoding and external type mappings remain
unchanged.

Include the Python timezone regression correction from #68381: the
single string output uses ARRAY<STRING>, so the lateral-view comparison
reaches execution instead of failing on a STRUCT-versus-STRING
comparison. Retain coverage for four session timezones, microseconds,
pre-epoch values, NULLs, UDF, UDTF and UDAF.

### Release note

Fix Python UDF timestamp conversion to preserve wall-clock values when
the Arrow protocol declares a fixed-offset timezone.

### Check List (For Author)

- Test
- [x] Unit Test: explicit schemas and independent converter instances,
nested/null values, UUID and fixed binary bytes, timestamp bindings, and
Iceberg writer statistics.
- [x] Regression test: Python UDF/UDTF/UDAF timezone comparisons and the
existing timestamp snapshot corrections.
- Behavior changed:
- [x] Yes: align Python UDF conversion with its Arrow timezone
declaration; reject invalid nested bindings before casts.
- Does this need documentation?
    - [x] No.

Validation: ASAN BE build and 310 focused tests passed (53 suites),
covering Arrow conversion, Parquet/ORC, Variant SerDe and Python. All 42
affected C++ files passed clang-format 16, and header hygiene passed.
The original UDTF declaration reproduced the SQL analysis error on an
isolated FE; the corrected declaration passed the same analysis. Groovy
and embedded Python checks passed. Full Python SQL and external-catalog
regressions remain for CI. clang-tidy was attempted: the new converter's
size warning was resolved; analysis remains blocked by a pre-existing
unmatched NOLINTEND in core/types.h.

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
yiguolei pushed a commit that referenced this pull request Sep 23, 2026
…68381)

### What problem does this PR solve?

Related PR: #67784. This is the second split, based on the primitives
merged in #68297.

Arrow batch conversion mixes protocol serialization with table-specific
UUID and Variant handling, while writers construct schemas separately.
Introduce explicit Doris, Python, Arrow Flight, Parquet, Hive, Iceberg
and Paimon convertors, instantiated with their own schema parameters and
timezone. Move table schema construction/decoding into the convertors
and route nested SerDe writes through the selected format.

Separate Parquet, Hive and Iceberg writers and migrate all callers.
Preserve current Parquet timestamp encoding, UUID/Variant layouts and
external type mappings. Keep Python's numeric conversion timezone
consistent with its declared Arrow schema, and reject incompatible
nested target schemas before casts or child access. No FE, Thrift, ORC
timestamp, or binary computation changes are included.

### Release note

Fix Python UDF timestamp conversion to preserve wall-clock values when
the Arrow protocol declares a fixed-offset timezone.

### Check List (For Author)

- Test
- [x] Unit Test: schema ownership/isolation, slices, invalid schemas,
nested/null values, UUID/Variant bytes, fixed-offset Python batches, and
Parquet timestamp representation.
- [x] Regression test: add Python UDF/UDTF/UDAF timezone assertions and
extract the corresponding existing Python snapshot corrections from
#67784.
- Behavior changed:
- [x] Yes: align Python UDF conversion with its Arrow timezone
declaration; report invalid nested schema bindings as errors.
- Does this need documentation?
    - [x] No.

Validation: ASAN BE build and 396 selected tests passed; the final
incremental build and 285 focused tests also passed. clang-format 16 and
Groovy/embedded Python syntax checks passed. Python SQL and
external-catalog end-to-end regressions were not run locally and remain
for CI.

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants