Conversation
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
|
run buildall |
|
/review |
There was a problem hiding this comment.
Found two blocking correctness and compatibility issues:
- Request-local Parquet schema synchronization destroys the physical STRUCT contract used by native Variant readers.
- The absent INT96 timezone field changes semantics across old and new BEs during a supported BE-first rolling upgrade.
Critical checkpoints:
- Arrow and target-specific writers: traced recursive null/complex handling, UUID/FIXED/Variant metadata, timestamp units, converter ownership, and production call sites; no distinct writer defect survived.
- Format-v2 readers: checked mapping, nested/filter/dictionary paths, statistics, caches, splits, and Hudi/Paimon routing; the two inline issues are the surviving defects.
- ORC rounding and pushdown: checked half-up carry, negative time, DST, SARG envelopes, and aggregate/statistics fallbacks; no reachable additional issue survived.
- FE, properties, and export: checked validation, catalog/TVF transport, lake-format exclusions, writer defaults, export persistence/replay, and mixed-version behavior; no distinct issue survived beyond the rolling-upgrade contract.
- Tests and regressions: reviewed the relevant BE, FE, and regression changes plus generated output updates. No additional user focus was supplied. Tests were not run, per the review instructions.
FE UT Coverage ReportIncrement line coverage |
|
run buildall |
|
/review |
|
Codex automated review failed and did not complete. Error: Invalid prompt: your prompt was flagged as potentially violating our usage policy. Please try again with a different prompt: https://platform.openai.com/docs/guides/reasoning#advice-on-prompting Please inspect the workflow logs and rerun the review after the underlying issue is resolved. |
|
run buildall |
|
/review |
There was a problem hiding this comment.
Found three blocking correctness and rolling-compatibility issues on the reviewed head:
- Default INT64 Parquet output gives timezone-free DATETIMEV2 values instant semantics, so cross-timezone readers shift them.
- The versioned INT96 contract is honored only by FileScannerV2; the supported V1 route returns different values for the same current-FE plan.
- Upgraded BEs reject old-FE CHAR(N) plans for Iceberg fixed(N) writes.
Coverage: reviewed all 156 changed paths and the related Arrow writer, nested schema, Paimon history, Parquet V1/V2, ORC rounding/SARG, external binary, Export replay, and mixed-version paths through three convergence rounds. The earlier Variant physical-STRUCT and old-FE missing-marker threads are fixed on this head and were not duplicated. No additional user focus was specified, so the full PR review was used. Builds and tests were not run because this review runner explicitly disallows them.
Requesting changes until the three inline issues are addressed.
|
run buildall |
|
/review |
There was a problem hiding this comment.
Requesting changes for five substantiated issues on exact head 489a3c5aa8373bee9fbe9abe1708ddf72acb804b.
Findings:
- P1: Paimon native-Parquet scans still select FileScannerV1 when
enable_file_scanner_v2=false, bypassing the new per-column TIMESTAMP/TIMESTAMP_LTZ contract and returning shifted values. - P1: Mapping-disabled ORC TIMESTAMP_INSTANT rounding carries in local civil time across DST transitions, returning the wrong DATETIMEV2 value.
- P1: The new plain Arrow-converter default deterministically breaks both pre-existing Iceberg Variant transformer tests; production Iceberg call sites do pass the target converter.
- P2: An ORC value immediately below year zero that rounds into the supported boundary is rejected before its carry is applied.
- P2: File-controlled ORC nanoseconds are checked with
DORIS_CHECK, converting malformed data into a fatal-coded failure and a process-fatal path in debug orexit_on_exceptionconfigurations instead of a checked corruption status.
Critical checkpoints:
- Architecture and lifecycle: Scanner selection, mixed native/JNI splits, request-local schemas, cache ownership, projection refresh, and teardown were traced. The Paimon scan-level/range-level format mismatch is the sole surviving scanner/lifecycle issue.
- Schema mapping and materialization: INT96 absent/empty/named semantics, nested and dictionary readers, Paimon history annotations, Variant physical schemas, and ORC row conversion were checked. The Paimon and ORC row-value issues above remain.
- Filtering, deletes, and pushdown: Parquet statistics do not accept INT96, row-group dictionary pruning is string-only, and the relevant non-fixed-zone ORC aggregate/SARG paths fall back. No additional false-pruning or delete-path issue survived.
- Format and external compatibility: Iceberg/Paimon writer recursion, UUID/fixed/VARBINARY bindings, FE/BE rolling-version behavior, Parquet output defaults, EXPORT persistence/replay, Hive/Hudi/JDBC mappings, and TVF exceptions were checked. The existing old-FE Iceberg fixed thread was treated as a hard duplicate fence; no distinct production writer issue remains beyond the Paimon/ORC findings.
- Performance and observability: Split/refinement, cache, prefetch, and profile-sensitive paths showed no material new issue.
- Tests: The two Iceberg Variant tests fail by static control-flow proof, and the ORC tests omit the mapping-disabled DST carry, rounding-into-range lower bound, and malformed-nanoseconds cases. No build was run because the review contract explicitly prohibits builds; evidence is static review only.
User focus: no additional focus was provided. Two complete review rounds converged, including separate risk challenges, and all Round 2 reviewers returned NO_NEW_VALUABLE_FINDINGS beyond these accepted comments.
FE UT Coverage ReportIncrement line coverage |
|
run buildall |
|
/review |
There was a problem hiding this comment.
Request changes: two P1 correctness regressions remain.
- ORC timestamp results are not scanner-independent: V2 now rounds/carries nanoseconds, while supported V1 query and load paths still truncate.
- Explicit legacy INT96 output now uses a timezone-naive Arrow staging schema, but the reader still applies the writer timezone, shifting cross-zone round trips and contradicting the added regression oracle.
Critical checkpoints reviewed: Paimon native/JNI routing; full, partial, predicate-only, and refreshed Parquet projections; request-local schema isolation; Variant physical carriers; timestamp statistics, SARG and aggregate safety; target-specific nested Arrow conversion; the versioned absent/empty/named INT96 contract on the supported BE-first rollout; and catalog-wide binary mapping. No additional distinct issue survived those checks.
I did not repeat the plain-ORC year-zero boundary issue or old-FE Iceberg fixed-width writer compatibility issue because live threads already cover them. No additional user focus was supplied.
Tests were not run because this review environment explicitly prohibits builds/tests; conclusions are from static path tracing against the authoritative PR bundle and the changed tests/golden outputs.
|
run buildall |
|
/review |
|
run buildall |
|
/review |
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
|
Codex automated review failed and did not complete. Error: Codex completed, but no new pull request review was submitted for the current head SHA. Please inspect the workflow logs and rerun the review after the underlying issue is resolved. |
There was a problem hiding this comment.
Automated review summary for exact head f68ba57
Review opinion: no new actionable or blocking findings.
I reviewed the complete authoritative 162-file diff and deduplicated against every existing inline thread. The review converged after two rounds of full-coverage and risk-focused passes; every candidate was independently resolved, dismissed with code evidence, or fenced by existing review context.
Critical checkpoints covered:
- ORC and Parquet timestamp row decoding, writer semantics, half-up rounding, bounds, named-zone/DST behavior, statistics, SARGs, page/footer metadata, dictionary and Bloom pruning, plus INT96 tri-state propagation and scanner routing.
- Paimon historical TIMESTAMP/TIMESTAMP_LTZ annotations through nested and filter-only projections, native/JNI range selection, late-filter refresh, request-local schema mutation, mixed-plan marker handling, and forced Scanner V2 routing.
- Plain, Iceberg, and Paimon Arrow adapters across nullable and nested ARRAY/MAP/STRUCT values, offsets/null maps, converter lifetimes, timestamp units/zones, UUID/fixed/VARBINARY, and Variant physical layouts.
- FE catalog/TVF/JDBC mappings, Hive/Hudi/Iceberg/Paimon timezone contracts, OUTFILE/EXPORT INT64 versus INT96 selection, persisted export replay, irrelevant non-Parquet property handling, and old/new FE-BE compatibility paths.
- Changed unit/regression tests and expected outputs, including explicit legacy-scope cases already discussed in existing threads.
The user focus file contained no additional focus points, so full PR scope was retained. Builds and tests were not run because the review instructions explicitly prohibited them; this conclusion is based on static code, control-flow, compatibility, and test-code review.
|
run buildall |
|
/review |
|
Review and CI follow-up:
Local validation passed: 698 related BE tests, 378 FE tests, FE Checkstyle, clang-format 16 for all 74 C/C++ files touched by the PR, and syntax parsing for all 12 updated Groovy suites. The INT96 writer test also passes in isolation. The three unmuted FE failures from the previous CI run passed locally without relaxing their timeouts. The coverage build failed while starting an incompatible Node binary; the current TeamCity build configuration already uses the glibc-217 distribution. No shared CI configuration was changed as part of this follow-up. Full external regression requires the new CI run; local validation does not establish that every pipeline is green. |
|
run buildall |
|
/review |
|
Codex automated review failed and did not complete. Error: Selected model is at capacity. Please try a different model. Please inspect the workflow logs and rerun the review after the underlying issue is resolved. |
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
|
run buildall |
|
/review |
|
Codex automated review failed and did not complete. Error: You've hit your usage limit. Visit https://chatgpt.com/codex/settings/usage to purchase more credits or try again at Sep 23rd, 2026 6:44 AM. The selected account is excluded until 2026-09-23T06:44:00Z. Please trigger /review again; another configured account may be available. |
|
Codex automated review failed and did not complete. Error: You've hit your usage limit. Visit https://chatgpt.com/codex/settings/usage to purchase more credits or try again at Sep 23rd, 2026 7:04 AM. The selected account is excluded until 2026-09-23T07:04:00Z. Please trigger /review again; another configured account may be available. |
|
run buildall |
|
/review |
|
Codex automated review failed and did not complete. Error: Selected model is at capacity. Please try a different model. Please inspect the workflow logs and rerun the review after the underlying issue is resolved. |
|
run buildall |
|
/review |
There was a problem hiding this comment.
Static review at exact head 1cb0641 (base 8d2018b).
Found one new P1 correctness/stability issue: the Paimon/Iceberg map Arrow writer dereferences a nullable key-column cast even when required map keys are represented by plain columns, which can crash a release BE. The inline comment points to the added helper and requests a guarded plain-key path plus regression coverage.
I traced the ORC timestamp SARG/zone-map paths, Parquet INT96 tri-state contract, nested/schema/cache ownership, catalog marker migration, JDBC/Hudi/MaxCompute/Trino transport, and Iceberg partition encoding. No additional changed-head defect was substantiated. The signed negative-offset boundary is already covered by the existing unresolved cast/parser thread and is not duplicated here.
This was a static-only review; no builds or tests were run.
| DCHECK(nested_values_column.is_nullable()); | ||
| const auto* keys_nullmap_data = | ||
| check_and_get_column<ColumnNullable>(nested_keys_column)->get_null_map_data().data(); | ||
| const auto& offsets = map_column.get_offsets(); |
There was a problem hiding this comment.
[P1] Guard required map keys before dereferencing the nullable cast
check_and_get_column<ColumnNullable>(nested_keys_column) is dereferenced unconditionally here, although map keys are not required to be nullable. DataTypeMap::create_column() and TableReader nullability alignment can produce a plain key column (for example a Paimon map whose key is non-nullable), so this returns null and crashes the BE in release builds before the row loop. Please handle plain keys without a null-map lookup (or normalize/validate the child and return a Status), and add a Paimon/Iceberg map writer test with a non-nullable key.
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
) ### What problem does this PR solve? This ports #68297 to `master`, preserving the first of five planned extractions from #67784. Binary `Field` values can retain references to released source storage, and Hive binary text needs its own Base64 contract. TIMESTAMPTZ output can lose historical offset seconds, format invalid NULL payloads, or fail again while reporting a boundary cast error. - Own long binary Field values while keeping short values inline. Preserve execution type lengths and decoder bytes, and add Hive Base64 and hexadecimal decoding support. - Explicitly reject unsupported binary hash keys, IN, aggregates, predicates and computed partition transforms. Keep the existing FE comparison/group/join restrictions and existing binary scalar functions. Reject unsupported collection kernels in each function's legality check before coercion. - Preserve historical second offsets in both TIMESTAMPTZ formatting and parsing. Skip masked NULL payloads, reject unrepresentable local years, and preserve cast error/NULL behavior at boundaries. Arrow convertor migration, Parquet/ORC semantics, external writer changes and catalog mapping migration belong to the subsequent extractions. This PR does not enable native VARBINARY storage. ### Master adaptation - Retain the fixed-offset normalization and tests already present on master. - Use the current void-returning `VInPredicate::_prepare_zonemap_min_max` interface in both the guard and its test. - Retain master header cleanup and existing timestamp-nanosecond tests. - Retain the existing master binary-literal encoder and its StringView input contract; the older std::string-based caller fix is not applicable. ### Testing - TIMESTAMPTZ regression follow-up: reproduced both binary-output and stream-load failures using the master PR CI artifact, regenerated the two snapshots through `run-regression-test.sh`, and passed both suites in comparison mode from each branch checkout. Explicit `Asia/Shanghai` session settings were verified with the server default session zone set to UTC. Only historical offset seconds changed in the generated results. - Function-local validation update: 18 FE tests passed with Checkstyle enabled, covering direct legality checks, nested/mixed/variadic VARBINARY arguments, both `collect_set` arities, supported ordinary types, SQL analysis, and existing array rewrites. The new direct-legality tests reproduced missing rejection before the change. - BE ASAN build and **199 tests passed** across 17 suites using `run-be-ut.sh`, including binary lifetime/SerDe/rejection, timestamp parsing/casts, and existing Arrow/Variant serialization coverage. - `VarBinaryUnsupportedCollectionTest`: **passed** (13 unsupported expressions plus supported byte-preserving collection analysis). The FE test reactor and repository Checkstyle passed after cleaning stale branch build artifacts. - Repository clang-format 16 check and build-header hygiene checks: **passed**; 31 changed C++ source/header files. - Groovy compilation of the three regression suites: **passed**. Live SQL regression execution remains pending CI. - clang-tidy was attempted but could not complete because master already contains an unmatched `NOLINTEND` in `be/src/core/types.h`. A diagnostic run with the compiler resource directory corrected reproduced that blocker; the other reported findings in `column_varbinary.cpp` were outside changed lines. This is not a clean clang-tidy result. The focused BE test source list and local test/build settings were restored before committing. No build configuration changes are included. ### Release note Fix binary value lifetime and serialization, reject unsupported binary computation paths, and preserve TIMESTAMPTZ historical offsets and boundary error behavior. ### Check List (For Author) - Test - [x] Regression test (three self-checking suites added; execution pending CI) - [x] Unit Test - Behavior changed: - [x] Yes. Binary rejection and timestamp boundary behavior are described above. - Does this need documentation? - [x] No. This fixes existing type behavior without introducing a configuration option. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label ### Scoped review follow-up This follow-up only fixes correctness/stability defects introduced by this PR. Compatibility preservation, pre-existing limitations, additional VARBINARY computation/validation, performance refactors, and test-style-only rewrites are excluded. - Separate historical TIMESTAMPTZ wire-offset parsing from session fixed-zone limits in both parser paths. - Use UTC diagnostics for the TIMESTAMP_NS cast/comparison failures affected by the new local-year formatting exception. - Validation: 31 focused ASAN BE tests passed, including ordinary DATE/DATETIME parsing. Three targeted tests failed before the fixes. clang-format 16 and build hygiene passed. Full clang-tidy remains affected by pre-existing diagnostics. - Branch-specific UTC/GMT normalization and FE folding fixes are handled in #68297; the corresponding master behavior predates this PR or already defers folding. ### CI test follow-up - Keep the binary literal test's owning Field alive while reading its StringView. Branch-4.1 now has the corresponding short/long embedded-NUL coverage using its execution API. - Replace the obsolete +15:00 rejection input with +24:00. Add generated historical-offset checks in both cast modes; all prior snapshot results are unchanged. - Validation: 34 focused ASAN BE tests passed on each branch. The lifetime error and the original SQL mismatch were reproduced. The complete cast regression suite passed in comparison mode from both branch checkouts against the reported master CI artifact. clang-format 16 passed; full clang-tidy remains blocked by pre-existing diagnostics. This follow-up changes tests only and retains the agreed scope: no compatibility work or additional binary computation support. Existing muted failures are outside this fix.
) ### What problem does this PR solve? This is the first of five planned extractions from #67784, targeting `branch-4.1`. Binary `Field` values can retain references to released source storage, and Hive binary text needs its own Base64 contract. TIMESTAMPTZ output can lose historical offset seconds, format invalid NULL payloads, or fail again while reporting a boundary cast error. - Own long binary Field values while keeping short values inline. Preserve execution type lengths and decoder bytes, fix binary literal encoding, and add Hive Base64 and hexadecimal decoding support. - Explicitly reject unsupported binary hash keys, IN, aggregates, predicates and computed partition transforms. Keep the existing FE comparison/group/join restrictions and existing binary scalar functions. Reject unsupported collection kernels before coercion. - Normalize fixed timezone offsets and preserve historical second offsets in both TIMESTAMPTZ formatting and parsing. Skip masked NULL payloads, reject unrepresentable local years, and preserve cast error/NULL behavior at boundaries. Arrow convertor migration, Parquet/ORC semantics, external writer changes and catalog mapping migration belong to the subsequent extractions. This PR does not enable native VARBINARY storage. ### Testing - TIMESTAMPTZ regression follow-up: reproduced both binary-output and stream-load failures using the master PR CI artifact, regenerated the two snapshots through `run-regression-test.sh`, and passed both suites in comparison mode from each branch checkout. Explicit `Asia/Shanghai` session settings were verified with the server default session zone set to UTC. Only historical offset seconds changed in the generated results. - Function-local validation update: 17 FE tests passed after a clean build with Checkstyle enabled. Coverage includes direct legality checks, nested/mixed/variadic VARBINARY arguments, both `collect_set` arities, supported ordinary types, SQL analysis, and existing array rewrites. Collection restrictions now live in each function's legality check before coercion; existing branch-specific argument rules are preserved. - Rebuilt the BE ASAN test target from this extraction: **184 tests passed**, zero failures. Coverage includes binary lifetime/SerDe/rejection paths, timestamp parsing/casts, hash and partition guards, and existing Arrow/Variant serialization tests. - `VarBinaryUnsupportedCollectionTest`: **passed** (13 unsupported collection expressions, plus existing byte-preserving array/collection analysis). - FE reactor `validate` with repository Checkstyle: **passed**. - clang-format 16 check on all 34 changed C++ source/header files: **passed**. - Groovy compilation of the three new regression suites: **passed**. Live SQL regression execution is pending CI. The local BE test source list was narrowed for the focused build and restored before committing. No build configuration changes are included. ### Release note Fix binary value lifetime and serialization, reject unsupported binary computation paths, and preserve TIMESTAMPTZ historical offsets and boundary error behavior. ### Check List (For Author) - Test - [x] Regression test (three self-checking suites added; execution pending CI) - [x] Unit Test - Behavior changed: - [x] Yes. Binary rejection and timestamp boundary behavior are described above. - Does this need documentation? - [x] No. This fixes existing type behavior without introducing a configuration option. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label ### Scoped review follow-up This follow-up only fixes correctness/stability defects introduced by this PR. Compatibility preservation, pre-existing limitations, additional VARBINARY computation/validation, and unrelated refactors are excluded. - Separate historical TIMESTAMPTZ wire-offset parsing from session fixed-zone limits in both parser paths. - Validate the complete UTC/GMT fixed offset and exclude rejected endpoint values from the timezone cache. - Decline FE string folding when the session-local year is outside the new BE display range. Preserve the CAST for BE evaluation in both cast modes instead of folding non-strict casts to NULL. - Validation: 29 focused ASAN BE tests and 15 FE tests passed. Four BE tests and the new FE boundary test failed before the fixes. clang-format 16 and FE Checkstyle passed. - The corresponding master follow-up is in #68301. Master already has different timezone normalization and FE folding behavior; its additional TIMESTAMP_NS error-reporting fix does not apply to branch-4.1. ### CI test follow-up - Keep the binary literal test's owning Field alive while reading its StringView. Branch-4.1 now has the corresponding short/long embedded-NUL coverage using its execution API. - Replace the obsolete +15:00 rejection input with +24:00. Add generated historical-offset checks in both cast modes; all prior snapshot results are unchanged. - Validation: 34 focused ASAN BE tests passed on each branch. The lifetime error and the original SQL mismatch were reproduced. The complete cast regression suite passed in comparison mode from both branch checkouts against the reported master CI artifact. clang-format 16 passed; full clang-tidy remains blocked by pre-existing diagnostics. This follow-up changes tests only and retains the agreed scope: no compatibility work or additional binary computation support. Existing muted failures are outside this fix.
) ### What problem does this PR solve? This is the first of five planned extractions from #67784, targeting `branch-4.1`. Binary `Field` values can retain references to released source storage, and Hive binary text needs its own Base64 contract. TIMESTAMPTZ output can lose historical offset seconds, format invalid NULL payloads, or fail again while reporting a boundary cast error. - Own long binary Field values while keeping short values inline. Preserve execution type lengths and decoder bytes, fix binary literal encoding, and add Hive Base64 and hexadecimal decoding support. - Explicitly reject unsupported binary hash keys, IN, aggregates, predicates and computed partition transforms. Keep the existing FE comparison/group/join restrictions and existing binary scalar functions. Reject unsupported collection kernels before coercion. - Normalize fixed timezone offsets and preserve historical second offsets in both TIMESTAMPTZ formatting and parsing. Skip masked NULL payloads, reject unrepresentable local years, and preserve cast error/NULL behavior at boundaries. Arrow convertor migration, Parquet/ORC semantics, external writer changes and catalog mapping migration belong to the subsequent extractions. This PR does not enable native VARBINARY storage. ### Testing - TIMESTAMPTZ regression follow-up: reproduced both binary-output and stream-load failures using the master PR CI artifact, regenerated the two snapshots through `run-regression-test.sh`, and passed both suites in comparison mode from each branch checkout. Explicit `Asia/Shanghai` session settings were verified with the server default session zone set to UTC. Only historical offset seconds changed in the generated results. - Function-local validation update: 17 FE tests passed after a clean build with Checkstyle enabled. Coverage includes direct legality checks, nested/mixed/variadic VARBINARY arguments, both `collect_set` arities, supported ordinary types, SQL analysis, and existing array rewrites. Collection restrictions now live in each function's legality check before coercion; existing branch-specific argument rules are preserved. - Rebuilt the BE ASAN test target from this extraction: **184 tests passed**, zero failures. Coverage includes binary lifetime/SerDe/rejection paths, timestamp parsing/casts, hash and partition guards, and existing Arrow/Variant serialization tests. - `VarBinaryUnsupportedCollectionTest`: **passed** (13 unsupported collection expressions, plus existing byte-preserving array/collection analysis). - FE reactor `validate` with repository Checkstyle: **passed**. - clang-format 16 check on all 34 changed C++ source/header files: **passed**. - Groovy compilation of the three new regression suites: **passed**. Live SQL regression execution is pending CI. The local BE test source list was narrowed for the focused build and restored before committing. No build configuration changes are included. ### Release note Fix binary value lifetime and serialization, reject unsupported binary computation paths, and preserve TIMESTAMPTZ historical offsets and boundary error behavior. ### Check List (For Author) - Test - [x] Regression test (three self-checking suites added; execution pending CI) - [x] Unit Test - Behavior changed: - [x] Yes. Binary rejection and timestamp boundary behavior are described above. - Does this need documentation? - [x] No. This fixes existing type behavior without introducing a configuration option. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label ### Scoped review follow-up This follow-up only fixes correctness/stability defects introduced by this PR. Compatibility preservation, pre-existing limitations, additional VARBINARY computation/validation, and unrelated refactors are excluded. - Separate historical TIMESTAMPTZ wire-offset parsing from session fixed-zone limits in both parser paths. - Validate the complete UTC/GMT fixed offset and exclude rejected endpoint values from the timezone cache. - Decline FE string folding when the session-local year is outside the new BE display range. Preserve the CAST for BE evaluation in both cast modes instead of folding non-strict casts to NULL. - Validation: 29 focused ASAN BE tests and 15 FE tests passed. Four BE tests and the new FE boundary test failed before the fixes. clang-format 16 and FE Checkstyle passed. - The corresponding master follow-up is in #68301. Master already has different timezone normalization and FE folding behavior; its additional TIMESTAMP_NS error-reporting fix does not apply to branch-4.1. ### CI test follow-up - Keep the binary literal test's owning Field alive while reading its StringView. Branch-4.1 now has the corresponding short/long embedded-NUL coverage using its execution API. - Replace the obsolete +15:00 rejection input with +24:00. Add generated historical-offset checks in both cast modes; all prior snapshot results are unchanged. - Validation: 34 focused ASAN BE tests passed on each branch. The lifetime error and the original SQL mismatch were reproduced. The complete cast regression suite passed in comparison mode from both branch checkouts against the reported master CI artifact. clang-format 16 passed; full clang-tidy remains blocked by pre-existing diagnostics. This follow-up changes tests only and retains the agreed scope: no compatibility work or additional binary computation support. Existing muted failures are outside this fix.
…68396) ### What problem does this PR solve? Related PR: #68381. This is the master version of the second split from #67784, based on the primitives merged in #68301. Arrow batch conversion mixes protocol serialization with table-specific UUID handling, while writers construct schemas separately. Introduce explicit Doris, Python, Arrow Flight, Parquet, Hive, Iceberg and Paimon convertors with instance-owned schema parameters and timezone. Move schema construction/decoding into the convertors and route nested SerDe writes through the selected format. Separate Parquet, Hive and Iceberg writers and migrate existing callers. Preserve master's tracked Arrow memory pools, Iceberg statistics and timestamp-nanosecond support. Master does not yet contain the Paimon write backend or physical Variant table writes present on branch-4.1; this pick adds the converter interfaces without importing those features. Parquet timestamp encoding and external type mappings remain unchanged. Include the Python timezone regression correction from #68381: the single string output uses ARRAY<STRING>, so the lateral-view comparison reaches execution instead of failing on a STRUCT-versus-STRING comparison. Retain coverage for four session timezones, microseconds, pre-epoch values, NULLs, UDF, UDTF and UDAF. ### Release note Fix Python UDF timestamp conversion to preserve wall-clock values when the Arrow protocol declares a fixed-offset timezone. ### Check List (For Author) - Test - [x] Unit Test: explicit schemas and independent converter instances, nested/null values, UUID and fixed binary bytes, timestamp bindings, and Iceberg writer statistics. - [x] Regression test: Python UDF/UDTF/UDAF timezone comparisons and the existing timestamp snapshot corrections. - Behavior changed: - [x] Yes: align Python UDF conversion with its Arrow timezone declaration; reject invalid nested bindings before casts. - Does this need documentation? - [x] No. Validation: ASAN BE build and 310 focused tests passed (53 suites), covering Arrow conversion, Parquet/ORC, Variant SerDe and Python. All 42 affected C++ files passed clang-format 16, and header hygiene passed. The original UDTF declaration reproduced the SQL analysis error on an isolated FE; the corrected declaration passed the same analysis. Groovy and embedded Python checks passed. Full Python SQL and external-catalog regressions remain for CI. clang-tidy was attempted: the new converter's size warning was resolved; analysis remains blocked by a pre-existing unmatched NOLINTEND in core/types.h. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label
…68381) ### What problem does this PR solve? Related PR: #67784. This is the second split, based on the primitives merged in #68297. Arrow batch conversion mixes protocol serialization with table-specific UUID and Variant handling, while writers construct schemas separately. Introduce explicit Doris, Python, Arrow Flight, Parquet, Hive, Iceberg and Paimon convertors, instantiated with their own schema parameters and timezone. Move table schema construction/decoding into the convertors and route nested SerDe writes through the selected format. Separate Parquet, Hive and Iceberg writers and migrate all callers. Preserve current Parquet timestamp encoding, UUID/Variant layouts and external type mappings. Keep Python's numeric conversion timezone consistent with its declared Arrow schema, and reject incompatible nested target schemas before casts or child access. No FE, Thrift, ORC timestamp, or binary computation changes are included. ### Release note Fix Python UDF timestamp conversion to preserve wall-clock values when the Arrow protocol declares a fixed-offset timezone. ### Check List (For Author) - Test - [x] Unit Test: schema ownership/isolation, slices, invalid schemas, nested/null values, UUID/Variant bytes, fixed-offset Python batches, and Parquet timestamp representation. - [x] Regression test: add Python UDF/UDTF/UDAF timezone assertions and extract the corresponding existing Python snapshot corrections from #67784. - Behavior changed: - [x] Yes: align Python UDF conversion with its Arrow timezone declaration; report invalid nested schema bindings as errors. - Does this need documentation? - [x] No. Validation: ASAN BE build and 396 selected tests passed; the final incremental build and 285 focused tests also passed. clang-format 16 and Groovy/embedded Python syntax checks passed. Python SQL and external-catalog end-to-end regressions were not run locally and remain for CI. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label
What problem does this PR solve?
Issue Number: DORIS-28352, DORIS-28353
Related PRs: #66891, #65446
Problem Summary:
This PR is based on #66891 and completes the timestamp and binary type contracts needed by Iceberg and Paimon on
branch-4.1.It fixes the Arrow writer and external timestamp semantics:
DATETIMEV2;TIMESTAMPTZcontinues to require instant semantics.Zalias toUTCin Arrow timestamp metadata so Arrow clients can resolve the timezone.DATETIMEV2(p)to PaimonTIMESTAMP(p)and DorisTIMESTAMPTZ(p)to PaimonTIMESTAMP_LTZ(p)without losing precision.DATETIMEV2as a timezone-naive logical timestamp andTIMESTAMPTZas an adjusted-to-UTC timestamp.FORMAT_JNI, on scanner V2 even whenenable_file_scanner_v2=false. Scanner V1 cannot honor their explicit wall-clock contract.TIMESTAMP_LTZchild from being materialized asDATETIMEV2.+08:00) instead of exporting cctz internalFixed/...names, so UDF/UDTF/UDAF batches agree with Python protocol metadata. Parquet, Hive and Iceberg retain their declared schema timezone labels too.It also makes catalog binary mappings binary-safe:
BINARYandUUID, PaimonBINARYandVARBINARY, Hivebinary, and JDBC binary types directly to DorisVARBINARY.enable.mapping.varbinarybehavior; the property no longer changes catalog type mapping.enable_mapping_varbinaryoption as an explicit opt-in, defaulting tofalse, because changing an ad-hoc TVF result schema also changes CTAS type inference.STRING, preventing Arrow Flight clients from rejecting valid binary payloads.It ports the timestamp correctness fixes from #65446 to the direct scanner architecture used by
branch-4.1:hive.parquet.time-zoneproperty for HMS catalogs and file TVFs.TIMESTAMPversusTIMESTAMP_LTZsemantics, including nested filter-only projections backed by unannotated INT96.TIMESTAMP_INSTANTcarry before timezone conversion, reject malformed nanoseconds as data errors, and keep timestamp statistics/SARG pruning conservative.enable_int96_timestamps=truesupport.branch-4.1.External-to-Doris type mapping changes
The comparisons below are against the pre-PR
branch-4.1implementation, not an intermediate revision of this PR. Previously, the catalog propertyenable.mapping.varbinarydefaulted tofalse. The new catalog mappings apply even when that property is omitted or explicitly set tofalse.Catalog data columns
false(default)trueBINARY(non-partition data column)STRINGVARBINARYVARBINARYBINARYSTRINGVARBINARYVARBINARYFIXED(N)CHAR(N)VARBINARY(N)VARBINARY(N)UUIDSTRINGVARBINARY(16)VARBINARY(16)BINARY(N)STRINGVARBINARY(N)VARBINARY(N)VARBINARY(N)STRINGVARBINARY(N)VARBINARY(N)BINARY,VARBINARYSTRINGVARBINARY(N)VARBINARY(N)TINYBLOB,BLOB,MEDIUMBLOB,LONGBLOBSTRINGVARBINARY(N)VARBINARY(N)BYTEASTRINGVARBINARY(N)VARBINARY(N)BLOBSTRINGVARBINARY(N)VARBINARY(N)BINARY,VARBINARY,IMAGESTRINGVARBINARY(N)VARBINARY(N)BLOBSTRINGVARBINARY(N)VARBINARY(N)Here,
Nis a byte length: the declared external width for Iceberg/Paimon andrequiredColumnSize()from JDBC metadata for the listed JDBC mappings. UnqualifiedVARBINARYabove denotes the maximum-length binary mapping. This list names the changed JDBC branches explicitly; it does not imply that every type in every JDBC connector is remapped.Timestamp mappings
The timestamp mapping option previously defaulted to false. Both catalog
enable.mapping.timestamp_tzand file TVFenable_mapping_timestamp_tzare now compatibility inputs only: omitting them or specifying false cannot change the mapping. Precision is capped at six fractional digits.TIMESTAMP WITH LOCAL TIME ZONEtimestamp with time zone/ adjust-to-UTC timestampTIMESTAMP_LTZ(p)TIMESTAMP(p)timestamptzTIMESTAMP(p) WITH LOCAL TIME ZONETIMESTAMP(p) WITH TIME ZONEdatetimeoffset(p)timestamp(p) with time zoneDateTime,DateTime64(p), including column/server timezone variantstimestamp-millis,timestamp-microslocal-timestamp-millis,local-timestamp-microsTIMESTAMPTIMESTAMP_INSTANTNested timestamp leaves follow the same rules. DATE and TIME are not remapped by this policy. SQL Server
TIMESTAMPis a binary rowversion, not a timestamp; its existing mapping is unchanged.The read/write paths are updated together with metadata: JNI instant carriers use UTC components, ClickHouse scalar and array timestamps are projected as epoch microseconds (including Query TVFs), MaxCompute instant writes retain microseconds, and Iceberg defaults preserve offsets. TIMESTAMPTZ preserves the instant, not the source's original zone identifier; its displayed offset follows the Doris session timezone.
JDBC predicates containing TIMESTAMPTZ literals, or calendar functions/casts involving TIMESTAMPTZ columns, are evaluated locally until the remote timezone semantics can be preserved. Timezone-independent direct NULL checks remain eligible for pushdown. This can increase scanned rows and also prevent limit pushdown. Trino connector domains retain UTC and fractional precision and reject non-representable short-timestamp endpoints.
Partition, nested-type and TVF boundaries
BINARYpartition columnfalse:VARCHAR(MAX_VARCHAR_LENGTH); withtrue:VARBINARYVARCHAR(MAX_VARCHAR_LENGTH)in both cases, preserving the HMS text partition-name/literal contractARRAY/MAP/STRUCT, IcebergLIST/MAP/STRUCT, and PaimonARRAY/MAP/ROWenable_mapping_varbinaryoption, defaultfalse; binary inference was opt-infalse; catalog migration does not enable itCompatibility and migration
true, including requests that supplyfalse.ALTER CATALOG PROPERTIESrecord settingenable.mapping.varbinary=trueand/orenable.mapping.timestamp_tz=truefor each existing external catalog whose corresponding marker is absent or not true. The migration runs before query readiness/checkpoints; already-enabled catalogs are skipped. Followers replay the same durable change.falsesetting.collect_set, array membership/position/distinct/remove/set operations and their related hash-based functions) are rejected during analysis with a clear unsupported-type error. Byte-agnostic array construction/element access andcollect_listremain available; binary values are never silently coerced to text to enable a kernel.[-0.999, 0)seconds (ORC-645). DATETIMEV2/TIMESTAMPTZ ORC writes now fail explicitly for that interval rather than silently producing positive timestamps; use Parquet for these values. Other negative timestamps remain supported. Existing files are not rewritten.Release note
External catalog binary types now map directly to Doris
VARBINARY. The formerenable.mapping.varbinarycatalog property no longer enables STRING fallback behavior. File TVFs retain the separateenable_mapping_varbinaryopt-in and continue to default to STRING.Arrow timestamp schemas now publish the UTC timezone as
UTCinstead ofZfor client compatibility.Parquet Export and Outfile now write INT64 logical timestamps by default:
DATETIMEV2is timezone-naive andTIMESTAMPTZis adjusted to UTC. Setenable_int96_timestamps=truefor readers that require INT96. Versioned scanner plans preserve raw INT96 wall-clock values by default, andhive.parquet.time-zonecan describe Hive files normalized by a known writer timezone. External instant types now always map to TIMESTAMPTZ, including Hudi, Iceberg and Paimon; local timestamp types retain DATETIMEV2 semantics. Scanner V2 rounds ORC timestamp nanoseconds to microseconds; instant carry is applied before timezone conversion, and malformed fractional values fail the scan without terminating the BE process.Check List (For Author)
Validation:
The converter refactor builds successfully under ASAN. Its 50 focused BE tests cover Arrow column bindings, Variant output, Parquet/Hive/Iceberg writers, internal schema initialization, malformed Paimon schema input, and instance-local timestamp bindings. The fixed-offset regression tests failed before the fix and passed afterward. clang-format 16 passed for all 16 C/C++ files affected by the timezone-label fix.
The earlier binary computation and review fixes passed 354 focused ASAN BE tests; one temporary-directory permission conflict was resolved by rerunning with an isolated directory.
Confirmed five regression tests failed before the fixes: malformed Paimon ARRAY/MAP arity and unsupported binary column hashing, hash-key selection, IN/NOT IN, and Iceberg bucket/truncate transforms.
Removed the new binary computation assertions from the SQL and Hive ORC regression cases. Retained coverage for byte transport, ordinary views, native materialization restrictions, nested binary values, and existing scalar functions. The long-value case verifies all 2,048 returned rows using integer-key filtering instead of a binary predicate. Both edited scripts compile with the regression framework's Groovy 4.0.19 version; end-to-end SQL reruns remain pending CI.
Standard CI and review are requested through separate
run buildalland/reviewcomments.Behavior changed:
Does this need documentation?
User-facing documentation should remove catalog binary and timestamp fallback options, describe mandatory instant mappings and native materialization restrictions, and retain the TVF
enable_mapping_varbinaryopt-in.