[fix](be) Fix Parquet timestamp decoding and export defaults - #65446
Open
Gabriel39 wants to merge 5 commits into
Open
[fix](be) Fix Parquet timestamp decoding and export defaults#65446Gabriel39 wants to merge 5 commits into
Gabriel39 wants to merge 5 commits into
Conversation
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Gabriel39
marked this pull request as ready for review
July 10, 2026 05:55
Contributor
Author
|
/review |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Automated review completed. I found one correctness issue that should be fixed before merge: plain ORC timestamp SARG pruning is not using the same session-timezone semantics as row decoding and aggregate min/max.
Critical checkpoint conclusions: the PR is focused on format v2 timestamp decoding and pruning; row decode, ORC statistics rounding, and Parquet INT96 row materialization were reviewed. No new concurrency, lifecycle, persistence, protocol-compatibility, or observability risk was found. The main gap is a parallel-path inconsistency in ORC timestamp SARG pruning, and the tests do not cover non-UTC plain ORC timestamp SARG behavior. git diff --check passed; build/unit tests were not run in this checkout because .worktree_initialized, thirdparty/installed, and thirdparty/installed/bin/protoc are missing.
Subagent conclusions: TSC-001 was accepted as MAIN-001 and submitted inline. OPT-001 was dismissed with evidence from the ORC C++ public statistics/writer shape. Convergence round 2 ended with both optimizer-rewrite and tests-session-config reporting NO_NEW_VALUABLE_FINDINGS for the same current ledger/comment set.
User focus: no additional user-provided focus was present.
Gabriel39
force-pushed
the
agent/fix-format-v2-timestamp-decoding
branch
2 times, most recently
from
July 10, 2026 08:07
e8f9938 to
4f5fa22
Compare
Contributor
Author
|
/review |
Contributor
Author
|
run buildall |
Contributor
|
Codex automated review failed and did not complete. Error: Review step was skipped (possibly timeout or cancelled) Please inspect the workflow logs and rerun the review after the underlying issue is resolved. |
Contributor
Author
|
/review |
Contributor
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
There was a problem hiding this comment.
Review result: request changes.
I reviewed the changed FE/thrift wiring, Parquet/ORC format-v2 reader paths, timestamp decoding/rounding behavior, and the added BE/FE tests. Validation was static because this runner is a shallow checkout and thirdparty/installed/bin/protoc is missing; I did verify the live PR head, GitHub file scope, existing inline comments, and git diff --check for the GitHub-listed changed files against the fetched base.
Critical checkpoints:
- Goal/test coverage: The PR mostly covers top-level Parquet INT96 timezone behavior and ORC nanosecond rounding, but it misses nested INT96 materialization and shorthand timezone propagation.
- Scope/focus: The implementation is mostly focused, but the new timezone contract is not applied uniformly across all Parquet INT96 value materialization paths.
- Concurrency/lifecycle: No new concurrent mutable state or lifecycle hazard found in the reviewed paths.
- Config/session propagation: Found one FE/BE normalization mismatch for
hive.parquet.time-zoneshorthand offsets. - Compatibility: The thrift field is optional, so old senders remain compatible; BE still validates invalid values. The accepted shorthand case is the compatibility gap.
- Parallel paths: Top-level INT96 and nested INT96 paths diverge; this is covered by an inline comment.
- Test results: I did not run builds/tests in this runner due missing thirdparty/protoc prerequisites. Static changed-file whitespace check passed.
- Observability/performance/persistence: No additional blocking issue found.
Subagent conclusions:
- optimizer-rewrite proposed the nested Parquet INT96 timezone gap; it was accepted as M1.
- tests-session-config proposed the FE/BE shorthand timezone normalization gap; it was accepted as M2.
- The existing ORC timestamp SARG timezone concern is already covered by discussion_r3556745988 and was not resubmitted.
- Convergence round C2 ended with both live subagents replying
NO_NEW_VALUABLE_FINDINGSfor the final M1/M2 comment set.
User focus: no additional user-provided review focus was supplied.
Gabriel39
force-pushed
the
agent/fix-format-v2-timestamp-decoding
branch
from
July 10, 2026 09:32
4f5fa22 to
05da9fa
Compare
Contributor
Author
|
run buildall |
Contributor
Author
|
/review |
Contributor
FE UT Coverage ReportIncrement line coverage |
Contributor
|
Codex automated review failed and did not complete. Error: You've hit your usage limit. Visit https://chatgpt.com/codex/settings/usage to purchase more credits or try again at 10:57 AM. Please inspect the workflow logs and rerun the review after the underlying issue is resolved. |
Contributor
Author
|
/review |
Contributor
FE UT Coverage ReportIncrement line coverage |
Contributor
There was a problem hiding this comment.
Review result: request changes.
I found one additional non-duplicate correctness issue: the new HMS Parquet INT96 timezone property is honored by scanner v2, but the legacy FileScanner path is still reachable and ignores the new scan parameter. Existing inline threads already cover the ORC plain TIMESTAMP SARG timezone issue, nested INT96 forwarding, and shorthand timezone canonicalization, so I did not duplicate those comments.
Critical checkpoint conclusions:
- Goal/test: the PR addresses format v2 Parquet INT96 and ORC timestamp rounding, with targeted BE/FE tests, but the new catalog property is not proven across the legacy scanner path.
- Scope/focus: the change is mostly focused, but it misses a parallel read path that users can still select.
- Concurrency/lifecycle: no new shared-state or lifecycle issue found in the reviewed changes.
- Config/session propagation: FE-to-BE propagation is present for scanner v2, but not for legacy scanner fallback/disablement.
- Compatibility/protocol: the new thrift field is optional and absence is handled; mixed-version field absence did not show a separate issue.
- Parallel paths: legacy Hive Parquet scanning remains inconsistent with scanner v2 for INT96 timezone semantics.
- Tests: new tests cover scanner v2 behavior, nested INT96, canonicalized timezone params, and ORC rounding; missing coverage for
enable_file_scanner_v2=false. Existing review context already requests non-UTC ORC TIMESTAMP SARG coverage. - Performance/observability: no additional issue found.
- Transaction/persistence/data writes: not applicable to this PR.
Subagent conclusions: OPT-1 became the inline comment below. OPT-2 and TSC-1 were verified as duplicates of existing GitHub thread discussion_r3556745988 and were not resubmitted. Convergence round 1 ended with both live subagents replying NO_NEW_VALUABLE_FINDINGS for the same ledger/comment set after the accepted candidate update.
User focus: no additional user-provided review focus.
Validation: static review only. This checkout is shallow/grafted and is missing .worktree_initialized, thirdparty/installed, and thirdparty/installed/bin/protoc, so I did not run BE/FE tests or style checks.
Contributor
TPC-H: Total hot run time: 29435 ms |
Contributor
TPC-DS: Total hot run time: 179312 ms |
Contributor
ClickBench: Total hot run time: 24.77 s |
Contributor
FE Regression Coverage ReportIncrement line coverage |
Gabriel39
force-pushed
the
agent/fix-format-v2-timestamp-decoding
branch
from
July 11, 2026 12:51
05da9fa to
5c21eda
Compare
Gabriel39
changed the title
Contributor
Author
|
run buildall |
Contributor
Author
|
/review |
Contributor
There was a problem hiding this comment.
Automated review completed for the current PR head.
No additional non-duplicate, diff-scoped blocking issues were found beyond the existing inline review context. I am not adding new inline comments in this review.
User focus: no additional user-provided review focus.
Critical checkpoint conclusions:
- Goal and tests: the patch focuses on Parquet INT96 timezone decoding and ORC timestamp rounding/SARG behavior; current head includes targeted FE and BE tests for catalog timezone canonicalization/propagation, scanner-v2 and legacy scanner paths, nested INT96 decode, ORC timestamp rounding, and conservative SARG behavior.
- Scope: reviewed changed FE scan parameter plumbing, thrift field addition, BE Parquet legacy/v2 readers, ORC decode/stat/SARG paths, and related tests.
- Parallel paths: scanner-v2 and legacy Parquet paths both propagate
hive.parquet.time-zone; nested INT96 decode also receives the configured timezone. - Config/protocol compatibility: the new scan parameter is an optional thrift field; FE canonicalizes shorthand timezone values before sending them and BE validates/uses the configured zone when present.
- Concurrency/lifecycle: no new shared concurrency or lifecycle risk was identified; timezone state is carried on per-reader state and used within reader lifetimes.
- Data correctness: ORC timestamp rounding and SARG bounds appear conservative in the reviewed paths, and Parquet INT96 decode uses the catalog timezone in the covered scalar/nested paths.
- Transactions/persistence/writes: not applicable to this read-path change.
- Observability/performance: no new logging/metric requirement was identified; pruning remains conservative for the timestamp edge cases reviewed.
- Validation: static review only. I did not run BE/FE builds or tests because this checkout is missing
thirdparty/installedandthirdparty/installed/bin/protoc.
Subagent conclusions:
- optimizer-rewrite: no candidates; convergence round returned
NO_NEW_VALUABLE_FINDINGS. - tests-session-config: no candidates; convergence round returned
NO_NEW_VALUABLE_FINDINGS.
Existing review threads were treated as already-submitted context and were not duplicated.
Contributor
FE UT Coverage ReportIncrement line coverage |
Contributor
Author
|
run buildall |
Contributor
Author
|
/review |
Issue Number: None Related PR: apache#65446 Problem Summary: Timestamp decoding had two correctness problems in scanner v2. Parquet INT96 does not carry a timezone annotation, so applying the Doris SQL session timezone shifted raw wall-clock fields written by Trino/UTC-style writers. ORC nanoseconds were truncated inconsistently across row decoding, statistics, and SARG predicates, which could also produce unsafe pruning at millisecond statistic boundaries. Add an explicit `hive.parquet.time-zone` property for scanner-v2 Parquet INT96 reads. By default scanner v2 preserves raw INT96 wall-clock fields; when configured, it interprets them with the requested unambiguous IANA timezone or valid UTC offset. Reject ambiguous short aliases such as `CST`, share timezone parsing across HMS catalogs and file TVFs, and propagate the setting through generic external-table and TVF interfaces for Hive and Hudi while leaving Iceberg semantics unchanged. Change Parquet export to write INT64 logical timestamps by default and require `enable_int96_timestamps` to be exactly `true` or `false`. Update export regressions to exercise the new default directly while retaining explicit INT96 only for legacy Hive interoperability coverage. Add an end-to-end regression that exports INT96 in one session timezone and reads it through scanner v2 in another session timezone with `hive.parquet.time-zone` configured. Apply half-up nanosecond rounding consistently to ORC rows and statistics, use conservative SARG bounds, and fall back to exact row filtering where negating a widened timestamp envelope would be unsafe. The legacy v1 reader remains unchanged by design. Parquet export now writes INT64 logical timestamps by default. Set `enable_int96_timestamps=true` to request INT96. When scanner v2 reads legacy INT96 files that were normalized by a writer timezone, set `hive.parquet.time-zone` to that timezone. The property accepts unambiguous IANA timezone IDs or valid UTC offsets; ambiguous short aliases such as `CST` are rejected. Existing Hive and Hudi catalogs containing normalized INT96 files should be checked during upgrade. - Test: - Build: `./build.sh --fe` passed on the designated remote host. - Checkstyle: `cd fe && mvn checkstyle:check -pl fe-core` passed on the designated remote host with 0 violations. - Unit Test: 39 targeted FE tests covering scan propagation, HMS/TVF timezone validation, Parquet properties, and export commands passed on the designated remote host. - Unit Test: targeted Parquet and ORC BE unit tests passed on the designated remote host. - Regression test: `test_outfile_parquet` passed on an isolated scanner-v2 cluster on the designated remote host and verified default Parquet export/stream-load round-trip results. - Regression test: `test_hive_read_parquet` and `test_hive_read_parquet_complex_type` passed on the designated remote host. - Regression test: `test_doris_int96_round_trip` passed in generated-output and comparison modes on the designated remote host, verifying that the configured compatibility timezone overrides a different read session timezone. - Regression test: `test_hdfs_parquet_group0` passed completely on an isolated scanner-v2 cluster on the designated remote host. - Regression test: `paimon_timestamp_types` passed completely for JNI and native readers on the isolated cluster on the designated remote host. - Regression test: TeamCity builds 994160 and 994162 produced matching millisecond-precision real results for the updated ARRAY, MAP, and STRUCT expectations. - Regression test: the three S3 outfile suites were attempted on the designated remote host but were blocked before the changed assertions because the configured OSS credential returned `InvalidAccessKeyId`. - Behavior changed: Yes. Scanner v2 preserves raw INT96 wall-clock values by default, Parquet export defaults to INT64 timestamps, timezone aliases are rejected, and ORC timestamp rounding and pruning are consistent and conservative. - Does this need documentation: Yes. apache/doris-website#3982
Contributor
|
Codex automated review failed and did not complete. Error: Review context preparation failed before Codex ran; inspect the 'Prepare authoritative PR context and required AGENTS guides' step. Please inspect the workflow logs and rerun the review after the underlying issue is resolved. |
Gabriel39
force-pushed
the
agent/fix-format-v2-timestamp-decoding
branch
from
August 3, 2026 08:35
a9258d3 to
ae911c1
Compare
Contributor
Author
|
run buildall |
Contributor
Author
|
/review |
Contributor
Author
|
run buildall |
Contributor
Author
|
/review |
Contributor
|
Codex automated review failed and did not complete. Error: Codex completed, but no new pull request review was submitted for the current head SHA. Please inspect the workflow logs and rerun the review after the underlying issue is resolved. |
Contributor
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
There was a problem hiding this comment.
Requesting changes for two correctness gaps introduced by the current patch:
- ORC
DATETIMEV2decoding now applies the UTC 0001-9999 bound before session-timezone conversion, rejecting valid local endpoint values in nonzero-offset zones. hive.parquet.time-zoneis validated on catalog creation but not on the detachedALTER CATALOGcandidate, so an invalid built-in HMS/Hudi value can be journaled and fail later scan planning.
I did not duplicate the live threads covering the ORC year-10000 carry, the unversioned old-FE/new-BE INT96 semantics, or the custom-provider property-routing gap.
Critical checkpoint conclusions
- Goal and proof: The PR aims to preserve Parquet/ORC timestamp semantics across Paimon schema history, catalog/TVF scan configuration, and Parquet export. The current-version Paimon and scan/export propagation largely accomplish that goal, but the two inline correctness gaps mean it is not complete. Existing tests cover creation, propagation, nested/current/historical Paimon schemas, ORC rounding/SARG behavior, and export output; they do not cover non-UTC Doris endpoint decoding or invalid
ALTER CATALOGupdates. - Scope and clarity: The FE, Thrift, BE reader, writer-option, and test changes are cross-cutting but generally follow the required end-to-end data flow. I found no unrelated source change. The two duplicated validation/domain rules called out inline should be unified with their owning lifecycle/type boundary.
- Concurrency: No new thread, lock, atomic, or shared mutable cache protocol is introduced. Parquet schema/type mutations are made on request/reader-owned copies, and no new lock-order or race issue was substantiated.
- Lifecycle and static initialization: New C++ constants are function-local
constexpr; there is no cross-TU initialization dependency or new ownership cycle. Provider classloader context, reader schema copies, and export-job lifecycle were traced. The catalog update lifecycle defect is the second inline issue. - Configuration behavior:
hive.parquet.time-zoneis a catalog/TVF property rather than a process-dynamic config. Create and TVF parsing reject invalid values, and catalog reset exposes accepted updates, but detached ALTER validation bypasses the parser. Custom-provider ownership/routing is already covered by a live thread. - Compatibility: The new optional Thrift fields are wired for current FE/BE versions. Missing versus explicit raw INT96 semantics are not rolling-upgrade safe, but that exact mixed-version defect is already covered by the live
parquet_scan.cpp:1165thread. Old persisted export jobs deserialize the absent boolean asfalse; no separate replay incompatibility was found. - Parallel paths: Catalog scans, plugin-driven scans, TVFs, Paimon current/history mappings, native Parquet row/statistics readers, ORC row/SARG/aggregate paths, direct export, and replayed export jobs were checked. No additional path-specific defect survived beyond the ALTER issue and existing live custom-provider thread.
- Conditional checks and error handling: Status/exception results in the changed paths are propagated. The ORC widened carry arithmetic is appropriate, but its UTC-domain conditional is applied too early for local
DATETIMEV2. Invalid property parsing fails loudly where it runs; the ALTER path currently omits it. - Test coverage: Coverage is broad, including BE/FE unit tests and regression suites, but the two inline cases need targeted negative/boundary tests. No concurrency test is needed because the patch adds no concurrent protocol.
- Expected results: The changed regression outputs consistently reflect the new default Parquet INT64 timestamp representation, and ordering/oracles inspected were deterministic. I did not find an internally inconsistent
.outchange. This was a static-only review; no build or test command was run, as required by the review contract. - Observability: The patch adds no long-running distributed state requiring new metrics. Existing failures propagate explicit status/exception text; no distinct logging or metric gap was substantiated.
- Persistence and replay:
ExportJobpersists the explicit writer option and preserves the absent-field default for older jobs. Catalog property updates are edit-logged/replayed through the existing path, but that makes the missing pre-journal ALTER validation user-visible and persistent, as called out inline. - Data writes and atomicity: The write-side behavior change is Parquet timestamp physical encoding for OUTFILE/EXPORT. Writer option propagation and replay were consistent, and no new transaction, crash-recovery, leak, or atomicity problem was found.
- FE-BE variables:
timestamp_is_adjusted_to_utcandhive_parquet_time_zonereach the relevant current-version producers and consumers, including nested Paimon mapping and native reader construction. The remaining mixed-version problem is the existing rolling-upgrade thread, not a new duplicate. - Performance: The added parsing occurs during validation/reader setup, schema transformations are request-scoped and bounded by projected schema size, and widened timestamp arithmetic is constant-time. No material CPU, allocation, or hot-loop regression was substantiated.
- Other correctness: Nullable/nested projection handling, COW/schema-copy ownership, ORC negative/DST/SARG fallbacks, aggregate fallback, and Paimon alias/history semantics were checked without another distinct issue.
ExportCommandis a Nereids command path, but this patch adds no optimizer/plan rewrite requiring a plan-tree finding.
The user focus file contained no additional guidance, so the whole PR was reviewed without a narrower focus.
| const __int128 epoch_microseconds = | ||
| static_cast<__int128>(seconds) * MICROS_PER_SECOND + rounded_microseconds; | ||
| if (epoch_microseconds < MIN_DORIS_TIMESTAMP_MICROS || | ||
| epoch_microseconds > MAX_DORIS_TIMESTAMP_MICROS) { |
Contributor
There was a problem hiding this comment.
[P2] Validate DATETIMEV2 after timezone conversion
This precheck uses the UTC 0001..9999 interval before the caller converts the instant to the session timezone. For example, epoch second 253402329599 with 999999000ns is outside the UTC maximum, but in a -08:00 session it is the valid local value 9999-12-31 23:59:59.999999; this helper now returns DataQualityError before the later target-local is_valid_date() check. The lower boundary is symmetric in +14:00, and the new boundary tests use UTC only. Please retain the widened overflow/carry guard, but validate DATETIMEV2 against the converted local value (while TIMESTAMPTZ can keep UTC-domain validation), with nonzero-offset endpoint tests. This is distinct from the existing carry thread: that case admitted an invalid year-10000 result, while this one rejects a valid target-local endpoint.
| super.checkProperties(); | ||
| String catalogType = getType(); | ||
| if ("hms".equalsIgnoreCase(catalogType) || "hudi".equalsIgnoreCase(catalogType)) { | ||
| String hiveParquetTimeZone = catalogProperty.getOrDefault( |
Contributor
There was a problem hiding this comment.
[P2] Apply this validation to ALTER CATALOG too
This parser runs during checkProperties(), but PluginDrivenExternalCatalog.validatePropertiesBeforeUpdate() builds the detached candidate and returns true after only the base checks plus ConnectorFactory.validatePropertiesForUpdate(). The Hive provider does not validate this key, so CatalogMgr skips the legacy checkProperties() branch and an ALTER such as hive.parquet.time-zone=CST can be journaled successfully; the next scan then throws when getConfiguredHiveParquetTimeZone() parses it. Please reuse this validation for the candidate update path (ideally in the owning provider so create and alter share it) and add an ALTER regression. This is distinct from the existing custom-provider routing thread: it affects invalid built-in HMS updates rather than dropping a valid custom-provider value.
Contributor
TPC-H: Total hot run time: 28891 ms |
Contributor
Author
|
run buildall |
Contributor
Author
|
/review |
Contributor
TPC-DS: Total hot run time: 168751 ms |
Contributor
ClickBench: Total hot run time: 23.96 s |
Contributor
There was a problem hiding this comment.
Automated static review completed at head 17a35689d143c0d6edc3db1413c64ad4039df6b7.
I found no new non-duplicate inline issue. Three existing unresolved discussions remain substantively blocking, so I am not duplicating them:
- Preserve legacy INT96 semantics for old-FE plans
- Validate DATETIMEV2 after timezone conversion
- Apply the timezone validation to ALTER CATALOG
The custom-provider routing discussion remains unresolved in the GitHub UI, but current-head provider-based routing appears to address it. The scanner-v1 divergence was previously raised in a discussion that is now marked resolved, so this review does not reopen it.
Critical checkpoint conclusions:
- Goal and proof: Current-version Parquet INT96 raw/configured decoding, recursive Paimon timestamp provenance, ORC half-up rounding and conservative pruning, provider/TVF propagation, and Parquet writer defaults were traced end to end. The three existing blockers above mean the compatibility and boundary-validation goal is not yet complete.
- Scope and focus: The change is broad across FE, Thrift, BE readers, writer options, and tests, but remains focused on timestamp semantics and export defaults.
review_focus.txtadds no narrower guidance. - Concurrency, lifecycle, and memory: Reader-local schema/native-field copies preserve cache ownership and pointer lifetimes; FE provider state is safely published. No new race, lock-order, static-initialization, cleanup, nullable-shape, or memory-safety defect was found.
- Configuration, compatibility, and parallel paths: CREATE and TVF parsing are fail-fast and canonicalized, while ALTER validation and old-FE/mixed-BE semantics remain covered by the existing blockers. Catalog, provider, TVF, Paimon current/history, scanner-v2, scanner-v1, row/dictionary/statistics, OUTFILE, EXPORT, and replay paths were all checked.
- Correctness and error handling: Status/exception propagation is explicit. ORC widened rounding, statistics, aggregate, SARG, negative/epoch, DST, and negated-predicate paths are conservative outside the already-reported local-endpoint defect. No additional wrong-result path survived duplicate suppression.
- Tests and expected results: The changed unit/regression tests directly cover ordinary rounding, configured/raw/nested INT96, Paimon semantics, strict properties, and default/explicit writer output. Missing old-wire, non-UTC endpoint, and invalid-ALTER cases correspond to the existing blockers. Expected-output changes were consistent and deterministically ordered where applicable.
- Observability and performance: No new attribution, diagnostic, retry, unbounded-work, or material hot-path allocation issue was found; parsing and recursive schema work occur at bounded validation/reader-setup boundaries.
- Persistence, transactions, and writes: The export option is persisted and reconstructed, and unfinished replayed jobs are canceled before registration. No additional journal, partial-write, commit, rollback, crash-recovery, cloud-mode, or merge-on-write issue was found.
Validation was static only, as required by the review contract; no build, format check, or test command was run.
Contributor
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
TPC-H: Total hot run time: 28612 ms |
Contributor
TPC-DS: Total hot run time: 169193 ms |
Contributor
ClickBench: Total hot run time: 25.19 s |
Contributor
FE UT Coverage ReportIncrement line coverage |
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
FE Regression Coverage ReportIncrement line coverage |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: None
Problem Summary:
Timestamp decoding had two correctness problems in scanner v2:
This PR follows Trino's explicit connector-configuration model and adds
hive.parquet.time-zonefor scanner v2:-12:00..+14:00: convert INT96TIMESTAMPvalues with that configured timezone.8:00is canonicalized to+08:00; ambiguous short aliases such asCSTare rejected.TIMESTAMPTZkeeps the same UTC instant and ignores this compatibility conversion.The legacy v1 Parquet reader is intentionally unchanged and can return different results. v1/v2 behavioral parity is outside this PR's scope, and switching to v1 is not the migration mechanism for this change.
Parquet Export and Outfile now write INT64 logical timestamps by default. Set
enable_int96_timestamps=trueonly when a legacy downstream reader such as Hive 2 or Hive 3 requires INT96. Doris INT96 export normalizes timestamps using the export session timezone; when scanner v2 reads that file back, configurehive.parquet.time-zonewith the export session timezone. An end-to-end regression exports underAsia/Shanghai, reads under anAmerica/Los_Angelessession, and verifies that the compatibility property restores the original wall-clock values.For existing Hive and Hudi catalogs, upgrading users should inspect representative INT96 files. If a legacy writer normalized timestamps using a known timezone, set the catalog property before relying on scanner-v2 results:
For ORC, this PR rounds nanoseconds to microseconds with second carry for both
DATETIMEV2andTIMESTAMPTZ, applies the same rounding to statistics, and uses conservative millisecond SARG boundaries. TimestampNE,NOT IN, and negated timestamp subtrees fall back to exact Doris row filtering because negating a widened statistics envelope is unsafe.Release note
Parquet Export and Outfile now write INT64 logical timestamps by default. Set
enable_int96_timestamps=trueto request INT96 for legacy readers. Scanner v2 preserves raw Parquet INT96 wall-clock values by default; sethive.parquet.time-zonefor Hive/Hudi files normalized by a known writer timezone. Existing Hive and Hudi catalogs containing normalized INT96 files should be checked during upgrade. FileScannerV1 remains unchanged. ORC timestamp nanoseconds are rounded consistently to microseconds and timestamp SARG pruning is conservative.Check List (For Author)
gabriel@10.26.20.3:/mnt/disk3/gabriel/Workspace/dev2/doris:./build.sh --fepassed.cd fe && mvn checkstyle:check -pl fe-corepassed with 0 violations.test_outfile_parquet,test_hive_read_parquet,test_hive_read_parquet_complex_type,test_hdfs_parquet_group0, andpaimon_timestamp_typespassed in their validated environments.test_doris_int96_round_trippassed in generated-output and comparison modes. It verifies that the configured compatibility timezone, rather than a different read session timezone, restores explicitly exported INT96 values.InvalidAccessKeyId; no expected output was generated manually.