[improvement](parquet) Optimize typed dictionary range filtering - #66036
Conversation
|
run buildall |
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
|
/review |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
Request changes. The typed range-comparison semantics and legacy dictionary-encoding normalization look sound, but the new direct path has two blocking edge cases: it can discard a hidden predicate column before residual/delete consumers finish with it, producing wrong rows, and it aborts on a legal all-NULL page in an otherwise dictionary-encoded chunk. I also found two non-blocking issues in the performance/benchmark contract.
Review checkpoints:
- Goal and tests: the dense INT/string happy paths demonstrate the intended dictionary-range optimization, but the added tests do not cover mixed residual/delete consumers or nullable multi-page all-NULL fragments. The author reports 179/179 ASAN unit tests and microbenchmarks; I did not run builds or tests because this review environment explicitly prohibits them.
- Scope and parallel paths: the change is focused. The residual/delete path is the critical parallel path and is incorrect for predicate-only payload elision. Sparse selections, ordinary nullable payloads, page-index gaps, non-empty multi-page batches, and zero survivors otherwise remain aligned by static trace.
- Concurrency and lifecycle: reader, decoder, typed-dictionary, and scratch state remain reader-local; I found no new shared-state race, lock-order issue, or cross-thread lifetime problem. The legal all-NULL fragment is the concrete cursor/lifecycle failure, including the projected-INT mode constraint described inline.
- Configuration, compatibility, and persistence: no configuration, FE/BE protocol variable, transaction, persistence, or storage-write change is involved. Legacy
PLAIN_DICTIONARYpages are normalized before the direct gate, and the admittedFieldcomparisons match normal Doris comparison ordering for compatible types. - Conditions and error handling: malformed dictionary IDs remain checked before output commit, but a legal zero-payload page is currently routed to a production
DORIS_CHECK. - Performance and observability: the direct-path counters are initialized and published coherently, but the survivor bitmap is redundantly recounted and the mandatory benchmark guide still documents the old 152-case registration contract.
- User focus: no additional user-provided focus was supplied.
At review time, compile, FE UT, Cloud UT, formatting, style, license, dependency, and secret checks pass; BE UT, macOS BE UT, performance, and regression checks are still pending.
| const uint16_t selected_rows_before = *selected_rows; | ||
| IColumn::Filter compact_filter; | ||
| bool used_filter = false; | ||
| const bool predicate_only = request.is_predicate_only(local_id); |
There was a problem hiding this comment.
[P1] Keep hidden values that later filters still consume
predicate_only only means the slot is not part of final output; it can still be referenced by a remaining residual or delete conjunct. This branch ignores residual_predicate_positions (unlike the fixed-width direct path) and passes nullptr, then installs defaults before later stages run. For example, with hidden dictionary column id, id > 2 plus residual id + score = 33 filters IDs first and then evaluates the residual with id=0, dropping the valid (3,30) row. Please retain/project the payload whenever any residual/delete expression references this position, and add the corresponding hidden-column test.
| *projected_directly = false; | ||
| *used_filter = false; | ||
| row_filter->clear(); | ||
| if (_current_encoding != tparquet::Encoding::RLE_DICTIONARY || _page_decoder == nullptr || |
There was a problem hiding this comment.
[P1] Handle all-NULL dictionary pages as a successful fragment
A nullable data page can legally contain only definition levels even when the Column Chunk otherwise uses RLE_DICTIONARY; load_page_data() deliberately installs EmptyValueSectionDecoder for that shape. It reports no dictionary, so this new gate returns used_filter=false after _read_dictionary_filter_values() has already consumed the definition levels, and the caller immediately hits DORIS_CHECK(used_filter). A clustered-null page in a chunk with non-NULL dictionary values can therefore terminate the BE on valid input. Please treat the zero-physical-value page as a successful all-false fragment that advances logical progress without dictionary IDs, while preserving (or making neutral) the requested projection mode across page fragments; otherwise projected INT batches can next fail the mode-consistency check. Add predicate-only and projected nullable-INT multi-page coverage with the all-NULL page on both sides of a non-NULL dictionary page.
| &projected_directly, &direct_filter_used)); | ||
| if (direct_filter_used) { | ||
| advance_selected_span(direct_rows_read); | ||
| const size_t survivor_count = |
There was a problem hiding this comment.
[P2] Reuse the survivor count from bitmap construction
filter_dictionary_indices() has just visited every selected row to build row_filter, but this recount scans the full bitmap again, and read_filter_columns() scans it once more with count_selected_rows(). Those extra O(selected_rows) memory passes run for every direct dictionary batch, including the predicate-only workload this PR is optimizing. Please accumulate/return the survivor count while building the bitmap and reuse it for reader statistics and scheduler selection updates.
| } | ||
| } | ||
| } | ||
| for (const int selectivity : {1, 10, 50, 90}) { |
There was a problem hiding this comment.
[P2] Update the benchmark's required registration contract
These additions move the deduplicated reader matrix to 159 cases (as the changed tests assert), but be/benchmark/parquet/AGENTS.md still tells benchmark users and reviewers to expect 152 at lines 49, 129, and 302. Following that mandatory smoke-validation guide will now flag the correct binary as inconsistent. Please update all three counts and the matrix description with this change.
TPC-H: Total hot run time: 29488 ms |
TPC-DS: Total hot run time: 178757 ms |
ClickBench: Total hot run time: 25.04 s |
|
run buildall |
|
/review |
|
run buildall |
|
/review |
There was a problem hiding this comment.
Request changes. On the authoritative snapshot 943c0b9, five additional issues remain: a valid zero-value page can terminate the BE, the exact benchmark commands match no registrations, the FLOAT/DOUBLE comparison path lacks an end-to-end value oracle, retained selected-ID scratch bypasses query memory accounting, and sparse selection ranges allocate per fragment. I did not repeat the four existing threads covering the hidden residual-predicate payload, the all-NULL-page crash, the repeated survivor recount, and the stale benchmark matrix/count.
Snapshot note: the live PR head advanced to a697fb9 after this bundle and convergence were completed. This review is explicitly attached to 943c0b9; the newer commit was not part of the authoritative diff and was not reviewed here.
Critical checkpoints:
- Goal and proof: the typed dictionary range fast path works for the covered INT32, BIGINT, and BYTE_ARRAY happy paths, but valid empty-page transitions and FLOAT/DOUBLE comparison coverage are incomplete.
- Scope and compatibility: the patch is cohesive and changes no storage or wire format. Legacy PLAIN_DICTIONARY normalization appears safe, but the optimized path still differs incorrectly from the generic path on the already-reported hidden-payload and all-NULL cases plus the new zero-value-page case.
- Concurrency and lifecycle: no new shared concurrency primitive or static cross-translation-unit state was introduced. Reader-local typed dictionary lifetime is sound; projection-mode transitions and retained scratch lifetime/accounting are not.
- Configuration, variables, transactions, and persistence: no new configuration, FE/BE variable, transaction boundary, or persistent write path is involved.
- Special conditions and parallel paths: type/encoding gates and page-index selected-span boundaries were checked. A zero-value V1/V2 page is a valid transition that reaches the new projection-mode check, while selected-range validation prevents a page-index gap from being the same defect.
- Tests: the new scans cover useful single-page integer/string paths, but not multi-page zero/all-NULL transitions, FLOAT/DOUBLE comparison results, or staged sparse-selection allocation behavior.
- Observability: the six new profile counters are wired from reader statistics through the scan profile.
- Performance and memory: the existing survivor-recount thread remains valid; the two new scratch findings separately cover untracked retained IDs and repeated sparse-range allocation.
- Documentation: the new benchmark name component invalidates the exact filters in the checked-in guide and README.
- User focus: no additional focus was supplied.
- Validation: static review only, as required by the review prompt; no local build or test was run. At the final live-state sweep, the newer head still had BE/compile/code-review/performance jobs pending.
| return Status::OK(); | ||
| } | ||
| if (*rows_read != 0) { | ||
| DORIS_CHECK_EQ(*projected_directly, loop_projected_directly); |
There was a problem hiding this comment.
[P1] Keep empty page transitions projection-mode neutral
A zero-value Data Page V1/V2 is a valid input that the native reader already skips. Here, after an earlier dictionary page has appended fixed-width survivors and set projected_directly=true, the zero-row page returns loop_rows=0, used_filter=true, and the default loop_projected_directly=false, so this check terminates the BE before the following page is read. This is distinct from the all-NULL-page thread because the zero-row page consumes no definition/value cursor. Please compare/update projection mode only for fragments that process logical rows, and add projected RLE_DICTIONARY/PLAIN_DICTIONARY V1/V2 coverage with a zero-value page between nonempty pages.
| "/sel_" + std::to_string(scenario.selectivity_percent) + "/" + | ||
| to_string(scenario.projection) + "/width_" + std::to_string(scenario.schema_width) + | ||
| "/predicate_" + std::to_string(scenario.predicate_position); | ||
| return to_string(scenario.operation) + "/" + to_string(scenario.encoding) + "/" + |
There was a problem hiding this comment.
[P2] Update the exact benchmark filters for the new name shape
Adding /<value_type>/ changes every reader registration to .../<encoding>/<value_type>/null_..., but the exact filters in be/benchmark/parquet/AGENTS.md:93 and be/benchmark/parquet/README.md:75,87 still use .../<encoding>/null_... and now select no cases. This is distinct from the existing stale-count/matrix thread: correcting those descriptions still leaves the documented comparisons as no-ops. Please update all exact filters and validate them with --benchmark_list_tests.
| return get_fixed_dictionary_raw_values<ColumnInt32>(dictionary, values, value_width); | ||
| case TYPE_BIGINT: | ||
| return get_fixed_dictionary_raw_values<ColumnInt64>(dictionary, values, value_width); | ||
| case TYPE_FLOAT: |
There was a problem hiding this comment.
[P2] Add end-to-end FLOAT/DOUBLE dictionary-filter coverage
This admits FLOAT and DOUBLE to an exact bitmap path whose covered conjunct may be removed, but the new comparison scans exercise only INT32, BIGINT, and BYTE_ARRAY. The pre-existing accept-all dictionary scan asserts only row count/broad counters, and the raw NaN unit test does not cover FLOAT/DOUBLE ID filtering, nullable placement, or fused projection. Please add differential FLOAT/DOUBLE comparison scans for projected and predicate-only modes, including NaN, signed zero, infinities, NULLs, and operand reversal, and assert output values plus direct-path counters.
| // Plain or Dictionary encoding. If the dictionary grows too big, the encoding will fall back to the plain encoding | ||
| std::unordered_map<int, std::unique_ptr<Decoder>> _decoders; | ||
| NullMap _nullable_selection_nulls; | ||
| std::vector<uint32_t> _selected_dictionary_indices; |
There was a problem hiding this comment.
[P2] Track the retained selected-ID scratch
This is a persistent per-leaf buffer, but std::vector growth does not pass through Doris Allocator::consume_memory; the checked-in malloc interposer only forwards allocations to jemalloc. The retained-byte bookkeeping here drives the scratch-release policy, not the query MemTracker. With the uint16_t batch bound this can retain about 256 KiB per dictionary predicate leaf, and the 4 MiB per-buffer release threshold means that normal maximum capacity survives until reader teardown, multiplying across wide scans. Please use allocator-aware storage such as DorisVector (updating the decoder interface) or explicitly charge/release capacity to the owning query tracker.
| dictionary_filter.size(), _page_decoder->dictionary_size()); | ||
| } | ||
|
|
||
| ParquetSelection selection; |
There was a problem hiding this comment.
[P2] Reuse sparse selection-range scratch
A staged earlier predicate can leave one disjoint physical run per surviving row. This local ParquetSelection then grows ranges to that fragmentation and destroys the capacity at the end of every page fragment, so later dictionary predicates repeatedly allocate/free the same range storage. The mandatory scanner guide requires selection ranges to live in persistent reader scratch specifically to avoid this hot-path allocation thrash. Please retain and clear a reusable allocator-aware selection (including its capacity in the existing active/retained scratch policy), and cover a two-predicate fragmented-selection benchmark.
There was a problem hiding this comment.
Automated review of current head a697fb98371101a109e3b3a32d7b7c75eee5c733: no additional inline comment remains after refreshing and deduplicating against the live PR state. The two candidates independently found in this run (per-fragment selection-range allocation and zero-match benchmark filters) are already covered by live discussions r3649660108 and r3649660099, so they were not reposted.
The PR remains not ready. Nine live inline discussions cover three correctness/test blockers and six performance, memory, coverage, or benchmark-contract issues:
- hidden predicate-only payload is discarded before residual/delete consumers;
- legal all-NULL and zero-value page transitions can terminate the BE;
- the survivor bitmap is redundantly recounted;
- retained selected-ID memory is not charged to the query tracker;
- fragmented selection ranges allocate and free per page fragment;
- FLOAT/DOUBLE dictionary filtering lacks an end-to-end value oracle;
- benchmark registration counts/matrix text are stale; and
- all checked-in exact reader benchmark filters match zero renamed cases.
Critical checkpoints:
- Goal and proof: the patch implements typed dictionary range evaluation and fused survivor projection for covered integer/string happy paths, but the live page-transition and hidden-payload failures prevent the goal from being safely accomplished. Added tests do not prove the all-NULL/zero-page, hidden residual/delete, or FLOAT/DOUBLE cases.
- Scope and focus: the production, profile, benchmark, test, and design changes are cohesive. No extra user focus was supplied. The checked-in benchmark commands and required matrix documentation were not updated consistently with the expanded type/name matrix.
- Concurrency: the changed reader, decoder, dictionary-cache, and scratch state is reader-local. No new shared-state race, lock-order problem, or deadlock risk was found.
- Lifecycle: dictionary generations, destinations, and normal cursor domains are otherwise coherent, but the live all-NULL/zero-page comments identify invalid projection/cursor lifecycle transitions. Persistent selected-ID and selection-range scratch also violates the intended memory/reuse lifecycle.
- Configuration: no configuration item or dynamic-reload path is added.
- Compatibility: no storage or wire format changes are introduced. Legacy
PLAIN_DICTIONARYnormalization and admitted comparison ordering appear compatible on the reviewed paths. - Parallel paths and conditions: dictionary, raw fixed-width, and generic typed comparison paths were traced, including literal reversal, nullable selection, sparse selection, page boundaries, and fallback. The live hidden-payload and empty-page conditions are the concrete divergences.
- Tests and results: author-reported tests and benchmarks were not independently executed. This workflow requires static review only and explicitly prohibits builds; the checkout also lacks initialized third-party build dependencies.
- Observability: the new direct-path profile counters are initialized and published through the scan profile. No additional logging/metric defect was found.
- Transactions, persistence, writes, and FE/BE variables: not applicable; this patch adds no transaction/EditLog behavior, persistent data mutation, storage write protocol, or FE/BE variable transport.
- Performance and memory: the intended dictionary filtering avoids full typed row materialization, but the live recount, untracked retained-ID capacity, and per-fragment range-allocation comments remain unresolved.
- Other issues: dictionary-ID validation, destination rollback, nullable append parity, string offset preflight, comparison orientation, NaN/string ordering, and typed dictionary invalidation were rechecked without finding another nonduplicate issue.
Validation: static review of the complete authoritative 21-file diff, mandatory scanner design/review guides, tests, and live PR review state. No local build or test was run.
TPC-H: Total hot run time: 29496 ms |
TPC-DS: Total hot run time: 177176 ms |
ClickBench: Total hot run time: 25.17 s |
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
github-actions
Bot
added
the
approved
|
PR approved by at least one committer and no changes requested. |
…che#66036) ### What problem does this PR solve? File Scanner V2 could only use row-level Parquet dictionary filtering efficiently for a narrow set of predicates and projected INT columns. Range predicates and other typed dictionaries could still pay per-entry generic expression evaluation, generic SerDe insertion, or row-sized predicate-column materialization. ### What is changed? - Enable exact typed dictionary evaluation for `=`, `!=`, `<`, `<=`, `>`, and `>=`, including symmetric literal-on-left comparisons. - Accept both `RLE_DICTIONARY` and legacy `PLAIN_DICTIONARY` data-page encodings for supported primitive columns. - Build numeric dictionary bitmaps over contiguous typed INT, BIGINT, FLOAT, and DOUBLE dictionary values, evaluating each conjunct once per dictionary generation. - Build string equality and range bitmaps by comparing dictionary slices directly, without per-entry `Field` construction or generic dictionary expression dispatch. - Decode selected dictionary IDs at the native page-reader layer and apply the per-entry bitmap without materializing a complete predicate value column. - Write all supported fixed-width survivors directly from the filter loop into the target column. - Gather string survivors with pre-sized character/offset buffers and one copy per selected dictionary value. - Add direct-path profile counters and focused INT32/BIGINT/BYTE_ARRAY dictionary microbench scenarios. ### Verification - ASAN focused tests: 14/14 passed, covering typed numeric/string range filters, literal-on-left normalization, fixed-width fused gather, compact string gather, and benchmark scenario registration. - Related ASAN suite: 487 applicable tests passed across Parquet, File Scanner V2, SerDe, column mapping, JSON, WAL, and remote reader coverage. Three unrelated Flight tests could not bind their fixed localhost endpoint because it was already occupied by a pre-existing service. - `git diff --check` passed. ### Microbenchmark: upstream master vs this PR Compared upstream master `7809a73814e` directly with this PR at `943c0b94ffc`. Both binaries use the same Release compiler options, benchmark harness, fixtures, and fixed CPU. Each binary received three warmups; measurements used interleaved master/PR ordering and 20 samples per scenario. Values below are median CPU time. `raw_rows`, `selected_rows`, and fixture sizes were identical for every pair. | Dictionary type | Selectivity | Projection | Master | This PR | Change | |---|---:|---|---:|---:|---:| | INT32 | 10% | predicate only | 1,332,627 ns | 329,241 ns | -75.3% | | INT32 | 50% | predicate only | 1,344,746 ns | 338,433 ns | -74.8% | | INT32 | 10% | predicate projected | 1,686,162 ns | 1,000,973 ns | -40.6% | | INT32 | 50% | predicate projected | 1,713,684 ns | 1,041,358 ns | -39.2% | | BIGINT | 10% | predicate only | 1,053,504 ns | 329,214 ns | -68.8% | | BIGINT | 50% | predicate only | 1,053,577 ns | 340,627 ns | -67.7% | | BIGINT | 10% | predicate projected | 1,367,541 ns | 996,056 ns | -27.2% | | BIGINT | 50% | predicate projected | 1,438,891 ns | 1,053,813 ns | -26.8% | | BYTE_ARRAY | 10% | predicate only | 2,007,277 ns | 386,762 ns | -80.7% | | BYTE_ARRAY | 50% | predicate only | 2,017,380 ns | 398,810 ns | -80.2% | | BYTE_ARRAY | 10% | predicate projected | 2,437,263 ns | 1,194,903 ns | -51.0% | | BYTE_ARRAY | 50% | predicate projected | 2,541,460 ns | 1,323,767 ns | -47.9% | All twelve direct master-to-PR scenarios improved. Predicate-only scans avoid row-sized materialization and generic per-entry dispatch; projected scans additionally avoid generic survivor insertion.
…che#66036) ### What problem does this PR solve? File Scanner V2 could only use row-level Parquet dictionary filtering efficiently for a narrow set of predicates and projected INT columns. Range predicates and other typed dictionaries could still pay per-entry generic expression evaluation, generic SerDe insertion, or row-sized predicate-column materialization. ### What is changed? - Enable exact typed dictionary evaluation for `=`, `!=`, `<`, `<=`, `>`, and `>=`, including symmetric literal-on-left comparisons. - Accept both `RLE_DICTIONARY` and legacy `PLAIN_DICTIONARY` data-page encodings for supported primitive columns. - Build numeric dictionary bitmaps over contiguous typed INT, BIGINT, FLOAT, and DOUBLE dictionary values, evaluating each conjunct once per dictionary generation. - Build string equality and range bitmaps by comparing dictionary slices directly, without per-entry `Field` construction or generic dictionary expression dispatch. - Decode selected dictionary IDs at the native page-reader layer and apply the per-entry bitmap without materializing a complete predicate value column. - Write all supported fixed-width survivors directly from the filter loop into the target column. - Gather string survivors with pre-sized character/offset buffers and one copy per selected dictionary value. - Add direct-path profile counters and focused INT32/BIGINT/BYTE_ARRAY dictionary microbench scenarios. ### Verification - ASAN focused tests: 14/14 passed, covering typed numeric/string range filters, literal-on-left normalization, fixed-width fused gather, compact string gather, and benchmark scenario registration. - Related ASAN suite: 487 applicable tests passed across Parquet, File Scanner V2, SerDe, column mapping, JSON, WAL, and remote reader coverage. Three unrelated Flight tests could not bind their fixed localhost endpoint because it was already occupied by a pre-existing service. - `git diff --check` passed. ### Microbenchmark: upstream master vs this PR Compared upstream master `7809a73814e` directly with this PR at `943c0b94ffc`. Both binaries use the same Release compiler options, benchmark harness, fixtures, and fixed CPU. Each binary received three warmups; measurements used interleaved master/PR ordering and 20 samples per scenario. Values below are median CPU time. `raw_rows`, `selected_rows`, and fixture sizes were identical for every pair. | Dictionary type | Selectivity | Projection | Master | This PR | Change | |---|---:|---|---:|---:|---:| | INT32 | 10% | predicate only | 1,332,627 ns | 329,241 ns | -75.3% | | INT32 | 50% | predicate only | 1,344,746 ns | 338,433 ns | -74.8% | | INT32 | 10% | predicate projected | 1,686,162 ns | 1,000,973 ns | -40.6% | | INT32 | 50% | predicate projected | 1,713,684 ns | 1,041,358 ns | -39.2% | | BIGINT | 10% | predicate only | 1,053,504 ns | 329,214 ns | -68.8% | | BIGINT | 50% | predicate only | 1,053,577 ns | 340,627 ns | -67.7% | | BIGINT | 10% | predicate projected | 1,367,541 ns | 996,056 ns | -27.2% | | BIGINT | 50% | predicate projected | 1,438,891 ns | 1,053,813 ns | -26.8% | | BYTE_ARRAY | 10% | predicate only | 2,007,277 ns | 386,762 ns | -80.7% | | BYTE_ARRAY | 50% | predicate only | 2,017,380 ns | 398,810 ns | -80.2% | | BYTE_ARRAY | 10% | predicate projected | 2,437,263 ns | 1,194,903 ns | -51.0% | | BYTE_ARRAY | 50% | predicate projected | 2,541,460 ns | 1,323,767 ns | -47.9% | All twelve direct master-to-PR scenarios improved. Predicate-only scans avoid row-sized materialization and generic per-entry dispatch; projected scans additionally avoid generic survivor insertion.
…che#66036) ### What problem does this PR solve? File Scanner V2 could only use row-level Parquet dictionary filtering efficiently for a narrow set of predicates and projected INT columns. Range predicates and other typed dictionaries could still pay per-entry generic expression evaluation, generic SerDe insertion, or row-sized predicate-column materialization. ### What is changed? - Enable exact typed dictionary evaluation for `=`, `!=`, `<`, `<=`, `>`, and `>=`, including symmetric literal-on-left comparisons. - Accept both `RLE_DICTIONARY` and legacy `PLAIN_DICTIONARY` data-page encodings for supported primitive columns. - Build numeric dictionary bitmaps over contiguous typed INT, BIGINT, FLOAT, and DOUBLE dictionary values, evaluating each conjunct once per dictionary generation. - Build string equality and range bitmaps by comparing dictionary slices directly, without per-entry `Field` construction or generic dictionary expression dispatch. - Decode selected dictionary IDs at the native page-reader layer and apply the per-entry bitmap without materializing a complete predicate value column. - Write all supported fixed-width survivors directly from the filter loop into the target column. - Gather string survivors with pre-sized character/offset buffers and one copy per selected dictionary value. - Add direct-path profile counters and focused INT32/BIGINT/BYTE_ARRAY dictionary microbench scenarios. ### Verification - ASAN focused tests: 14/14 passed, covering typed numeric/string range filters, literal-on-left normalization, fixed-width fused gather, compact string gather, and benchmark scenario registration. - Related ASAN suite: 487 applicable tests passed across Parquet, File Scanner V2, SerDe, column mapping, JSON, WAL, and remote reader coverage. Three unrelated Flight tests could not bind their fixed localhost endpoint because it was already occupied by a pre-existing service. - `git diff --check` passed. ### Microbenchmark: upstream master vs this PR Compared upstream master `7809a73814e` directly with this PR at `943c0b94ffc`. Both binaries use the same Release compiler options, benchmark harness, fixtures, and fixed CPU. Each binary received three warmups; measurements used interleaved master/PR ordering and 20 samples per scenario. Values below are median CPU time. `raw_rows`, `selected_rows`, and fixture sizes were identical for every pair. | Dictionary type | Selectivity | Projection | Master | This PR | Change | |---|---:|---|---:|---:|---:| | INT32 | 10% | predicate only | 1,332,627 ns | 329,241 ns | -75.3% | | INT32 | 50% | predicate only | 1,344,746 ns | 338,433 ns | -74.8% | | INT32 | 10% | predicate projected | 1,686,162 ns | 1,000,973 ns | -40.6% | | INT32 | 50% | predicate projected | 1,713,684 ns | 1,041,358 ns | -39.2% | | BIGINT | 10% | predicate only | 1,053,504 ns | 329,214 ns | -68.8% | | BIGINT | 50% | predicate only | 1,053,577 ns | 340,627 ns | -67.7% | | BIGINT | 10% | predicate projected | 1,367,541 ns | 996,056 ns | -27.2% | | BIGINT | 50% | predicate projected | 1,438,891 ns | 1,053,813 ns | -26.8% | | BYTE_ARRAY | 10% | predicate only | 2,007,277 ns | 386,762 ns | -80.7% | | BYTE_ARRAY | 50% | predicate only | 2,017,380 ns | 398,810 ns | -80.2% | | BYTE_ARRAY | 10% | predicate projected | 2,437,263 ns | 1,194,903 ns | -51.0% | | BYTE_ARRAY | 50% | predicate projected | 2,541,460 ns | 1,323,767 ns | -47.9% | All twelve direct master-to-PR scenarios improved. Predicate-only scans avoid row-sized materialization and generic per-entry dispatch; projected scans additionally avoid generic survivor insertion.
## Proposed changes Backport the requested changes to `branch-4.1` in master merge order, skipping changes already present in this PR: 1. #62438 2. #65329 (merged prerequisite for the nested-schema cases) 3. #65960 4. #65965 5. #65972 6. #65998 7. #66002 8. #65992 9. #66021 10. #66036 11. #66008 12. #66073 13. #66056 (explicitly requested; current open-PR head, appended after the merged sequence) The branch-specific compatibility commits preserve the selected master behavior on `branch-4.1`, including master wire IDs for file formats and the merged Paimon test helper prerequisite. No regression expected output, test assertion, or test input was changed to make validation pass. ## Validation - Full BE ASAN build passed. - Full FE build passed. - Targeted BE ASAN unit tests: 332 tests from 13 suites passed. - Targeted FE Iceberg unit tests: 41 passed, 0 failed. - Iceberg write regression: 20 suites, 0 failed, 0 fatal. - `PaimonScanNodeTest`: all 16 test bodies completed with 0 assertion failures; the class reports one Mockito teardown error because #66008 left four now-unused stubs on master. The still-open #65867 contains the upstream test-only cleanup commit. This PR intentionally does not alter those test cases. - Final rebase against the latest `branch-4.1` completed; the branch was already up to date. - Working-tree, formatting, and sensitive-information audits completed. Existing EOF blank lines in picked regression output files are preserved. --------- Co-authored-by: daidai <changyuwei@selectdb.com> Co-authored-by: Mingyu Chen (Rayner) <yunyou@selectdb.com>
3 participants
What problem does this PR solve?
File Scanner V2 could only use row-level Parquet dictionary filtering efficiently for a narrow set of predicates and projected INT columns. Range predicates and other typed dictionaries could still pay per-entry generic expression evaluation, generic SerDe insertion, or row-sized predicate-column materialization.
What is changed?
=,!=,<,<=,>, and>=, including symmetric literal-on-left comparisons.RLE_DICTIONARYand legacyPLAIN_DICTIONARYdata-page encodings for supported primitive columns.Fieldconstruction or generic dictionary expression dispatch.Verification
git diff --checkpassed.Microbenchmark: upstream master vs this PR
Compared upstream master
7809a73814edirectly with this PR at943c0b94ffc. Both binaries use the same Release compiler options, benchmark harness, fixtures, and fixed CPU. Each binary received three warmups; measurements used interleaved master/PR ordering and 20 samples per scenario. Values below are median CPU time.raw_rows,selected_rows, and fixture sizes were identical for every pair.All twelve direct master-to-PR scenarios improved. Predicate-only scans avoid row-sized materialization and generic per-entry dispatch; projected scans additionally avoid generic survivor insertion.