[opt](nereids) scale num_nulls in col stats when partition pruned by englefly · Pull Request #62265 · apache/doris

englefly · 2026-04-09T06:51:34Z

What problem does this PR solve?

For an OlapTable, when only a subset of partitions is selected, the num_nulls value in column statistics needs to be scaled proportionally.
Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

…um_nulls value in column statistics needs to be scaled proportionally.

englefly · 2026-04-09T06:51:43Z

run buildall

Thearas · 2026-04-09T06:52:07Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

englefly · 2026-04-10T02:17:54Z

run buildall

hello-stephen · 2026-04-10T03:37:19Z

FE UT Coverage Report

Increment line coverage 14.29% (1/7) 🎉
Increment coverage report
Complete coverage report

englefly · 2026-04-16T05:41:40Z

/review

englefly · 2026-04-16T05:41:51Z

run nonConcurrent

github-actions

Blocking findings:

StatsCalculator.computeOlapScan() now rescales numNulls even when getColumnStatistic(..., selectedPartitionNames) has already returned the merged stats for exactly the selected partitions. With enable_partition_analyze=true, that undercounts nulls by the prune ratio and feeds wrong stats into IS NULL / join estimation.
The added regression only covers the fallback-to-table-stats path and misses the partition-stats-enabled path where the regression occurs.

Critical checkpoints:

Goal of change: scale numNulls for partition-pruned scans when only table-level column stats are available. Current code only partially accomplishes this: it fixes the fallback path but breaks the exact partition-stats path. The new regression proves the fallback case only.
Minimality: The patch is small, but the unconditional rescaling is broader than needed. It should be limited to the table-level fallback path.
Concurrency: No new concurrency concerns in this patch.
Lifecycle/static initialization: Not applicable.
Configuration: No new config item, but behavior depends on existing enable_partition_analyze; the enabled path regresses.
Compatibility: No FE/BE protocol, storage-format, or rolling-upgrade compatibility change.
Parallel code paths: Applicable. There are two functional paths here: merged partition stats and table-level fallback. The fix must distinguish them; only the fallback path should be rescaled.
Special conditional checks: The new scale logic is not sufficient because it does not distinguish fallback stats from exact merged partition stats.
Test coverage: Insufficient. The added regression exercises the default enable_partition_analyze=false path only; it needs a case for enable_partition_analyze=true, preferably selecting multiple partitions so the exact merged numNulls is observable.
Observability: No additional observability appears necessary for this optimizer change.
Transaction/persistence: Not applicable.
Data writes/modifications: Not applicable.
FE-BE variable passing: Not applicable.
Performance: No meaningful performance concern found; the blocking issue is correctness.
Other issues: No additional blocking issues beyond the incorrect rescaling and missing coverage above.

github-actions · 2026-04-16T05:51:53Z

+                colStatsBuilder.normalizeAvgSizeByte(slot.getDataType());
+                //scale null_num
+                double scale = tableRowCount == 0 ? 1 : selectedPartitionsRowCount / tableRowCount;
+                colStatsBuilder.setNumNulls(colStatsBuilder.getNumNulls() * scale);


When enable_partition_analyze is on and every selected partition has cached stats, getColumnStatistic(..., selectedPartitionNames) already returns the merged stats for exactly those partitions. Scaling numNulls again by selectedPartitionsRowCount / tableRowCount undercounts nulls by the prune ratio. For example, selecting two 4-row partitions that each contain one null produces a merged numNulls = 2, but this code changes it to 8 / 12 * 2 = 1.333.... That regresses scan stats for IS NULL filters and downstream selectivity that depends on null counts. The scaling needs to happen only on the table-level fallback path, not when merged partition stats are available.

github-actions · 2026-04-16T05:51:53Z

+        def colStats = sql "show column stats ptable";
+        def memo = sql "explain memo plan select * from ptable where d='2017-01-01'"
+        // check numNulls=1.0000 for column val, which is scaled from 3 to 1 according to the partition pruning result.
+        assertTrue(memo.toString().contains("val#1 -> ndv=1.0000, min=2.000000(2), max=2.000000(2), count=4.0000, numNulls=1.0000"),


This regression only covers the default enable_partition_analyze = false path, so it never exercises the merged partition-stats branch that this PR also changes. With enable_partition_analyze = true, the new unconditional scaling above turns an exact merged numNulls into a smaller value. Please add a case that enables partition analyze (and restores it in finally, like query_p0/stats/partition_col_stats.groovy) and verifies the selected-partition numNulls stays exact instead of being rescaled.

hello-stephen · 2026-04-16T07:34:49Z

FE Regression Coverage Report

Increment line coverage 1.02% (8/787) 🎉
Increment coverage report
Complete coverage report

englefly · 2026-04-17T01:26:57Z

run nonConcurrent

hello-stephen · 2026-04-17T03:32:40Z

FE Regression Coverage Report

Increment line coverage 0.22% (4/1838) 🎉
Increment coverage report
Complete coverage report

github-actions · 2026-04-20T08:25:52Z

PR approved by at least one committer and no changes requested.

ignore

…2265) ### What problem does this PR solve? For an OlapTable, when only a subset of partitions is selected, the num_nulls value in column statistics needs to be scaled proportionally.

…n pruned #62265 (#62694) Cherry-picked from #62265 Co-authored-by: minghong <zhouminghong@selectdb.com>

…n pruned #62265 (#62693) Cherry-picked from #62265 Co-authored-by: minghong <zhouminghong@selectdb.com>

…ache#62265) ### What problem does this PR solve? For an OlapTable, when only a subset of partitions is selected, the num_nulls value in column statistics needs to be scaled proportionally.

### What problem does this PR solve? Related PR: #62265 Problem Summary: `test_scale_num_nulls` checks that table-level `numNulls=3` is scaled to `1` when partition pruning selects 4 of 12 rows. The regression case executes `EXPLAIN MEMO PLAN` immediately after insert and analyze, but Cloud P0 reports the physical partition row count asynchronously. The planner can therefore observe a transient selected-partition row count of zero, while waiting for the real row count can take up to two minutes. This change: - adds deterministic FE unit coverage that invokes `StatsCalculator.computeOlapScan()` with a three-partition OLAP scan; - fixes the table row count at 12, selected-partition row count at 4, and cached table-level column statistics at `ndv=1`, `min=max=2`, and `numNulls=3`; - verifies the final scan statistics have `ndv=1`, `min=max=2`, `count=4`, and scaled `numNulls=1`; - removes the timing-dependent Groovy regression case. The unit test exercises the production partition-pruning statistics path using fixed in-memory inputs. It has no cluster, physical row-count reporting, polling, sleep, network, or storage dependency. Production behavior is unchanged.

…est (#65656) ### What problem does this PR solve? Related PR: #65445 Problem Summary: Backport #65445 to branch-4.1. The regression case `test_scale_num_nulls` depends on asynchronously visible physical row counts. When the physical row count is still 0, Nereids scales the selected-partition column statistics to 0 even though the injected partition/table statistics are correct, making the case flaky. Replace the environment-dependent Groovy regression case with a deterministic FE unit test that directly constructs the partition/table statistics and verifies selected-partition scaling, including row count, NDV, min/max, count, and numNulls. ### Coverage rationale - This is a branch backport of merged PR #65445. The same planner-level coverage concern was raised in [the upstream review thread](#65445 (comment)), then explicitly resolved by a maintainer before the PR was approved and merged. - The independent behavior introduced with #62265 is selected-partition statistics scaling. The replacement test invokes the real `StatsCalculator.computeOlapScan()` path and verifies row count, NDV, min/max, count, and `numNulls`. SQL predicate-to-partition selection remains covered by `PruneOlapScanPartitionTest` on this release branch. - The removed cluster case sampled the strict physical partition row count immediately after insert/analyze. In Cloud P0 that value can remain 0 for up to two minutes even when table-level column statistics are correct. Retaining the case or waiting for that report would preserve the flake or add up to two minutes to the suite. - `SHOW COLUMN STATS` was only failure diagnostics, and memo text formatting was not an independent assertion target of this case. ### Release note None ### Check List (For Author) - Test - [ ] Regression test - [x] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason Local validation: - `StatsCalculatorTest#testComputeOlapScanScalesNumNullsForSelectedPartitions`: passed (1 test) - `StatsCalculatorTest`: passed (17 tests) - `mvn -pl fe-core -am checkstyle:check`: passed (0 violations) - Behavior changed: - [x] No. - [ ] Yes. - Does this need documentation? - [x] No. - [ ] Yes. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label

For an OlapTable, when only a subset of partitions is selected, the n…

2fd32f3

…um_nulls value in column statistics needs to be scaled proportionally.

morrySnow added dev/4.0.x dev/4.1.x labels Apr 9, 2026

morrySnow reviewed Apr 9, 2026

View reviewed changes

Comment thread fe/fe-core/src/main/java/org/apache/doris/nereids/stats/StatsCalculator.java

fmt

8cdb531

github-actions Bot previously requested changes Apr 16, 2026

View reviewed changes

starocean999 approved these changes Apr 20, 2026

View reviewed changes

feiniaofeiafei approved these changes Apr 20, 2026

View reviewed changes

github-actions Bot added the approved Indicates a PR has been approved by one committer. label Apr 20, 2026

englefly merged commit 336bc56 into apache:master Apr 22, 2026
32 of 33 checks passed

englefly deleted the partition-null-num branch April 22, 2026 03:27

github-actions Bot mentioned this pull request Apr 22, 2026

branch-4.0: [opt](nereids) scale num_nulls in col stats when partition pruned #62265 #62693

Merged

github-actions Bot mentioned this pull request Apr 22, 2026

branch-4.1: [opt](nereids) scale num_nulls in col stats when partition pruned #62265 #62694

Merged

yiguolei pushed a commit that referenced this pull request Apr 23, 2026

branch-4.1: [opt](nereids) scale num_nulls in col stats when partitio…

c7fa091

…n pruned #62265 (#62694) Cherry-picked from #62265 Co-authored-by: minghong <zhouminghong@selectdb.com>

yiguolei added dev/4.1.1-merged and removed dev/4.1.x labels Apr 23, 2026

yiguolei pushed a commit that referenced this pull request May 7, 2026

branch-4.0: [opt](nereids) scale num_nulls in col stats when partitio…

c8c9de2

…n pruned #62265 (#62693) Cherry-picked from #62265 Co-authored-by: minghong <zhouminghong@selectdb.com>

yiguolei added dev/4.0.6-merged and removed dev/4.0.x labels May 7, 2026

yiguolei mentioned this pull request May 20, 2026

4.1.1 Release Notes #63426

Open

morningman mentioned this pull request Jun 3, 2026

4.0.6 Release Notes #64065

Open

shuke987 mentioned this pull request Jul 10, 2026

[test](nereids) Replace flaky null stats case with unit test #65445

Merged

12 tasks

This was referenced Jul 15, 2026

branch-4.0: [test](nereids) Replace flaky null stats case with unit test #65655

Open

branch-4.1: [test](nereids) Replace flaky null stats case with unit test #65656

Merged

Uh oh!

Conversation

englefly commented Apr 9, 2026

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

englefly commented Apr 9, 2026

Uh oh!

Thearas commented Apr 9, 2026

Uh oh!

Uh oh!

englefly commented Apr 10, 2026

Uh oh!

hello-stephen commented Apr 10, 2026

FE UT Coverage Report

Uh oh!

englefly commented Apr 16, 2026

Uh oh!

englefly commented Apr 16, 2026

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Apr 16, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Apr 16, 2026

Choose a reason for hiding this comment

Uh oh!

hello-stephen commented Apr 16, 2026

FE Regression Coverage Report

Uh oh!

englefly commented Apr 17, 2026

Uh oh!

hello-stephen commented Apr 17, 2026

FE Regression Coverage Report

Uh oh!

github-actions Bot commented Apr 20, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants