Skip to content

refactor: moving WindowTopN before EnsureRequirements - #24191

Merged
alamb merged 8 commits into
apache:mainfrom
saadtajwar:chore/moving-windowtopn-before-enforcesorting
Aug 11, 2026
Merged

refactor: moving WindowTopN before EnsureRequirements#24191
alamb merged 8 commits into
apache:mainfrom
saadtajwar:chore/moving-windowtopn-before-enforcesorting

Conversation

@saadtajwar

@saadtajwar saadtajwar commented Aug 8, 2026

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

The WindowTopN physical optimizer rule currently runs after EnsureRequirements, which means it must pattern-match through SortExec nodes that EnsureRequirements inserts:

FilterExec(rn <= K)                                                                                                                                                               
  [optional ProjectionExec]                                                                                                                                                       
    BoundedWindowAggExec(ROW_NUMBER PARTITION BY ... ORDER BY ...)
      SortExec(partition_keys, order_keys)   ← inserted by EnforceSorting

By moving WindowTopN to run before EnsureRequirements, we can simplify the logic to avoid pattern-matching through SortExec node and instead use the ORDER BY and PARTITION BY on the BoundedWindowAggExec

What changes are included in this PR?

Moved WindowTopN to run before EnsureRequirements, and updated its logic to not expect a SortExec and instead use the ORDER BY and PARTITION BY on the BoundedWindowAggExec

Are these changes tested?

Yes

Are there any user-facing changes?

These are just optimizer changes (order of optimizations and internal logic of WindowTopN), no API changes

@github-actions github-actions Bot added optimizer Optimizer rules core Core DataFusion crate sqllogictest SQL Logic Tests (.slt) labels Aug 8, 2026
@saadtajwar saadtajwar changed the title chore: moving WindowTopN before EnsureRequirements refactor: moving WindowTopN before EnsureRequirements Aug 8, 2026
@codecov-commenter

codecov-commenter commented Aug 8, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 90.00000% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 80.98%. Comparing base (d5bd10d) to head (7a5bccd).

Files with missing lines Patch % Lines
datafusion/physical-optimizer/src/window_topn.rs 88.88% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #24191      +/-   ##
==========================================
- Coverage   80.98%   80.98%   -0.01%     
==========================================
  Files        1106     1106              
  Lines      383495   383498       +3     
  Branches   383495   383498       +3     
==========================================
  Hits       310590   310590              
+ Misses      54572    54571       -1     
- Partials    18333    18337       +4     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@saadtajwar

Copy link
Copy Markdown
Contributor Author

@SubhamSinghal & @2010YOUY01 - ready for review if you two have the time! Thanks so much in advance!

@saadtajwar

Copy link
Copy Markdown
Contributor Author

Thank you @jayzhan211 for the review! Please let me know if there's anything else needed on my end to get this in :)

@alamb
alamb added this pull request to the merge queue Aug 11, 2026
Merged via the queue into apache:main with commit 0ef1aaa Aug 11, 2026
40 checks passed
kosiew pushed a commit to kosiew/datafusion that referenced this pull request Aug 12, 2026
## Which issue does this PR close?

- Closes apache#21594

## Rationale for this change

The WindowTopN physical optimizer rule currently runs after
EnsureRequirements, which means it must pattern-match through SortExec
nodes that EnsureRequirements inserts:

```
FilterExec(rn <= K)                                                                                                                                                               
  [optional ProjectionExec]                                                                                                                                                       
    BoundedWindowAggExec(ROW_NUMBER PARTITION BY ... ORDER BY ...)
      SortExec(partition_keys, order_keys)   ← inserted by EnforceSorting
```

By moving WindowTopN to run before EnsureRequirements, we can simplify
the logic to avoid pattern-matching through `SortExec` node and instead
use the `ORDER BY` and `PARTITION BY` on the `BoundedWindowAggExec`

## What changes are included in this PR?

Moved `WindowTopN` to run before `EnsureRequirements`, and updated its
logic to not expect a `SortExec` and instead use the `ORDER BY` and
`PARTITION BY` on the `BoundedWindowAggExec`

## Are these changes tested?

Yes

## Are there any user-facing changes?

These are just optimizer changes (order of optimizations and internal
logic of `WindowTopN`), no API changes
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

core Core DataFusion crate optimizer Optimizer rules sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Simplify WindowTopN by moving it before EnforceSorting

4 participants