Skip to content

branch-4.1: [improvement](topn) Add option to skip file cache writes in lazy materialization - #66414

Merged
yiguolei merged 1 commit into
apache:branch-4.1from
bobhan1:backport/4.1/pr-65021-topn-lazy-file-cache
Aug 5, 2026
Merged

branch-4.1: [improvement](topn) Add option to skip file cache writes in lazy materialization#66414
yiguolei merged 1 commit into
apache:branch-4.1from
bobhan1:backport/4.1/pr-65021-topn-lazy-file-cache

Conversation

@bobhan1

@bobhan1 bobhan1 commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Issue Number: N/A

Related PR: #65021

Problem Summary:

Backport #65021 to branch-4.1. This adds an option for TopN lazy materialization phase 2 to read remote cache misses without populating File Cache, and exposes the corresponding profile metrics.

The cherry-pick conflicts were adapted to the branch-4.1 layouts:

  • Keep the existing adjacent-file batching and compile checks in RowIDFetcher, while propagating file_cache_miss_policy to Doris-format reads.
  • Keep the single-row Segment API and preserve the copied IOContext when setting reader type, stats, and cache-miss policy.
  • Use the branch-4.1 VariableMgr.VarAttr annotation for the new session variable.

Release note

Add enable_topn_lazy_mat_phase2_no_write_file_cache to skip File Cache population for remote misses during TopN lazy materialization phase 2.

Check List (For Author)

  • Test
    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

Validation:

  • ./build.sh --fe -j100

  • ./build.sh --be --clean -j100

  • ./run-be-ut.sh --run --filter='MaterializationSharedStateTest.*:BlockFileCacheTest.get_downloaded_blocks_if_fully_covered_is_read_only:BlockFileCacheTest.cached_remote_file_reader_remote_only_on_miss:BlockFileCacheTest.fd_cache_remove:BlockFileCacheTest.fd_cache_evict' -j100 (9 tests passed)

  • ./run-regression-test.sh --run -d cloud_p0/cache/topn_lazy_file_cache -s test_topn_lazy_mat_phase2_no_write_file_cache -g docker -runMode=cloud -dockerSuiteParallel 1 (1 suite passed)

  • Behavior changed:

    • No.
    • Yes. When enabled, TopN lazy materialization phase 2 serves remote misses without writing the fetched data into File Cache.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@bobhan1
[improvement](topn) Add option to skip file cache writes in topn lazy…
2b9dc61 
… materialization (apache#65021)

This PR adds an opt-in session variable for TopN lazy materialization
phase-2 file-cache miss handling.

- add `enable_topn_lazy_mat_phase2_no_write_file_cache`
- keep the option disabled by default, so existing behavior is unchanged
- when the option is enabled, phase-2 lazy materialization uses
already-downloaded file-cache blocks for full local hits and reads
remote data directly on cache miss without writing the file cache
- expose aggregate and per-BE phase-2 rows, segments, and file-cache
counters in the `MaterializeNode` profile
- accumulate per-BE stats across multiple phase-2 fetch calls
- cover both row-store and column-store fetch paths

- `./build.sh --be --fe --cloud -j100`
- before packaging, verified no `output/fe/conf/fe_custom.conf`, no
`output/be/conf/be_custom.conf`, empty `output/fe/doris-meta`, and empty
`output/be/storage`
- for docker regression image, verified FE/BE debug-point config and BE
Java support config:
  - `output/fe/conf/fe.conf`: `enable_debug_points=true`
- `output/be/conf/be.conf`: `enable_debug_points=true`,
`enable_java_support=false`
- `docker build -f docker/runtime/doris-compose/Dockerfile -t
bh-cluster-2 .`
- `env -u HTTP_PROXY -u HTTPS_PROXY -u http_proxy -u https_proxy -u
ALL_PROXY -u all_proxy ./run-regression-test.sh --run -d
cloud_p0/cache/topn_lazy_file_cache -s
test_topn_lazy_mat_phase2_no_write_file_cache -g docker -runMode=cloud
-dockerSuiteParallel 1`
  - `All suites success`
  - `Test 1 suites, failed 0 suites, fatal 0 scripts, skipped 0 scripts`
- `./run-be-ut.sh --run
--filter=MaterializationSharedStateTest.*:BlockFileCacheTest.get_downloaded_blocks_if_fully_covered_is_read_only:BlockFileCacheTest.cached_remote_file_reader_remote_only_on_miss:BlockFileCacheTest.fd_cache_remove:BlockFileCacheTest.fd_cache_evict
-j100`
  - `PASSED 9 tests`

(cherry picked from commit f45888f)
@hello-stephen

hello-stephen commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@bobhan1 bobhan1 changed the title [branch-4.1][improvement](topn) Add option to skip file cache writes in lazy materialization branch-4.1: [improvement](topn) Add option to skip file cache writes in lazy materialization Aug 4, 2026
@bobhan1
bobhan1 marked this pull request as ready for review August 4, 2026 03:58
@bobhan1
bobhan1 requested a review from yiguolei as a code owner August 4, 2026 03:58
@bobhan1

bobhan1 commented Aug 4, 2026

Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen

hello-stephen commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 77.62% (1904/2453)
Line Coverage 64.46% (34017/52774)
Region Coverage 64.46% (17208/26694)
Branch Coverage 53.95% (9206/17064)

@hello-stephen

hello-stephen commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 100.00% (2/2) 🎉
Increment coverage report
Complete coverage report

@bobhan1

bobhan1 commented Aug 4, 2026

Copy link
Copy Markdown
Contributor Author

run external

@hello-stephen

hello-stephen commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

BE UT Coverage Report

Increment line coverage 67.90% (275/405) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 58.31% (24601/42188)
Line Coverage 42.46% (247011/581721)
Region Coverage 38.52% (194966/506126)
Branch Coverage 39.84% (89089/223591)

@hello-stephen

hello-stephen commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 89.63% (363/405) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 73.53% (30294/41202)
Line Coverage 57.47% (332461/578516)
Region Coverage 54.32% (276484/509027)
Branch Coverage 55.25% (123600/223694)

@hello-stephen

hello-stephen commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 0.00% (0/70) 🎉
Increment coverage report
Complete coverage report

@yiguolei
yiguolei merged commit bfc2255 into apache:branch-4.1 Aug 5, 2026
32 of 35 checks passed
@bobhan1 bobhan1 mentioned this pull request Aug 5, 2026
Merged
11 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@yiguolei yiguolei yiguolei approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@bobhan1 @hello-stephen @yiguolei