[feature](cloud) Support file cache write index only - #64995
Merged
Conversation
…file cache write index only (apache#9450) pick selectdb/selectdb-core#9096 - Cherry-pick `a37184e9f097e789c4f4e0b40725a98fc49f2851` from `tag-selectdb-cloud-26.0.3-minimax`. - Support file cache write index-only behavior and related segment index/footer preload plumbing. - Adapt conflicts for this branch local writer, packed file, and vertical compaction interfaces. - Preserved this branch existing index writer interface instead of importing source-branch-only `IndexFileWriterPtr` / `create_index_file_writer` plumbing. - Preserved packed-file behavior and added branch-local `PackedAppendContext::write_file_cache` compatibility with legacy default behavior. - Adapted the new vertical compaction test to this branch 4-argument `RowsetWriter::add_columns` API. - Did not introduce `enable_file_cache_write_cumu_compaction_index_only` or `enable_file_cache_write_base_compaction_index_only`, because this branch did not originally have those configs. - `git diff HEAD^ HEAD --check` - `rg -n "\\bIndexFileWriterPtr\\b|has_ann_index|create_index_file_writer|enable_file_cache_write_(base|cumu)_compaction_index_only|compaction_output_write_index_only|should_enable_compaction_cache_index_only" be/src be/test regression-test docker/runtime/doris-compose/command.py -S` - `./run-be-ut.sh --run --filter=CloudFileCacheWriteIndexOnlyConfigTest.* -j120` - `./run-be-ut.sh --run --filter=CloudFileCacheWriteIndexOnly* -j120` - `./build.sh --be --fe --cloud -j120` - Rebuilt `foundationdb/foundationdb:7.1.26-single-layer` from remote registry layers to avoid the local containerd overlay mount failure. Verified the imported image has one rootfs layer. - Rebuilt `bh-cluster-2` with cloud FE enterprise guard jar present in `output/fe/lib/fe-enterprise.jar`, so FE can load `org.apache.doris.cluster.ClusterGuard` in cloud docker mode. - `env -u HTTP_PROXY -u HTTPS_PROXY -u http_proxy -u https_proxy -u ALL_PROXY -u all_proxy DORIS_FDB_IMAGE=foundationdb/foundationdb:7.1.26-single-layer ./run-regression-test.sh --run -d regression-test/suites/cloud_p0/cache/write_index_only -runMode=cloud` - Result: `Test 3 suites, failed 0 suites, fatal 0 scripts, skipped 0 scripts`
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
bobhan1
marked this pull request as ready for review
June 30, 2026 06:23
bobhan1
requested review from
Gabriel39,
airborne12,
csun5285,
eldenmoon,
gavinchou,
liaoxin01,
luwei16,
morningman and
yiguolei
as code owners
June 30, 2026 06:23
Contributor
Author
|
run buildall |
Contributor
Author
|
/review |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Automated review for PR #64995.
I found one issue in the new unit coverage: the empty-range loader test exits through the cloud-mode config gate before it reaches the branch it is intended to protect. The inline comment has the specific evidence.
Review notes:
- Correctness: traced the BE/cloud storage, writer, packed-file, and cache-read paths; no production correctness issue was accepted.
- Optimizer/rewrite scope: no FE/Nereids, join, aggregate, distribution, or planner-facing semantic changes were present.
- Duplicate preflight: no existing inline review comments or replies were found for this PR.
- Validation:
git diff --check ac877e5cc4f27de76753794a4948d55bd425bf0a eaa67b1020c70f1844c0b86b6bd70938e1476698 -- <changed files>was clean. I did not run builds or tests because this checkout is missing.worktree_initializedandthirdparty/installed. - Convergence: both live subagents reviewed the current ledger and final comment set and replied
NO_NEW_VALUABLE_FINDINGS.
Contributor
Author
|
run performance |
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: The empty-range file cache loader unit test expected load_segment_index_to_file_cache to skip opening segment data, but it did not enable cloud mode, so the function returned at the cloud-mode config gate before reaching the empty-range guard. Enable cloud mode in the fixture, restore it in teardown, and make the S3 open path fail if an empty range ever tries to open the segment.
### Release note
None
### Check List (For Author)
- Test: Unit Test
- `./run-be-ut.sh --run --filter=CloudFileCacheWriteIndexOnlyConfigTest.* -j100`
- Behavior changed: No
- Does this need documentation: No
Contributor
Author
|
run buildall |
Contributor
Author
|
/review |
Contributor
There was a problem hiding this comment.
Reviewed the GitHub-authoritative PR file set for PR 64995 and the relevant surrounding BE/cloud/cache/storage paths. I do not have a new inline issue to add.
Checkpoint conclusions:
- Goal and coverage: the change adds cloud index-only file-cache behavior for rowset output, avoids segment-data cache writes, and preloads segment footer/common-index ranges after segment files close. The added BE tests and cloud regression suites cover option splitting, loader gates, vertical writer and vertical compaction paths, V1/V2 inverted index writers, packed-file behavior, compaction output, schema change, and profile counters.
- Scope: the implementation is focused on cloud config, file-writer options, cached reader stats, packed-file lookup, rowset/segment writer close paths, and tests.
- Concurrency/lifecycle: the new preload task vectors are protected by local mutexes and are drained after writer finalization/close ordering. Packed-file lookup continues to use PackedFileManager locking. I did not find a new lock-order or async lifetime issue.
- Configuration: the new mutable cloud config is read at writer-option and loader execution points. Existing compaction index-only settings and the new global mode have explicit precedence tests.
- Compatibility: no storage format or thrift/protocol compatibility issue found; segment/index ranges are used only for cache warming and read accounting.
- Parallel paths: checked normal segment writer, vertical segment writer, vertical beta rowset writer, load, compaction, schema-change/local-rowset skip, packed files, and separate inverted-index files.
- Tests/session/config: the existing inline thread about the empty-range test was treated as already-known review context. In the current head that test sets cloud mode and fails if the open-file path is reached, so I did not duplicate the comment.
User focus: no additional user-provided review focus was present.
Subagent conclusions: optimizer-rewrite found no optimizer/rewrite candidate. tests-session-config found no new test/config candidate beyond the existing empty-range thread. After the main ledger update and empty proposed final comment set, both live subagents returned NO_NEW_VALUABLE_FINDINGS in convergence round C1.
Validation: git diff --check is clean for the 31 files returned by the live GitHub PR files API. I did not run BE unit tests, regression tests, or a full build in this review runner.
Contributor
TPC-H: Total hot run time: 29268 ms |
Contributor
TPC-DS: Total hot run time: 172487 ms |
Contributor
ClickBench: Total hot run time: 25.3 s |
Contributor
TPC-H: Total hot run time: 30033 ms |
Contributor
TPC-DS: Total hot run time: 174087 ms |
Contributor
ClickBench: Total hot run time: 25.25 s |
Contributor
Author
|
run nonConcurrent |
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
gavinchou
approved these changes
Jul 1, 2026
Contributor
|
skip check_coverage |
github-actions
Bot
added
the
approved
Contributor
|
PR approved by at least one committer and no changes requested. |
Contributor
|
PR approved by anyone and no changes requested. |
16 tasks
yiguolei
pushed a commit
that referenced
this pull request
Jul 13, 2026
) ### What problem does this PR solve? Issue Number: N/A Related PR: #64995 Problem Summary: Pick master commit `73c481f65848ed5f449fad2ac9e7c59cf54bcf9e` to `branch-4.1` to support file cache write index only in cloud mode. Adaptations for `branch-4.1`: - Keep the existing `IndexFileWriter` constructor shape on `branch-4.1`. - Drop the unused master-only `row_binlog_segment_writer.h` include. Validation note: - The docker regression was run with an uncommitted local workaround following #65142, because `branch-4.1` does not have that docker-compose fix yet. ### Release note Support file cache write index only in cloud mode. ### Check List (For Author) - Test <!-- At least one of them must be included. --> - [x] Regression test - [x] Unit Test - [x] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason <!-- Add your reason? --> Manual test: - `git diff --check upstream/branch-4.1...HEAD` - `./run-be-ut.sh --run --filter=CloudFileCacheWriteIndexOnly* -j100` - `./build.sh --be --fe --cloud -j100` - `docker build -f docker/runtime/doris-compose/Dockerfile -t bh-cluster-2 .` - `./run-regression-test.sh --run -d regression-test/suites/cloud_p0/cache/write_index_only -g docker -runMode=cloud -dockerSuiteParallel 1` - Behavior changed: - [ ] No. - [x] Yes. Support file cache write index only in cloud mode. - Does this need documentation? - [x] No. - [ ] Yes. <!-- Add document PR link here. eg: apache/doris-website#1214 --> ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
16 tasks
morningman
pushed a commit
that referenced
this pull request
Jul 28, 2026
) ### What problem does this PR solve? Issue Number: N/A Related PR: #64995
11 tasks
bobhan1
added a commit
to apache/doris-website
that referenced
this pull request
Jul 28, 2026
## Summary - document the global `enable_file_cache_write_index_file_only` policy, including its scope, precedence, expected behavior, configuration example, and operational trade-offs - document the Base and Cumulative Compaction-specific index-only policies and clarify that they refine existing cache-write decisions - add the related index and Segment footer/index profile metrics - update the File Cache internals page to keep the Compaction behavior description consistent ## Related Doris PRs - apache/doris#60609 - apache/doris#64995 ## Scope This PR updates the current and 4.x File Cache documentation in English, Chinese, and Japanese. Japanese source pages are updated under `ja-source`. ## Validation - `git diff --check` - reviewed the documented configuration names, precedence, and behavior against the current Doris master implementation and the related PR descriptions - verified that the added current and 4.x content is aligned across all three languages - Docusaurus build not run, following the request to keep this round limited to documentation content ## Versions - [x] dev - [x] 4.x - [ ] 3.x - [ ] 2.1 or older (not covered by version/language sync gate) ## Languages - [x] Chinese - [x] English - [x] Japanese candidate translation needed ## Docs Checklist - [x] Checked by AI - [ ] Test Cases Built - [x] Updated required version and language counterparts, or explained why not - [x] If only one language changed, confirmed whether source/translation counterparts need sync
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary:
Purpose
This PR adds a new BE config,
enable_file_cache_write_index_file_only, to make cloud rowset writes and compaction outputs write only index-related content into file cache. This avoids actively putting segment data into file cache during write, reducing cache pollution from large data pages.The main target scenario is a limited local cache where query performance should prioritize indexes and segment metadata. With this mode enabled, the write path protects inverted index files and segment index/footer ranges, while normal data pages are still loaded and cached by the query read path when needed.
High-Level Changes
enable_file_cache_write_index_file_only, defaultfalse.SEGMENT_FILEdoes not create a full-file cache builder and does not allow adaptive write in index-only mode.INVERTED_INDEX_FILEis still written into file cache through the direct write path.SegmentIndexFileCacheLoaderafter segment close.CachedRemoteFileReader::_update_statswith a read-type enum:Behavior
When
enable_file_cache_write_index_file_only=true:Interaction With Existing Configs
The write-side file-cache behavior is decided in this order:
enable_file_cache=falseis the highest-priority global switch.enable_file_cache=trueandenable_file_cache_write_index_file_only=trueenables global forced index-only mode.SegmentIndexFileCacheLoaderafter segment close.INVERTED_INDEX_FILEdirect write path.enable_file_cache_write_base_compaction_index_onlyandenable_file_cache_write_cumu_compaction_index_only.enable_file_cache_adaptive_write, compaction keep configs, hit-ratio thresholds, or request/sessionwrite_file_cache.enable_file_cache=true,enable_file_cache_write_index_file_only=false, and a compaction-level index-only config is enabled applies index-only behavior only to the matching compaction output.enable_file_cache_write_base_compaction_index_only=trueonly forces base compaction output to use index-only file-cache writes.enable_file_cache_write_cumu_compaction_index_only=trueonly forces cumulative compaction output to use index-only file-cache writes.write_file_cache, adaptive write, compaction keep, and hit-ratio decisions.enable_file_cache=trueand all three index-only configs arefalsepreserves existing behavior.write_file_cache,enable_file_cache_adaptive_write, compaction keep configs, schema-change hit ratios, and existing rowset/file-writer decisions continue to control whether cache is written.With
enable_file_cache_adaptive_writeenable_file_cache_adaptive_write=truekeeps the existing adaptive write behavior. Segment data may be written into file cache if the cache-space check passes.enable_file_cache_adaptive_write=true, segment data is still not written into file cache, while segment index/footer and independent inverted index files are still written.With Compaction Output Keep Configs
Related configs include:
enable_file_cache_keep_base_compaction_outputfile_cache_keep_base_compaction_output_min_hit_ratioWhen index-only is disabled, existing compaction output cache behavior is preserved.
When index-only is enabled, compaction output is forced into index-only behavior:
With Base/Cumulative Compaction Index-Only Configs
Related configs include:
enable_file_cache_write_base_compaction_index_onlyenable_file_cache_write_cumu_compaction_index_onlyThese two configs are narrower compaction-output policies and only take effect when
enable_file_cache_write_index_file_only=false.enable_file_cache_write_base_compaction_index_only=true, base compaction output uses index-only behavior: output segment data is not actively cached, while output segment index/footer ranges and independent inverted index files are written into file cache.enable_file_cache_write_cumu_compaction_index_only=true, cumulative compaction output uses index-only behavior: output segment data is not actively cached, while output segment index/footer ranges and independent inverted index files are written into file cache.enable_file_cache_write_index_file_only=true, the global index-only config takes precedence and the base/cumulative compaction-specific configs no longer change the final behavior.enable_file_cache=falseis still the highest-priority switch and disables all of these file-cache writes.With Schema Change Output Thresholds
file_cache_keep_schema_change_output_min_hit_ratiokeeps its existing semantics.With
disable_file_cache/ Requestwrite_file_cacheenable_file_cache=falsedisables everything and has the highest priority.write_file_cache=false.disable_file_cache=truemakes request-sidewrite_file_cache=false, as long as BEenable_file_cache=trueand index-only is enabled:disable_file_cache=truepreserves existing behavior and load output does not actively write file cache.With Packed File
SegmentIndexFileCacheLoader;With
enable_flush_file_cache_asyncenable_flush_file_cache_async.With
index_percentinfile_cache_pathSegmentIndexFileCacheLoadermarks index-related reads as index data so that they go to theFileCacheType::INDEXqueue when possible.After enabling index-only mode,
index_percentshould be reviewed based on index size. The defaultindex_percent=5may be insufficient for large indexes.Behavior Matrix
Release note
Add BE config
enable_file_cache_write_index_file_onlyto support index-only file-cache writes for cloud rowset writes and compaction outputs.Check List (For Author)
Test
env -u HTTP_PROXY -u HTTPS_PROXY -u http_proxy -u https_proxy -u ALL_PROXY -u all_proxy ./run-regression-test.sh --run -d regression-test/suites/cloud_p0/cache/write_index_only -g docker -runMode=cloud -dockerSuiteParallel 1Test 3 suites, failed 0 suites, fatal 0 scripts, skipped 0 scripts./run-be-ut.sh --run --filter=CloudFileCacheWriteIndexOnlyConfigTest.*:CloudFileCacheWriteIndexOnlyTest.*:CompactionFileCacheTest.*:CloudCompactionTest.should_cache_compaction_output -j100PASSED 35 testsbuild-support/clang-format.shbuild-support/check-format.sh./build.sh --be --fe --cloud -j100../load_docker_conf.shdocker build -f docker/runtime/doris-compose/Dockerfile -t bh-cluster-2 .Behavior changed:
enable_file_cache_write_index_file_only=true, write-side cloud file-cache behavior becomes index-only: segment data is not actively written into file cache, while segment index/footer ranges and independent inverted index files are written into file cache. The config defaults tofalse, so existing behavior is preserved by default.Does this need documentation?
Check List (For Reviewer who merge this PR)