Skip to content

branch-4.0: pick #59776, #61318 - #61930

Merged
yiguolei merged 2 commits into
apache:branch-4.0from
mymeiyi:branch-4.0-pick-60543-2
Mar 31, 2026
Merged

branch-4.0: pick #59776, #61318#61930
yiguolei merged 2 commits into
apache:branch-4.0from
mymeiyi:branch-4.0-pick-60543-2

Conversation

@mymeiyi

@mymeiyi mymeiyi commented Mar 31, 2026

Copy link
Copy Markdown
Contributor

pick:

  1. modify CloudTabletStatMgr to reduce memory ([fix](fe) modify CloudTabletStatMgr to reduce memory #59776)
  2. modify CloudTabletRebalancer and CloudTabletStatMgr to reduce memory ([fix](cloud) modify CloudTabletRebalancer and CloudTabletStatMgr to reduce memory #61318)

mymeiyi added 2 commits March 30, 2026 14:05
@mymeiyi
[fix](fe) modify CloudTabletStatMgr to reduce memory (apache#59776)
59a21f5
@mymeiyi
[fix](cloud) modify CloudTabletRebalancer and CloudTabletStatMgr to r…
e60a409 
…educe memory (apache#61318)

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Reduce FE memory by
1. moving top-N table stats filtering from PrometheusMetricVisitor into
CloudTabletStatMgr so it's computed once per stat cycle instead of per
Prometheus scrape,
2. removing the unused beToTablets field from InfightTask to avoid
retaining a large map reference
3. changing InfightTablet.tabletId from Long to long to avoid boxing
overhead.

None

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
@mymeiyi
mymeiyi requested a review from yiguolei as a code owner March 31, 2026 06:30
Copilot AI review requested due to automatic review settings March 31, 2026 06:30
@hello-stephen

hello-stephen commented Mar 31, 2026

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

Copilot AI reviewed Mar 31, 2026
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Backport to branch-4.0 of two upstream fixes aimed at reducing FE memory overhead in cloud-mode tablet/table statistics and rebalancing, primarily by avoiding per-scrape recomputation and unnecessary object/collection retention.

Changes:

  • Move Prometheus “top-N table stats” filtering and total table size computation into CloudTabletStatMgr (computed once per stat cycle, reused across scrapes).
  • Reduce object/collection overhead in cloud rebalancing inflight tracking (remove unused beToTablets reference, avoid Long boxing for tablet ids).
  • Replace several boxed Long statistic fields/counters with primitive long to reduce boxing/memory overhead.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
fe/fe-core/src/main/java/org/apache/doris/metric/PrometheusMetricVisitor.java Switch Prometheus table-stats output to consume pre-filtered stats + precomputed totals from CloudTabletStatMgr.
fe/fe-core/src/main/java/org/apache/doris/cloud/catalog/CloudTabletRebalancer.java Remove unused inflight task map reference; change inflight tablet id to primitive long.
fe/fe-core/src/main/java/org/apache/doris/catalog/TabletStatMgr.java Use primitive long for local aggregation counters to reduce boxing.
fe/fe-core/src/main/java/org/apache/doris/catalog/OlapTable.java Convert OlapTable.Statistics numeric fields/ctor params to primitive long.
fe/fe-core/src/main/java/org/apache/doris/catalog/CloudTabletStatMgr.java Store only top-N table stats for Prometheus, expose total table size, and avoid building/storing a full per-table stats map.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread fe/fe-core/src/main/java/org/apache/doris/catalog/CloudTabletStatMgr.java
Comment on lines +173 to 174
List<OlapTable.Statistics> newCloudTableStatsList = new ArrayList<>();
for (Long dbId : dbIds) {

Copilot AI Mar 31, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

updateStatInfo() collects statistics for all tables into newCloudTableStatsList and only filters to the top-N at the end. If the number of tables is very large, this can still create a large temporary list/array each stat cycle. Consider maintaining the top-N PriorityQueue incrementally during the scan so you never retain references for every table at once.

Copilot uses AI. Check for mistakes.
Comment thread fe/fe-core/src/main/java/org/apache/doris/catalog/CloudTabletStatMgr.java
Comment on lines +370 to +372
public List<OlapTable.Statistics> getCloudTableStats() {
return this.cloudTableStatsList;
}

Copilot AI Mar 31, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

getCloudTableStats() returns the internal mutable List instance. To prevent accidental external mutation (and potential ConcurrentModificationException during iteration), consider returning an unmodifiable view or a defensive copy, and/or storing an immutable list in the field.

Copilot uses AI. Check for mistakes.
@mymeiyi

mymeiyi commented Mar 31, 2026

Copy link
Copy Markdown
Contributor Author

run buildall

@yiguolei
yiguolei merged commit 47e68de into apache:branch-4.0 Mar 31, 2026
30 of 35 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot
Copilot review effort
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments
@yiguolei yiguolei yiguolei approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@mymeiyi @hello-stephen @yiguolei