[fix](cloud) modify CloudTabletRebalancer and CloudTabletStatMgr to reduce memory by mymeiyi · Pull Request #61318 · apache/doris

mymeiyi · 2026-03-13T09:37:21Z

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Reduce FE memory by

moving top-N table stats filtering from PrometheusMetricVisitor into CloudTabletStatMgr so it's computed once per stat cycle instead of per Prometheus scrape,
removing the unused beToTablets field from InfightTask to avoid retaining a large map reference
changing InfightTablet.tabletId from Long to long to avoid boxing overhead.

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

hello-stephen · 2026-03-13T09:37:40Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

Copilot

Pull request overview

This PR refactors how cloud table statistics are prepared for Prometheus output by moving aggregation/filtering responsibilities from the Prometheus visitor into CloudTabletStatMgr, and includes a small cleanup in the cloud tablet rebalancer.

Changes:

Move “top-N by data size” filtering for per-table metrics from PrometheusMetricVisitor into CloudTabletStatMgr.
Add totalTableSize tracking + getter in CloudTabletStatMgr so Prometheus can emit a total even when per-table stats are filtered.
Simplify CloudTabletRebalancer inflight tracking by switching tabletId to primitive long and removing an unused task field/parameter.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 4 comments.

File	Description
`fe/fe-core/src/main/java/org/apache/doris/metric/PrometheusMetricVisitor.java`	Uses `CloudTabletStatMgr`’s pre-filtered table stats and new total size getter instead of computing/filtering locally.
`fe/fe-core/src/main/java/org/apache/doris/cloud/catalog/CloudTabletRebalancer.java`	Removes unnecessary boxing for `tabletId` and drops an unused `beToTablets` field/arg from inflight task handling.
`fe/fe-core/src/main/java/org/apache/doris/catalog/CloudTabletStatMgr.java`	Adds total table size state and implements top-N filtering during stats update to support Prometheus output efficiently.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review. Take the survey.

+        filterTopTableStatsByDataSize(newCloudTableStatsList);
+        this.totalTableSize = totalTableSize;


mymeiyi · 2026-03-13T09:51:10Z

/review

github-actions

Code Review Summary

PR Goal: Reduce FE memory by (1) moving top-N table stats filtering from PrometheusMetricVisitor into CloudTabletStatMgr so it's computed once per stat cycle instead of per Prometheus scrape, (2) removing the unused beToTablets field from InfightTask to avoid retaining a large map reference, and (3) changing InfightTablet.tabletId from Long to long to avoid boxing overhead.

Critical Checkpoint Conclusions

Does the code accomplish its goal? Is there a test that proves it? The code accomplishes the goal. The filtering logic is correctly moved, beToTablets is confirmed unused and safely removed, and the Long-to-long change is correct (including equals/hashCode adjustments). However, no tests are added or referenced -- none of the test checkboxes in the PR template are checked.
Is the modification minimal and focused? Yes -- three small, related memory-reduction changes across three files.
Concurrency? Both totalTableSize and cloudTableStatsList are volatile and written by a single daemon thread. The reference-swap pattern is safe. Minor note: a Prometheus scrape can observe the new filtered list with the old totalTableSize (or vice versa), but this is benign staleness.
Lifecycle / static init? No issues.
Configuration items? No new configs. Uses existing prom_output_table_metrics_limit.
Incompatible changes? None.
Parallel code paths? The filtering was in PrometheusMetricVisitor and is now in CloudTabletStatMgr. The move is complete -- old code is fully removed.
Special conditional checks? See inline comment about PriorityQueue with limit <= 0.
Test coverage? No tests added. A unit test for filterTopTableStatsByDataSize with various edge cases (empty list, list smaller than limit, list larger than limit, limit=0) would be valuable.
Observability? No regression -- same Prometheus metrics are output.
Performance? Good improvements: avoids Long boxing, drops unnecessary Map reference from in-flight tasks, and pre-filters the stats list once per cycle instead of per scrape.
Other issues? One edge-case bug found -- see inline comment.

…educe memory

mymeiyi · 2026-03-13T10:04:15Z

run buildall

doris-robot · 2026-03-13T10:30:50Z

TPC-H: Total hot run time: 27302 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6b6dbfa0f96ecb851efd8564d80c22cfd61a8000, data reload: false

------ Round 1 ----------------------------------
orders	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	17607	4478	4313	4313
q2	q3	10652	791	512	512
q4	4679	384	258	258
q5	7575	1205	1026	1026
q6	182	191	157	157
q7	784	885	679	679
q8	9298	1528	1406	1406
q9	4861	4778	4776	4776
q10	6242	1907	1664	1664
q11	464	257	255	255
q12	727	584	486	486
q13	18018	2909	2178	2178
q14	225	235	223	223
q15	q16	747	757	669	669
q17	736	875	480	480
q18	5918	5406	5208	5208
q19	1238	998	626	626
q20	542	492	384	384
q21	4760	1987	1687	1687
q22	479	365	315	315
Total cold run time: 95734 ms
Total hot run time: 27302 ms

----- Round 2, with runtime_filter_mode=off -----
orders	Doris	NULL	NULL	150000000	42	6422171781	NULL	22778155	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	4672	4585	4554	4554
q2	q3	3870	4332	3827	3827
q4	965	1266	827	827
q5	4124	4431	4362	4362
q6	185	183	148	148
q7	1780	1694	1515	1515
q8	2549	2715	2646	2646
q9	7832	7506	7704	7506
q10	3724	4055	3623	3623
q11	532	445	430	430
q12	508	610	454	454
q13	2715	3043	2307	2307
q14	284	292	271	271
q15	q16	737	765	751	751
q17	1213	1414	1428	1414
q18	7173	6809	6710	6710
q19	991	980	924	924
q20	2055	2153	2012	2012
q21	4105	3748	3588	3588
q22	493	469	442	442
Total cold run time: 50507 ms
Total hot run time: 48311 ms

doris-robot · 2026-03-13T10:41:33Z

TPC-DS: Total hot run time: 167620 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6b6dbfa0f96ecb851efd8564d80c22cfd61a8000, data reload: false

query5	4337	627	495	495
query6	329	227	207	207
query7	4206	468	263	263
query8	347	276	241	241
query9	8737	2776	2776	2776
query10	548	389	359	359
query11	6986	5087	4876	4876
query12	191	126	130	126
query13	1271	463	350	350
query14	5761	3723	3424	3424
query14_1	2817	2779	2899	2779
query15	217	191	174	174
query16	1028	465	460	460
query17	1109	717	625	625
query18	2575	450	360	360
query19	211	211	182	182
query20	135	130	127	127
query21	216	132	116	116
query22	13240	14011	14570	14011
query23	16483	15834	15653	15653
query23_1	15652	15437	15213	15213
query24	7155	1609	1225	1225
query24_1	1222	1216	1253	1216
query25	538	456	403	403
query26	1229	276	143	143
query27	2761	481	294	294
query28	4429	1879	1876	1876
query29	812	564	473	473
query30	301	229	195	195
query31	1006	936	867	867
query32	79	68	67	67
query33	519	331	275	275
query34	889	885	541	541
query35	641	670	613	613
query36	1088	1147	1002	1002
query37	129	94	83	83
query38	2958	2987	2910	2910
query39	862	832	814	814
query39_1	780	782	796	782
query40	228	149	133	133
query41	62	59	57	57
query42	256	256	253	253
query43	243	259	230	230
query44	
query45	200	186	181	181
query46	873	975	609	609
query47	2133	2192	2052	2052
query48	317	313	227	227
query49	630	453	375	375
query50	737	273	218	218
query51	4046	4057	4075	4057
query52	259	265	258	258
query53	288	338	285	285
query54	294	269	261	261
query55	88	89	83	83
query56	313	322	312	312
query57	1918	1880	1470	1470
query58	288	278	264	264
query59	2809	2928	2731	2731
query60	342	333	324	324
query61	149	149	146	146
query62	621	583	556	556
query63	334	283	281	281
query64	4990	1275	980	980
query65	
query66	1427	458	351	351
query67	24321	24269	24092	24092
query68	
query69	406	323	293	293
query70	937	948	889	889
query71	331	310	304	304
query72	2744	2620	2515	2515
query73	549	538	323	323
query74	9621	9604	9389	9389
query75	2890	2757	2483	2483
query76	2308	1025	691	691
query77	377	416	327	327
query78	10939	11115	10430	10430
query79	1106	785	589	589
query80	764	616	529	529
query81	497	256	222	222
query82	1357	157	117	117
query83	381	256	238	238
query84	250	122	95	95
query85	849	500	438	438
query86	381	315	299	299
query87	3168	3130	3029	3029
query88	3539	2680	2658	2658
query89	431	371	347	347
query90	2010	177	176	176
query91	172	169	132	132
query92	77	80	68	68
query93	887	845	496	496
query94	445	312	302	302
query95	580	405	317	317
query96	629	521	229	229
query97	2472	2495	2399	2399
query98	248	242	221	221
query99	1016	1015	892	892
Total cold run time: 247977 ms
Total hot run time: 167620 ms

hello-stephen · 2026-03-13T11:27:10Z

FE UT Coverage Report

Increment line coverage 4.17% (1/24) 🎉
Increment coverage report
Complete coverage report

github-actions · 2026-03-13T15:33:46Z

PR approved by at least one committer and no changes requested.

github-actions · 2026-03-13T15:33:49Z

PR approved by anyone and no changes requested.

dataroaring

LGTM

…educe memory (apache#61318) Issue Number: close #xxx Related PR: #xxx Problem Summary: Reduce FE memory by 1. moving top-N table stats filtering from PrometheusMetricVisitor into CloudTabletStatMgr so it's computed once per stat cycle instead of per Prometheus scrape, 2. removing the unused beToTablets field from InfightTask to avoid retaining a large map reference 3. changing InfightTablet.tabletId from Long to long to avoid boxing overhead. None - Test  - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason  - Behavior changed: - [ ] No. - [ ] Yes.  - Does this need documentation? - [ ] No. - [ ] Yes.  - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label

pick: 1. modify CloudTabletRebalancer and CloudTabletStatMgr to reduce memory (#61318) 2. cloud reduce get_tablet_stats rpc to meta_service (#60543) 3. checkpoint save cloud tablet stats to image (#60705)

pick: 1. modify CloudTabletStatMgr to reduce memory (#59776) 2. modify CloudTabletRebalancer and CloudTabletStatMgr to reduce memory (#61318)

### What problem does this PR solve? Issue Number: None Related PR: apache#61318 Problem Summary: Cloud tablet route rebuilding repeatedly boxes primitive backend, table, partition, index, and tablet IDs while inserting the same logical IDs into current and future global, table, and partition indexes. Hoist boxing to traversal callers and preserve selected boxed tablet IDs through direct transfers, warmup moves, and warmup rollbacks so all route index families reuse immutable Long references without changing routing or scheduling semantics. A single-threaded JDK 17 allocation model that keeps eager container candidates in both variants estimates that 4 million tablets across 4 clusters reduce cumulative allocation from 25.47 GiB to 19.76 GiB (22.42%) and approximate post-full-GC retained heap from 6.80 GiB to 4.94 GiB (27.37%). For IDs outside the Long cache, each direct incremental move additionally avoids two tablet wrapper allocations and one duplicate retained tablet wrapper; warmup and rollback each avoid one tablet wrapper allocation. These are path-level model and identity-test results, not production RSS measurements. ### Release note None ### Check List (For Author) - Test: Unit Test and Manual test\n - ./run-fe-ut.sh --run org.apache.doris.cloud.catalog.CloudTabletRebalancerTest (13 tests passed)\n - mvn checkstyle:check -pl fe-core (0 violations)\n - Multi-scale JDK 17 eager-container allocation model at 0.2M, 0.4M, 0.8M, and 3.2M tablet-cluster pairs, three runs per scale\n- Behavior changed: No\n- Does this need documentation: No

Copilot AI review requested due to automatic review settings March 13, 2026 09:37

mymeiyi requested review from dataroaring, gavinchou and w41ter as code owners March 13, 2026 09:37

Copilot started reviewing on behalf of mymeiyi March 13, 2026 09:38 View session

Copilot AI reviewed Mar 13, 2026

View reviewed changes

mymeiyi force-pushed the cloud-fe-memory branch from 8f18a44 to 98d541e Compare March 13, 2026 09:48

github-actions Bot reviewed Mar 13, 2026

View reviewed changes

Comment thread fe/fe-core/src/main/java/org/apache/doris/catalog/CloudTabletStatMgr.java

[fix](cloud) modify CloudTabletRebalancer and CloudTabletStatMgr to r…

6b6dbfa

…educe memory

mymeiyi force-pushed the cloud-fe-memory branch from 98d541e to 6b6dbfa Compare March 13, 2026 10:00

gavinchou approved these changes Mar 13, 2026

View reviewed changes

github-actions Bot added the approved Indicates a PR has been approved by one committer. label Mar 13, 2026

github-actions Bot added the reviewed label Mar 13, 2026

dataroaring approved these changes Mar 14, 2026

View reviewed changes

dataroaring added dev/4.0.x dev/4.1.x labels Mar 14, 2026

dataroaring merged commit c28fe0c into apache:master Mar 14, 2026
31 of 33 checks passed

github-actions Bot added the dev/4.0.x-conflict label Mar 14, 2026

mymeiyi mentioned this pull request Mar 30, 2026

branch-4.0: pick #60543, #60705 #61870

Closed

mymeiyi mentioned this pull request Mar 30, 2026

branch-4.1: pick #61318, #60543, #60705 #61874

Merged

yiguolei added dev/4.1.0-merged and removed dev/4.1.x labels Mar 31, 2026

mymeiyi mentioned this pull request Mar 31, 2026

branch-4.0: pick #59776, #61318 #61930

Merged

yiguolei pushed a commit that referenced this pull request Mar 31, 2026

branch-4.0: pick #59776, #61318 (#61930)

47e68de

pick: 1. modify CloudTabletStatMgr to reduce memory (#59776) 2. modify CloudTabletRebalancer and CloudTabletStatMgr to reduce memory (#61318)

mymeiyi added dev/4.0.6-merged and removed dev/4.0.x dev/4.0.x-conflict labels May 28, 2026

morningman mentioned this pull request Jun 3, 2026

4.0.6 Release Notes #64065

Open

		filterTopTableStatsByDataSize(newCloudTableStatsList);
		this.totalTableSize = totalTableSize;

Uh oh!

Conversation

mymeiyi commented Mar 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

hello-stephen commented Mar 13, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mymeiyi commented Mar 13, 2026

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Code Review Summary

Critical Checkpoint Conclusions

Uh oh!

Uh oh!

mymeiyi commented Mar 13, 2026

Uh oh!

doris-robot commented Mar 13, 2026

Uh oh!

doris-robot commented Mar 13, 2026

Uh oh!

hello-stephen commented Mar 13, 2026

FE UT Coverage Report

Uh oh!

github-actions Bot commented Mar 13, 2026

Uh oh!

github-actions Bot commented Mar 13, 2026

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

mymeiyi commented Mar 13, 2026 •

edited

Loading