columnar hub: share process-global ColumnarMetaCache across snaps (#11022)

[ti-chi-bot]

This is an automated cherry-pick of #11022

What problem does this PR solve?

Issue Number: close #11021

Problem Summary:

On disagg columnar CN Hub, each snap historically constructed its own ColumnarMetaCache via ColumnarMetaCache::default() (mem_limit/500 per snap). Concurrent region snaps therefore held independent meta caches, amplifying TableMeta memory even when keys (file_id, table_id) overlap. This diverges from WN's process-global meta cache.

What is changed and how it works?

columnar hub: share ColumnarMetaCache at CloudHelper and size it at mem/500

• Keep one ColumnarMetaCache on CloudHelper and clone() it into each SnapCtx (same pattern as columnar_file_cache / other process-global caches).
• Size the shared cache at SysQuota::memory_limit_in_bytes() / 500 (~0.2% of process memory). Per-snap default() used mem/500 each.
• Does not include contrib/cloud-storage-engine weight/metrics changes; those can land separately.

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)
• No code

Manual test (lab CN tiflash-5035 / 5036, table widecol.widecol_test_wide_500gb):

1. Case 1 (N=1 large range) and Case 6 (N=4 disjoint ranges) from CN Hub creates per-snap ColumnarMetaCache, amplifying TableMeta memory under disagg columnar #11021.
2. Per-snap baseline: Σ reported_size_bytes ≈ 1.2GB for ~5070 parses while last-instance weighted_size / entries only reflected one snap (~7MiB / ~30).
3. Shared + small cap (64MiB) experiment: process-level gauges (weighted≈cap, same entries on both CNs) and thrash (parse ≫ unique files).

Side effects

• Performance regression: Consumes more CPU
• Performance regression: Consumes more Memory
• Breaking backward compatibility

Documentation

• Affects user behaviors
• Contains syntax changes
• Contains variable changes
• Contains experimental features
• Changes MySQL compatibility

Release note

columnar hub: share one process-global ColumnarMetaCache across snaps to avoid per-snap TableMeta memory amplification

Summary by CodeRabbit

Performance Improvements

• Improved columnar data access by sharing metadata caching across readers and snapshot operations.
• Cache capacity now adapts to the configured memory limit, reducing repeated metadata loading and supporting more efficient memory usage.

Observability

• Added clearer status information when heap profiling is activated or deactivated.
• Expanded Grafana monitoring with columnar storage metrics for cache usage, remote reads, prefetching, snapshots, memory waits, and read duration.
• Improved dashboard defaults for monitoring columnar storage activity.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

🗂️ Base branches to auto review (3)

• release-8.5
• release-7.5
• release-8.1

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: ca06e86c-7542-40eb-9231-9780f92fdaa9

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: JaySon-Huang, JinheLin

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [JaySon-Huang,JinheLin]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ti-chi-bot]

[LGTM Timeline notifier]

Timeline:

• 2026-08-10 03:34:23.930761957 +0000 UTC m=+3017449.966857023: ☑️ agreed by JaySon-Huang.
• 2026-08-10 03:39:25.93925549 +0000 UTC m=+3017751.975350546: ☑️ agreed by JinheLin.