columnar: gc ia meta & schema files in cloud_helper (#10988)

[ti-chi-bot]

This is an automated cherry-pick of #10988

What problem does this PR solve?

Issue Number: close #10995
ref https://github.com/tidbcloud/cloud-storage-engine/issues/5686
close https://github.com/tidbcloud/cloud-storage-engine/issues/5716

Problem Summary:
The disk or memory can be keep increasing in the long run with no background gc.

What is changed and how it works?

1. Add gc for ia_meta files in disk. (In TiKV there has a background gc task to scan the local meta files and do the gc)
2. Add gc for schema_files map in memory.
3. Decrease the ia mem_cap default value from 20% to 10%.
4. Update kvengine to latest cloud-storage-engine commit b49d94ae594914225390e3914f92025f1486d95b to fix the columnar file decryption with re-mapped file id during packed restoration.

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)
• No code

Side effects

• Performance regression: Consumes more CPU
• Performance regression: Consumes more Memory
• Breaking backward compatibility

Documentation

• Affects user behaviors
• Contains syntax changes
• Contains variable changes
• Contains experimental features
• Changes MySQL compatibility

Release note

None

Summary by CodeRabbit

• New Features
  • Added a configuration option to cap cached schema keyspaces.
  • Added an optional background cleanup loop to evict stale schema files and perform IA metadata GC.
• Bug Fixes
  • Improved schema cache eviction by updating keyspace recency on snapshot access.
  • Refined shutdown coordination so the heartbeat and background cleanup stop reliably.
• Tests
  • Extended coverage for stale-version pruning, recency updates, and GC eviction (with and without keyspace limits), plus adjusted default assertions.
• Chores
  • Updated the cloud storage engine submodule and refreshed dependency pinning/config defaults.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

🗂️ Base branches to auto review (3)

• release-8.5
• release-7.5
• release-8.1

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 0ee9e9bb-d675-4be9-bc9a-672658acb59c

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: JaySon-Huang, yongman

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [JaySon-Huang,yongman]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ti-chi-bot]

[LGTM Timeline notifier]

Timeline:

• 2026-07-30 03:46:23.072269702 +0000 UTC m=+2067769.108364768: ☑️ agreed by yongman.
• 2026-07-30 03:47:17.479324154 +0000 UTC m=+2067823.515419210: ☑️ agreed by JaySon-Huang.