Skip to content

feat(os): enable four opt-in guest kernel performance features - #1504

Open
kvinwang wants to merge 1 commit into
nextfrom
feat/guest-kernel-perf
Open

kvinwang wants to merge 1 commit into
nextfrom
feat/guest-kernel-perf

Conversation

@kvinwang

@kvinwang kvinwang commented Oct 29, 2026 •

Copy link
Copy Markdown
Collaborator
Option Effect Opt-in
READ_ONLY_THP_FOR_FS=y Read-only file mappings (program/library text) can be collapsed into huge pages. Without it, code runs from 4 KiB pages, and every iTLB miss pays the two-dimensional (guest + EPT) page walk MADV_HUGEPAGE / MADV_COLLAPSE, or THP always
LRU_GEN=y Multi-Gen LRU, built in but off (LRU_GEN_ENABLED unset) echo y > /sys/kernel/mm/lru_gen/enabled
TCP_CONG_BBR=y Built in, so a container can select it in its own netns without loading a module. Default stays cubic net.ipv4.tcp_congestion_control=bbr
DEFERRED_STRUCT_PAGE_INIT=y Struct pages are initialised by parallel kthreads after SMP bring-up always; it only moves boot work

All four are added to both backends and asserted in parity.json.

Testing:

  • make os-image passes, and the built .config has all four options.
  • A/B in a QEMU/KVM guest (non-TD, Xeon Gold 6530 host), using this config with and without the four options:
    • MADV_COLLAPSE on a read-only executable mapping of a 64 MiB ext4 file: EINVAL → 0, FilePmdMapped: 65536 kB.
    • lru_gen/enabled absent → 0x0000, and switches to 0x0007.
    • bbr is available and selectable.
    • With 256 GiB and 64 vCPUs, the kernel reaches userspace in 0.80–0.90 s instead of 2.54–2.67 s.
  • TDX guest (dev image, 32 vCPUs, 128 GiB, ZFS data disk, KMS and gateway on):
    • Boots; node 0 deferred pages initialised in 23ms. Run /init at 7.23–7.24 s vs 7.44–8.15 s for next without this PR.
    • MADV_COLLAPSE with data checks before and after collapse, and after a write through a new O_WRONLY open, on ZFS (64 MiB–1 GiB), ext4, the squashfs rootfs and an overlay2-on-ZFS file in an unprivileged container: all consistent. On ZFS (no large folios) the first writable open drops the collapsed folios; ext4 keeps them.
    • The khugepaged path (MADV_HUGEPAGE) on ZFS collapses as well.
    • 300 rounds of four MADV_COLLAPSE loops racing a writer on one ZFS file (~26k collapses): no mapping/pread mismatch, no kernel warnings, zpool scrub clean.
    • An unprivileged container sets bbr in its netns; the host stays on cubic. lru_gen toggles 0x0000 ↔ 0x0007.

Base automatically changed from feat/guest-thp to next October 30, 2026 00:35
@kvinwang
kvinwang force-pushed the feat/guest-kernel-perf branch from 782e517 to 6405796 Compare October 30, 2026 00:35
- READ_ONLY_THP_FOR_FS: khugepaged can collapse read-only file mappings
  (program and library text) into huge pages, for regions marked
  MADV_HUGEPAGE or everything under THP "always". Without it, large-code
  workloads run their text from 4 KiB pages, and every iTLB miss pays the
  two-dimensional (guest + EPT) page walk.
- LRU_GEN: Multi-Gen LRU, built in but off by default (LRU_GEN_ENABLED unset);
  a workload under memory pressure turns it on through
  /sys/kernel/mm/lru_gen/enabled.
- TCP_CONG_BBR: built in rather than a module, so a container can select it
  in its own network namespace without the privilege to load a module. The
  default stays cubic.
- DEFERRED_STRUCT_PAGE_INIT: struct pages for most of memory are initialised
  by parallel kthreads after SMP bring-up instead of on the boot CPU, which
  shortens boot of large guests.

None of them changes behaviour unless a workload opts in, except the deferred
initialisation, which only moves work at boot. Added to both backends and
asserted in parity.json.

Signed-off-by: Kevin Wang <wy721@qq.com>
@kvinwang
kvinwang force-pushed the feat/guest-kernel-perf branch from 6405796 to bcd2035 Compare October 30, 2026 02:23

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant