Skip to content

fix: valgrind on ARM#21

Merged
not-matthias merged 7 commits into
masterfrom
cod-2985-arm-flamegraph-failures
Jul 8, 2026
Merged

fix: valgrind on ARM#21
not-matthias merged 7 commits into
masterfrom
cod-2985-arm-flamegraph-failures

Conversation

@not-matthias

@not-matthias not-matthias commented Jun 30, 2026

Copy link
Copy Markdown
Member

FYI: fde493a is 100% Claude generated and only used for debugging

@codspeed-hq

codspeed-hq Bot commented Jun 30, 2026

Copy link
Copy Markdown

Merging this PR will degrade performance by 90.69%

❌ 1 regressed benchmark
✅ 29 untouched benchmarks
⏩ 100 skipped benchmarks1

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Benchmark BASE HEAD Efficiency
test_valgrind[valgrind.codspeed, echo Hello, World!, full-no-inline] 575 ms 6,179.2 ms -90.69%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing cod-2985-arm-flamegraph-failures (aa6643d) with master (ce9d871)

Open in CodSpeed

Footnotes

  1. 100 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

@not-matthias
not-matthias force-pushed the cod-2985-arm-flamegraph-failures branch 6 times, most recently from 5e34b22 to 771904a Compare July 3, 2026 08:54
@not-matthias
not-matthias marked this pull request as ready for review July 3, 2026 08:55
@greptile-apps

greptile-apps Bot commented Jul 3, 2026

Copy link
Copy Markdown

Greptile Summary

This PR fixes ARM64 call-graph correctness in valgrind-codspeed by addressing several interrelated issues: the VEX IR mislabelled every unconditional B instruction as Ijk_Call, corrupting recursive and cyclic call graphs; the frame-pointer fallback unwinder was missing for JIT frames; TLS descriptor resolvers were not being skipped like PLT stubs; and NixOS debug-info directories were not searched. A new callgrind-utils Rust crate is introduced to parse Callgrind output and build call graphs and flamegraphs, with an extensive integration test suite.

  • VEX fix (guest_arm64_toIR.c): B (plain branch) now emits Ijk_Boring instead of Ijk_Call, so only BL is treated as a call.
  • ARM64 call-stack recording (callstack.c, m_stacktrace.c): guest X30 is read as the architectural return target for real calls; a frame-pointer chain unwinder is added as CFI fallback; the +1 bias fix in reconstruct_call_stack_from_native correctly undoes the symbolization -1 offset.
  • callgrind-utils crate (parser, flamegraph, perf_map, redact): full Callgrind output parser with folded-stack flamegraph generation, JIT perf-map symbolisation, and snapshot regression tests for ARM64-specific edge cases.

Confidence Score: 5/5

Safe to merge. All ARM64 logic changes are well-reasoned, thoroughly commented, and covered by integration test snapshots.

The core C changes (VEX IR, callstack, stacktrace, fn, readelf) are correct and well-documented. The new callgrind-utils Rust crate has solid test coverage with snapshot tests for ARM64-specific edge cases. The only issue found is stylistic: eprintln! in test files without a tracing dependency, which does not affect correctness.

No files require special attention. Test files use eprintln! against the team style rule but this is a minor style concern.

Important Files Changed

Filename Overview
VEX/priv/guest_arm64_toIR.c Corrects B vs BL classification: plain unconditional branch now emits Ijk_Boring instead of Ijk_Call, fixing corrupted call graphs on ARM64.
callgrind/callstack.c ARM64-specific return-address recording reads guest X30 directly instead of statically computing the post-call instruction address; reconstruct_call_stack_from_native undoes the -1 symbolization bias with a +1 correction.
coregrind/m_stacktrace.c Adds AAPCS64 frame-pointer chain fallback (X29/X30) when CFI is missing; guards are aligned with the existing amd64 %rbp fallback pattern.
callgrind/fn.c ARM64 TLS descriptor resolvers (dl_tlsdesc*) are now treated as skip-on-PLT trampolines, preventing phantom recursion nodes in CPython flamegraphs.
coregrind/m_debuginfo/readelf.c Adds NIX_DEBUG_INFO_DIRS and extrapath lookup before the FHS /usr/lib/debug fallback for build-id-based debug files; correctly parses colon-separated dir lists.
callgrind-utils/build.rs Build script that auto-configures and builds the in-repo Callgrind before tests; correctly relies on configure.ac reading CAPSTONE_DIR from the environment when --with-capstone is not passed.
callgrind-utils/src/parser/mod.rs Full Callgrind output parser handling name-compression ID spaces, cost-line layout, and call edge extraction; rule-compliant multi-clause guard is extracted to has_valid_token_count.
callgrind-utils/src/flamegraph.rs DFS-based folded-stack flamegraph builder with budget pruning; multi-clause guard correctly extracted to should_prune.
callgrind-utils/tests/rust_callgraph.rs Integration tests for Rust fixture call graphs; uses eprintln! for tool-not-found skip messages, violating the team's no-eprintln rule.

Sequence Diagram

%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
    participant VEX as "VEX IR (arm64_toIR)"
    participant CG as "Callgrind (callstack/fn)"
    participant ST as "Stacktrace (m_stacktrace)"
    participant ELF as "readelf.c"
    participant CU as "callgrind-utils (Rust)"

    Note over VEX: B gets Ijk_Boring, BL gets Ijk_Call

    VEX->>CG: "jk_Call event (BL only)"
    CG->>ST: "VG_get_StackTrace for seed"
    ST-->>CG: "ips[] with pc-1 bias"
    Note over CG: "+1 correction restores ret_addr"
    Note over CG: "read guest X30 for real calls"

    Note over ST: "CFI unwind attempt"
    ST->>ST: "use_CF_info success: ips[i]=pc-1"
    ST->>ST: "CFI fail: walk AAPCS64 X29 chain"

    Note over CG: "_dl_tlsdesc_* skipped like PLT"

    ELF->>ELF: "NIX_DEBUG_INFO_DIRS then extrapath then /usr/lib/debug"

    CU->>CU: "parse Callgrind .out to CallGraph"
    CU->>CU: "symbolize JIT frames via perf map"
    CU->>CU: "fold stacks to flamegraph SVG"
Loading
%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
    participant VEX as "VEX IR (arm64_toIR)"
    participant CG as "Callgrind (callstack/fn)"
    participant ST as "Stacktrace (m_stacktrace)"
    participant ELF as "readelf.c"
    participant CU as "callgrind-utils (Rust)"

    Note over VEX: B gets Ijk_Boring, BL gets Ijk_Call

    VEX->>CG: "jk_Call event (BL only)"
    CG->>ST: "VG_get_StackTrace for seed"
    ST-->>CG: "ips[] with pc-1 bias"
    Note over CG: "+1 correction restores ret_addr"
    Note over CG: "read guest X30 for real calls"

    Note over ST: "CFI unwind attempt"
    ST->>ST: "use_CF_info success: ips[i]=pc-1"
    ST->>ST: "CFI fail: walk AAPCS64 X29 chain"

    Note over CG: "_dl_tlsdesc_* skipped like PLT"

    ELF->>ELF: "NIX_DEBUG_INFO_DIRS then extrapath then /usr/lib/debug"

    CU->>CU: "parse Callgrind .out to CallGraph"
    CU->>CU: "symbolize JIT frames via perf map"
    CU->>CU: "fold stacks to flamegraph SVG"
Loading

Reviews (5): Last reviewed commit: "feat(callgrind-utils): add perf_map symb..." | Re-trigger Greptile

Comment thread coregrind/m_stacktrace.c
Comment thread callgrind-utils/src/flamegraph.rs Outdated
Comment thread callgrind-utils/src/parser/mod.rs Outdated
@not-matthias
not-matthias force-pushed the cod-2985-arm-flamegraph-failures branch 2 times, most recently from 9db3f68 to 0a322a7 Compare July 3, 2026 12:02

@art049 art049 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks basically ok. That's a huge testing infra addition so I hope we can manage and maintain it down the line.

The AArch64 B{L} decoder tagged the whole opcode group as Ijk_Call,
but only BL (bit 31 = 1, writes the link register) is a call; a plain
B (bit 31 = 0) is an ordinary unconditional branch.

Mislabelling B as a call made Callgrind treat every branch to a
function epilogue or tail target as a call. At -O0 a conditional like
`return n < 2 ? n : fib(...)` compiles the base case to `b <epilogue>`,
so each base case was counted as a recursive call -- inflating
recursive/cyclic call graphs and inventing phantom self-edges on arm64
(e.g. fib recursion 64 -> 98; mutual is_even/is_odd gaining self-loops).

Align plain B with B.cond and the register-indirect JMP, which already
use Ijk_Boring. Fixes the callgrind-utils recursion/mutual snapshot
failures.
On arm64, bl/blr write the return address to X30 and SP does not move
across a call/return pair, so unlike on x86 the return detector cannot
fall back on SP progress and depends entirely on each frame's recorded
ret_addr. When a call entered skipped code (a libc PLT hop), the
skipped->nonskipped jump was pushed with setup_bbcc's spliced
'nonskipped' source BB, whose last jump is the very call that created
the skip frame below: the emulated frame duplicated that frame's
statically computed return address, the callee's single ret popped only
the top entry, and the leaked equal-SP skip frame starved the pop
budget of the next same-SP return. Misclassified returns were then
re-promoted into phantom calls back into the live caller, cloning
non-recursive functions as bogus 'N recursion levels
(complex_fractal_benchmark'2) and misattributing follow-up work
("free calls X").

Record the guest X30 -- the architectural return target -- for frames
entered by a real call, and record ret_addr = 0 for emulated/spliced
pushes so the return matcher absorbs them down the same-SP run and pops
the group at the frame of the real call underneath, which also restores
the pre-call nonskipped state. x86 keeps the static computation and is
behaviorally unchanged.

Verified by the arm64_plt_phantom_recursion / arm64_free_tailcall_phantom
fixture snapshots (flipped to the correct shapes), the structural guards
in arm64_fractal_alloc_no_free_misattribution, the no-phantom-clone
assertion in rust_fixture_full_trace, and an unchanged in-tree
'vg_regtest callgrind' pass/fail set.
VG_(get_StackTrace) on arm64 was CFI-only, so Callgrind's OFF->ON
shadow-stack seeding stopped at the first frame without unwind info --
notably CPython's -X perf JIT trampolines, which have no FDEs but keep
the AAPCS64 frame chain alive (that is their design: perf/samply walk
them by FP). The truncated seed left the seeded context stack one entry
deep after the innermost frame popped; bbcc.c's underflow check then
misread the fn-stack base sentinel as a signal marker on every return,
and handleUnderflow fabricated named nodes for obj-skipped interpreter
functions with inverted, full-cost edges (_ctypes_callproc ->
PyCFuncPtr_call -> _TAIL_CALL_* -> ... as the graph root). x86_64 never
hit this because its unwinder already has the %rbp fallback.

Follow the frame records {saved X29, saved X30} when CFI fails, with
guards: record in-stack and 8-aligned, SP must progress, next record
strictly higher (saved X29 == 0 accepted as chain terminator), pc 0/1
stops the walk. Caller IPs keep the CFI path's -1 bias.

Callgrind's seeder correspondingly records ret_addr = ips[frame+1] + 1,
undoing that bias so the arm64 return matcher (exact-X30 matching, no
SP movement on bl/ret) can pop seeded frames.

An A/B run confirmed raising CLG_RECON_MAX_FRAMES alone does not help:
the seed was CFI-truncated at ~8 frames, nowhere near the 256 cap.

Regression coverage: callgrind-utils/tests/objskip_seed_underflow.rs, a
minimal fixture whose asm trampoline maintains FP but has no CFI and
which starts instrumentation two obj-skipped frames deep; it asserts no
skipped frame leaks into the folded graph and that the workload parents
under the trampoline.
Every TLSDESC __thread access blr's into the dynamic linker's resolver,
which rets straight back into the middle of the accessing function. When
the access is made from an obj-skipped object (CPython under
pytest-codspeed), the skipped->nonskipped splice pushed the resolver
frame with ret_addr 0; its mid-function return could never match, the
RET-w/o-CALL promotion re-entered the skipped object with nonskipped
pointing at the resolver, and skipped cost plus call edges piled up
under _dl_tlsdesc_return -- pulling nearly whole Python flamegraphs
under that node, plus inverted return-direction edges.

Mark _dl_tlsdesc_* skipped (gated on --skip-plt, arm64 only), the same
transparent-trampoline class as _dl_runtime_resolve; pop_on_jump cannot
apply since these exit via a plain ret to a non-entry address. Skip
pushes are never spliced and record the architectural X30, so the
return pops cleanly and cost keeps flowing to the real non-skipped
caller.

Regression: callgrind-utils/tests/arm64_tls_access.rs (plain +
--obj-skip runs of the shared-lib TLS fixture); both tests fail on the
unfixed tool.
find_debug_file() only searched the /usr/lib/debug tree, which does not
exist on NixOS: Nix ships separate debug outputs under their own store
paths. Factor the build-id .build-id/xx/yyyy.debug probe into
try_buildid_dir() and honour NIX_DEBUG_INFO_DIRS -- the established
colon-separated convention also used by the nixpkgs gdb/lldb wrappers --
so split debug info resolves inside the dev-shell.
…megraph output

New Rust crate (edition 2024) that reads a Callgrind .out profile and
extracts call-graph topology (costs/addresses ignored), serializing to
canonical index-ref JSON for stable cross-platform callgraph diffing and
to folded/flamegraph stacks.

Node identity is the {object,file,function} tuple so same-named statics
stay distinct. Edges are emitted only on calls= lines; name compression
across the three ID spaces, the cfl/cfi alias, inline fi/fe
callee-context inheritance, and multi-part merge are handled. A redaction
pass strips volatile addresses/paths so snapshots stay portable.

Fixtures are compiled and profiled by the in-repo Callgrind through an
rstest harness (vg-in-place, --instr-atstart=no plus client requests
keep loader/libc frames out): base graphs (recursion, chain, diamond,
mutual), arm64 unwind fixtures (tail calls, recursion, TLS access,
alloc/free cycles, longjmp, phantom recursion), the objskip seeding
underflow regression, and the arm64 TLS-descriptor regression.
Folded-stack snapshots and structural assertions; clippy and rustfmt
clean.
…frames

Add the `perf_map` module: `CallGraph::symbolize_perf_map` resolves Callgrind's anonymous `0x...` JIT nodes to `py::<qualname>:<file>` via CPython's `/tmp/perf-<pid>.map`, written under `python3 -X perf`. Add the `fractal.py` fixture and `python_fractal_callgraph` test that profiles it live and snapshots the folded stacks and canonical JSON.
@not-matthias

Copy link
Copy Markdown
Member Author

@greptile do a final extensive review to ensure that no issues are in the code. ignore the commits for the callgrind-utils test infrastructure.

@not-matthias
not-matthias force-pushed the cod-2985-arm-flamegraph-failures branch from 0a322a7 to aa6643d Compare July 8, 2026 09:51
@not-matthias
not-matthias merged commit aa6643d into master Jul 8, 2026
7 of 9 checks passed
@not-matthias
not-matthias deleted the cod-2985-arm-flamegraph-failures branch July 8, 2026 10:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants