Phase 11.12: pack slow_path_allocs into combined counter - #65
Merged
Merged
Conversation
Disassembly of `_malloc` on the Phase 11.11 baseline showed the
BASIC tier `medium_allocs` residual cost concentrated at two
adjacent counter stores on the small-refill slow path:
- `stats.slow_path_allocs++` at the entry to `small_refill`
(ldr/add/str on field 0x2388).
- `stats.fast_path_allocs += refill_count` at the refill site
(ldr/add/str on adjacent field 0x2380).
`medium_allocs` (4 KiB allocations) hits `small_refill` more
often than `small_allocs` because each chunk yields fewer
objects per refill, so the per-refill counter cost is the
residual.
Pack the two fields into one 64-bit `FrontendStats::packed_allocs`:
- bits 0-47: cumulative_allocs (fast + slow combined)
- bits 48-63: slow-path call count
At the refill site the two stores collapse into ONE packed `+=`:
stats.packed_allocs +=
static_cast<uint64_t>(refill_count) + PACKED_ALLOCS_SLOW_INC;
The two lanes occupy disjoint bit ranges so the packed `+=` is
correct as long as neither lane overflows its sub-field width.
The 16-bit slow lane saturates at 65535 refills (~16M allocs
per thread for the smallest sizeclasses); effectively unbounded
for any realistic workload on an observability surface.
The `FullAllocStats` FFI struct is unchanged: at aggregation
time `stats_export.cc` decodes the packed word back into the
public `fast_path_allocs` and `slow_path_allocs` fields. The
`FrontendStatsGlobal` thread-exit aggregator drops to a single
`fetch_add` for the combined counter.
Bench results (apple silicon, paired OFF/BASIC):
group | OFF (ns) | BASIC (ns) | ratio |
small_allocs | ~203.7 | ~203.7 | 1.00 |
medium_allocs | ~1039 | ~1032 | 0.99 |
mixed | ~612 | ~612 | 1.00 |
vs Phase 11.11 baseline (medium 1.122) -- medium drops to 0.99
(within bench noise of stats-off), all groups <= 1.02.
Disassembly delta: the 3-inst `slow_path_allocs++` block at the
entry to the inlined `small_refill` is gone; the
`fast_path_allocs +=` becomes a 6-inst packed update with one
constant materialization for `1ULL << 48`. Net -1 inst in the
inlined body and -1 STORE to a separate counter field per
slow-path call.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Phase 11.11 disassembly diff isolated the BASIC tier
medium_allocsresidual to two adjacent counter store-bursts on thesmall_refillslow path:stats.slow_path_allocs++at the entry tosmall_refill(3 inst on field offset 0x2388)stats.fast_path_allocs += refill_countat the refill site (3 inst on adjacent offset 0x2380)medium_allocs(4 KiB) hitssmall_refillmore often thansmall_allocsbecause each chunk yields fewer objects per refill, so the per-refill counter cost dominates the residual.This PR packs both counters into one 64-bit
FrontendStats::packed_allocs:At the refill site, the two stores collapse into ONE packed
+=:The two lanes occupy disjoint bit ranges; the packed
+=is correct as long as neither lane overflows its sub-field width. The 16-bit slow lane saturates at 65535 refills (~16M allocs per thread) -- unbounded for any realistic workload on an observability surface.FullAllocStatsABI is unchanged:stats_export.ccdecodes the packed word back into the publicfast_path_allocs/slow_path_allocsfields at snapshot time.Disassembly delta (
_mallocbody, arm64, BASIC=ON)Phase 11.11 (parent commit
337bd4d):Phase 11.12:
Net change: the 3-instruction
slow_path_allocs++block at the inlinedsmall_refillentry is gone; thefast_path_allocs +=becomes a 6-inst packed update (one extramovfor the1ULL<<48constant materialization). Net -1 inst in the inlined_mallocbody and -1 STORE to a separate counter field per slow-path call -- the cache-line write reduction is the win that shows up at bench time.Bench results (apple silicon, paired OFF/BASIC)
vs Phase 11.11 baseline (
medium_allocswas 1.122), all three groups land at or below the 1.02 BASIC bar;medium_allocsis now within the noise envelope of stats-off.Test plan
cmake -B build -DSNMALLOC_STATS_BASIC=ON && cmake --build build -j-- clean build./build/func-fast_path_counters-fast-- passes (fast_path_allocs >= N - K,slow_path_allocs >= 1)./build/func-statistics-check-- passescargo test --features stats-basic-- 33 passedcargo bench --bench stats_bench [--features stats-basic]-- five paired passes; ratios aboveotool -tvVdiff of_mallocbody shows the slow_path_allocs++ block gone, packed-update merged into the refill-site storeDetails and a re-run recipe live in
docs/heap-profiling-benchmarks.mdunder "Phase 11.12 -- packed slow_path counter".ClickUp: 86aj12be5