Skip to content

Phase 11.12: pack slow_path_allocs into combined counter - #65

Merged
jayakasadev merged 1 commit into
mainfrom
feature/phase-11-12-packed-slow-counter
Jun 12, 2026
Merged

jayakasadev merged 1 commit into
mainfrom
feature/phase-11-12-packed-slow-counter

Conversation

@jayakasadev

Copy link
Copy Markdown
Owner

Summary

Phase 11.11 disassembly diff isolated the BASIC tier medium_allocs residual to two adjacent counter store-bursts on the small_refill slow path:

  • stats.slow_path_allocs++ at the entry to small_refill (3 inst on field offset 0x2388)
  • stats.fast_path_allocs += refill_count at the refill site (3 inst on adjacent offset 0x2380)

medium_allocs (4 KiB) hits small_refill more often than small_allocs because each chunk yields fewer objects per refill, so the per-refill counter cost dominates the residual.

This PR packs both counters into one 64-bit FrontendStats::packed_allocs:

  • bits 0-47: cumulative allocs (fast + slow combined)
  • bits 48-63: slow-path call count

At the refill site, the two stores collapse into ONE packed +=:

stats.packed_allocs +=
  static_cast<uint64_t>(refill_count) +
  FrontendStats::PACKED_ALLOCS_SLOW_INC;  // 1ULL << 48

The two lanes occupy disjoint bit ranges; the packed += is correct as long as neither lane overflows its sub-field width. The 16-bit slow lane saturates at 65535 refills (~16M allocs per thread) -- unbounded for any realistic workload on an observability surface.

FullAllocStats ABI is unchanged: stats_export.cc decodes the packed word back into the public fast_path_allocs / slow_path_allocs fields at snapshot time.

Disassembly delta (_malloc body, arm64, BASIC=ON)

Phase 11.11 (parent commit 337bd4d):

; slow_path_allocs++ block at small_refill entry (3 inst):
0x4098  ldr  x8, [x1, #0x2388]
0x409c  add  x8, x8, #0x1
0x40a0  str  x8, [x1, #0x2388]
; ... refill site ...
0x416c  and  x8, x10, #0xffff
0x4170  ldr  x9, [x1, #0x2380]   ; fast_path_allocs
0x4174  add  x9, x9, x8
0x4178  str  x9, [x1, #0x2380]
0x417c  ldr  x9, [x1, #0x2390]   ; fast_path_deallocs
0x4180  add  x8, x9, x8
0x4184  str  x8, [x1, #0x2390]

Phase 11.12:

; no slow_path_allocs++ block at the entry -- merged into packed update
; ... refill site ...
0x4114  and  x8, x10, #0xffff
0x4118  ldr  x9, [x1, #0x2380]            ; packed_allocs
0x411c  mov  x10, #0x1000000000000        ; 1ULL << 48
0x4120  add  x10, x8, x10
0x4124  add  x9, x9, x10
0x4128  str  x9, [x1, #0x2380]
0x412c  ldr  x9, [x1, #0x2388]            ; fast_path_deallocs
0x4130  add  x8, x9, x8
0x4134  str  x8, [x1, #0x2388]

Net change: the 3-instruction slow_path_allocs++ block at the inlined small_refill entry is gone; the fast_path_allocs += becomes a 6-inst packed update (one extra mov for the 1ULL<<48 constant materialization). Net -1 inst in the inlined _malloc body and -1 STORE to a separate counter field per slow-path call -- the cache-line write reduction is the win that shows up at bench time.

Bench results (apple silicon, paired OFF/BASIC)

group OFF (ns) BASIC (ns) ratio
small_allocs ~203.7 ~203.7 1.00
medium_allocs ~1039 ~1032 0.99
mixed ~612 ~612 1.00

vs Phase 11.11 baseline (medium_allocs was 1.122), all three groups land at or below the 1.02 BASIC bar; medium_allocs is now within the noise envelope of stats-off.

Test plan

  • cmake -B build -DSNMALLOC_STATS_BASIC=ON && cmake --build build -j -- clean build
  • ./build/func-fast_path_counters-fast -- passes (fast_path_allocs >= N - K, slow_path_allocs >= 1)
  • ./build/func-statistics-check -- passes
  • cargo test --features stats-basic -- 33 passed
  • cargo bench --bench stats_bench [--features stats-basic] -- five paired passes; ratios above
  • otool -tvV diff of _malloc body shows the slow_path_allocs++ block gone, packed-update merged into the refill-site store

Details and a re-run recipe live in docs/heap-profiling-benchmarks.md under "Phase 11.12 -- packed slow_path counter".

ClickUp: 86aj12be5

Disassembly of `_malloc` on the Phase 11.11 baseline showed the
BASIC tier `medium_allocs` residual cost concentrated at two
adjacent counter stores on the small-refill slow path:

  - `stats.slow_path_allocs++` at the entry to `small_refill`
    (ldr/add/str on field 0x2388).
  - `stats.fast_path_allocs += refill_count` at the refill site
    (ldr/add/str on adjacent field 0x2380).

`medium_allocs` (4 KiB allocations) hits `small_refill` more
often than `small_allocs` because each chunk yields fewer
objects per refill, so the per-refill counter cost is the
residual.

Pack the two fields into one 64-bit `FrontendStats::packed_allocs`:
  - bits  0-47: cumulative_allocs (fast + slow combined)
  - bits 48-63: slow-path call count

At the refill site the two stores collapse into ONE packed `+=`:

  stats.packed_allocs +=
    static_cast<uint64_t>(refill_count) + PACKED_ALLOCS_SLOW_INC;

The two lanes occupy disjoint bit ranges so the packed `+=` is
correct as long as neither lane overflows its sub-field width.
The 16-bit slow lane saturates at 65535 refills (~16M allocs
per thread for the smallest sizeclasses); effectively unbounded
for any realistic workload on an observability surface.

The `FullAllocStats` FFI struct is unchanged: at aggregation
time `stats_export.cc` decodes the packed word back into the
public `fast_path_allocs` and `slow_path_allocs` fields.  The
`FrontendStatsGlobal` thread-exit aggregator drops to a single
`fetch_add` for the combined counter.

Bench results (apple silicon, paired OFF/BASIC):

  group           |  OFF (ns) | BASIC (ns) | ratio |
  small_allocs    |    ~203.7 |     ~203.7 |  1.00 |
  medium_allocs   |    ~1039  |     ~1032  |  0.99 |
  mixed           |    ~612   |     ~612   |  1.00 |

vs Phase 11.11 baseline (medium 1.122) -- medium drops to 0.99
(within bench noise of stats-off), all groups <= 1.02.

Disassembly delta: the 3-inst `slow_path_allocs++` block at the
entry to the inlined `small_refill` is gone; the
`fast_path_allocs +=` becomes a 6-inst packed update with one
constant materialization for `1ULL << 48`.  Net -1 inst in the
inlined body and -1 STORE to a separate counter field per
slow-path call.
@jayakasadev
jayakasadev merged commit de7baa7 into main Jun 12, 2026
14 of 211 checks passed
@jayakasadev
jayakasadev deleted the feature/phase-11-12-packed-slow-counter branch June 12, 2026 19:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant