Repository navigation
linalg: gate par_chunks_mut/par_bin launches on two thresholds of work - #2986
Conversation
5d7368a to
0b29317
Compare
0b29317 to
982cf20
Compare
|
🔴 Bench vs main — 2 speed regression(s) · Reference: 2026-10-12 morning nightly run (0d old) · full report → run Speed — evaltime · prefill · decode
Improvements
|
| Δ | metric | device | main → PR |
|---|---|---|---|
| nemotron_3_5_asr_streaming_0_6b_f32f32_preprocessor_1s load · cuda |
jetson-orin-nx |
43 ms → 54 ms | |
| speaker_id RSS @ load · pulse8 |
cortex-a53 |
19.2 MB → 23.2 MB | |
| en_tdnn_15M_nnef load+optimize · pulse_240ms |
beaglev-ahead |
1.01 s → 1.1 s | |
| en_tdnn_15M_nnef load · pulse_240ms |
beaglev-ahead |
712 ms → 780 ms | |
| en_tdnn_15M load · pulse_120ms |
beaglev-ahead |
2.05 s → 2.21 s | |
| en_tdnn_15M load+optimize · pulse_120ms |
beaglev-ahead |
2.4 s → 2.58 s | |
| mobilenet_v2_1 load · pass_mt |
beaglev-ahead |
1.34 s → 1.44 s | |
| en_tdnn_lstm_bn_q7 RSS @ load · pulse_240ms |
cortex-a53 |
24.9 MB → 26.4 MB |
Bench numbers come from shared CI hardware and do produce false positives: a flagged
row is not on its own evidence of a real regression, and a clean report is not proof
there is none. Contributors do not need to chase what is reported here — leave the
reading of it to the maintainers, who will say if something needs action.
|
@kali WDYT? does it need some work ? |
|
/ci bench |
|
when in doubt... |
|
Reference: 2026-10-12 morning nightly run (0d old) · full report → run Speed — evaltime · prefill · decode no inference-speed regressions Improvements
+1 more improvement(s)
|
| Δ | metric | device | main → PR |
|---|---|---|---|
| speaker_id RSS @ load · pulse8 |
cortex-a53 |
19.2 MB → 23.1 MB | |
| en_tdnn_lstm_bn_q7 RSS @ load · pulse_240ms |
cortex-a55 |
24.7 MB → 26.7 MB |
Bench numbers come from shared CI hardware and do produce false positives: a flagged
row is not on its own evidence of a real regression, and a clean report is not proof
there is none. Contributors do not need to chase what is reported here — leave the
reading of it to the maintainers, who will say if something needs action.
❌ CI / cli: failure
|
|
|
982cf20 to
efc4104
Compare
|
@kali attempting a couple of improvements, commit to this or a follow up PR |
|
you can stack it up here. |
|
Reference: 2026-10-12 morning nightly run (0d old) · full report → run Speed — evaltime · prefill · decode no inference-speed regressions Improvements
+3 more improvement(s)
|
| Δ | metric | device | main → PR |
|---|---|---|---|
| en_tdnn_lstm_bn_q7 RSS @ load · pulse_240ms |
cortex-a55 |
24.7 MB → 26.3 MB |
Bench numbers come from shared CI hardware and do produce false positives: a flagged
row is not on its own evidence of a real regression, and a clean report is not proof
there is none. Contributors do not need to chase what is reported here — leave the
reading of it to the maintainers, who will say if something needs action.
|
@kali bench please |
e054c73 to
58829fa
Compare
|
🔴 Bench vs main — 1 speed regression(s) · Reference: 2026-10-20 morning nightly run (0d old) · full report → run Speed — evaltime · prefill · decode
Improvements
|
| Δ | metric | device | main → PR |
|---|---|---|---|
| mdl_en_2019_Q3_librispeech_onnx RSS @ ready · 2600ms |
i9-9900k |
27.1 MB → 31.4 MB | |
| nemotron_3_5_asr_streaming_0_6b_f32f32_joint_pass RSS @ ready · cpu |
i9-9900k |
55.7 MB → 61.9 MB | |
| en_tdnn_15M load · pulse_240ms |
beaglev-ahead |
1.58 s → 1.71 s | |
| en_tdnn_15M load+optimize · pulse_240ms |
beaglev-ahead |
1.81 s → 1.95 s | |
| voicecom_float load+optimize · 2sec |
cortex-a55 |
96 ms → 102 ms | |
| voicecom_fake_quant load+optimize · 2sec |
cortex-a55 |
155 ms → 164 ms |
Bench numbers come from shared CI hardware and do produce false positives: a flagged
row is not on its own evidence of a real regression, and a clean report is not proof
there is none. Contributors do not need to chase what is reported here — leave the
reading of it to the maintainers, who will say if something needs action.
…ement threshold and always split the work into 4 * threads pieces, so a tensor barely over the threshold was shredded into chunks far smaller than the threshold and the launch cost more than the work saved. A launch now needs at least two threshold-sized chunks, and the split is bounded by the work present as well as by 4 * threads, so large tensors keep the old granularity for load balance. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
… three thresholds large only that many chunks, so on a four-core part half the cores sat idle once the launch was already paid; MobileNet's late layers live in that band and regressed on Cortex-A7 (+7.2% pass_mt) and on M4 (+2.5-4.2%). A launch now gives every thread a chunk, growing with the work up to 4 * threads, in both par_chunks_mut and par_bin. Depthwise convolution still decides whether to launch on its output size, so small planes stay inline, but once a plane is large enough to launch it counts a whole kernel of multiply-adds per output point, so the split is sized by the work rather than the output. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
DepthWise::threading_work reads current_threading_element_threshold, but tract-core only enables multithread-mm in its dev-dependencies, so the library failed to build without it. The threshold is a plain atomic, so its static and getter are now available whatever the features; the setter stays gated.
|
|
58829fa to
6f31af3
Compare
|
🔴 Bench vs main — 1 speed regression(s) · Reference: 2026-10-23 morning nightly run (0d old) · full report → run Speed — evaltime · prefill · decode
Improvements
|
| Δ | metric | device | main → PR |
|---|---|---|---|
| sd15_dummy_f16_vae_decoder_512px heap @ ready · metal |
apple-m1-max |
101 MB → 12.3 GB | |
| sd15_dummy_f32_vae_decoder_512px heap @ ready · metal |
apple-m1-max |
200 MB → 12.4 GB | |
| sd15_dummy_f16_unet_step_b2 heap @ ready · metal |
apple-m1-max |
1.73 GB → 5.98 GB | |
| sd15_dummy_f32_unet_step_b2 heap @ ready · metal |
apple-m1-max |
3.45 GB → 7.7 GB | |
| en_tdnn_8M load · pulse_180ms |
cortex-a55 |
940 ms → 995 ms | |
| en_tdnn_8M load+optimize · pulse_180ms |
cortex-a55 |
1.08 s → 1.15 s |
Bench numbers come from shared CI hardware and do produce false positives: a flagged
row is not on its own evidence of a real regression, and a clean report is not proof
there is none. Contributors do not need to chase what is reported here — leave the
reading of it to the maintainers, who will say if something needs action.
par_chunks_mut/par_binlaunched on any tensor over the element threshold and always split into4 * threadspieces — a tensor barely past the threshold was shredded into sub-threshold chunks and the launch cost more than it saved. A launch now needs at least two threshold-sized chunks of work. Once launched, every thread gets a chunk and the split grows with the work up to4 * threads.The first commit split by threshold count alone, which left cores idle on tensors two or three thresholds large: bench-vs-main flagged mobilenet_v2
pass_mt+7.2% on Cortex-A7, and M4 showed the same direction. The second commit restores the per-thread floor and lets depthwise conv size its split by its real work (output × kernel taps). The launch decision itself stays on the output size, so small depthwise planes keep running inline.Benchmarks (Apple M4, 5-round interleaved A/B, medians vs main;
0= all cores; measured before the rebase, on main12e9af25)Single-thread is unchanged (±0.4%, noise). MobileNet rows are parity within run noise; the first commit's MobileNet regression is gone. Tests cover the gate (
one_threshold_stays_inline,two_thresholds_fill_every_thread,several_thresholds_split_once_per_thread). Cortex-A7 is for the bench bot to confirm.🍍