perf(cpu_x86): qwen35 Q8_0 prefill ~20-25 tok/s on Zen 5, no thread scaling, GEIST_FORCE_ISA has no effect #504

Activity

1. geisten

   added

   bugSomething isn't workingSomething isn't working

   on Sep 27, 2026
2. geisten commented on Sep 27, 2026

   

   geisten

   on Sep 27, 2026

   OwnerAuthor

   More actions

   Full timing harness used for the table above:

   #define _POSIX_C_SOURCE 200809L
   #include "geist.h"
   #include <stdio.h>
   #include <stdlib.h>
   #include <string.h>
   #include <time.h>
   static double now(void) { struct timespec t; clock_gettime(CLOCK_MONOTONIC, &t); return t.tv_sec + t.tv_nsec / 1e9; }
   static void run(struct geist_model *m, struct geist_backend *be, const char *label, const char *prompt) {
       struct geist_session_opts o = {0};
       struct geist_session *s = nullptr;
       double a = now();
       if (geist_session_create(m, be, &o, &s) != GEIST_OK) exit(2);
       double b = now();
       if (geist_session_set_prompt(s, prompt) != GEIST_OK) exit(3);
       double c = now();
       geist_token_t tok;
       for (int i = 0; i < 8; i++) if (geist_session_decode_step(s, &tok) != GEIST_OK) exit(4);
       double d = now();
       printf("%-12s prompt_bytes=%zu session_create=%.2fs set_prompt=%.2fs decode8=%.2fs\n", label, strlen(prompt), b - a, c - b, d - c);
       geist_session_destroy(s);
   }
   int main(int argc, char **argv) {
       struct geist_backend *be = nullptr; struct geist_model *m = nullptr;
       double a = now();
       if (geist_backend_create("auto", nullptr, nullptr, &be) != GEIST_OK) return 1;
       if (geist_model_load(argv[1], be, &m) != GEIST_OK) return 1;
       printf("model_load=%.2fs backend=%s arch=%s\n", now() - a, geist_backend_name(be), geist_model_arch(m));
       static char big[4000]; big[0] = 0;
       while (strlen(big) < 2400) strcat(big, "Die Station braucht Energie und Reparaturen. ");
       run(m, be, "short#1", "Hallo, wie geht es dir?");
       run(m, be, "short#2", "Hallo, wie geht es dir?");
       run(m, be, "long~600tok", big);
       run(m, be, "long#2", big);
   }

   Build: gcc-14 -std=c23 -O2 -fopenmp -I include prefill.c <build>/libgeist.a -lm -o prefill · Run: OMP_NUM_THREADS=N ./prefill Qwen3.5-0.8B-Q8_0.gguf
3. geisten commented on Sep 27, 2026

   

   geisten

   on Sep 27, 2026

   OwnerAuthor

   More actions

   Reproduced on current `main` (commit `7a0b421`), same GGUF (`qwen3.5-0.8b-q8_0.gguf`, matches your sha256), `OMP_NUM_THREADS=16`:

   ```
   pp128 19.8 t/s (6.48 s)
   pp512 20.3 t/s (25.2 s)
   ```

   Confirms your report exactly — throughput is flat regardless of prompt length (linear time growth, no batching benefit).

   Root cause

   `cpu_x86_resolve_weight` (`src/backends/cpu_x86/backend.c:430`) only overrides linear_m1/linear_mN for Q4_K, Q6_K and I2_S. Every other dtype — including Q8_0 — falls through to `cpu_scalar_resolve_weight`, whose `cpu_scalar_w_quant_mN` (`src/backends/cpu_scalar/weight_resolve.c:209`) is explicitly the reference-correctness kernel: no SIMD, double-precision scalar accumulation, and no OpenMP at all — the whole for (j = 0; j < n_out; j++) loop is single-threaded. This is a documented, intentional gap, not a regression: the comment right above the dtype switch in `cpu_x86_resolve_weight` says outright "M>1 stays on cpu_scalar's slow path for now — Phase 1b wires the W4A8 prefill kernel." Q8_0 was never Phase-1b'd.

   This answers all four checklist items:

   • Batched or GEMV-per-token? Technically one call over the whole prefill batch (dequant is amortized once per output row, reused across all m input rows) — but the arithmetic is scalar double with zero threading, so it behaves like a GEMV-per-token in wall-clock terms.
   • Thread scaling: confirmed absent — the hot loop has no #pragma omp at all. Whatever scaling you saw (~11%) comes from other, smaller parallel regions elsewhere in the forward pass, not this one.
   • GEIST_FORCE_ISA no-op: confirmed — this path never reaches the cpu_x86 ISA-tier dispatcher (the W4A8/W8A8 dispatcher kernel_w4a8.c uses); it bypasses it entirely via the un-overridden cpu_scalar fallback.
   • Not specific to qwen35's DeltaNet layers — every FFN/attention projection weight in a Q8_0-quantized model of any architecture hits the same path on cpu_x86.

   Fix scope

   A real fix means a genuine cpu_x86 Q8_0 prefill kernel: block-symmetric int8, a natural fit for AVX-512 VNNI (vpdpbusd, same primitive already used for Q4_K→W4A8 and Q6_K→W8A8) with an AVX2 fallback tier, plus OpenMP over n_out (matching the pattern in #499's attention kernel and the existing W4A8/W8A8 files). Comparable scope to the existing W8A8 kernel work, not a small patch — same family of fix as #295 (int8 mN GEMM for prefill), which is likely the right issue to fold this into rather than duplicating tracking.
4. geisten commented on Sep 27, 2026

   

   geisten

   on Sep 27, 2026

   OwnerAuthor

   More actions

   Re-measured on current main (7a0b421) with the same harness, same host and model. No change:

   OMP_NUM_THREADS	long prompt (2430 B) set_prompt	decode 8 tokens
   1	23.97 s	0.87 s
   16	21.41 s	0.88 s
   unset (32)	22.15 s	1.41 s

   So the issue is present on both v0.11.0 and main.
5. geisten

   added 2 commits that reference this issue on Sep 27, 2026

   cpu_x86: generic multi-threaded linear for dtypes without a native ke…

   ...

   cfa1ca5

   cpu_x86: decode on physical cores when SMT is on (region hooks)

   ...

   4d8b701
6. geisten commented on Oct 1, 2026

   

   geisten

   on Oct 1, 2026

   OwnerAuthor

   More actions

   Additional end-to-end data from a consumer (pinned 7a0b421, qwen3.5-0.8b-q8_0.gguf, GEMM_PROVIDER=native, TARGET=linux with -march=x86-64-v3 -fopenmp), one action request ≈ 700 prompt tokens + ~11 decoded tokens:

   Host	Prefill P50 / P95	Decode per token
   Apple M1 Max (cpu_neon, Accelerate)	2.5 s / 2.6 s	~25 ms
   Ryzen 9 9950X, 16 CPUs (cpu_x86)	16.1 s / 17.4 s	~200 ms

   Observations that match the root cause above and add one point:

   • max_threads 2 / 4 / 8 / 16: no change in prefill or decode (within noise).
   • GEMM_PROVIDER=openblas (OpenBLAS 0.3.30, OpenMP) instead of native: no change either, so the dense fp32 sgemm is not the bottleneck.
   • Decode is ~8× slower too, not only prefill — consistent with Q8_0 linear_m1 also falling through to cpu_scalar on x86.

   Impact for us: on weaker x86 hardware (Steam Deck class, Zen 2 4C/8T) a single request will likely exceed a 30 s budget with Q8_0. A cpu_x86 Q8_0 kernel for both linear_m1 and linear_mN (AVX2 tier first, VNNI optional) would cover it; until then switching the model to Q4_K is our workaround candidate.
7. geisten commented on Oct 1, 2026

   

   geisten

   on Oct 1, 2026 – with Claude

   OwnerAuthor

   More actions

   Likely fixed in code by #512 (merged 2026-09-29, after the 7a0b421 measurements): native Q8_0 kernels on cpu_x86 (AVX2 m1/mN with OpenMP plus AVX-512 VNNI, linear_q8_0.c), the VNNI tile choice now honours GEIST_FORCE_ISA (linear_q8_0.c:249-265), and decode runs one thread per physical core (threads.c).

   Remaining before closing: re-run the Qwen3.5-0.8B Q8_0 harness on main ≥ bce22ae on the Zen 5 box and add the number to #364. Note that GEIST_FORCE_ISA=scalar still runs the AVX2 Q8_0 kernel (AVX2 is the build baseline); it only chooses between VNNI and AVX2.

   ---

   Generated by Claude Code
8. geisten commented on Oct 3, 2026

   

   geisten

   on Oct 3, 2026 – with Claude

   OwnerAuthor

   More actions

   I re-measured this on the Zen 5 box (Ryzen 9 9950X, the self-hosted geist-vulkan runner) with vulkan-ab (run, via #591).

   Setup:

   • Backend cpu_x86.
   • Synthetic Q8_0 model from the smollm2-360m-q8_0 preset. The real Qwen3.5 GGUF can't be downloaded there.
   • 7a0b421 (the commit the table above was measured on) against current main.
   • 6 cycles, with a control copy of the baseline binary.

   	7a0b421	main	change
   prefill, 128 tokens	4225 ms (30 tok/s)	49 ms (2600 tok/s)	−98.8 %, 6/6
   prefill, 512 tokens	16969 ms (30 tok/s)	196 ms (2610 tok/s)	−98.8 %, 6/6
   decode, 16 tokens	1070 ms	81 ms (5 ms/token)	−92.4 %, 6/6

   The control stays within ±11 %, which is the resolution of this run.

   The old prefill rate was flat, about 30 tok/s whatever the prompt length, matching what this issue reported. Main now prefills about 2600 tok/s and does it in batches.

   #512's native Q8_0 kernels cover every Q8_0 projection whatever the architecture, so qwen35 goes through the same kernels. The checklist items are covered as follows:

   • Batching: yes.
   • Kernel tier: VNNI or AVX2. GEIST_FORCE_ISA picks between them; scalar still runs AVX2, the build baseline.
   • Thread count: decode uses one thread per physical core.

   Closing. If HELIO still sees a slow first reply on the real model, please reopen with the new numbers.

   ---

   Generated by Claude Code
9. geisten

   closed this as completedon Oct 3, 2026
10. geisten

    added a commit that references this issue on Oct 3, 2026

    ci(vulkan-ab): default to #504's case, Q8_0 on the Zen 5 CPU before a…

    ...

    18b21fb