Intel arc 0.1.39 + perf fixes

[maxfridbe]

Follow-up to #423 on top of the 0.1.39 release. Every change is in sycl/ or the Intel docs; no shared upstream file is touched.

1. The 0.1.39 SYCL port did not build (fixes)

Two upstream changes merged for the release did not reach the port's copies of the files they touch. On the B70 (oneAPI 2026.1, AOT bmg-g31) sycl/ failed with 3 errors:

• cpu: preserve full thread affinity on Windows and Linux #626 (thread affinity, ec35cf0): pin_current_thread returns a ThreadAffinity, but sycl/src/core/session.cpp / session.hpp still held a long long.
• Several conversations at once: batch slots, a pipelined layer split, per-stage dense weights #559 (per-stage dense layers, ba707d8 / e50f266): NativeDense::load takes layer_lo / layer_hi, but the port's native_dense.cpp kept the old signature.

With those two 3-way merged, 0.1.39 builds with 0 errors. On the card, 21 of 25 kernel tests pass, and benchy matches the 0.1.38 port's speeds. Of the four failures:

• iq_parity and ple_parity need model fixtures.
• s2_expert_grouped_parity covers the s2 path (it fails on the 0.1.38 port too).
• conversation_snapshot_test aborted with munmap_chunk(): invalid pointer. The port's conversation_state.cpp (last synced at 0.1.31) and the test (0.1.32) predate Several conversations at once: batch slots, a pipelined layer split, per-stage dense weights #559's parking changes, while the port builds against upstream's conversation_snapshot.hpp, which Several conversations at once: batch slots, a pipelined layer split, per-stage dense weights #559 changed. Merged here as well.

Not ported yet: #559's batch slots (--batch / "parallel"). The port's verify.cpp / verify.hpp do not have them. It is opt-in, so default use is not affected.

2. Prompt-path speedups

Measured on a B70 with each model's served config. Decode speed is unchanged.

• Lend mirror. The slots a prompt borrows have their experts re-read from the GGUF for every chunk and again for the refill. Those experts are now kept in pinned RAM and DMA'd. Same bytes, identical outputs. About 2 GB more RAM and about 2 s more at start. STRATA_LEND_MIRROR=0 turns it off.
• QSA block scores as oneMKL GEMM tiles. The old kernel used one sub-group per (query, block) pair, so its cost grew with the square of the context. Now the pooled keys are multiplied against the batch's indexer queries in fp32 tiles, followed by a small relu-sum kernel. 7-8x faster at 40K-256K, relative error 2-4e-7 vs FP64. STRATA_SELECT_GEMM=0 restores the old kernel.
• Prompt attention scores one work-item per cell (128-cell chunks), instead of a 5-step shuffle tree per head. 1.5x faster. Decode keeps the old kernel. STRATA_ATTN_PERCELL=0 restores the old kernel.

The last two are FP32 in a different summation order, so they are not bitwise, like the CUDA build's tensor-core paths. Greedy outputs follow the same text and part ways at a near-tie after tens of tokens.

Time to first token, Coder IQ1_M on its served 256K config (measured on the 0.1.38 base):

Prompt	Before	After
2,185 tokens	4.9 s	4.6 s
40K	40.1 s	33.6 s
128K	140.9 s	111.2 s
256K	328.8 s	235.1 s

IQ2_XS and Swift 1.5 at 128K: 191 -> 156 s.

Things tried that did not help are recorded in sycl/TODO.md: 15 attention variants, including two XMX versions, and expert grouping on the GPU (its "6.5%" was mostly the profiler's own cost).

3. benchy v1: a standard bench for Intel results

sycl/benchy.sh runs every set-up model on fixed prompts (sycl/bench/v1: 20 / 2,185 / 8K / 40K / 128K / 256K tokens) with its own serve config, from a cold page cache. It reports:

• speed: PP, TTFT, TG, drafts accepted;
• resources: experts in VRAM and in the RAM mirror, VRAM, RAM, SSD reads, load time, power;
• the system and each config's args.

docs/INTEL_PERFORMANCE.md holds every measured number: current results, dated history, the B50 / B580 / 2x B70 results from #423, and "Submitting numbers", which asks testers to post benchy's report plus their config.

4. Smaller

• setup_intel.py --check sized each model by the first card only. With two cards it now reports what fits across both.
• strata-sycl.sh: with no ONEAPI_DEVICE_SELECTOR set, a --layer-split gets level_zero:gpu (the image pins level_zero:0). A set selector is still passed through.
• Duplicate stage_room fix removed: where INTEL B70 32gb support #423's fix and yours overlapped, this keeps yours.
• Docs: an accuracy pass on docs/INTEL.md (build section, switches, a corrected "gather-bound" claim), plus the probe programs and three kernel benches (attn_bench, sel_scores_bench, xmx_int8_bench).

Testing

• The 0.1.39 build fix: built AOT for the B70, kernel tests run, benchy v1 on all three models.
• This exact branch: a full B70 run (build, ctest, benchy v1) is in progress. I will post its report as a comment.

[maxfridbe]

Update: conversation_snapshot_test is fixed in 5780652 and passes on the B70 (2,169 checks, no copy-engine resets). A queue memcpy with no device end (pinned host ↔ pageable host) hangs the Arc's copy engine (Engine reset: engine_class=bcs; repeated runs wedged the card). Host-only copies now go through the CPU. The engine's conversation_kv_save/restore had the same copy for --kv-resident sessions, used by the opt-in conversation cache (#596). Now 22 of 25 kernel tests pass.

[maxfridbe]

b56c45e: iq_parity and ple_parity are registered as the top-level CMakeLists does. The port ran iq_parity --selftest, which read the flag as its fixture dir. With generated fixtures, all 10 types pass on the B70. ple_parity needs the bench/micro captures, which aren't in the repo.
s2_expert_grouped_parity: under -fp-model=precise the new grouped kernels' acc += dw * dx * v became an FMA, while the old kernels' chunk_dot product plus a separate add did not. nvcc contracts both, which is why it was bitwise on CUDA. The new kernels no longer contract, so the test is bitwise here too.
5780652 (earlier): conversation_snapshot_test. Host-only queue copies hung the Arc's copy engine; they now use the CPU.

[joeyjoe02]

The testing and write-up below were done by Claude on my machine, at my request. I'm not a developer; I'm posting it because it may help with the conversation cache on Arc.

Tested this branch (baa84d9) on one Arc Pro B60 24 GB (8086:e211, OCuLink PCIe 4.0 x4, NEO 26.31, oneAPI 2026.1.1, AOT bmg-g21), with #866 and #889 applied on top. Flash-Next GSQ-RCO IQ2_XS, --max-context 65536 --kv int8 --spec 4 with MTP, --stream-experts --vram-reserve-mib 1024, 11,935 experts resident and 12,643 in the RAM mirror, per-layer waits (no STRATA_VERIFY_NO_HOST).

Good news first:

• Builds with 0 errors; conversation_snapshot_test passes on the B60 (2,169 checks).
• Prompt reading is faster than the 7ba023e port: a 21.5K-token cold prompt 29.9 s vs 35.2 s. Decode unchanged (~22 tok/s).

The opt-in conversation cache still produces token-0 output on this card (--conversation-cache-mib 8192 --conversation-cache-slots 4):

• A synthetic test (four ~21K-token conversations taking turns, 20 tokens each) was clean: every return to a parked conversation reused its prompt (1.0-1.6 s instead of ~30 s).
• A real agent workload (two Hermes Agent sessions with tools taking turns, 8 turns, run twice): run 2 was clean, but in run 1 every reply of the second session was !!!!… (256 tokens, drafts accepted 126 of 126), including its first request, which was not a restore: it came right after the first session was parked. The first session's later restores were fine.
• Without the conversation cache, the same workload on the 7ba023e port (sycl: keep cudaStreamQuery's answer in the per-layer ring waits #866 + sycl: read doorbell flags with an uncached L1+L3 hint (the GPU never saw the host's store on an Arc Pro B60) #889) was clean in 2 of 2 runs and in 59 benchmark runs, so it looks specific to parking (capture) on this card rather than the per-layer waits.

Engine log lines around the first bad reply, if useful:

conversation cache: parked 23036 tokens in 150.9 ms; parked=4 bytes=3798174316 evictions=5 snapshot_bytes=1695236392 reused_kv_bytes=343928448
prompt 18889 tokens = 18870 reused + 19 read in 688 ms (27.6 tok/s), 256 generated in 5848 ms (43.8 tok/s), drafts accepted 126 of 126, 6 checkpoints

(that pair is from the 7ba023e run, where I first saw it; this branch shows the same pattern, the bad replies all 256 generated ... drafts accepted 126 of 126.)

Happy to rerun anything on the B60 (with STRATA_VERIFY_TRACE or similar) if it helps narrow it down. For now I'm running with the conversation cache off.

🤖 Generated with Claude Code

[joeyjoe02]

This testing and write-up were done by Claude on my machine, at my request. I'm not a developer; I'm posting it because the benchy v1 tables here have no B60 rows yet.

benchy v1 on one Arc Pro B60 24 GB (8086:e211, OCuLink PCIe 4.0 x4, Ryzen AI 9 HX 470, NEO 26.31, oneAPI 2026.1.1, AOT bmg-g21). Original Flash-Next IQ2_XS, --max-context 65536 --kv int8 --spec 4 with MTP, --stream-experts --vram-reserve-mib 1024: 11,935 experts in VRAM, the rest in the RAM mirror, per-layer waits (no STRATA_VERIFY_NO_HOST). The v1 prompts and run shape from perf_matrix.py (fresh engine, --max-new 256 --greedy --stats), two runs per size, the mean shown (the two runs agree within ~1% on PP and within ~10% on TG, which moves with draft acceptance).

Input tokens	This branch (baa84d9) PP / TTFT	7ba023e port PP / TTFT	TG this branch	TG 7ba023e
20	28.4 / 0.9	22.9 / 1.0	31.7	31.6
2,185	359.3 / 6.3	322.1 / 7.0	38.5	35.0
8,000	684.6 / 11.9	575.3 / 14.1	36.2	38.0
40,000	809.8 / 49.7	666.6 / 60.2	32.8	33.6

So on the B60 this branch reads prompts ~19-22% faster at 8K-40K, and TG is the same within noise.

For context, #868 has 2x B60 (--layer-split) results and a single-B60 run on 6f32ec0 (Coder IQ1_M, 11.9 tok/s with experts mirrored). These are single-B60 numbers on this branch with benchy v1, so they line up with the B70 / B65 tables.

Differences from a standard benchy run, so the numbers are read correctly:

• Warm page cache (it ran without root, so no drop_caches). That should only affect load time, not PP/TG.
• Run natively (no container), with the served config's args (64K context, so no 128K/256K rows).
• Both builds carry sycl: keep cudaStreamQuery's answer in the per-layer ring waits #866 and sycl: read doorbell flags with an uncached L1+L3 hint (the GPU never saw the host's store on an Arc Pro B60) #889 (the doorbell read fix). Without sycl: read doorbell flags with an uncached L1+L3 hint (the GPU never saw the host's store on an Arc Pro B60) #889, decode on this card is about half.

Correction to my comment above: the "~22 tok/s" decode figure there came from a short synthetic prompt where drafts are rarely accepted. On these prompts it's 32-38 tok/s, and real agent conversations on this card average ~41 tok/s.

🤖 Generated with Claude Code

[TheEmeryverse]

Tested the PR #809 branch on an Arc B580 (G21) with Qwen3.8 IQ2_XS.

Startup: The checked-in wrapper forces STRATA_VERIFY_NO_HOST=1 and doesn’t set STRATA_MIRROR_MIB. Startup reported 21,216 experts neither resident nor mirrored. With our B580 wrapper (STRATA_MIRROR_MIB=8192, leaving STRATA_VERIFY_NO_HOST unset), 5,989 experts were mirrored in 8 GiB and the model served a request. Since both settings differed, this doesn’t isolate the effect of the verification flag.

Benchmarks: In one synthetic QSA run at 32,768 cells / 256 queries, the current kernel took 5.895 ms; TF32 GEMM took 0.336 ms (17.52×), with relative score error of 2.25e-7 and identical selected IDs. At 128K context, across five timing repetitions, per-cell attention took 0.981 ms versus 0.990 ms for the engine kernel, with exact output.

Across two valid runs per configuration on a 2,185-token prompt with 256 greedy output tokens, the candidate stack (PR #809 plus #866/#889 and local B580 safety changes) averaged 314.01 tok/s prefill and 31.18 tok/s decode, versus 311.96 and 31.18 for the existing B580 build; TTFT was 7.138 s versus 7.184 s. These small differences are within noise and don’t establish an end-to-end gain.

The up-projection XMX GEMM completed but was 0.25–0.33× as fast as dequantize plus oneMKL. The int8 DPAS prototype was 0.56–0.73× as fast as the reference, including input quantization. Q6_K pre-unpack parity, MMVQ, and down-projection XMX runs encountered UR_RESULT_ERROR_DEVICE_LOST; Xe engine resets were logged during these tests. B580 testing of the experimental paths is still in progress.

[Niko1221]

Heads-up: today we cleaned up this repository's commit history, and GitHub closed this PR automatically when main was force-pushed (that is the "base branch was force-pushed" event above). It is not a rejection.

GitHub cannot reopen it, because its branch is built on the old history. If you want to keep it going: start a new branch from the new main, cherry-pick your own commits onto it (git fetch upstream && git switch -c my-fix upstream/main && git cherry-pick <your commits>), and open a new PR. Or simply leave it: when we take your change for a release we port it onto the new main ourselves and keep you as its author.

Sorry for the noise, and thank you for the contribution.