feat(core): move to ryzenai-corelib 0.33.1, run every corelib model behind one engine, prefer AMD Q4 GGUFs, rename rai to core

[chizamd]

Follow-up to #748. Moves the corelib backend from ryzenai-corelib 0.11 to 0.33.1, adds Qwen3.6-35B-A3B and Qwen3.5-4B, runs Gemma 4 at 16k and Qwen3.5 at 32k, has every corelib model (Phi-4, Qwen3, Qwen3.5, Qwen3.6, Gemma 4) run as corelib's own model behind one shared engine, makes flm pull prefer AMD's Q4 GGUFs with a public Q8_0 fallback, and renames the backend from rai to core.

Breaking rename (commit 6). The rai backend added in #748 becomes core, and the aie_next platform becomes dev04: tags move from *-rai:* to *-core:* (e.g. phi4-mini-it-rai:4b → phi4-mini-it-core:4b), FLM_ENABLE_RAI → FLM_ENABLE_CORE, the *-rai presets → *-core, FLM_RAI_* → FLM_CORE_*, home_install.sh --rai → --core, and --backend rai → --backend core. No old name is kept as an alias. ryzenai-corelib and RYZENAI_CORELIB_* name the external library and stay.

Commits:

1. corelib 0.26, Qwen3.6, Gemma 4 at 16k on the existing per-op engines.
2. One engine for corelib's models. Qwen3, Qwen3.5 and Gemma 4 stop driving corelib op by op and run as ryzenai_corelib_model_*, like Qwen3.6.
3. corelib 0.28. Qwen3.5's KV cache is paged up to 32768, and a later turn's prefill chunks start on 4096-row page boundaries.
4. corelib 0.29.2. No ABI change; corelib speeds up Gemma 4's per-layer-embedding projection and runs Qwen3.6's in-page prefill chunks on the NPU.
5. corelib 0.29.4. No ABI change; corelib refuses a Qwen3.6 capacity past 131072, the most it has verified on the NPU, so Qwen3.6's limit drops from 262144 to 131072.
6. Rename rai → core and aie_next → dev04, as above. Directories and files carrying either name move with them; no behaviour change.
7. corelib 0.29.7. No ABI change. src/test/phi4_core's version-pin probe can fail again (it used to pass by failing on a missing include path).
8. flm pull prefers AMD Q4 GGUFs for the core models and falls back to the public Q8_0 source. Adds qwen3.5-core:4b.
9. Phi-4 on corelib's phi4mini model, Q8_0 or Q4_G128; the last per-op engine is removed. Also fixes the token history keeping an EOS the engine never ran.
10. corelib 0.33.1. model_forward, forward_options and create_stream change shape.

Models

Every tag below runs as a corelib model through the shared engine.

Tag	GGUF (preferred → fallback)	Context	Change
qwen3.6-moe-core:35b-a3b	unsloth Q8_0	default 4096, up to 131072	New. Qwen3.6-35B-A3B, text only
gemma4-it-core:e2b gemma4-it-core:e4b	AMD Q4_G32 → unsloth Q8_0	default and max 16384	Was 4096; E4B is offered again
qwen3-core:0.6b	unsloth Q8_0	4096	Moved off the per-op engine
qwen3-core:1.7b 4b 8b	AMD Q4_G128 → unsloth Q8_0	4096	Moved off the per-op engine
qwen3.5-core:9b	AMD Q4_G128 → unsloth Q8_0	default 4096, up to 32768	Moved off the per-op engine; was capped at 4096
qwen3.5-core:4b	AMD Q4_G128 → unsloth Q8_0	default 4096, up to 32768	New. A bare qwen3.5-core now resolves to 4b
phi4-mini-it-core:4b	AMD Q4_G128 → unsloth Q8_0	4096	Moved off the per-op engine; reads Q8_0 or Q4_G128

What changes

corelib 0.33.1 (version pinned exactly)

• 0.26: create_tensor_window takes strides, linear_attention_bf16 takes rows, and models are built with model_weights_create_path + model_create(name, weights, options).
• 0.30: create_stream takes a priority list of PDIs per phase. OpenModelStream, which nothing calls any more, passes its two PDIs as one-element lists.
• 0.31 / 0.33: model_forward takes images, options, out_token and the logits as arguments, and forward_options gains a leading size that corelib refuses when short. CorelibModel sets size and passes its logits buffer only when it wants logits.
• 0.28, 0.29.x and the rest of 0.30–0.33 add nothing FastFlowLM has to meet. 0.31–0.33 add Qwen3.6 image input, which FastFlowLM does not pass.
• docs/core-setup-linux.md: RYZENAI_CORELIB_FALLBACK_PROFILE is now RYZENAI_CORELIB_REPORT_FALLBACKS, and RYZENAI_CORELIB_TRACE_XRT is now RYZENAI_CORELIB_TRACE=xrt.

Packaging

• corelib installs ryzenai_corelib plus one ryzenai_corelib_model_<name> library per model and loads them only from its own directory. A Windows core build copies them beside flm.exe; home_install.sh copies them beside corelib on Linux.
• RYZENAI_CORELIB_DD_ROOT / RYZENAI_CORELIB_KERNELS_ROOT are no longer needed, and the installer stops staging DynamicDispatch.

One engine: flm::core::CorelibModel (include/core/corelib_model.hpp, common/core/corelib_model.cpp)

• A causal_lm over a corelib model. Each model is a short ModelSpec naming what differs:
  • Continuation: where a multi-row pass may start past position 0 — anywhere (Qwen3.5, Qwen3.6), only at 0 (Qwen3 and Phi-4, so a later prompt re-runs the conversation), or at 0 / a 4096-row page boundary (Gemma 4).
  • Rewind: kSeek for attention-only models (every pass states its position; rewind just moves it), kSnapshot for Qwen3.6 (model_state_save/restore, reused only if its history is a prefix of the current one), kReplay for Qwen3.5 (reset and replay).
• Token ids and the context limit are checked before corelib is called. A failed forward, restore or reset poisons the engine.
• Logits are fetched only for the row that gets sampled.

One backend: flm::core::Factory(BackendSpec) (include/core/corelib_model_backend.hpp)

• Refuses preemption and a context outside the model's rule, reads the package's BOS/EOS ids (tokenizer_config.json + config.json, e.g. Gemma 4 → {1, 106}; Phi-4 adds <|end|>, which the package does not name), and resolves the GGUF the catalog points at before the runtime or weights are touched. Image requests on core return HTTP 400.

Preferred AMD sources in flm pull (model_list.hpp, pull/)

• A catalog entry may carry preferred_source, a JSON merge patch over the entry that sets its own url, files, size, quantization level and model_info_key. Each AMD source is pinned to a commit, and its GGUF to a sha256 in model_info.json (keyed <tag>@amd); the tokenizer and config files keep their existing pinned sources.
• flm pull sends a HEAD request for the preferred GGUF and uses it on 2xx/3xx; otherwise it prints Falling back to: <url> and pulls the entry's own source. Files already on disk are kept without probing.
• Outside a pull, an entry resolves to its preferred source exactly when every file only that source has is on disk, so run, serve, list and check follow whatever was pulled.
• The AMD repos currently need a Hugging Face token (HF_TOKEN, HUGGING_FACE_HUB_TOKEN, HF_TOKEN_PATH, HF_HOME/token or ~/.cache/huggingface/token). It is sent only for a "use_hf_token": true source, only to https://huggingface.co/, with TLS verification on; public sources never get it. Without access, a pull falls back to the public Q8_0.

Page-aligned prefill chunks (AutoModel::_chunked_insert, model_backend.hpp)

• New ModelBackend::aligns_prefill_chunks(), default false, so FastFlowLM's own engines are unaffected. The corelib-model backend returns true: a later turn's prompt is chunked on multiples of 4096 counted from position 0 instead of from the turn's start, so every chunk after the first starts on a page boundary and corelib runs it as whole pages on the NPU. Prompts carrying a multimodal payload keep the old chunking.

EOS in the token history (AutoModel::_shared_generate)

• An EOS the engine never ran (sampled straight from the prefill, or ending a decode on a backend that declines the post-EOS forward, which every core backend does) is no longer appended to the token history. Before, the next turn's prefix match treated it as already held, so the engine never saw the end-of-turn token between the reply and the next user message. It is now prefilled with the next turn. A frontend test covers both cases on core and the unchanged behaviour on FastFlowLM's own engines.

Removed

• Every per-op engine (Phi-4, Qwen3, Qwen3.5, Gemma 4) with its GGUF reader, shape plan, host math and packed-weight cache; rai/gguf_file, rai/weight_cache, core/kernel_grid.hpp, core/weight_source.hpp, the Q8/RMSNorm parts of the host ops, and the corelib entry points nothing calls.
• No core model has a packed-weight cache any more: corelib packs weights on every load. Old *.rai-weights.bin/.json and phi4-*-weights.* files can be deleted.

Shared

• Sampler::sampler_topk_apply selects top-k with a k-entry heap in one pass instead of partial_sorting a vocab-sized array each token. Same ranking and lowest-id tie rule; checked against the old code on 63 inputs (vocab 1..248320, k = 1/2/20/40, heavy ties).

Docs: src/common/models/README.md gains a section on models corelib runs whole; docs/core-setup-linux.md and the model pages cover the version, the AMD and public sources, and the dropped weight cache.

Results

All on a dev04 NPU, Windows.

Unit tests (fake corelib, 0.33.1 headers): core_model, phi4_core, gemma4_core and model_backend pass; test_model_downloader includes offline file:// tests for the preferred-source logic. test_real_corelib against the 0.33.1 library loads it, matches the version and resolves every entry point. flm.exe builds with FLM_ENABLE_CORE=ON against 0.33.1.

Shared engine vs the per-op engines (measured at commit 4, corelib 0.29.2) — "before" is the first commit's build on corelib 0.26: the per-op engines for Phi-4, Qwen3, Qwen3.5 and Gemma 4, and Qwen3.6 on corelib's model API. Both builds run interleaved, Gemma 4 twice each, with the NPU checked idle before and after every run. Long-prompt TTFT is the median of three ~1.1k-token prompts; decode is the median over three short prompts, 64 greedy tokens each.

Model	Load, this PR vs before	Long-prompt TTFT, this PR vs before	Decode tok/s, this PR vs before	Greedy text
gemma4-it-core:e2b	8.1–11.4 s vs 3.3–4.2 s	0.64–0.68 vs 0.80–0.82 s	34.2–36.2 vs 37.1–38.0	1 of 3 prompts differs (known near-tie token)
gemma4-it-core:e4b	11.2–16.4 s vs 3.4–3.6 s	1.02–1.06 vs 1.33–1.35 s	23.5–23.9 vs 23.6–24.2	identical
qwen3.6-moe-core:35b-a3b	83.2 vs 84.4 s	1.45 vs 14.18 s	19.1 vs 20.4	identical
qwen3.5-core:9b	22.5 vs 25.1 s	42.7 vs 49.3 s	19.0 vs 19.3	identical
qwen3-core:0.6b	3.2 vs 1.8 s	0.24 vs 0.25 s	116.4 vs 121.8	identical

Loads are slower because corelib packs the weights on every load and this path has no packed-weight cache.

Phi-4 on corelib's phi4mini vs its per-op engine (commit 9, corelib 0.29.7):

Run	Load	~1.1k-token TTFT	Decode tok/s	Passkey at ~3k tokens
per-op, Q8_0, cache cold / warm	61.7 s / 2.0 s	0.61 s	40.8 / 44.4	correct
phi4mini, Q8_0	6.4–7.6 s	0.59–0.60 s	39.0–42.6	correct
phi4mini, AMD Q4_G128	2.0–2.6 s	0.60–0.61 s	39.8–42.6	correct

Q8_0 gives the same greedy text on both engines. A second turn, which the per-op engine refused (Phi-4 prefill must start at logical position zero), now works, and a two-turn passkey is recalled on Q8_0 and Q4_G128.

corelib 0.33.1 vs 0.29.7, and 0.29.7 vs 0.29.4 (each pair run interleaved): greedy text and TTFTs match on every model, Phi-4 on both its Q8_0 and Q4_G128 files. Decode and load differences followed run order rather than the build: rerun with the order swapped, the two builds match.

Two-turn continuation at --ctx-len 16384: turn 1 is ~6.8k tokens carrying a passkey; turn 2 resends it with the reply and adds ~6.9k tokens, chunked on page boundaries (1395 / 4096 / 1414).

Model	Turn 1 TTFT	Turn 2 TTFT	Passkey
qwen3.6-moe-core:35b-a3b	6.4 s	7.5 s (14.3 s on corelib 0.28)	correct
gemma4-it-core:e2b	6.7 s	17.5 s	correct

The same chunks and TTFTs within 0.1 s on 0.29.7 and 0.33.1.

Qwen3.6 at --ctx-len 131072 (corelib 0.29.4, 0.29.7 and 0.33.1): turn 1 is 49,917 tokens with a passkey, turn 2 ends at 75,442 tokens; every build recalls the passkey with the same seven turn-2 chunks, turn-2 TTFT 60–65 s.

Measured on the second commit (corelib 0.26):

• Switching e2b → qwen3 → qwen3.5 → e4b → phi4 → e2b in one flm serve process: every answer matches a fresh run, no errors.
• Gemma 4 E2B at 16k: passkey recalled; a short prompt after a full paged cache matches a fresh run.

Long-context passkey recall (passkey at 45% depth, measured on the first commit):

Tag	Context	Prompt tokens	Prefill	Decode tok/s	Answer
gemma4-it-core:e2b	16384	11,329	13.5 s (838 tok/s)	28.9	correct
gemma4-it-core:e4b	16384	11,329	13.4 s	—	correct
qwen3.6-moe-core:35b-a3b	32768	23,714	932 s (25 tok/s)	19.7	correct

Gemma 4 on corelib's built-in model. On corelib 0.26 it was slower than the per-op engine (prefill ~1.5–2×, decode 24–29% slower), because of corelib's host per-layer-embedding projection. corelib 0.29 fixes that projection: prefill is now faster than the per-op engine, and decode is equal on E4B and ~5–8% lower on E2B. Loads stay slower, as above.

Not done yet

• Rebuild and rerun the unit tests on top of current main (merge was clean; the merged tree has not been built)
• Linux build of flm_core and the home_install.sh --core smoke test
• Qwen3.5 at 32k end to end
• On-NPU runs of the AMD Q4 GGUFs other than Phi-4's