Re-enable Ornith 1.0 9B in Settings UI once multi-step tool-calling slowdown is addressed #1477

Overview

Ornith 1.0 9B is fully wired up in the backend (catalog entry, download, XML tool-call parsing, <think> reasoning routing, response-leak fix) but is intentionally hidden from the Settings UI — removed from EXPOSED_GGUF_MODEL_IDS in packages/desktop-app/src-tauri/src/commands/chat_models.rs — pending a decision on how to handle its multi-step tool-calling slowdown.

Why it's hidden

Live testing during #1465 found that Ornith's recurrent (SSM/Gated Delta Net) layers cannot do partial KV-cache reuse — every ReAct loop iteration re-decodes the entire growing prompt from scratch, rather than reusing prior turns' work like Ministral/Gemma do. This is a confirmed upstream, currently-unresolved ggml-org/llama.cpp limitation affecting all Qwen 3.5-based hybrid models. Also documented in the ORNITH_1_9B catalog entry's doc comment in model_manager.rs.

Upstream tracking issues (ggml-org/llama.cpp) — check these for resolution before re-evaluating:

• Eval bug: Qwen3.5 always re-processes the full prompt ggml-org/llama.cpp#19858
• Eval bug: Qwen3.5-35B-a3b full prompt re-processing with Claude Code ggml-org/llama.cpp#20003
• Eval bug: forcing full prompt re-processing due to lack of cache data in Qwen3.5 27B ggml-org/llama.cpp#20153
• Eval bug: Qwen 3.5 Full prompt re-processing on every conversation turn ggml-org/llama.cpp#20225
• Eval bug: Gemma 4 & Qwen 3.5 full prompt reprocessing from system prompt in OpenCode and Pi Coding Agent but only with llama.ccp server as backend. ggml-org/llama.cpp#21912

Related upstream work-in-progress (partial fixes, not yet covering our case):

• completion : fix prompt cache for recurrent models ggml-org/llama.cpp#19045 — "completion: fix prompt cache for recurrent models" (merged)
• Enable Partial Prompt Cache Reuse for Recurrent Models via State Checkpointing ggml-org/llama.cpp#19264 — "Enable Partial Prompt Cache Reuse for Recurrent Models via State Checkpointing" (closed as resolved, but doesn't fully cover the hybrid multi-turn ReAct case)
• Hybrid model cache: add --checkpoint-every-nb ggml-org/llama.cpp#20087 — "Hybrid model cache: add --checkpoint-every-nb"

Practical impact: a turn requiring multiple tool-call retries pays a full-prompt decode cost on every iteration. One observed real turn took ~108s across 4 iterations. Single-shot inference (no retries) is fine — the eval's "flat speed across context depths" finding is real, it just doesn't extend to multi-iteration agentic use as originally assumed.

Options to consider before re-enabling

• Wait for upstream. The PRs/discussions above show ggml-org/llama.cpp maintainers are actively working on recurrent-model prompt-cache/checkpointing. A future vendored llama.cpp version may resolve this with no changes needed on our side — re-check the issues above before doing anything else.
• Reduce ReAct retry cost some other way — e.g. improving tool-call reliability/prompting so fewer retries happen in practice (Ornith's update_task_status/search_nodes argument-guessing behavior was already one contributor to multi-iteration turns during testing).
• Ship it anyway, clearly labeled. The UI warning banner and doc comment already written (still present in the code, just gated off) could be turned back on as-is if the trade-off is judged acceptable for a "fast single-shot, slow multi-step" positioning — this doesn't require waiting on anything, just a product decision.
• Re-run the eval properly. The original Evaluate Ornith 1.0 as native local model for mode 2a #1464 evaluation's Sub-experiment D never actually tested repeated re-prompting within one context (it created a fresh LlamaContext per single-shot measurement) — if Ornith is reconsidered, the eval methodology should be fixed first so this gap doesn't recur for the next model.

To re-enable

Single-line change: add "ornith-1-9b-q4km" back to EXPOSED_GGUF_MODEL_IDS in packages/desktop-app/src-tauri/src/commands/chat_models.rs. Everything else (catalog entry, frontend ModelFamily type/filter, UI warning banner) is already in place and does not need to be re-implemented.

Related

• Add Ornith 1.0 to model catalog (catalog entry, think-block routing, update_node prompt) #1465 — original Ornith catalog integration (implemented, then partially hidden)
• Evaluate Ornith 1.0 as native local model for mode 2a #1464 — original evaluation (methodology gap noted above)