From 33504e1a2b100227a3c5b96a8beb51e6a7074017 Mon Sep 17 00:00:00 2001 From: Anish Sarkar <104695310+AnishSarkar22@users.noreply.github.com> Date: Sat, 19 Sep 2026 12:59:12 +0530 Subject: [PATCH 01/60] feat(spec): add llama.cpp runtime spec --- plans/community-local/00-umbrella-plan.md | 7 +- .../api/05a-model-recommendations.md | 6 + .../api/05d-llmfit-catalog-expansion.md | 7 + .../api/07-llamacpp-runtime.md | 524 ++++++++++++++++++ 4 files changed, 541 insertions(+), 3 deletions(-) create mode 100644 plans/community-local/api/07-llamacpp-runtime.md diff --git a/plans/community-local/00-umbrella-plan.md b/plans/community-local/00-umbrella-plan.md index abc8a5dc15..ddb78e66c9 100644 --- a/plans/community-local/00-umbrella-plan.md +++ b/plans/community-local/00-umbrella-plan.md @@ -29,6 +29,7 @@ Same phase number = integrate together. | **4** | [`04-studio.md`](frontend/04-studio.md) ◐ | [`04-studio.md`](api/04-studio.md) ◐ | [`04-studio.md`](worker/04-studio.md) ◐ | | **5** | [`05-install-ux.md`](frontend/05-install-ux.md) ◐ | [`05a-model-recommendations.md`](api/05a-model-recommendations.md) ✓ + [`05b-openai-compatible-connections.md`](api/05b-openai-compatible-connections.md) ✓ + [`05c-packaging.md`](api/05c-packaging.md) ◐ | [`05-packaging.md`](worker/05-packaging.md) ✓ | | **6** | import, license, egress settings ([`00d-pivot-plan.md`](00d-pivot-plan.md)) ✓ | `modules/migration/`, `modules/license/`, keychain, auto-update ([`00d-pivot-plan.md`](00d-pivot-plan.md)) ◐ | — (import reuses `ingest_document`) | +| **7** | catalog rework, fit badges, HF search (in [`api/07-llamacpp-runtime.md`](api/07-llamacpp-runtime.md)) | [`07-llamacpp-runtime.md`](api/07-llamacpp-runtime.md) — replace Ollama with llama.cpp | — | **Demo:** phase 3 all streams. **Ship:** phase 6 = SurfSense 2.0.0 (1.0.x tags are taken by the old project versioning; see [`00d-pivot-plan.md`](00d-pivot-plan.md)). @@ -102,9 +103,9 @@ Docker Compose, Postgres, Zero, Redis, Celery, LangGraph, git KB, scrapers, MCP, | **HTTP stack** | **FastAPI** + uvicorn | Same stack as cloud backend; native OpenAPI for frontend; SSE streaming for chat. PyInstaller risk is handled in [`api/00-spike.md`](api/00-spike.md) — not a reason to downgrade. | | **Retrieval** | **FTS5 + sqlite-vec hybrid, cosine rescore** | Semantic + keyword from day one: both legs widen recall, cosine orders. Embeddings on ingest must be queried properly — not keyword-only, not in-memory scan over BLOBs. | | **Embed provider** | Bundled bge-small-en-v1.5 int8, in-process on onnxruntime | 384-dim, ~66MB, runs offline on CPU with no model server. Docling parses, Chonkie chunks. Remote embedding is a later opt-in, not a launch dependency. | -| **Generation architecture** | llmfit catalog/advisor + curated-model policy + runtime adapters; Ollama local, multiple OpenAI-compatible connections remote | [`api/05a-model-recommendations.md`](api/05a-model-recommendations.md) and [`api/05b-openai-compatible-connections.md`](api/05b-openai-compatible-connections.md). llmfit supplies broad model metadata, hardware detection, and fit estimates; it is not a `Generator`, installer, or inference runtime. Local adapters resolve trusted artifacts and Ollama is first. Remote endpoints are named connection instances with their own URL and optional key; models are discovered live, not synchronized into SQLite. `SelectedModel(role)` stores the provider, connection identity when remote, and exact model id. SurfSense selects endpoints but does not load-balance their replicas. | -| **llmfit integration** | **Pinned official binary, short-lived JSON CLI, normalized behind `ModelAdvisor`** | The API runs `llmfit --json system` and `llmfit --max-context 8192 --json fit`, caches one scan, and exposes only SurfSense DTOs. No fork, patch, Python import, or permanent llmfit server. CLI label/machine-code differences are normalized at one seam and fixture-tested before a version bump. Failure removes ranking, not installed-model selection or chat. | -| **Ollama runtime** | **Bundled as a supervised sidecar (packaged); dev uses the developer's own `ollama serve`** | Chat can't depend on a daemon the user may not have installed. The packaged app ships the standalone Ollama archive (`electron/scripts/fetch-ollama.mjs` stages it, electron-builder carries it in `resources/ollama`) and Electron runs it as a third sidecar on a chosen port, passing `SURFSENSE_LOCAL_OLLAMA_BASE_URL` to the API. Models aren't shipped — the user pulls into the writable data dir after install. The Python sidecars and Ollama share one supervisor (`electron/src/main/sidecars/`): the supervisor spawns/reaps, one spec file per sidecar carries its identity. Boot now starts four — `api`, `worker-ingest`, `worker-studio`, and Ollama when packaged — and `watchImageModel` adds a fifth, sd-server, later, because only the API knows which image model was chosen. Ollama is best-effort at boot (only the API gates the window; its state surfaces via `/llm/providers`). | +| **Generation architecture** *(Phase 7 supersedes the local half — [`api/07-llamacpp-runtime.md`](api/07-llamacpp-runtime.md): llama.cpp is the one local runtime, llmfit moves to build time, the catalog opens to every GGUF on Hugging Face)* | llmfit catalog/advisor + curated-model policy + runtime adapters; Ollama local, multiple OpenAI-compatible connections remote | [`api/05a-model-recommendations.md`](api/05a-model-recommendations.md) and [`api/05b-openai-compatible-connections.md`](api/05b-openai-compatible-connections.md). llmfit supplies broad model metadata, hardware detection, and fit estimates; it is not a `Generator`, installer, or inference runtime. Local adapters resolve trusted artifacts and Ollama is first. Remote endpoints are named connection instances with their own URL and optional key; models are discovered live, not synchronized into SQLite. `SelectedModel(role)` stores the provider, connection identity when remote, and exact model id. SurfSense selects endpoints but does not load-balance their replicas. | +| **llmfit integration** *(Phase 7: the binary stops shipping. llmfit runs in CI to rank the curated manifest; it is never a gate, a hardware source, or on a request path)* | **Pinned official binary, short-lived JSON CLI, normalized behind `ModelAdvisor`** | The API runs `llmfit --json system` and `llmfit --max-context 8192 --json fit`, caches one scan, and exposes only SurfSense DTOs. No fork, patch, Python import, or permanent llmfit server. CLI label/machine-code differences are normalized at one seam and fixture-tested before a version bump. Failure removes ranking, not installed-model selection or chat. | +| **Ollama runtime** *(**superseded by Phase 7** — [`api/07-llamacpp-runtime.md`](api/07-llamacpp-runtime.md) replaces it with `llama-server` in router mode: 11–31 MB instead of 501 MB, CUDA inside the Windows installer, and no MLX until after launch)* | **Bundled as a supervised sidecar (packaged); dev uses the developer's own `ollama serve`** | Chat can't depend on a daemon the user may not have installed. The packaged app ships the standalone Ollama archive (`electron/scripts/fetch-ollama.mjs` stages it, electron-builder carries it in `resources/ollama`) and Electron runs it as a third sidecar on a chosen port, passing `SURFSENSE_LOCAL_OLLAMA_BASE_URL` to the API. Models aren't shipped — the user pulls into the writable data dir after install. The Python sidecars and Ollama share one supervisor (`electron/src/main/sidecars/`): the supervisor spawns/reaps, one spec file per sidecar carries its identity. Boot now starts four — `api`, `worker-ingest`, `worker-studio`, and Ollama when packaged — and `watchImageModel` adds a fifth, sd-server, later, because only the API knows which image model was chosen. Ollama is best-effort at boot (only the API gates the window; its state surfaces via `/llm/providers`). | | **Persistence** | SQLAlchemy 2.0 + Alembic, same as cloud | Models are the source of truth. `versions/` ships as PyInstaller data, resolved from the package's own `__file__` — de-risked in [`api/00-spike.md`](api/00-spike.md). | | **Migrations** | **Hand-written; autogenerate is off** | Autogenerate cannot see a rename — it emits drop + add, which deletes a column's data silently. The target database is one user's laptop, unbacked and uninspectable, so every revision is written and read by a person. `env.py` carries no `target_metadata`, so `--autogenerate` cannot be used by accident. Mature SQLite-backed apps make the same call — hand-written revisions throughout. | | **Schema owner** | Alembic only; **never** `create_all` | Cloud's `create_all`-on-startup races its own migrations and breaks releases. Local has one path to a schema, and a test fails if models and migrations drift. | diff --git a/plans/community-local/api/05a-model-recommendations.md b/plans/community-local/api/05a-model-recommendations.md index 57e4cf5ec5..68cdede83b 100644 --- a/plans/community-local/api/05a-model-recommendations.md +++ b/plans/community-local/api/05a-model-recommendations.md @@ -135,6 +135,12 @@ the server and is never accepted from the renderer. ### llama.cpp, later +> **Now.** [`07-llamacpp-runtime.md`](07-llamacpp-runtime.md) is this section, +> specified. It goes further than "adding a runtime": llama.cpp becomes the +> *only* local runtime, llmfit leaves the installer for CI, and the Explore +> bucket becomes a Hugging Face search over every GGUF rather than the subset +> llmfit happens to know. + A future adapter resolves a canonical model and quantization to a verified GGUF, downloads atomically, starts `llama-server`, and implements the existing `Generator` stream. Adding it changes the runtime registry and packaging, not diff --git a/plans/community-local/api/05d-llmfit-catalog-expansion.md b/plans/community-local/api/05d-llmfit-catalog-expansion.md index 720f6f49ce..0b75f9e611 100644 --- a/plans/community-local/api/05d-llmfit-catalog-expansion.md +++ b/plans/community-local/api/05d-llmfit-catalog-expansion.md @@ -4,6 +4,13 @@ > Owns the `hf.co/` fallback `ollama_name` synthesis in > `backend/modules/llm/recommendations/llmfit.py`'s `_parse_model()`. +> **Superseded by [`07-llamacpp-runtime.md`](07-llamacpp-runtime.md).** The +> `hf.co/` synthesis this document specifies exists only to push GGUF +> models through Ollama; with llama.cpp as the runtime it is deleted, along with +> `trusted_quant` and the Marginal hedge. The Windows GPU measurements below +> stay — they are the evidence that llmfit reports `llama.cpp` with GGUF-shaped +> quants on real hardware, which is what the replacement relies on. + ## Background `surfsense_local`'s "Local" model tab shows far fewer models than the hardware diff --git a/plans/community-local/api/07-llamacpp-runtime.md b/plans/community-local/api/07-llamacpp-runtime.md new file mode 100644 index 0000000000..5d401156e1 --- /dev/null +++ b/plans/community-local/api/07-llamacpp-runtime.md @@ -0,0 +1,524 @@ +# API — Phase 7: Replace Ollama with llama.cpp + +> Owns: `modules/llm/providers/`, the runtime sidecar, the model catalog, and +> packaging for all three targets. Supersedes the Ollama runtime decision in +> [`../00-umbrella-plan.md`](../00-umbrella-plan.md) and the `hf.co/` fallback in +> [`05d-llmfit-catalog-expansion.md`](05d-llmfit-catalog-expansion.md). Extends +> [`05a-model-recommendations.md`](05a-model-recommendations.md) — its +> "llama.cpp, later" section is now this document. + +## Goal + +One local runtime: `llama-server` in router mode. Any GGUF model the user wants, +not a curated subset. Recommendations that come from the machine's own allocator +rather than an advisor's estimate. + +Ingestion, the 384-dimensional bge-small index, retrieval, and chat SSE do not +change. + +## Why + +Ollama's library holds 240 models. A measured scan on this branch resolved 138 +of 9,590 llmfit rows to an installable Ollama artifact — 1.4%. The +`ollama_name` gate in `OllamaRuntime.resolve()` drops every model Ollama's own +registry does not carry, including the ~1,500 per scan that llama.cpp could run +directly from a Hugging Face GGUF. + +llama.cpp runs anything in GGUF: **204,797 repos on Hugging Face**, bounded only +by its 152 supported architectures. The runtime also ships smaller (11–31 MB +versus Ollama's 501 MB staged payload), fits models to the machine itself +(`--fit`, default on), and exposes multimodal and structured-output contracts +Ollama's native API does not. + +## Decisions + +| Decision | Choice | Why | +|---|---|---| +| **Runtime** | `llama-server` router mode, single sidecar | `--models-dir`, `--models-max 1`, LRU eviction, one PID for the supervisor to reap. Maps 1:1 onto every Ollama call the adapter makes. | +| **MLX** | **Not shipped.** Revisit after launch | Ollama switched its Apple Silicon engine to MLX in Mar 2026, so the swap costs Mac speed on models under ~14B. MLX's format covers 23,985 HF repos against GGUF's 204,797 — one format and the full catalog wins. Mac users who want MLX point a connection at LM Studio; see **The Mac path**. | +| **llmfit** | **Build-time only. Not shipped.** | It ranks the curated entries in CI and writes the numbers into the manifest. It is not a gate, not a hardware source, and not in any request path. Removes 27 MB, a pinned version, and five failure modes per installer. | +| **Gating** | Only physics refuses | Weights + minimum KV exceeding VRAM **plus** RAM. `can_install` never reads a fit level. Everything else is a badge. | +| **Catalog** | Two tiers — curated manifest, and Hugging Face search | Curated is the offline product and ships frozen. Search is a network feature that is simply absent airgapped. | +| **Fit estimate** | Our own, from the GGUF header | llmfit cannot score a model outside its database (`llmfit plan` states the precondition). Search needs an estimate anyway; once it exists, llmfit's is redundant *and* less accurate here. | +| **Hardware budget** | `ggml_backend_dev_memory()` via `ctypes`, from the shipped libs | The allocator's own view. Falls back to `llama-server --list-devices`, then OS APIs. | +| **Downloads** | **SurfSense fetches the GGUF**, not `POST /models` | `llama-server` is a second process we do not proxy; an in-process fetch is the only place `egress.require()` actually holds. Also buys resume, checksums, and the header as the file lands. | +| **Quantization** | Pinned per entry; two variants on the two largest | A pinned file is what "tested by SurfSense" can honestly claim. See **The quantization ladder**. | +| **Context** | Set at load, grown on occupancy. Floor 16K | llama.cpp fixes context at load; `num_ctx()`'s per-request sizing has no equivalent. 3K history + ~8K grounding + reply. | +| **Windows CUDA** | **In the installer.** No post-install download | Airgapped. Dropping Ollama frees ~500 MB; CUDA 13.4 + cudart costs 573 MB. Net ≈ +100 MB, under the 2 GB `makensis` ceiling that forced `pruneCudaRunners()`. | +| **CUDA version** | 13.4 only | CUDA 13 requires Turing (7.5)+. Shipping 12.4 as well for Pascal costs another 645 MB — that is what made Ollama's archive 1.8 GB. Pascal falls through to Vulkan automatically via backend scoring. | +| **Backend selection** | Runtime, by ggml | Releases are built `GGML_BACKEND_DL=ON`. A failed `dlopen` is skipped silently, so a missing Vulkan loader degrades to CPU rather than failing. | +| **Prompt tier fallback** | Keys on **loopback**, not provider name | `provider == "ollama" → COMPACT` breaks the moment a Mac user runs a 4B through LM Studio. `host_destination()` already computes loopback. | + +## Boundaries + +```text + hardware budget ── ggml_backend_dev_memory() ← the allocator, not an advisor + │ + model facts ───── GGUF header (local file, or HTTP Range on huggingface.co) + │ + ▼ + FitEstimator ── advisory only + │ weights + KV(window) + overhead vs VRAM + RAM + │ → fits-gpu | needs-ram | too-big | PhysicsRefusal + ▼ + CatalogService + ├── curated manifest, ranked by llmfit AT BUILD TIME, ships frozen + └── search HF /api/models?filter=gguf + /tree/main + │ + ▼ + LlamaCppProvider ── Generator + ModelStore + LocalRuntime + │ + ▼ + llama-server router ── --fit places layers; the real answer +``` + +The estimator is advisory at every stage above the runtime. `--fit`'s allocation +is authoritative at launch, and generation is ground truth after it. Unknown +shapes round **up**; never underestimate memory. + +## The quantization ladder + +Measured: architecture fields are identical across quantizations of the same +model (`Q4_K_M`, `Q8_0`, `f16` of SmolLM2-135M all report `layers=30 +kv_heads=3 emb=576 ctx=8192`; only `general.file_type` differs). KV cost is +architecture-derived and quant-independent. So **one header read prices every +quantization of a model**, and the weights bytes come free from `tree/main`. + +Per-machine quant selection is therefore possible at no extra cost. We pin +anyway, for three reasons that are policy rather than capability: the +"tested by SurfSense" claim covers files we ran, support needs reproducible +bytes, and an unconstrained fit-maximiser hands small machines a Q2_K build +that answers noticeably worse (llmfit currently recommends exactly that for +Qwen3-14B on an 8 GB Mac). + +The shipped eight, measured at Q4_K_M and a 16K window: + +| model | file GB | layers | kv_heads | needs @16K | +|---|---|---|---|---| +| Qwen3 0.6B | 0.40 | 28 | 8 | 2,354 MiB | +| Qwen3 1.7B | 1.11 | 28 | 8 | 3,032 | +| Qwen3 4B | 2.50 | 36 | 8 | 4,630 | +| Qwen2.5-Coder 7B | 4.68 | 28 | 4 | 5,966 | +| Qwen3 8B | 5.03 | 36 | 8 | 7,043 | +| Qwen3 14B | 9.00 | 40 | 8 | 10,969 | +| Qwen3 32B | 19.76 | 64 | 8 | 22,047 | +| Qwen2.5-Coder 32B | 19.85 | 64 | 8 | 22,132 | + +What each machine gets: + +```text +machine 0.6B 1.7B 4B C7B 8B 14B 32B C32B → recommended +M2 8 GB GPU GPU GPU RAM RAM RAM — — → Qwen3 4B (831 MiB spare) +RTX 3050 6 GB GPU GPU GPU RAM RAM RAM RAM RAM → Qwen3 4B (490 MiB spare) +M4 16 GB GPU GPU GPU GPU GPU RAM RAM RAM → Qwen3 8B (3,857 MiB spare) +RTX 4070 12 GB GPU GPU GPU GPU GPU GPU RAM RAM → Qwen3 14B (31 MiB spare) +RTX 4090 24 GB GPU GPU GPU GPU GPU GPU GPU GPU → Qwen3 32B (953 MiB spare) +M4 Max 64 GB GPU GPU GPU GPU GPU GPU GPU GPU → Qwen3 32B (26,000 MiB spare) +``` + +Every machine gets a GPU-resident pick, so eight rungs strand nobody. Three gaps: + +- **The top collapses.** 24 GB and 64 GB — 2.7× apart — get the same file, and + the larger wastes 26 GB. A higher quant does not fix this; 32B at Q8 is ~35 GB + and still leaves room. **The top needs a bigger model.** +- **A gap at 16 GB.** 14B misses by 69 MiB (0.6%), so a 16 GB Mac drops to 8B + with 3.9 GB spare. +- **The 4070 row is decided by an uncalibrated constant.** 31 MiB of margin at + 1,024 MiB overhead; at Hermes' measured 1.5 GiB it does not fit and that + machine drops a rung. + +Manifest changes, in priority order: + +1. **A rung above 32B** — an 80B-A3B-class MoE. The only thing that serves + 48–64 GB machines. MoE needs a `decode_fraction` field (active bytes read per + token) or the speed prediction is wrong by an order of magnitude. +2. **A second variant on 8B and 32B** (`Q6_K`), filling the 16 GB gap and giving + the 4090 somewhere to go. Two extra pins, not fifty. +3. **A vision entry.** All eight are text-only, in an app with a PDF pipeline. +4. **Calibrate the overhead constant** before shipping. It decides the 4070 row. + +## The Mac path + +Dropping Ollama costs Apple Silicon speed on models under ~14B. Accepted, +because the escape hatch already ships and does not leave the machine: + +- `host_destination()` returns `None` for loopback — *"nothing leaves the + machine"* — so `require()` no-ops and no egress prompt appears. +- `api_key_ciphertext` is nullable and `_headers(None)` returns `{}` — keyless + local endpoints work end to end. +- `connection-form.tsx` already ships `LM Studio (local) → http://localhost:1234/v1` + and `Ollama (local) → http://localhost:11434/v1`. + +LM Studio runs MLX on Apple Silicon. A Mac user who wants it installs LM Studio +and picks the preset. **Keep the Ollama preset** — it is the user-managed MLX +path now, not dead code. + +## Runtime contract + +| Ollama today | llama-server router | +|---|---| +| `GET /` | `GET /health` | +| `GET /api/tags` | `GET /models` (+ `architecture.input_modalities`) | +| `POST /api/show` | `GET /props?model=…` — `chat_template_caps`, `modalities`, `n_ctx` | +| `POST /api/pull` | we fetch `resolve/main/{file}` ourselves | +| blob cleanup on cancel | **deleted** — `cleanup_cancelled_download()`, ~40 lines | +| `DELETE /api/delete` | `DELETE /models?model=…` | +| `POST /api/chat` | `POST /v1/chat/completions` — compose `OpenAICompatibleChatProvider` | + +Sidecar flags: `--models-dir /models --port --models-max 1 +--sleep-idle-seconds 300 --no-ui --jinja --reasoning-format deepseek`. + +`--no-ui` because llama-server ships its own web UI. `--reasoning-format deepseek` +routes `` blocks to `message.reasoning_content`; without it a thinking +model's trace enters `parts[]` and `resolve_citations()` rewrites `[n]` tokens +that appeared inside the reasoning. + +## Phases + +Phases 7.0–7.5 keep Ollama registered and working. `REGISTRY` carries both +adapters, so each ships independently and is reversible. **7.6 is the only +irreversible phase and the only one that touches user data.** + +### 7.0 — De-Ollama the domain + +No runtime change. `ScoredModel.ollama_name` → `artifacts: tuple[GgufArtifact, ...]` +where `GgufArtifact(repo, quantization, mmproj)`. `OllamaProvider` renders +`hf.co/{repo}:{quant}` itself, so the runtime's private vocabulary stops leaking +upward. + +Deleted: `_fallback_ollama_name()`, `clean_runtime_name()`, the `trusted_quant` +gate, the `FitLevel.MARGINAL` relabel, the `:latest` workaround. All five exist +only because Ollama's library was in the way. + +llmfit adapter switches to `recommend --force-runtime llamacpp --output-llamacpp +-n --no-dashboard`. Bump `CACHE_VERSION`. + +**Tests:** `llmfit.py` fixtures. No UI change. + +### 7.1 — GGUF header reader + +`modules/llm/gguf/header.py`, stdlib only. Local file or `Range: bytes=0-4194303` +against `huggingface.co`. Yields architecture, `block_count`, `context_length`, +`embedding_length`, `head_count_kv`, key/value lengths, `sliding_window`, +`expert_count`, `n_vocab`, chat template, exact tensor bytes. + +Verified: HF returns `206` with `accept-ranges: bytes`. A 135M model's metadata +ends at 1.77 MB and its tensor table at 1.79 MB, in 2.5 s. Budget 4 MB and +retry wider on a truncated parse — the vocab token list is what makes it large. + +**Tests:** a checked-in truncated GGUF; a fake HTTP server asserting the Range +header and a truncated-response retry. + +### 7.2 — Hardware budget and estimator + +`modules/llm/hardware.py` — `ctypes` into `libggml`, then `--list-devices`, then +OS APIs. First hit wins, cached; warm the probe in the background at first +launch, not on the path of the first render (a cold `ggml_backend_load_all()` on +macOS compiles 20 Metal shader libraries, measured at ~20 s once, 180 ms after). + +Two budget modes. **Capacity** (total − margin) for catalog pricing; **live** +(free now) for launch decisions. Pricing against live-free while a model is +loaded makes every row read as too large. + +`modules/llm/fit.py` — `weights + KV(window) + overhead` vs `VRAM + RAM`. +Per-layer KV from the header. Split today's single `recommendation_reserve_gb: 2.0` +(flagged `ponytail` as uncalibrated) into a measured runtime overhead, a device +margin, and a UMA headroom. + +> **ponytail:** the runtime overhead constant is a placeholder until measured +> against real RSS on all three targets. On a 5.4 GB budget it moves rows across +> the fits/spills line. + +**Tests:** decision-table over constructed profiles — no GGUF parsing in the fit +tests, so the estimator is testable independently of the reader. + +### 7.3 — `LlamaCppProvider` + +Registered alongside Ollama. `REGISTRY` already takes two; `CatalogService` +already loops `self._runtimes`. This is where 05a's acceptance criterion — +"adding a fake llama.cpp runtime requires no change to llmfit parsing, curated +partitioning, or the frontend catalog schema" — becomes a real test. + +Chat composes `OpenAICompatibleChatProvider` rather than reimplementing +streaming. Install fetches the GGUF into `--models-dir` with real filenames, not +llama.cpp's content-addressed HF cache layout (`LLAMA_CACHE` points at the app +data dir so nothing writes to `~/.cache`). Context set via `POST /models/load` +with `args`, on a ladder grown at turn boundaries. + +**Tests:** a fake llama-server (aiohttp) covering health, models, props, load, +chat stream, delete; install against a fake HF serving a real small GGUF. + +### 7.4 — Catalog: curated + search + +Manifest `schema_version` 3: `artifacts.llamacpp = {repo, quant, mmproj}`, plus +`quality`, `estimated_tps`, `prefill_tps`, `ttft_ms`, `use_case`, +`capability_ids`, `license`, `decode_fraction` — all written by a CI job running +llmfit. Ships frozen; airgapped means no refresh path. + +Search: `?filter=gguf&search=&sort=downloads&limit≤50`, 300 s TTL cache (HF +allows 500 requests per 5 minutes), `tree/main?recursive=true` for quants. Group +split parts (`-00001-of-00003`); exclude `mmproj*` and `*draft*` from the quant +list. + +Gates: architecture in the 152-name list · chat template present · repo not +`gated` · disk space · physics. Nothing else. + +Deleted: the collision-resolution block, `_placeholder_row()`, +`_synthetic_curated_model()`, `_runtime_target_model()`, the disk scan cache, +`CACHE_VERSION`, and the `scanned` flag through `AdvisorCatalog` → +`CatalogResult` → `RecommendationCatalogRead` → the frontend. + +**Recommendation policy**, its own module and its own test file: + +```text +fitting = entries the estimator does not refuse +resident = fitting, weights entirely in VRAM +pleasant = resident, felt_cost(representative RAG turn) under floor + +pleasant → max(quality, -size) reason: best-quality-resident + or speed-gated-quality if the floor + eliminated a higher-quality candidate +resident → max(speed) reason: fastest-resident +else → no recommendation; spilled entries stay installable +``` + +Quality is editorial and static per pinned file. Speed is physics per machine. +Predictions order candidates and gate the floor; they are never displayed. + +> **RAG is prefill-dominated.** `HISTORY_BUDGET_TOKENS = 3000` plus ~24k +> characters of grounding is roughly 8,000 prefill tokens against ~300 decoded — +> the inverse of an agent's ratio. Decode is memory-bound; prefill is +> compute-bound, which is why CUDA leads Vulkan ~36–40% on `pp512` and ~10% on +> `tg128`. A decode-only prediction mis-ranks for this app. Use llmfit's +> `prefill_tps`/`ttft_ms` from the manifest until measured class constants exist. + +**Tests:** policy decision-table across the six hardware profiles above. + +### 7.5 — Packaging, three targets + +`fetch-llamacpp.mjs`, pinned build number **and SHA-256** (`fetch-llmfit.mjs` +verifies a checksum; `fetch-ollama.mjs` does not — close that gap here, since +llama.cpp has no stable channel). + +| Target | Asset | Size | +|---|---|---| +| darwin-arm64 | `bin-macos-arm64.tar.gz` | 11 MB | +| win32-x64 | `bin-win-vulkan-x64.zip` + `cuda-13.4` + `cudart` | 31 + 150 + 423 MB | +| linux-x64 | `bin-ubuntu-vulkan-x64.tar.gz` | 30 MB | + +Verified: the Vulkan archive is the CPU archive plus exactly one file +(`libggml-vulkan.so` / `ggml-vulkan.dll`), and all CPU micro-architecture +variants ship inside — 15 on Windows, 10 on Linux. `ggml_backend_load_best()` +searches the executable's own directory and probes `cuda` before `vulkan`, so +the CUDA DLLs sit in the same flat folder and win automatically. + +cudart carries `cudart64_*.dll`, `cublas64_*.dll`, `cublasLt64_*.dll`; all must +be beside `llama-server.exe` or it fails at launch with a missing-DLL error. + +Prune to `llama-server` plus its libraries: 24 executables → 1, which also +shrinks the macOS notarization surface. + +- **Linux:** `libggml-vulkan.so` has `DT_NEEDED: libvulkan.so.1`. deb adds + `libvulkan1` to `Depends`; AppImage bundles the loader or accepts CPU (the ICD + must come from the host driver). glibc floor is **2.34**. +- **Pin `ubuntu-22.04`.** `ubuntu-latest` migrates to 26.04 between 19 Oct and + 19 Nov 2026 and will silently raise the AppImage's glibc floor. The + PyInstaller binaries already require 2.39 from the current runner, so this is + a pre-existing portability bug that llama.cpp does not cause and this phase + should fix. +- **macOS:** router mode spawns grandchildren under the hardened runtime. +- **Windows:** verify `taskkill /t /f` on the router PID reaps its model workers. + +**Tests:** staged-checksum verification; per-OS smoke running `--list-devices` +from the final packaged resource path. + +### 7.6 — Remove Ollama completely + +~440 references across 66 files. The only irreversible phase. + +**Delete:** + +```text +backend/modules/llm/providers/ollama/ provider.py, catalog.py, __init__.py +backend/tests/unit/llm/recommendations/test_ollama_runtime.py +backend/tests/unit/llm/test_ollama_num_ctx.py +electron/src/main/sidecars/ollama.ts +electron/scripts/fetch-ollama.mjs +electron/ollama/ 501 MB staged +``` + +**Rewrite:** `shared/config.py` (`ollama_base_url`/`ollama_models_dir` → +`llamacpp_*`), `sidecars/types.ts` (`SidecarContext.ollama*`), +`sidecars/python.ts` (stops passing `OLLAMA_*`), `electron-builder.yml`, +`electron/package.json` (`build:ollama` and the `dist` chain), +`.github/workflows/release-local.yml`, `modules/llm/router.py`, +`modules/llm/resolution.py`, `modules/llm/selection.py`, `modules/chat/errors.py`. + +**Frontend:** `selected-roles.tsx` (`llamacpp` → "Local"; a loopback connection +is arguably also "Local"), `chat-error-notice.tsx`. **Keep** +`connection-form.tsx`'s Ollama preset. + +**Egress:** `ollama_pull → registry.ollama.ai` becomes two destinations against +`huggingface.co` — `model_download` (a repo you named) and `model_search` (text +you typed). Same host, different consent. `ollama_pull_host()`'s `hf.co/` +special-case disappears. + +**Three things that will bite:** + +1. **Do not touch revisions `0004` and `0009`.** They mention Ollama and are + applied history on every installed machine. A new revision does the data work. + `tests/integration/test_migrations.py` asserts historical schema — read it + first. +2. **Existing users have a broken selection.** `SelectedModel(provider="ollama")` + makes `resolve_generation()` raise `unknown provider` and chat dies. The + migration clears it and the UI states why; the weights are in a blob format + we no longer manage, so there is no silent remap. +3. **`/ollama/` is orphaned and gigabytes.** Offer to delete it. Leaving + it silently is not acceptable in a local-first app. + +**Definition of done:** + +```bash +grep -rn -i ollama surfsense_local \ + --exclude-dir=node_modules --exclude-dir=release --exclude-dir=.venv +``` + +returns only the `connection-form.tsx` preset and its test, revisions `0004` +and `0009`, and the new migration's `WHERE destination = 'ollama_pull'`. + +**Tests:** a migration test over a database seeded with an Ollama selection and +an `ollama_pull` egress row. + +### 7.7 — Capabilities and multimodal + +`Capabilities(inputs, tools, system_role, typed_content, context_tokens)` at the +`Generator` seam, from `GET /models.architecture.input_modalities` and +`GET /props.chat_template_caps` — template-derived truth, not a name regex. +`supports_system_role` matters immediately: `build_messages()` always emits a +system message and silently degrades on templates without it. + +`Message.content` widens from `str` to `str | tuple[ContentPart, ...]`. The +adapter **downgrades at the seam**: a text-only model gets the image replaced by +its extracted text, never an error, so `modules/chat` never learns about +modality. + +Retrieval stays text. Ingestion already writes page files; `image_url.url` +accepts a **local file path**, so the path Docling wrote is handed over directly +with no base64 round-trip. `--mmproj-auto` is on by default with `-hf`, so a +vision model brings its own projector. + +> **ponytail:** [#19980](https://github.com/ggml-org/llama.cpp/issues/19980) — +> `--fit` does not account for mmproj VRAM, so vision models can OOM on tight +> machines. Add the projector bytes to our own sum. + +Touches `Hit`, the chunk model, and citations. Ships after the swap, not with it. + +### 7.8 — Constrained decoding and prompt consolidation + +69 prompt files exist (23 cases × 3 tiers); **48 demand JSON**. +`response_format: {"type": "json_schema", "schema": {…}}` masks every token that +would produce invalid JSON, so malformed output becomes mechanically impossible. + +Two caveats to design around now: + +- The schema is **not** injected into the prompt. Fields still get described — + once, not once per tier, because compliance no longer depends on persuasion. +- [#29006](https://github.com/ggml-org/llama.cpp/issues/29006): json_schema on + the chat endpoint returns 400 for some templates while equivalent GBNF on + `/completion` succeeds. Studio needs that fallback path. + +`params_b` becomes exact from the GGUF header, retiring `_estimated_b()`'s +blob-size division and its `gemma3:4b` reads-as-5.5B `ponytail`. + +Tiers do not disappear, but what they carry changes: after constrained decoding +and template caps, a tier carries **reasoning depth only**, not format +compliance. That plausibly collapses three tiers to two. + +## Failure behavior + +- Runtime missing or crashed: catalog stays visible, installs disabled, an + already-selected remote connection still answers. +- No Vulkan loader: `dlopen` fails, backend skipped silently, CPU runs. +- HF unreachable: curated and installed render from the manifest and disk; + search reports the destination is unavailable, not an error. +- Header read truncated: retry wider once, then fall back to a file-size estimate + with the badge marked approximate. +- Physics refusal: state required and available bytes and name a smaller model. +- Cancelled download: `POST /models/unload`, partial file removed, never selected. + +## Tests + +- GGUF reader: truncated file, oversized vocab, split parts, Range retry. +- Estimator: decision-table over constructed profiles; f16 vs q8_0 KV (a 1 GB + swing on an 8 GB machine, measured). +- Recommendation policy: the six hardware profiles above, asserting both pick + and reason key. +- Catalog: curated renders with no network; search gated on egress; arch, + template, and gated-repo rejections. +- Runtime: fake llama-server for health/models/props/load/chat/delete. +- Migration: an Ollama selection and an `ollama_pull` row, upgraded. +- Packaging: staged checksum; `--list-devices` from the packaged path per OS. + +## Acceptance + +- A clean machine on all three OSes opens the catalog with a hardware line and + badged rows, with no scan button and no network. +- The recommended model installs and answers with citations. +- A model found through search — one Ollama's library never carried — installs + and answers. +- A Windows machine with an RTX card runs on CUDA, from the installer alone. +- A machine with no usable GPU runs on CPU with no error. +- Airgapped: curated install from a local `.gguf`, chat, and Studio all work. +- `grep -i ollama` returns only the intended survivors. +- Quit leaves no orphan `surfsense-*` or `llama-server` process. + +## Appendix — measurements + +Taken on an M2 / 8 GB against llama.cpp `b11043` and llmfit `1.1.11`, except +where noted. + +| Measurement | Value | +|---|---| +| GGUF repos on Hugging Face | 204,797 | +| llmfit embedded database | 11,271 models, 3,454 publishers | +| Ollama library | 240 models | +| llmfit rows with an `ollama_name` | 138 of 9,590 (1.4%) | +| llmfit rows resolvable under `--force-runtime llamacpp` | 1,109 of 6,804 fitting | +| llama.cpp supported architectures | 152 (`LLM_ARCH_*`) | +| `llmfit --json system` | 0.68 s | +| `llmfit --json fit` (full scan) | **0.84 s**, 9,590 rows, 17 MB JSON | +| `llmfit recommend -n 20000 --force-runtime llamacpp` | 0.72 s, 6,804 rows | +| `llama-server --list-devices` | 174 ms cold, 60 ms warm | +| `ggml_backend_dev_memory()` via ctypes | 180 ms warm, ~20 s first call on Metal | +| GGUF header over HTTP Range | 1.79 MB, 2.5 s (135M model) | +| macOS DMG today | 1,507 MB | +| `electron/ollama/` staged (macOS) | 501 MB — of which 380 MB is MLX, 66 MB the Go binary | + +Hardware-dependence of llmfit fields, from two simulated machines +(`--memory 8G --ram 8G` versus `--memory 80G --ram 256G`, 3,047 models in common): + +| Field | Changes with hardware | +|---|---| +| `parameter_count`, `params_b`, `context_length`, `license`, `use_case`, `capability_ids`, `runtime` | **no** — identical | +| `best_quant`, `memory_required_gb`, `estimated_tps`, `fit_level`, `score_components.quality` | yes | + +Holding the quantization fixed (2,424 models): **quality differs 0/2424, memory +differs 0/2424.** Quality and memory are functions of *(model, quantization)*, +not of hardware — hardware only enters by choosing the quantization. That is +what makes a pinned manifest's numbers deterministic and build-time computable. + +Device query versus advisor, same machine: + +| | llmfit | `ggml_backend_dev_memory()` | +|---|---|---| +| Memory budgeted | 8.0 GB | **5,461 MiB** (Metal working set) | +| Context priced | 8,192 | 16,384 (our real turn) | +| Qwen3 8B verdict | **Perfect** | **uses system RAM** | + +## References + +- [llama.cpp server README](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md) — router mode, `/models`, `/models/sse`, `/props` +- [`ggml-backend-reg.cpp`](https://github.com/ggml-org/llama.cpp/blob/master/ggml/src/ggml-backend-reg.cpp) — `ggml_backend_load_best()`, silent skip on `dlopen` failure +- [`common/jinja/caps.h`](https://github.com/ggml-org/llama.cpp/blob/master/common/jinja/caps.h) — `chat_template_caps` fields +- [PR #17485](https://github.com/ggml-org/llama.cpp/pull/17485) — automatic `n_gpu_layers`, graceful degradation +- [NousResearch/hermes-agent](https://github.com/NousResearch/hermes-agent) `hermes_cli/local_runtime/` — MIT; the closest shipped analogue. Four curated entries, one pinned variant each, `variants[-1]` unconditionally, headroom buys context not quantization. From c125c378efddf604bff45884604282fcae7885ee Mon Sep 17 00:00:00 2001 From: Anish Sarkar <104695310+AnishSarkar22@users.noreply.github.com> Date: Sat, 19 Sep 2026 14:33:46 +0530 Subject: [PATCH 02/60] feat(spec): update llamacpp spec --- .../api/07-llamacpp-runtime.md | 594 +++++++++++++++++- 1 file changed, 561 insertions(+), 33 deletions(-) diff --git a/plans/community-local/api/07-llamacpp-runtime.md b/plans/community-local/api/07-llamacpp-runtime.md index 5d401156e1..975ba1c595 100644 --- a/plans/community-local/api/07-llamacpp-runtime.md +++ b/plans/community-local/api/07-llamacpp-runtime.md @@ -36,10 +36,15 @@ Ollama's native API does not. |---|---|---| | **Runtime** | `llama-server` router mode, single sidecar | `--models-dir`, `--models-max 1`, LRU eviction, one PID for the supervisor to reap. Maps 1:1 onto every Ollama call the adapter makes. | | **MLX** | **Not shipped.** Revisit after launch | Ollama switched its Apple Silicon engine to MLX in Mar 2026, so the swap costs Mac speed on models under ~14B. MLX's format covers 23,985 HF repos against GGUF's 204,797 — one format and the full catalog wins. Mac users who want MLX point a connection at LM Studio; see **The Mac path**. | -| **llmfit** | **Build-time only. Not shipped.** | It ranks the curated entries in CI and writes the numbers into the manifest. It is not a gate, not a hardware source, and not in any request path. Removes 27 MB, a pinned version, and five failure modes per installer. | -| **Gating** | Only physics refuses | Weights + minimum KV exceeding VRAM **plus** RAM. `can_install` never reads a fit level. Everything else is a badge. | +| **llmfit** | **Authoring-time only. Not shipped, and not in CI.** | A person runs `scripts/refresh_curated_models.py` when adding or changing a curated entry — a few times a year — and commits the numbers. llmfit is not a gate, not a hardware source, not in any request path, and not in the build. Removes 27 MB, a pinned version, and five failure modes per installer. | +| **Gating** | Only physics refuses | `can_install = state != TOO_BIG`, and `TOO_BIG` *is* the physics refusal — weights + KV at the 16K floor exceeding VRAM **plus** RAM. No other fit level gates anything; `PARTIAL` installs like `FITS`. Eligibility (architecture, chat template, gated repo) blocks separately and is not a fit state. See **Fit states**. | | **Catalog** | Two tiers — curated manifest, and Hugging Face search | Curated is the offline product and ships frozen. Search is a network feature that is simply absent airgapped. | | **Fit estimate** | Our own, from the GGUF header | llmfit cannot score a model outside its database (`llmfit plan` states the precondition). Search needs an estimate anyway; once it exists, llmfit's is redundant *and* less accurate here. | +| **`rank`** | **Curated entries only. A searched model never carries one, from llmfit or anywhere else** | A preference order over models we tested, **for this app's job** — answering from the user's documents with citations that resolve — not general capability. An integer typed by a person, with llmfit proposing. Called `rank`, not `quality`, because it is only ever a sort key: nothing reads its magnitude, so it must not imply a measurement we do not have. Attaching one to an arbitrary repo attaches a base model's score to a derivative that behaves differently: of the top 100 GGUF repos by downloads, 38 match an llmfit entry, and those matches include `Huihui-Qwen3.8-27B-abliterated-GGUF` and `Qwen3.8-27B-Uncensored-GGUF` resolving to the base model's score. A wrong number people trust is worse than a blank they investigate. Search rows are **described, not judged**. | +| **Search order** | `sort=downloads`, descending | The only HF sort that behaves as a default. Presented as a popularity fact, never as an endorsement. See **Search ordering**. | +| **Fit badge** | **Every row, both tiers** | Fit is subtraction, not judgement — the opposite of `rank`. Coarse from file size in the search list, exact from the header on open, exact from `shape` offline for curated. Three states, no `unknown`. | +| **Curated manifest** | **Stays, schema 3, authored by hand** | It is the whole product airgapped, the first-run default, and the only tier the recommendation reads. `scripts/refresh_curated_models.py` writes it; a person commits it; **llmfit never runs in CI**. | +| **Manifest validation** | Existing unit tests. **No new CI** | Pydantic (`extra="forbid"`, `model_validator`) plus `test_curated_models.py` already reject malformed manifests. A dead pin is a runtime failure, gracefully handled — CI cannot prevent a repo disappearing after the build anyway. `validated: true` is the real check, and it is a human one. | | **Hardware budget** | `ggml_backend_dev_memory()` via `ctypes`, from the shipped libs | The allocator's own view. Falls back to `llama-server --list-devices`, then OS APIs. | | **Downloads** | **SurfSense fetches the GGUF**, not `POST /models` | `llama-server` is a second process we do not proxy; an in-process fetch is the only place `egress.require()` actually holds. Also buys resume, checksums, and the header as the file lands. | | **Quantization** | Pinned per entry; two variants on the two largest | A pinned file is what "tested by SurfSense" can honestly claim. See **The quantization ladder**. | @@ -59,10 +64,10 @@ Ollama's native API does not. ▼ FitEstimator ── advisory only │ weights + KV(window) + overhead vs VRAM + RAM - │ → fits-gpu | needs-ram | too-big | PhysicsRefusal + │ → FITS | PARTIAL | TOO_BIG (TOO_BIG = the physics refusal) ▼ CatalogService - ├── curated manifest, ranked by llmfit AT BUILD TIME, ships frozen + ├── curated manifest: shape derived + rank by hand, ships frozen └── search HF /api/models?filter=gguf + /tree/main │ ▼ @@ -91,32 +96,37 @@ bytes, and an unconstrained fit-maximiser hands small machines a Q2_K build that answers noticeably worse (llmfit currently recommends exactly that for Qwen3-14B on an 8 GB Mac). -The shipped eight, measured at Q4_K_M and a 16K window: +**The two Qwen2.5-Coder entries are dropped.** SurfSense has no coding job — the +Studio formats are summary, flashcards, mindmap, quiz, podcast, office, web and +visuals, and chat is document Q&A with citations. A coding model was taking two +of eight slots in a list whose real gaps are a rung for 48-64 GB machines and a +vision entry. Anyone who wants one searches for it, gets a fit badge, and +installs it — that is what the search tier is for. + +The shipped six, measured at Q4_K_M and a 16K window: | model | file GB | layers | kv_heads | needs @16K | |---|---|---|---|---| | Qwen3 0.6B | 0.40 | 28 | 8 | 2,354 MiB | | Qwen3 1.7B | 1.11 | 28 | 8 | 3,032 | | Qwen3 4B | 2.50 | 36 | 8 | 4,630 | -| Qwen2.5-Coder 7B | 4.68 | 28 | 4 | 5,966 | | Qwen3 8B | 5.03 | 36 | 8 | 7,043 | | Qwen3 14B | 9.00 | 40 | 8 | 10,969 | | Qwen3 32B | 19.76 | 64 | 8 | 22,047 | -| Qwen2.5-Coder 32B | 19.85 | 64 | 8 | 22,132 | What each machine gets: ```text -machine 0.6B 1.7B 4B C7B 8B 14B 32B C32B → recommended -M2 8 GB GPU GPU GPU RAM RAM RAM — — → Qwen3 4B (831 MiB spare) -RTX 3050 6 GB GPU GPU GPU RAM RAM RAM RAM RAM → Qwen3 4B (490 MiB spare) -M4 16 GB GPU GPU GPU GPU GPU RAM RAM RAM → Qwen3 8B (3,857 MiB spare) -RTX 4070 12 GB GPU GPU GPU GPU GPU GPU RAM RAM → Qwen3 14B (31 MiB spare) -RTX 4090 24 GB GPU GPU GPU GPU GPU GPU GPU GPU → Qwen3 32B (953 MiB spare) -M4 Max 64 GB GPU GPU GPU GPU GPU GPU GPU GPU → Qwen3 32B (26,000 MiB spare) +machine 0.6B 1.7B 4B 8B 14B 32B → recommended +M2 8 GB GPU GPU GPU RAM RAM — → Qwen3 4B (831 MiB spare) +RTX 3050 6 GB GPU GPU GPU RAM RAM RAM → Qwen3 4B (490 MiB spare) +M4 16 GB GPU GPU GPU GPU RAM RAM → Qwen3 8B (3,857 MiB spare) +RTX 4070 12 GB GPU GPU GPU GPU GPU RAM → Qwen3 14B (31 MiB spare) +RTX 4090 24 GB GPU GPU GPU GPU GPU GPU → Qwen3 32B (953 MiB spare) +M4 Max 64 GB GPU GPU GPU GPU GPU GPU → Qwen3 32B (26,000 MiB spare) ``` -Every machine gets a GPU-resident pick, so eight rungs strand nobody. Three gaps: +Every machine gets a GPU-resident pick, so six rungs strand nobody. Three gaps: - **The top collapses.** 24 GB and 64 GB — 2.7× apart — get the same file, and the larger wastes 26 GB. A higher quant does not fix this; 32B at Q8 is ~35 GB @@ -134,7 +144,8 @@ Manifest changes, in priority order: token) or the speed prediction is wrong by an order of magnitude. 2. **A second variant on 8B and 32B** (`Q6_K`), filling the 16 GB gap and giving the 4090 somewhere to go. Two extra pins, not fifty. -3. **A vision entry.** All eight are text-only, in an app with a PDF pipeline. +3. **A vision entry.** All six are text-only, in an app with a PDF pipeline. + Vision is a **capability**, not a separate list — see **Ranking**. 4. **Calibrate the overhead constant** before shipping. It decides the 4070 row. ## The Mac path @@ -250,19 +261,457 @@ chat stream, delete; install against a fake HF serving a real small GGUF. ### 7.4 — Catalog: curated + search -Manifest `schema_version` 3: `artifacts.llamacpp = {repo, quant, mmproj}`, plus -`quality`, `estimated_tps`, `prefill_tps`, `ttft_ms`, `use_case`, -`capability_ids`, `license`, `decode_fraction` — all written by a CI job running -llmfit. Ships frozen; airgapped means no refresh path. +**`curated-models.json` stays.** It is the only tier that works with no network, +so it is the entire product airgapped, the default on first run, and the only +thing the recommendation can read. It ships frozen in the build; there is no +refresh path. + +Manifest `schema_version` **3**. One entry, before and after: + +```jsonc +// v2 — today +{ + "model_id": "Qwen/Qwen3-8B", + "family": "Qwen3", + "minimum_fit": "good", + "minimum_context": 8192, + "allowed_quantizations": ["Q4_K_M"], + "artifacts": { "ollama": { "name": "qwen3:8b", "quantization": "Q4_K_M" } }, + "label": "Qwen3 8B", + "parameter_count": "8B", + "size_bytes": 5225374496 +} + +// v3 +{ + "model_id": "Qwen/Qwen3-8B", + "family": "Qwen3", + "label": "Qwen3 8B", + "parameter_count": "8B", + + // ── DERIVED. The script writes all of this. No human input, ever. ── + "artifacts": { + "llamacpp": { + "repo": "unsloth/Qwen3-8B-GGUF", + "file": "Qwen3-8B-Q4_K_M.gguf", + "quantization": "Q4_K_M", + "size_bytes": 5027784512, + "mmproj": null + } + }, + // Estimator inputs, read from the real GGUF header at authoring time. + "shape": { + "architecture": "qwen3", + "block_count": 36, + "head_count_kv": 8, + "key_length": 128, + "value_length": 128, + "context_length": 40960, + "n_vocab": 151936 + }, + "capabilities": [], // user-facing only. Today: "vision" or nothing. + + // ── JUDGEMENT. Two fields, typed by a person. ── + "rank": 83, // our preference order for document Q&A. + // Not a benchmark. Never displayed. + "validated": true, // someone downloaded this exact file and ran it + + "decode_fraction": 1.0 // 1.0 dense; the active slice for MoE (not derivable +} // from the header — set by hand, see below) +``` + +**The split is the point.** Everything above the line comes free from the GGUF +header and the Hugging Face listing; everything below it is three small decisions +a person makes in a minute. That is what keeps the manifest maintainable at +twenty entries instead of six — the tedious half scales automatically, the half +that needs a brain stays tiny. + +**`shape` is what makes the curated tier work offline.** Pricing a model means +reading its GGUF header, and a curated entry is not downloaded yet — so without +these fields an airgapped machine has no fit badge on the one tier it can use. +Committing what the header said at authoring time is what lets `weights + +KV(16K) + overhead` run against the manifest alone, on first paint, with no +network. It is the same reason Hermes carries estimator inputs inline. + +`decode_fraction` is the fraction of the build's bytes read per decoded token — +`1.0` dense, the active slice for MoE. Without it an MoE entry's speed +prediction is wrong by an order of magnitude, which matters the moment the +80B-A3B-class rung lands. + +`validated: true` means a person downloaded that exact file and ran it. False is +allowed and honest; it is not a gate. + +Dropped from v2: `minimum_fit` (a gate — only physics gates now), +`allowed_quantizations` (redundant once `artifacts.llamacpp` names the file), +`minimum_context` (the context ladder and the model's own `context_length` +replace it), and top-level `advisor_providers` (scoped an llmfit scan that no +longer runs). + +#### Authoring + +A script run by hand, **never CI**. The manifest is **source, not build output**. + +```bash +$ uv run scripts/refresh_curated_models.py \ + --add Qwen/Qwen3-Next-80B-A3B \ + --repo unsloth/Qwen3-Next-80B-A3B-GGUF --quant Q4_K_M + + resolving unsloth/Qwen3-Next-80B-A3B-GGUF … Q4_K_M found, 45.2 GB + reading GGUF header (4 MB range) … qwen3next, 48 layers, 2 kv-heads, ctx 262144 + llmfit … best_quant=Q4_K_M ✓ matches pin + llmfit proposes rank 94 (would be highest in the list — currently 92) + + ⚠ MoE detected — decode_fraction not derivable from the header. Set it manually. + ⚠ validated=false until you run this file. + + wrote curated-models.json (+1 entry) +``` -Search: `?filter=gguf&search=&sort=downloads&limit≤50`, 300 s TTL cache (HF -allows 500 requests per 5 minutes), `tree/main?recursive=true` for quants. Group -split parts (`-00001-of-00003`); exclude `mmproj*` and `*draft*` from the quant -list. +The person then downloads it, chats with it, confirms citations resolve, sets +`decode_fraction` and `validated: true`, and commits. Twenty minutes, most of it +waiting for the download. + +```text +every build + read the committed JSON. No llmfit, no network, no subprocess. +``` + +**What triggers a refresh** — events, not a schedule; realistically four or five +a year: + +| Trigger | Cadence | +|---|---| +| A model worth recommending ships (a new family, a good MoE, a vision entry) | the main one | +| A quantizer deletes or re-uploads a pinned file, so users hit a 404 | reactive | +| The pinned llama.cpp build is bumped and pins want re-verifying | on bump | +| A hardware class is under-served (see **The quantization ladder**) | occasional | +| Someone reports a bad recommendation | reactive | + +**The budget flags are mandatory.** Run bare, llmfit detects whatever laptop the +script is on and grades a quantization nobody pinned, so the numbers would +describe a different file than the one in `artifacts`. The `best_quant` +assertion catches that at authoring time, where a person is already looking. + +> **ponytail:** llmfit has no way to request a score at a named quantization — +> `plan --quant` returns memory requirements but carries no quality field. Until +> it does, the declared budget is chosen so llmfit lands on the pinned quant, and +> the assertion above is what keeps the two in step. + +Numbers are source rather than build output for three reasons beyond speed: a +rank moving 78 → 94 appears in a pull request where someone notices, a tag +rebuilds to the same manifest forever, and the cadence is honest — these change +when someone adds a model, not when someone cuts a release. + +#### Ranking + +`rank` orders the curated rows and selects the ★. It does **two** things and +nothing else. It is never displayed, never compared outside the app, +and never applied to a searched model. + +Because it is only ever a sort key, it does not need to be a measurement — it +needs to be an **order**. Hence `rank`, not `quality`: calling it quality would +imply we measured something we did not. + +**`rank` means "good at this app's job"** — answering from the user's documents +with citations that resolve — not general capability. That distinction does real +work. llmfit rates Qwen2.5-Coder 7B at **89** against Qwen3 8B's **83**, because +it is measuring coding ability; for document Q&A the coder is the weaker model +and should rank *below* it. A proposal that grades the wrong task is exactly what +the human review step exists to correct. + +**llmfit proposes; a person decides.** It carries real model-specific signal — +grouping ~4,000 scored models by *(parameter count, quantization)*, 224 of 230 +groups show quality varying between models of identical size and quant, so it is +not parameter count in disguise. But it is unreliable on models it does not know +well: + +``` +22.1B @ Q6_K + 100.0 mconcat/Qwen3.5-27B-Claude-4.6-Opus-Reasoning-Distilled-NV + 100.0 crushleorey/Qwopus3.6-27B-v2-NVFP4 + 88.0 upstage/solar-pro-preview-instruct +``` + +Community fine-tunes scoring a perfect 100, above published models. That number +is closer to the name than to measured behaviour. It behaves on models it knows +properly — the shipped six come back `38 · 53 · 68 · 83 · 90 · 92`, monotonic +and sanely spread — which is exactly the population the curated tier draws from. + +Three checks before committing a rank: + +1. within a family, rank rises with parameter count — the invariant in **Tests** +2. anything at or near 100 is suspect; nothing worth curating maxes a scale +3. crossing families, cross-check a public leaderboard — that is the comparison + llmfit is least reliable at +4. a specialist (a coder, a reasoning-tuned variant) is ranked for *this* task, + not for the task it was tuned on + +If llmfit has no entry, or proposes something absurd, **type the integer by +hand**. The field is an int; nothing requires llmfit to have produced it. A +number written by someone who ran the model beats a scraped 100. + +**Parameter count is not a sufficient substitute.** Within the shipped six it +happens to work, since they are one family of dense models. It breaks the moment +the list grows: MoE (80B total, ~3B active reads nothing like a dense 80B), +a second dense family where sizes stop being comparable, and any specialist whose +size says nothing about how it handles documents. + +**One list, not tracks.** SurfSense has one job — the Studio formats are summary, +flashcards, mindmap, quiz, podcast, office, web and visuals, and chat is document +Q&A. All twenty-three prompt cases are the same shape: read documents, write +prose or JSON. A single ranking is therefore meaningful, and splitting it would +import a general-catalog framing this app does not need. + +**Vision is a capability, not a rank position.** Nobody chooses a vision model +*instead of* a general one; they need one when the documents are images. It is +already in the manifest, derived from the header and `input_modalities` (and +gated on `typed_content` — see **7.7**): + +```jsonc +"capabilities": ["vision"] +``` + +So the ★ stays "the best-ranked model that fits", and a vision entry surfaces +when it is relevant (*your PDFs contain scanned pages, and this model can read +them*), never as a competitor in the ordering. `vision` is the only capability +that reaches the UI; everything else `chat_template_caps` reports is an internal +constraint. + +Tracks would only be warranted if SurfSense gained jobs wanting genuinely +different models — a conversational model for podcasts against an analytical one +for chat, say. Nothing today does. + +**If llmfit disappears tomorrow:** committed ranks are unaffected, the schema is +unaffected (`rank` is an int), the refresh script loses its proposer and a person +types the number, and the app never knew about it. That is the test a long-term +dependency should pass, and the reason to keep it outside the product. + +> **Long-term destination, not phase 7.** llmfit grades general capability. +> SurfSense needs something narrower: *does this model answer from the user's +> documents, and cite the right chunks?* No external leaderboard measures that. +> The honest end state is a small internal eval — 20-40 fixed questions over a +> fixed document set, run through the real chat path, scored on whether the +> answer came from the sources and whether `[n]` citations resolved — with `rank` +> set from that. It measures the actual pipeline, catches the RAG failure that +> matters most (answering confidently *without* the sources), reruns in minutes, +> and would also tell you whether the compact/capable/frontier tier split still +> earns its keep after 7.8 lands. Nothing measures that today. + +**Staleness is acceptable, by design.** Airgapped means no refresh path, so a +shipped list ages. That would be fatal if curated were the only way to get a +model. It is not: 204,797 models are one search away, badged and installable, +with no rank attached. So curated ages into "a starting point we tested a while +ago" rather than a boundary — which is what makes a few-times-a-year cadence a +choice instead of a liability. + +**Search:** `?filter=gguf&search=&sort=downloads&direction=-1&limit≤50`, 300 s +TTL cache (HF allows 500 requests per 5 minutes), `tree/main?recursive=true` for +quants. Group split parts (`-00001-of-00003`); exclude `mmproj*` and `*draft*` +from the quant list. Gates: architecture in the 152-name list · chat template present · repo not `gated` · disk space · physics. Nothing else. +**Gate on the header, not on Hugging Face's tags.** Counted on `library=gguf`: + +| `pipeline_tag` | GGUF repos | +|---|---| +| `text-generation` | 37,849 | +| `image-text-to-text` | 5,327 | +| `any-to-any` | 315 | +| `audio-text-to-text` | 27 | +| `automatic-speech-recognition` | 547 — not chat | +| `feature-extraction` | 681 — embeddings, exclude | + +That totals ~43,500 of 204,797, so **roughly 80% of GGUF repos carry no useful +pipeline tag at all** — filtering the search by tag would hide four-fifths of the +catalog, including working chat models whose uploader left the field blank. The +two header gates do the job structurally instead: a supported architecture +excludes Whisper-type models, and a present chat template excludes embedding +models. Pipeline tags are worth offering as an optional facet ("vision models +only"); they are not the gate. + +**Search ordering.** All five HF sort fields work on `filter=gguf`; only one is +usable as a default: + +| `sort=` | Top result | Verdict | +|---|---|---| +| `downloads` | `unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF` | **default** | +| `likes` | `unsloth/Qwen3.8-27B-GGUF` | surfaces uncensored derivatives in the top few | +| `trendingScore` | `prism-ml/Ternary-Bonsai-2-27B-gguf` | same | +| `lastModified` | `tapiocaTakeshi/Qubit` | surfaces half-finished uploads | +| `createdAt` | `mradermacher/HuatuoGPT-3-9B-i1-GGUF` | same | + +Every order surfaces abliterated and uncensored fine-tunes somewhere near the +top. That is what an open catalog means, and it is not a reason to reintroduce +grading — the curated tier is what serves someone who does not want to evaluate +models, and it is the only tier the recommendation reads. Copy must present the +count as popularity, not endorsement. + +**What a search row shows** — facts about the file, never a judgement of it: +fit verdict · size per quantization · architecture · context length · chat +template present · vision capability · license · gated · downloads, likes, +last updated · **provenance**. + +Provenance is free and worth surfacing: 32 of the top 40 GGUF repos carry a +`base_model:quantized:` tag, so a row can read *"quantized from +`Qwen/Qwen3.8-27B`"*. That is a statement about where the file came from, not a +grade — and it is most of what a quality score was doing for a user anyway, +letting them recognise an unfamiliar repo as a repackaging of a model they know. +The 8 of 40 with no such tag are informative by their absence. + +**Fit badges appear on both tiers. Quality appears on neither.** These look +alike and are not: + +| | What it is | Needs | Where | +|---|---|---|---| +| fit badge | subtraction — does this file fit in this memory | device memory + file size | **every row, both tiers** | +| `rank` | a preference order over models we tested | a person to have decided | curated only, and never displayed | + +"This 40 GB file will not fit in your 8 GB of memory" is no more a judgement than +"this file is 40 GB", and it is the single most useful thing to tell someone +browsing 204,797 models they have no size intuition for. Withholding it does not +protect them; it makes them download 40 GB to find out. + +Two stages, so the list stays cheap: + +```text +search list file size is already in the HF response + → coarse badge from size alone, no extra request + +open a model read 2-4 MB of its GGUF header (2.5 s) + → exact badge, conversation memory included + +curated shape is in the manifest + → exact badge offline, on first paint +``` + +#### Fit states + +Badge vocabulary changes meaning with the swap. It stops being llmfit's +five-level score (`perfect`/`good`/`marginal`/`too_tight`/`unknown`) and becomes +a three-state statement about **where the weights will live**. There is no +`unknown`: every row has a file size, so every row has a badge. + +Let `need = weights + KV(window) + overhead`, `vram` = usable device memory from +the budget probe, `ram` = host memory available to the budget. + +| State | Condition | What happens at load | +|---|---|---| +| `FITS` | `need ≤ vram` | every layer on the device, full speed | +| `PARTIAL` | `vram < need ≤ vram + ram` | `--fit` places some layers on the CPU; runs, slower | +| `TOO_BIG` | `need > vram + ram` at the **16K floor** | physics refusal — the only state that blocks install | + +`TOO_BIG` is evaluated at the floor, not at the requested window: a model that +will not fit at 16K cannot be rescued by a smaller context, and the remedy to +offer is a smaller quantization or a smaller model. + +**Wording is per platform. The state set is not.** One label set is wrong on two +of the three targets, so the copy branches on two facts the budget already +carries: `uma`, and whether any non-CPU device exists. + +A badge is a **verdict plus one plain line of why**. The verdict is what someone +choosing a model needs (fast, slower, or impossible); the mechanism is the +explanation, not the headline. + +**Discrete GPU** (Windows or Linux with an NVIDIA or AMD card) + +```text +● Full speed Runs entirely on your graphics card +◐ Reduced speed Too big for your graphics card, so part runs on the processor +○ Won't fit Needs about 21 GB. This PC has 13.6 GB +``` + +**Apple Silicon** (`uma`) + +```text +● Full speed Runs entirely on the GPU +◐ Reduced speed Too big for the GPU, so part runs on the CPU +○ Won't fit Needs about 21 GB. This Mac has 13.6 GB +``` + +**No GPU device** (`--list-devices` prints `(none)`) + +```text +● Works here Runs on your processor +○ Won't fit Needs about 21 GB. This PC has 16 GB +``` + +Terse variant, if rows are tight: verdict as the badge, reason as dimmed +trailing text, full explanation in the row's detail view. + +```text +● Full speed +◐ Reduced speed (part runs on the processor) +○ Won't fit (needs 21 GB, you have 13.6 GB) +``` + +**"Full speed", not "Fast".** Fast is a promise the badge cannot keep: a 32B +running entirely on a 4090 is still slower than a 4B. *Full speed* is relative +to the model, which is exactly what the state means — as fast as this machine +can run this particular model. Same reason *Reduced speed* beats *Slower*: it +says reduced from what it could be, not slow in absolute terms. + +**"Works here" on a machine with no GPU.** Technically that case is `FITS`, but +"Full speed" reads as a boast about a slow situation when there is no faster +alternative to contrast with. The useful information is simply: yes, you can run +this. That is the one place the verdict word differs and not just the +explanation. + +On Apple Silicon there is no separate system RAM to spill into. A model over +Metal's working set is not copied anywhere; llama.cpp runs some layers on the CPU +backend against the same physical memory, so "uses system RAM" would describe a +transfer that does not happen. With no GPU at all, `FITS` means the processor and +`PARTIAL` is unreachable because there is nothing to spill from. + +> **Copy rule for every user-facing string in this phase: no em dashes and no +> hyphens.** Use commas, full stops or parentheses. Applies to badges, empty +> states, error copy and the recommendation line. + +> **Optional refinement, not a design change.** `PARTIAL` spans a wide range: 5% +> spilled is barely noticeable, 70% crawls. The number is already known +> (`need - vram`), so the reason line can be graded without adding a state — +> *"A little too big for the GPU. Most of it still fits."* against *"Well over +> your GPU's memory. Expect it to be slow."* Same badge, sharper sentence. Ships +> after the first version if wanted. + +> **Decision: `PARTIAL` stays available on unified memory.** Hermes' +> `_uma_budget()` sets `ram_available_bytes = 0`, so on Apple Silicon their +> physics check is `need ≤ vram` alone and the middle state cannot occur — on an +> 8 GB M2 that refuses Qwen3 8B (7,043 MiB against ~5,461) outright. We do not +> copy that. It genuinely runs with CPU layers, and refusing it removes a real +> option from the users with the least choice. On UMA the refusal threshold is +> total RAM, not the device working set. + +**One modifier, not a fourth state.** A search row badged from file size alone, +and a row whose header read was truncated, carry the same three states at lower +precision — rendered as approximate (a `~`, or a lighter treatment) and resolving +to the firm badge once the header lands. + +**Eligibility is not fit.** These block or warn independently and must not be +rendered as fit states — a model can be `FITS` and still gated: + +| Condition | Effect | +|---|---| +| architecture not in llama.cpp's 152-name list | not installable; no amount of memory helps | +| no chat template in the header | installable, warns — will produce garbage in a chat UI | +| repo is `gated` | needs a Hugging Face account before the download resolves | + +#### List order + +Curated sorts by `(fit state, -rank, model_id)` — fit coarsely, rank finely — +which is what today's `_sort_key` already does. Sorting by rank alone would put a +`TOO_BIG` 32B at the top of an 8 GB machine's screen. + +The recommendation is the top row of the `FITS` bucket, subject to the speed +floor. Search sorts by downloads. +**Neither list displays a rank**; rank only orders curated rows and selects the +recommendation. + +**A searched model can be installed, selected, and used. It can never be +recommended**, because recommending requires ranking and ranking requires a +`rank` that only the manifest carries. The two tiers differ in kind, not degree. + Deleted: the collision-resolution block, `_placeholder_row()`, `_synthetic_curated_model()`, `_runtime_target_model()`, the disk scan cache, `CACHE_VERSION`, and the `scanned` flag through `AdvisorCatalog` → @@ -275,14 +724,14 @@ fitting = entries the estimator does not refuse resident = fitting, weights entirely in VRAM pleasant = resident, felt_cost(representative RAG turn) under floor -pleasant → max(quality, -size) reason: best-quality-resident - or speed-gated-quality if the floor - eliminated a higher-quality candidate +pleasant → max(rank, -size) reason: best-rank-resident + or speed-gated-rank if the floor + eliminated a higher-ranked candidate resident → max(speed) reason: fastest-resident else → no recommendation; spilled entries stay installable ``` -Quality is editorial and static per pinned file. Speed is physics per machine. +Rank is editorial and static per pinned file. Speed is physics per machine. Predictions order candidates and gate the floor; they are never displayed. > **RAG is prefill-dominated.** `HISTORY_BUDGET_TOKENS = 3000` plus ~24k @@ -394,8 +843,52 @@ an `ollama_pull` egress row. `Capabilities(inputs, tools, system_role, typed_content, context_tokens)` at the `Generator` seam, from `GET /models.architecture.input_modalities` and `GET /props.chat_template_caps` — template-derived truth, not a name regex. -`supports_system_role` matters immediately: `build_messages()` always emits a -system message and silently degrades on templates without it. + +**Two kinds, two sources, and only one of them reaches the UI.** + +```text +GET /models → architecture.input_modalities what the model can ACCEPT +GET /props → chat_template_caps what the TEMPLATE supports +``` + +| | Field | Surfaced? | +|---|---|---| +| **Capability** | `vision` | **yes** — a row badge and a search facet | +| **Constraint** | `supports_system_role` | no — changes how the request is built | +| **Constraint** | `supports_typed_content` | no — precondition for vision | +| Recorded, unused | `tools`, `tool_calls` | no | +| Recorded, unused | reasoning support | no | + +One struct is fine; one *list that reaches the renderer* is not. +`supports_system_role` is meaningless to a person and `typed_content` is an +implementation detail. The manifest's `capabilities` array carries only the +user-facing set, which today means `["vision"]` and nothing else. + +**`supports_system_role` matters immediately.** `build_messages()` always emits a +system message and silently degrades on templates that do not support one. That +is a live bug the swap fixes. + +**Vision needs both halves.** "The model accepts images" is the obvious check and +it is not sufficient: + +```python +can_see = Modality.IMAGE in caps.inputs and caps.typed_content +``` + +A model can accept images architecturally while its chat template takes only +string content, leaving no way to hand it one. Checking `input_modalities` alone +produces a vision badge on a model that cannot be sent a picture. + +**What is deliberately absent.** `audio` and `video` are not modelled, because +nothing can feed them: Docling's `allowed_formats` in `worker/ingestion/parsing.py` +is PDF, DOCX, PPTX, XLSX, HTML, CSV, MD and **IMAGE**. Podcasts are audio *out* +and Kokoro owns that; audio *in* would need ingestion to accept and transcribe +media first, at which point `audio` becomes relevant and not before. `tts` and +`embedding` are fixed (Kokoro, and bge-small against a 384-dimensional index). + +`IMAGE` being in that list already is what makes vision worth doing at all: +SurfSense accepts standalone images today, on top of PDF pages that are +effectively pictures. A vision model has something to look at on day one. `Message.content` widens from `str` to `str | tuple[ContentPart, ...]`. The adapter **downgrades at the seam**: a text-only model gets the image replaced by @@ -443,22 +936,57 @@ compliance. That plausibly collapses three tiers to two. search reports the destination is unavailable, not an error. - Header read truncated: retry wider once, then fall back to a file-size estimate with the badge marked approximate. -- Physics refusal: state required and available bytes and name a smaller model. +- `TOO_BIG` (the physics refusal): state required and available bytes and name a + smaller quantization or a smaller model. Never a bare disabled control. - Cancelled download: `POST /models/unload`, partial file removed, never selected. +- **Dead curated pin** (the repo or file 404s at download): say the model is no + longer available from this source, drop it from the recommendation, and fall + through to the next-best entry. This must be handled here rather than checked + at build time — a repo can disappear between the release and a user's first + install, so no pre-flight check can prevent it. ## Tests - GGUF reader: truncated file, oversized vocab, split parts, Range retry. - Estimator: decision-table over constructed profiles; f16 vs q8_0 KV (a 1 GB swing on an 8 GB machine, measured). +- Fit states: each of `FITS` / `PARTIAL` / `TOO_BIG` at its boundary, ±1 byte; + `TOO_BIG` evaluated at the 16K floor, not the requested window; `PARTIAL` + reachable on a `uma` budget (the Hermes divergence — a regression here silently + refuses Qwen3 8B on an 8 GB Mac); `PARTIAL` unreachable with no GPU device; + `can_install` true for `PARTIAL` and false only for `TOO_BIG`; copy resolves to + the right platform wording for all three budget shapes. - Recommendation policy: the six hardware profiles above, asserting both pick - and reason key. + and reason key; and that a searched row is never eligible to be the pick. - Catalog: curated renders with no network; search gated on egress; arch, - template, and gated-repo rejections. + template, and gated-repo rejections; a search row carries no `rank` field at + all — asserted on the serialized response, so it cannot be reintroduced + silently. +- Manifest (extends `test_curated_models.py`, which already covers unique ids, + display metadata, and each rejection path): every entry has a `shape`, a + `rank` and a pinned `quantization`; every `shape.architecture` is + in the 152-name list; a curated row prices and badges with no network at all. +- **`rank` rises with parameter count within a family.** Today this passes + trivially — the shipped six are one family and score `38 · 53 · 68 · 83 · 90 · + 92`. The first failure is the signal to look, and it is expected: a + mixture-of-experts, a second dense family, or a specialist ranked for document + Q&A rather than for what it was tuned on will all break size-ordering + legitimately. The assertion exists to make that a decision, not a drift. +- A curated entry with a `vision` capability is never preferred over a + higher-ranked text model for the ★; capabilities do not enter the ordering. +- Capabilities: `vision` requires **both** `IMAGE in inputs` and + `typed_content`; a model with image input and a string-only template reports + no vision badge. `supports_system_role` false routes the system message into + the first user turn rather than dropping it. Only the user-facing set reaches + the serialized response — `system_role` and `typed_content` never appear in + an API payload. - Runtime: fake llama-server for health/models/props/load/chat/delete. - Migration: an Ollama selection and an `ollama_pull` row, upgraded. - Packaging: staged checksum; `--list-devices` from the packaged path per OS. +No new CI job. The build stays install → freeze → package → sign; neither llmfit +nor `huggingface.co` appears in it. + ## Acceptance - A clean machine on all three OSes opens the catalog with a hardware line and From e5f6ec1f8129a03ed3fdc6f9610a7baa5addd459 Mon Sep 17 00:00:00 2001 From: Anish Sarkar <104695310+AnishSarkar22@users.noreply.github.com> Date: Sat, 19 Sep 2026 15:30:22 +0530 Subject: [PATCH 03/60] feat(spec): drop llmfit and Ollama from the llama.cpp plan Phase 7 now replaces the Ollama sidecar with a packaged llama-server and stops shipping llmfit entirely. Hardware comes from ggml_backend_dev_memory() via ctypes; scoring moves offline to a maintainer machine and ships as data in curated-models.json. - 07-llamacpp-runtime: add Contracts, authoring and appendix sections; pin one quantization per entry, fix context at load, defer manifest growth and speed gating out of v1 - 05c-packaging: replace the llmfit section with llama-server staging, split the catalog and Hugging Face search routes, make .gguf import a first-class offline path - 05b-connections: rename provider ollama -> llamacpp, note revision 0012 rebuilds selected_models for the CHECK constraint - 00c-data-model: rename egress destinations to model_download and model_search - 00b-diagrams, frontend/01-shell, frontend/05-install-ux: follow the same rename and drop the scan step from the install flow --- plans/community-local/00b-diagrams.md | 41 +- plans/community-local/00c-data-model.md | 6 +- .../api/05b-openai-compatible-connections.md | 14 +- plans/community-local/api/05c-packaging.md | 125 ++++-- .../api/07-llamacpp-runtime.md | 399 ++++++++++++++++-- plans/community-local/frontend/01-shell.md | 6 +- .../community-local/frontend/05-install-ux.md | 157 ++++--- 7 files changed, 567 insertions(+), 181 deletions(-) diff --git a/plans/community-local/00b-diagrams.md b/plans/community-local/00b-diagrams.md index 7c9bd8d1b4..94c980b194 100644 --- a/plans/community-local/00b-diagrams.md +++ b/plans/community-local/00b-diagrams.md @@ -2,6 +2,10 @@ > Workstreams: [`frontend/`](frontend/), [`api/`](api/), [`worker/`](worker/). +> Updated for [`api/07-llamacpp-runtime.md`](api/07-llamacpp-runtime.md): +> **Phase 7** replaces the Ollama sidecar with `llama-server` and removes the +> bundled llmfit binary. + ## Process layout at runtime ```text @@ -10,7 +14,7 @@ │ • BrowserWindow → frontend/dist (workspace UI) │ │ • spawn surfsense-api │ │ • spawn surfsense-worker │ -│ • packaged Ollama sidecar; bundled llmfit executable │ +│ • packaged llama-server sidecar (router mode) │ │ • on quit: SIGTERM workers, wait, exit │ └─────────────────────────────────────────────────────────────┘ │ │ @@ -20,7 +24,8 @@ │ FastAPI :8xxx │ │ huey_consumer -w1│ │ LLM catalog │ │ Docling + embed │ │ runtime adapters│ │ │ -│ llmfit JSON CLI │ │ │ +│ GGUF header + │ │ │ +│ ggml budget │ │ │ │ SQLite R/W │ │ │ └────────┬────────┘ └────────┬─────────┘ │ │ @@ -42,20 +47,18 @@ ▼ ▼ ▼ ┌───────────┐ ┌─────────────┐ ┌───────────┐ │ A Airgap │ │ B Slim + │ │ C BYO │ - │ full pack │ │ on-demand │ │ Ollama/ │ + │ full pack │ │ on-demand │ │ endpoint │ └───────────┘ └─────────────┘ └───────────┘ ▼ - llmfit hardware scan + ggml device query (~180 ms, no scan) ▼ - Curated + Explore - (Curated and Installed render with no - scan; only Explore needs one) + Curated (offline) + HF search (network) + every row badged FITS / PARTIAL / TOO_BIG ▼ one-click Download & Use ▼ runtime install plan - ├─ Ollama tag (v1) - └─ verified GGUF (later) + └─ pinned GGUF (repo + file + quant) ▼ Create workspace → upload document ``` @@ -64,15 +67,15 @@ ```text LOCAL REMOTE -llmfit catalogue + fit provider_connections - │ │ - ▼ ▼ -SurfSense adapter + policy live GET /models per endpoint +curated manifest + HF search provider_connections │ │ ▼ ▼ -runtime artifact resolver SelectedModel(connection_id, name) - ├── Ollama ── pull tag ── chat ├── generation - └── llama.cpp (future) ── chat │ └── core vLLM/gateway +fit estimator (ggml budget + live GET /models per endpoint + GGUF header), rank from manifest │ + │ ▼ + ▼ SelectedModel(connection_id, name) +GgufArtifact resolver ├── generation + └── llama.cpp ── fetch file ── chat │ └── core vLLM/gateway │ └── /chat/completions └── image_generation ├── vLLM-Omni/gateway @@ -81,8 +84,10 @@ runtime artifact resolver SelectedModel(connection_id, name) └── /images on 404/405 ``` -llmfit never handles a SurfSense chat request. It can disappear and an already -installed model still answers through its runtime adapter. Remote model lists +llmfit is not in the app at all: it runs on a maintainer's machine a few times a +year and its numbers ship as data in the manifest. The fit badge comes from the +runtime's own allocator plus the model's GGUF header, so it works offline and +covers every GGUF, not a curated subset. Remote model lists are fetched live and never copied into SQLite. Two endpoints with the same model id remain distinct because selection identity includes the connection id. diff --git a/plans/community-local/00c-data-model.md b/plans/community-local/00c-data-model.md index 193928167e..e6a3298294 100644 --- a/plans/community-local/00c-data-model.md +++ b/plans/community-local/00c-data-model.md @@ -265,7 +265,7 @@ and the Fernet key comes from the OS keychain through Electron. `selected_models` stores one active model per `role`: `generation` or `image_generation`. `provider` chooses the runtime adapter, `name` is the exact -model id, and nullable `connection_id` identifies the remote endpoint. Ollama +model id, and nullable `connection_id` identifies the remote endpoint. Local **and `sdcpp`**, the bundled local image runtime admitted by revision `0009`, use no connection; an OpenAI-compatible selection requires one. The FK uses `ON DELETE CASCADE`, so disconnecting an endpoint clears only roles that use it. @@ -275,7 +275,7 @@ which is computed on read rather than stored, so retuning a threshold needs no migration ([`api/05a-model-recommendations.md`](api/05a-model-recommendations.md)). The offerable local catalog, remote `/models` responses, hardware profile, -llmfit scores, install plans, capabilities, and curated models are **not** +Fit estimates, install plans, capabilities, and curated models are **not** stored. Local recommendations are recomputed from the packaged inputs; remote inventory is fetched live. Persisting either would create synchronization work without improving inference. See @@ -291,7 +291,7 @@ without improving inference. See | `selected_models` | chosen model per role (above) | | `onboarding_completion` | the durable "model onboarding is done" marker (revision `0003`); selecting or clearing a model never writes it | | `license_state` | singleton (`CHECK id = 1`): the imported certificate, when it was imported, and `clock_watermark`, the highest instant ever seen. Plan and expiry are re-derived from the certificate on every read rather than stored (revision `0006`) | -| `egress_destinations` | one row per destination — `ollama_pull`, `image_model_pull`, or `host:` for a BYO provider — with `enabled` defaulting to **false** and `last_call_at` (revision `0008`) | +| `egress_destinations` | one row per destination — `model_download` and `model_search` (both `huggingface.co`, renamed from `ollama_pull` by revision `0012`, see [`api/07-llamacpp-runtime.md`](api/07-llamacpp-runtime.md)), `image_model_pull`, or `host:` for a BYO provider — with `enabled` defaulting to **false** and `last_call_at` (revision `0008`) | `workspaces` also gained a nullable `cloud_id` in revision `0005`: cloud-to-local import looks a workspace up by it and reuses the existing row rather than diff --git a/plans/community-local/api/05b-openai-compatible-connections.md b/plans/community-local/api/05b-openai-compatible-connections.md index 6500a3997a..34ee647077 100644 --- a/plans/community-local/api/05b-openai-compatible-connections.md +++ b/plans/community-local/api/05b-openai-compatible-connections.md @@ -37,7 +37,7 @@ The durable state stays minimal: ## Terms and boundaries ```text -provider protocol implementation, e.g. ollama or openai_compatible +provider protocol implementation, e.g. llamacpp or openai_compatible provider connection one named remote endpoint and its optional bearer secret model an id returned live by that endpoint, or entered manually role the one active model for generation or image_generation @@ -103,7 +103,7 @@ Keep one table and one row per role: | Column | Contract | |---|---| | `role` | primary key: `generation` or `image_generation` | -| `provider` | `ollama`, `openai_compatible`, or **`sdcpp`** — the bundled local image runtime, added in revision `0009` | +| `provider` | `llamacpp`, `openai_compatible`, or **`sdcpp`** — the bundled local image runtime, added in revision `0009`. Was `ollama` until revision `0012` ([`07-llamacpp-runtime.md`](07-llamacpp-runtime.md)) | | `connection_id` | nullable FK to `provider_connections.id`, `ON DELETE CASCADE` | | `name` | exact provider model id | | `params_b`, `vendor`, `line` | nullable model fingerprint, added in revision `0010`; feeds the prompt tier ([`05a`](05a-model-recommendations.md)) | @@ -111,9 +111,9 @@ Keep one table and one row per role: Invariants: -- `ollama` **and `sdcpp`** selections have `connection_id = NULL`; revision +- `llamacpp` **and `sdcpp`** selections have `connection_id = NULL`; revision `0009`'s check constraint reads - `(provider IN ('ollama','sdcpp') AND connection_id IS NULL) OR (provider = 'openai_compatible' AND connection_id IS NOT NULL)`; + `(provider IN ('llamacpp','sdcpp') AND connection_id IS NULL) OR (provider = 'openai_compatible' AND connection_id IS NOT NULL)`; **`text_enum` renders this as a CHECK constraint, which SQLite cannot alter in place — revision `0012` rebuilds the table**; - `sdcpp` is valid for `image_generation` only, never for chat; - `openai_compatible` selections require a connection id; - deleting a connection removes every role selection that uses it; @@ -179,7 +179,7 @@ Centralize selection resolution: ```text resolve generation selection - ├── ollama + no connection_id → supervised Ollama provider + ├── llamacpp + no connection_id → supervised llama-server provider └── openai_compatible + connection_id → load connection → chat provider resolve image_generation selection @@ -414,7 +414,7 @@ Hand-written Alembic revision `0004`: 1. create `provider_connections`; 2. rebuild `selected_models` with `connection_id`, the new role value, and the foreign key; -3. preserve valid Ollama generation selections; +3. preserve valid local generation selections; 4. remove stale rows whose provider is `openrouter`; 5. drop `provider_credentials`, which also removes residual OpenRouter keys; 6. keep `onboarding_completion`. @@ -448,7 +448,7 @@ Delete: ## Tests -- Migration: OpenRouter credential/selection removed, Ollama selection and +- Migration: OpenRouter credential/selection removed, local selection and onboarding completion preserved, foreign-key cascade works. - Connections: multiple rows of the same provider, duplicate-label rejection, optional key, key redaction, candidate-before-replace update, URL validation, diff --git a/plans/community-local/api/05c-packaging.md b/plans/community-local/api/05c-packaging.md index d950879fe4..f741636f5d 100644 --- a/plans/community-local/api/05c-packaging.md +++ b/plans/community-local/api/05c-packaging.md @@ -1,5 +1,11 @@ # API — Phase 5: Packaging +> **Phase 7 supersedes the generation-runtime half.** +> [`07-llamacpp-runtime.md`](07-llamacpp-runtime.md) replaces the bundled Ollama +> with `llama-server` and **stops shipping llmfit entirely**. The llmfit and +> first-run-download sections below are rewritten to match; the PyInstaller, +> sqlite-vec, Alembic and model-pack sections are unaffected and still govern. + ## Goal Installers for Paths A / B / C. @@ -13,9 +19,10 @@ Installers for Paths A / B / C. - CI: win/mac/linux; smoke `/health`. - Models outside exe — shipped as `extraResources`, `SURFSENSE_LOCAL_MODELS_DIR` pointed at them; `scripts/fetch_embedding_model.py` fetches the same files for CI. -- A pinned official llmfit executable plus the versioned curated-model - manifest; normalized recommendation/install routes are specified in - [`05a-model-recommendations.md`](05a-model-recommendations.md). +- A pinned `llama-server` build plus the versioned curated-model manifest; + catalog and install routes are specified in + [`07-llamacpp-runtime.md`](07-llamacpp-runtime.md) (which supersedes the + route table in [`05a-model-recommendations.md`](05a-model-recommendations.md)). ## Files no import statement names @@ -85,28 +92,51 @@ build would have added 3GB to every installer. The chat model is the one heavy file (1.4GB+), and bundling it would double the installer for something the app does not need to *start*. So it is not shipped. The installer carries only what runs offline — code, Docling/OCR weights, the -bge embedder, the Ollama binary, the pinned llmfit binary, and -`curated-models.json` — and ingestion and search work the moment the app opens. - -### llmfit executable - -- Pin one llmfit release and its SHA-256 checksums in the staging script; never - download "latest" during an app build. -- Stage the official archive for each supported OS/architecture under - `resources/llmfit/`, retaining upstream license and notices. -- The API receives an absolute `SURFSENSE_LOCAL_LLMFIT_PATH` from Electron. - Development may resolve `llmfit` from `PATH`; packaged builds must not. -- Invoke the binary without a shell and with a bounded timeout. Its model - catalog is read-only inside application resources. -- Code signing, notarization, and antivirus smoke tests include both llmfit and - Ollama. A missing or quarantined llmfit binary degrades recommendations but - must not prevent the API from starting. -- CI verifies the staged checksum and runs `llmfit --json system` from the final - packaged resource path on a clean supported target. - -llmfit's license and model-license metadata are not permission to redistribute -model weights. SurfSense ships no generation weights and the runtime downloads -the artifact selected by the server-side install resolver. +bge embedder, the pinned `llama-server` build, and `curated-models.json` — and +ingestion and search work the moment the app opens. **llmfit is not shipped** +(see below), which removes 27 MB from every target. + +### llmfit is not packaged + +**Superseded by phase 7.** llmfit ran on the user's machine to detect hardware +and score models. Both jobs moved: + +- **Hardware** now comes from `ggml_backend_dev_memory()` in the shipped ggml + libraries, via `ctypes` — the allocator's own view, ~180 ms, no subprocess. +- **Scoring** happens on a maintainer's machine, a few times a year, and the + numbers are committed into `curated-models.json` as source. + +So there is no `resources/llmfit/`, no `SURFSENSE_LOCAL_LLMFIT_PATH`, no +subprocess to time out, no checksum to stage, and nothing for code signing or +antivirus smoke tests to cover. `llmfit --json system` is not run from a packaged +path because the binary is not in the package. + +The rule that replaced it: **anything that returns the same answer on every +machine is computed once and shipped as data; anything that differs per machine +is measured on that machine by the runtime that will do the work.** + +### llama-server executable + +- Pin one llama.cpp build number **and its SHA-256** in `fetch-llamacpp.mjs`; + never fetch a floating tag. llama.cpp publishes ~10 builds a day with no + stable channel, so the pin matters more here than it did for Ollama — + `fetch-ollama.mjs` verified no checksum at all. +- Stage one archive per target under `resources/llamacpp/`, retaining upstream + licence and notices. Prune to `llama-server` plus the libraries it links: the + macOS tarball ships 24 executables and only one is needed, which also shrinks + the notarization surface. +- Sizes and per-OS details, including CUDA inside the Windows installer, are in + [`07-llamacpp-runtime.md`](07-llamacpp-runtime.md) under **Packaging, three + targets**. +- Code signing, notarization and antivirus smoke tests cover `llama-server` and + its libraries. Note it spawns **child processes** in router mode, so the + hardened runtime needs testing against a grandchild, not just the sidecar. +- CI runs `llama-server --list-devices` from the final packaged resource path on + each clean target. + +Model licences are not permission to redistribute weights. SurfSense ships no +generation weights; the app downloads the file its manifest pins, or one the +user finds by search. ### First-run model download @@ -114,28 +144,33 @@ The generation model arrives in first-run setup with progress through the normalized catalog boundary ([`05a-model-recommendations.md`](05a-model-recommendations.md)): -- `GET /llm/system` — normalized local hardware and scan status. -- `GET /llm/catalog` — Curated, Explore, and installed entries with opaque - catalog ids. Curated and installed need no hardware scan; only Explore and the - fit badges do. -- `POST /llm/install` — resolves the trusted runtime artifact server-side, - streams download progress, and optionally selects it. -- `GET/PUT /llm/selection/generation` — reads or validates the selected - provider/runtime model. - -A fully offline machine keeps chat through an "import a local model" option on -the same screen (a file, or a model already in Ollama), so airgapped is an opt-in -path rather than a cost every installer pays. Import validates and moves the -artifact atomically through the chosen runtime adapter; the renderer never -provides an arbitrary download URL. +- `GET /llm/system` — the hardware budget, from the runtime's allocator. No + scan, no status flag, no button. +- `GET /llm/catalog` — curated plus installed, every row badged. **No network.** +- `GET /llm/search` — the Hugging Face tier, paged and egress-gated. Separate + route so the page never blocks on it. +- `POST /llm/install` — fetches the pinned file server-side, streams progress, + and optionally selects it. +- `GET/PUT /llm/selection/generation` — reads or validates the selection. + +A fully offline machine keeps chat through **Add a .gguf file** on the same +screen. With search unavailable, that and the packaged manifest are the only two +routes to a model, so it is a first-class control rather than an advanced +option. Import validates the GGUF header, rejects an unsupported architecture, +and links the file into the models directory; the renderer never provides an +arbitrary download URL. ## Acceptance - Clean VM Path B: install → wizard → upload → chat. -- Hardware-ranked catalog works without a separately installed llmfit. -- Tampering with the staged llmfit archive fails checksum verification. -- Removing llmfit after install leaves ingestion and an already selected model - operational, with recommendation UI degraded explicitly. -- Download & Use resolves an Ollama artifact, completes, and selects the exact +- The catalog renders with a hardware line and badged rows on first paint, with + no network call and no binary beyond `llama-server`. +- Tampering with the staged llama.cpp archive fails checksum verification. +- `llama-server --list-devices` succeeds from the packaged resource path on each + target. +- Download & Use fetches the pinned GGUF, completes, and selects the exact installed model; no generation weights are present before that action. -- Quit → no orphan `surfsense-*` processes. +- With networking disabled: curated rows badge correctly and a `.gguf` imported + from disk reaches chat. +- Quit → no orphan `surfsense-*` or `llama-server` processes, including the + model workers the router spawns. diff --git a/plans/community-local/api/07-llamacpp-runtime.md b/plans/community-local/api/07-llamacpp-runtime.md index 975ba1c595..72ee954224 100644 --- a/plans/community-local/api/07-llamacpp-runtime.md +++ b/plans/community-local/api/07-llamacpp-runtime.md @@ -43,12 +43,16 @@ Ollama's native API does not. | **`rank`** | **Curated entries only. A searched model never carries one, from llmfit or anywhere else** | A preference order over models we tested, **for this app's job** — answering from the user's documents with citations that resolve — not general capability. An integer typed by a person, with llmfit proposing. Called `rank`, not `quality`, because it is only ever a sort key: nothing reads its magnitude, so it must not imply a measurement we do not have. Attaching one to an arbitrary repo attaches a base model's score to a derivative that behaves differently: of the top 100 GGUF repos by downloads, 38 match an llmfit entry, and those matches include `Huihui-Qwen3.8-27B-abliterated-GGUF` and `Qwen3.8-27B-Uncensored-GGUF` resolving to the base model's score. A wrong number people trust is worse than a blank they investigate. Search rows are **described, not judged**. | | **Search order** | `sort=downloads`, descending | The only HF sort that behaves as a default. Presented as a popularity fact, never as an endorsement. See **Search ordering**. | | **Fit badge** | **Every row, both tiers** | Fit is subtraction, not judgement — the opposite of `rank`. Coarse from file size in the search list, exact from the header on open, exact from `shape` offline for curated. Three states, no `unknown`. | +| **Recommendation** | Highest `rank` among `FITS` entries. **No speed gate in v1** | A latency gate needs a prefill model, and llmfit's `prefill_tps` is an estimate for a generic machine. Rather than invent a threshold, v1 starves the gate: the ★ is the best-ranked entry that fits entirely on the device, tie-broken toward the smaller build. Additive once `llama-bench pp512` numbers exist per device class. | | **Curated manifest** | **Stays, schema 3, authored by hand** | It is the whole product airgapped, the first-run default, and the only tier the recommendation reads. `scripts/refresh_curated_models.py` writes it; a person commits it; **llmfit never runs in CI**. | +| **Why `rank` can be authored at all** | Quality is a function of *(model, quantization)*, not of hardware | Measured twice: across 2,424 models at a fixed quant, quality differs 0/2424; and on one model swept 12G to 48G, quality does not move once `best_quant` stops moving. Hardware reaches quality through exactly one channel, the quantization, and pinning the file closes it. So the number is a property of a file, computable once and shipped as data, and a scan on the user's machine has nothing to discover. This is the load-bearing fact under the whole authoring design. See **Appendix**. | +| **Which llmfit field** | `score_components.quality` only. **`score` is discarded** | `score` blends fit, speed and context into quality, so three of its four terms describe a machine. In the same sweep it runs 65.1, 63.8, 63.6, 66.2, 63.6, 67.4 — non-monotonic and hardware-dependent. The composite is a statement about a laptop; the component is a statement about a file. Same reason `fit_level`, `memory_required_gb`, `estimated_tps` and `best_quant` are all read and thrown away. | +| **Scoring quantization** | The pinned one, found by a ladder sweep. A miss is a hard failure | llmfit cannot be asked for a score at a named quantization, and quality tracks `best_quant` exactly, so the budget decides which file gets graded. Qwen3 8B is **78** at the pinned Q4_K_M and **83** at Q8_0. See **Scoring at the pinned quantization**. | | **Manifest validation** | Existing unit tests. **No new CI** | Pydantic (`extra="forbid"`, `model_validator`) plus `test_curated_models.py` already reject malformed manifests. A dead pin is a runtime failure, gracefully handled — CI cannot prevent a repo disappearing after the build anyway. `validated: true` is the real check, and it is a human one. | | **Hardware budget** | `ggml_backend_dev_memory()` via `ctypes`, from the shipped libs | The allocator's own view. Falls back to `llama-server --list-devices`, then OS APIs. | | **Downloads** | **SurfSense fetches the GGUF**, not `POST /models` | `llama-server` is a second process we do not proxy; an in-process fetch is the only place `egress.require()` actually holds. Also buys resume, checksums, and the header as the file lands. | -| **Quantization** | Pinned per entry; two variants on the two largest | A pinned file is what "tested by SurfSense" can honestly claim. See **The quantization ladder**. | -| **Context** | Set at load, grown on occupancy. Floor 16K | llama.cpp fixes context at load; `num_ctx()`'s per-request sizing has no equivalent. 3K history + ~8K grounding + reply. | +| **Quantization** | **One pinned file per entry in v1** | A pinned file is what "tested by SurfSense" can honestly claim. Per-machine selection is possible at no extra cost (one header read prices every quant), so a second variant on the largest entries is a cheap follow-on, not v1. See **The quantization ladder**. | +| **Context** | **Fixed at load. Floor 16K, capped at the model's own `context_length`** | llama.cpp fixes context at load, so `num_ctx()`'s per-request sizing has no equivalent. The floor is 3K history plus ~8K grounding plus a reply. Growing on occupancy is additive later; it needs a mid-conversation reload and two constants with nothing measured behind them. | | **Windows CUDA** | **In the installer.** No post-install download | Airgapped. Dropping Ollama frees ~500 MB; CUDA 13.4 + cudart costs 573 MB. Net ≈ +100 MB, under the 2 GB `makensis` ceiling that forced `pruneCudaRunners()`. | | **CUDA version** | 13.4 only | CUDA 13 requires Turing (7.5)+. Shipping 12.4 as well for Pascal costs another 645 MB — that is what made Ollama's archive 1.8 GB. Pascal falls through to Vulkan automatically via backend scoring. | | **Backend selection** | Runtime, by ggml | Releases are built `GGML_BACKEND_DL=ON`. A failed `dlopen` is skipped silently, so a missing Vulkan loader degrades to CPU rather than failing. | @@ -137,11 +141,15 @@ Every machine gets a GPU-resident pick, so six rungs strand nobody. Three gaps: 1,024 MiB overhead; at Hermes' measured 1.5 GiB it does not fit and that machine drops a rung. -Manifest changes, in priority order: +**Manifest growth is follow-on work, not phase 7.** v1 ships these six re-pinned +to GGUF; the runtime swap is the deliverable and the search tier covers the gaps +meanwhile. In priority order, once it lands: 1. **A rung above 32B** — an 80B-A3B-class MoE. The only thing that serves - 48–64 GB machines. MoE needs a `decode_fraction` field (active bytes read per - token) or the speed prediction is wrong by an order of magnitude. + 48–64 GB machines. An MoE is the right shape for them: 80B of memory but ~3B + read per token, so it is the only way to spend that memory without the machine + crawling. Needs the `decode_fraction` field, which is computed from the header + (see **Authoring**) — without it the speed prediction is wrong by roughly 10×. 2. **A second variant on 8B and 32B** (`Q6_K`), filling the 16 GB gap and giving the 4090 somewhere to go. Two extra pins, not fifty. 3. **A vision entry.** All six are text-only, in an app with a PDF pipeline. @@ -184,6 +192,193 @@ routes `` blocks to `message.reasoning_content`; without it a thinking model's trace enters `parts[]` and `resolve_citations()` rewrites `[n]` tokens that appeared inside the reasoning. +## Contracts + +> **Provenance.** The shapes and runtime calls below are grounded: the +> llama-server README at `b11043`, `nm` on the shipped libraries plus a live +> `ctypes` call, real GGUF headers read over HTTP Range, and live Hugging Face +> responses. **The HTTP routes are proposed, not reported** — no such surface +> exists yet. They follow this codebase's conventions: the `/llm` prefix, `*Read` +> Pydantic response models in `modules/llm/schemas.py`, and the NDJSON +> `{"type": …}` stream frame that `_event()` already emits. + +### Domain shapes + +```python +@dataclass(frozen=True) +class GgufArtifact: + """One downloadable build. `file` is required: a repo holds ~20 quants, and + repo + quantization does not name one (Q4_K_M vs UD-Q4_K_M vs a split set).""" + repo: str # "unsloth/Qwen3-8B-GGUF" + file: str # "Qwen3-8B-Q4_K_M.gguf" + quantization: str # "Q4_K_M" + size_bytes: int + mmproj: str | None = None # vision projector, fetched alongside + + +@dataclass(frozen=True) +class ModelShape: + """Estimator inputs. Committed to the manifest for curated entries, read from + the header for everything else. Names mirror the GGUF metadata keys.""" + architecture: str # general.architecture + block_count: int # .block_count + head_count_kv: int # .attention.head_count_kv + key_length: int # .attention.key_length + value_length: int # .attention.value_length + context_length: int # .context_length + n_vocab: int + sliding_window: int = 0 # .attention.sliding_window + expert_count: int = 0 # .expert_count; > 0 means MoE + + +@dataclass(frozen=True) +class HardwareBudget: + usable_vram_bytes: int # capacity (total − margin) or live (free now) + total_device_bytes: int + ram_available_bytes: int + uma: bool # unified memory: selects badge copy and headroom + has_gpu: bool # False when --list-devices prints "(none)" + + +class FitState(StrEnum): + FITS = "fits" + PARTIAL = "partial" + TOO_BIG = "too_big" + + +@dataclass(frozen=True) +class FitVerdict: + state: FitState + need_bytes: int # weights + KV(window) + overhead + budget_bytes: int # vram, or vram + ram for the TOO_BIG comparison + approximate: bool = False # priced from file size alone, header not read yet + + +@dataclass(frozen=True) +class ContentPart: + kind: Literal["text", "image"] + text: str | None = None + path: str | None = None # local file path; image_url.url accepts one +``` + +`Capabilities` is defined in **7.7**. `Message.content` widens to +`str | tuple[ContentPart, ...]` there and nowhere else. + +### HTTP routes — proposed + +**Two catalog endpoints, deliberately.** Curated plus installed renders offline +and instantly; search needs `huggingface.co` and is paged. One response covering +both would either block on the network or return partial results behind a flag, +which is the `scanned` flag this phase deletes. + +| Route | Returns | Network | +|---|---|---| +| `GET /llm/system` | `HardwareBudget` plus device list | none | +| `GET /llm/catalog` | `{curated: [...], installed: [...]}`, every row badged | **none** | +| `GET /llm/search?q=&limit=&cursor=` | HF hits, coarse badge from file size | huggingface.co | +| `GET /llm/search/{repo:path}` | quant list with exact sizes, exact badge after the header read | huggingface.co | +| `POST /llm/install` | NDJSON progress, existing `_event()` frame shape | huggingface.co | +| `DELETE /llm/models/{model_id:path}` | `ModelDeleteRead`, `selection_cleared` | none | + +`GET /llm/search/{repo}` is where the 2–4 MB header read happens, so the trigger +is **opening a search result**, not hovering or typing. The list-level badge +stays `approximate` until then. + +Deleted with the Ollama adapter: `GET /llm/providers/{provider}/catalog` and +`POST /llm/providers/{provider}/pull`. Both are Ollama-shaped, and their +replacements are the rows above. + +Unchanged: `GET /llm/providers`, `GET|PUT /llm/selection/{role}`, +`GET|POST /llm/onboarding`, and everything under `/llm/connections`. + +### Identifiers + +| | Value | +|---|---| +| Registry key and `selected_models.provider` | `"llamacpp"` | +| Alembic revision for 7.6 | `0012` (head is `0011_document_cancelled_status`) | +| Egress destinations | `model_download`, `model_search` (both `huggingface.co`) | + +`"ollama"` appears at **18 non-test Python sites**; each becomes `"llamacpp"` or +is deleted with the adapter. + +### Out of scope + +Local image generation is untouched. `modules/llm/router.py` carries four image +routes (`/image/local`, `/image/local/runtime`, `/image/local/{name}`, +`/image/local/{name}/install`) and `resolve_image_generation()` dispatches on +`sdcpp.PROVIDER` through `OpenAICompatibleImageProvider`. None of that changes, +and the sd-server sidecar keeps its own lifecycle. The only shared file is +`modules/llm/router.py`, which is edited around them. + +Also unchanged: ingestion, the bge-small 384-dimensional index, retrieval, the +chat SSE protocol, Kokoro, and the OpenAI-compatible connection path. + +### Constants: what v1 carries, and what it deliberately does not + +A constant here is a fixed number compiled into the fit path. Not user-facing, +not per-machine: decided once, written down, and wrong for everybody if wrong at +all. Earlier drafts of this phase needed five. **v1 carries one.** + +**Prediction constants are not runtime constants.** Almost everything below only +decides what the screen says. `--fit` does its own allocation at load and moves +layers if our sum was optimistic, so an inaccurate prediction produces a wrong +*label*, not a broken model. The single exception is the context floor, which is +passed to the runtime as `-c` and genuinely changes behaviour — which is why it +is derived from the codebase rather than chosen. + +#### In v1 + +| Constant | Kind | Basis | +|---|---|---| +| **Context floor, 16K** | **runtime** | `HISTORY_BUDGET_TOKENS = 3000` plus ~8,000 tokens of grounding (~24k characters from `build_context`) plus a reply. Grounded, not chosen. | +| **Runtime overhead** | prediction | **unmeasured.** See below. | + +> **ponytail: runtime overhead.** The `overhead` term in +> `need = weights + KV(window) + overhead` is a placeholder. Hermes measured +> 1.5 GiB, but on a 32 GiB discrete card; today's `recommendation_reserve_gb: 2.0` +> is self-flagged as uncalibrated. On a 5.4 GB budget the difference between +> 1.0 and 1.5 GiB moves rows across the `FITS`/`PARTIAL` line, and on a 12 GB +> NVIDIA card it decides whether Qwen3 14B is recommended at all (31 MiB of +> margin at 1,024 MiB overhead). **Measure real RSS after loading three or four +> models on each target before 7.4 ships.** It is the only unmeasured number in +> the fit path, and the only one in this phase that a person still has to go and +> find out. + +#### Not in v1, and why + +Each of these was in an earlier draft. Each was dropped because it would have +meant writing an invented number into a spec where it would be indistinguishable +from a measured one. That is the specific failure this phase exists to avoid: +llmfit called Qwen3 8B a "Perfect" fit for an 8 GB Mac because it was working +from a memory figure that was not true on that machine. + +| Constant | What it would enable | Why not now | What it needs first | +|---|---|---|---| +| `felt_cost` | a formula predicting how slow a model *feels* on this machine | the recommendation is "highest `rank` that fits"; no latency judgement is made | a prefill model — `llama-bench pp512` per device class. llmfit's `prefill_tps` is an estimate for a generic machine. | +| **Speed floor** | refusing to star a model that fits but crawls | same | a `felt_cost` that means something, plus a threshold from real sessions rather than copied from an agent app | +| **Ladder rungs** | starting narrow so a **bigger model stays GPU-resident**, widening only when a conversation needs it. The benefit is memory, not long chats: Qwen3 4B on an 8 GB M2 is `FITS` at 16K and `PARTIAL` at 40K, purely from KV cache. | context is fixed at load | evidence that mid-conversation reloads are worth the complexity for document Q&A | +| **Growth threshold** | when to step up a rung | same | measured occupancy over real sessions | +| `decode_fraction` | correct speed prediction for mixture-of-experts models | **nothing to apply it to** — no MoE entry ships in v1. The value itself is computable today, from `expert_used_count` and the tensor table. | only the 80B-A3B-class rung | + +#### When they come back + +Each is additive and none requires a migration or a contract change. + +- **The speed gate** (`felt_cost` + floor) lands with the first real prefill + measurements. It slots into the recommendation policy as a filter over the + `FITS` set; the ordering and the badges are untouched. The reasoning about + *why* it must measure prefill and not decode is kept under the policy block so + it is not rediscovered. +- **The context ladder** (rungs + threshold) lands if fixed-at-load proves + limiting on large machines. It changes `POST /models/load` timing only. +- **`decode_fraction`** lands with the MoE rung, as a manifest field that + defaults to `1.0` for every dense entry already shipped. + +Shipping without them is not a gap to apologise for. Four of the five were +numbers nobody had measured, and a confident wrong badge is worse than a narrower +feature that tells the truth. + ## Phases Phases 7.0–7.5 keep Ollama registered and working. `REGISTRY` carries both @@ -208,14 +403,17 @@ llmfit adapter switches to `recommend --force-runtime llamacpp --output-llamacpp ### 7.1 — GGUF header reader -`modules/llm/gguf/header.py`, stdlib only. Local file or `Range: bytes=0-4194303` +`modules/llm/gguf/header.py`, stdlib only. Local file or `Range: bytes=0-8388607` against `huggingface.co`. Yields architecture, `block_count`, `context_length`, `embedding_length`, `head_count_kv`, key/value lengths, `sliding_window`, `expert_count`, `n_vocab`, chat template, exact tensor bytes. Verified: HF returns `206` with `accept-ranges: bytes`. A 135M model's metadata -ends at 1.77 MB and its tensor table at 1.79 MB, in 2.5 s. Budget 4 MB and -retry wider on a truncated parse — the vocab token list is what makes it large. +ends at 1.77 MB and its tensor table at 1.79 MB, in 2.5 s. **Bigger models need +more:** Qwen3-Coder-30B-A3B's metadata alone runs to 5.94 MB, with 579 tensor +entries after it, so a 4 MB budget fails on it. Budget **8 MB** and retry wider +on a truncated parse. The vocabulary token list is what makes the metadata +large, so the figure scales with vocab size rather than with the model. **Tests:** a checked-in truncated GGUF; a fake HTTP server asserting the Range header and a truncated-response retry. @@ -253,8 +451,18 @@ partitioning, or the frontend catalog schema" — becomes a real test. Chat composes `OpenAICompatibleChatProvider` rather than reimplementing streaming. Install fetches the GGUF into `--models-dir` with real filenames, not llama.cpp's content-addressed HF cache layout (`LLAMA_CACHE` points at the app -data dir so nothing writes to `~/.cache`). Context set via `POST /models/load` -with `args`, on a ladder grown at turn boundaries. +data dir so nothing writes to `~/.cache`). + +**Context is fixed at load, not grown.** `POST /models/load {"args": ["-c", N]}` +with `N` the largest window that keeps the verdict at **`FITS`** — not the +largest that fits at all. KV cache is allocated upfront and competes with the +weights for device memory, so maximising context silently demotes models out of +GPU residency: Qwen3 4B on an 8 GB M2 is `FITS` at 16K (4,632 MiB) and `PARTIAL` +at its native 40K (6,468 MiB). Floored at 16K (3,000 +history tokens plus ~8,000 of grounding plus a reply) and capped at the model's +own `context_length`. No ladder, no reload mid-conversation. Growing on +occupancy is additive later; it would mean reloading the model between turns and +two constants copied from Hermes with no evidence they suit RAG. **Tests:** a fake llama-server (aiohttp) covering health, models, props, load, chat stream, delete; install against a fake HF serving a real small GGUF. @@ -312,12 +520,12 @@ Manifest `schema_version` **3**. One entry, before and after: "capabilities": [], // user-facing only. Today: "vision" or nothing. // ── JUDGEMENT. Two fields, typed by a person. ── - "rank": 83, // our preference order for document Q&A. + "rank": 78, // our preference order for document Q&A. // Not a benchmark. Never displayed. "validated": true, // someone downloaded this exact file and ran it - "decode_fraction": 1.0 // 1.0 dense; the active slice for MoE (not derivable -} // from the header — set by hand, see below) + "decode_fraction": 1.0 // 1.0 dense; the active slice for MoE, computed +} // from the header (see below) ``` **The split is the point.** Everything above the line comes free from the GGUF @@ -333,8 +541,21 @@ Committing what the header said at authoring time is what lets `weights + KV(16K) + overhead` run against the manifest alone, on first paint, with no network. It is the same reason Hermes carries estimator inputs inline. -`decode_fraction` is the fraction of the build's bytes read per decoded token — -`1.0` dense, the active slice for MoE. Without it an MoE entry's speed +`decode_fraction` is the fraction of the build's bytes read per decoded token: +`1.0` for a dense model, the active slice for MoE. **It is computed, not typed.** +The header carries `.expert_count` and `.expert_used_count` (verified: +128 and 8 on Qwen3-Coder-30B-A3B), and expert tensors are identifiable in the +tensor table the reader already parses — `blk.N.ffn_{down,gate,up}_exps.weight`, +144 of that model's 579 tensors. So: + +``` +decode_fraction = (non_expert_bytes + expert_bytes × used_count / expert_count) + ÷ total_bytes +``` + +Sanity check: Qwen3-Coder-**30B-A3B** is 30B total against 3B active, so the +formula should land near 0.1, which is the range Hermes hand-authored (0.08 to +0.15) for comparable models. Without it an MoE entry's speed prediction is wrong by an order of magnitude, which matters the moment the 80B-A3B-class rung lands. @@ -357,11 +578,13 @@ $ uv run scripts/refresh_curated_models.py \ --repo unsloth/Qwen3-Next-80B-A3B-GGUF --quant Q4_K_M resolving unsloth/Qwen3-Next-80B-A3B-GGUF … Q4_K_M found, 45.2 GB - reading GGUF header (4 MB range) … qwen3next, 48 layers, 2 kv-heads, ctx 262144 - llmfit … best_quant=Q4_K_M ✓ matches pin + reading GGUF header (8 MB range) … qwen3next, 48 layers, 2 kv-heads, ctx 262144 + llmfit … sweeping budgets for the quantization ladder + Q3_K_M 88 · Q4_K_M 94 · Q5_K_M 96 · Q6_K 97 · Q8_0 97 + pinned quant Q4_K_M found in ladder ✓ llmfit proposes rank 94 (would be highest in the list — currently 92) - ⚠ MoE detected — decode_fraction not derivable from the header. Set it manually. + MoE detected: 128 experts, 8 used → decode_fraction 0.09 ⚠ validated=false until you run this file. wrote curated-models.json (+1 entry) @@ -387,15 +610,56 @@ a year: | A hardware class is under-served (see **The quantization ladder**) | occasional | | Someone reports a bad recommendation | reactive | -**The budget flags are mandatory.** Run bare, llmfit detects whatever laptop the -script is on and grades a quantization nobody pinned, so the numbers would -describe a different file than the one in `artifacts`. The `best_quant` -assertion catches that at authoring time, where a person is already looking. +#### Scoring at the pinned quantization -> **ponytail:** llmfit has no way to request a score at a named quantization — -> `plan --quant` returns memory requirements but carries no quality field. Until -> it does, the declared budget is chosen so llmfit lands on the pinned quant, and -> the assertion above is what keeps the two in step. +**llmfit has no flag for this, so the script has to work for it.** There is no +way to ask for a score at a named quantization: `recommend` and `info` grade +whatever `best_quant` the detected hardware selects, and `plan --quant` accepts a +quantization but returns only `weight_gb`, `kv_cache_gb`, `total_vram_gb`, +`estimated_tps` and `recommended_gpu` — **no quality field**. Verified on +`1.1.11`. + +That matters because quality tracks `best_quant` exactly (see **Appendix**). +Score at the wrong budget and you have graded a different file than the one in +`artifacts`. Qwen3 8B is 78 at the pinned Q4_K_M and 83 at Q8_0; run the script +on a workstation and you commit 83 for a file that behaves like 78. + +**The procedure: build the model's ladder, then index into it.** Do not search +for a single budget. Sweep the device budget across the model's whole +quantization ladder, record the `(best_quant → quality)` pair at each step, and +look up the pinned quantization in the resulting map. + +``` +for budget in multiplicative_steps(lower, upper, ratio=1.01): + row = llmfit recommend --force-runtime llamacpp --no-dashboard \ + --memory {budget} --ram 128G -n 20000 + ladder[row.best_quant] = row.score_components.quality + +rank_proposal = ladder[pinned_quant] # KeyError is a hard failure +``` + +Three properties this needs, each learned from a measured failure: + +- **Multiplicative steps at 1 % or finer.** The window in which a given + quantization is `best_quant` scales with the model, so fixed steps are wrong at + both ends. Qwen3 0.6B reads `Q3_K_M` at 1400M, `Q4_K_M` at 1420M and `Q5_K_M` + at 1500M — a Q4_K_M window roughly 6 % wide, which a 1G-step sweep skips + entirely and silently. Qwen3 32B's is 22G to 24G, with 21G reading Q3_K_M and 25G reading Q5_K_M. +- **`--ram` pinned high and constant.** It is swept as one variable, not two. +- **A missing rung is a hard failure, never a fallback.** If the pinned + quantization never appears in the ladder, the script **exits non-zero and says + so**. It must not fall back to the nearest rung or to the bare-hardware score: + that is precisely the silent mis-grade this section exists to prevent. The + operator then types the rank by hand, which is always allowed. + +The `best_quant` assertion in the transcript above is the visible half of this. +The ladder sweep is what makes it something the script can actually assert +rather than something the operator has to remember. + +> **ponytail:** this whole subsection is a workaround for a missing llmfit flag. +> If `plan --quant` ever grows a quality field, or `recommend` grows +> `--quant`, the sweep collapses to one call and this text should be deleted +> rather than kept for history. Numbers are source rather than build output for three reasons beyond speed: a rank moving 78 → 94 appears in a pull request where someone notices, a tag @@ -434,9 +698,16 @@ well: Community fine-tunes scoring a perfect 100, above published models. That number is closer to the name than to measured behaviour. It behaves on models it knows -properly — the shipped six come back `38 · 53 · 68 · 83 · 90 · 92`, monotonic +properly — the shipped six come back `33 · 48 · 63 · 78 · 85 · 92`, monotonic and sanely spread — which is exactly the population the curated tier draws from. +> **Those figures are at the pinned Q4_K_M**, recovered by the ladder sweep +> above. An earlier draft of this document carried `38 · 53 · 68 · 83 · 90 · 92`, +> which is the same six graded at whatever `best_quant` a large budget selected — +> Q8_0 for five of them. The order was unaffected, so no pick or test changed, +> but the numbers described files the manifest does not ship. **Any rank quoted +> anywhere must name the quantization it was taken at.** + Three checks before committing a rank: 1. within a family, rank rises with parameter count — the invariant in **Tests** @@ -720,19 +991,26 @@ Deleted: the collision-resolution block, `_placeholder_row()`, **Recommendation policy**, its own module and its own test file: ```text -fitting = entries the estimator does not refuse -resident = fitting, weights entirely in VRAM -pleasant = resident, felt_cost(representative RAG turn) under floor - -pleasant → max(rank, -size) reason: best-rank-resident - or speed-gated-rank if the floor - eliminated a higher-ranked candidate -resident → max(speed) reason: fastest-resident -else → no recommendation; spilled entries stay installable +resident = curated entries whose state is FITS +resident → max(rank, -size) reason: best-rank-resident +else → no recommendation; PARTIAL entries stay installable, + just never starred ``` -Rank is editorial and static per pinned file. Speed is physics per machine. -Predictions order candidates and gate the floor; they are never displayed. +**No speed gate in v1.** An earlier draft gated on predicted latency, which +required two constants with nothing measured behind them. The policy is now +"highest rank among the entries that fit entirely on the device", and the tie +break prefers the smaller build, which leaves more room for context. + +The gate is additive later. What it needs first is a prefill model: a handful of +`llama-bench pp512` runs per device class, since llmfit's `prefill_tps` and +`ttft_ms` are estimates for a generic machine and this app is prefill-dominated. + +> **Why prefill, when the gate lands.** `HISTORY_BUDGET_TOKENS = 3000` plus ~24k +> characters of grounding is roughly 8,000 prefill tokens against ~300 decoded, +> the inverse of an agent's ratio. Decode is memory-bound; prefill is +> compute-bound, which is why CUDA leads Vulkan ~36-40% on `pp512` and ~10% on +> `tg128`. A decode-only prediction mis-ranks for this app. > **RAG is prefill-dominated.** `HISTORY_BUDGET_TOKENS = 3000` plus ~24k > characters of grounding is roughly 8,000 prefill tokens against ~300 decoded — @@ -967,11 +1245,16 @@ compliance. That plausibly collapses three tiers to two. `rank` and a pinned `quantization`; every `shape.architecture` is in the 152-name list; a curated row prices and badges with no network at all. - **`rank` rises with parameter count within a family.** Today this passes - trivially — the shipped six are one family and score `38 · 53 · 68 · 83 · 90 · - 92`. The first failure is the signal to look, and it is expected: a + trivially — the shipped six are one family and score `33 · 48 · 63 · 78 · 85 · + 92` at the pinned Q4_K_M. The first failure is the signal to look, and it is expected: a mixture-of-experts, a second dense family, or a specialist ranked for document Q&A rather than for what it was tuned on will all break size-ordering legitimately. The assertion exists to make that a decision, not a drift. +- **Authoring: every committed `rank` was taken at that entry's pinned + quantization.** The ladder sweep records which rung it read; the script fails + non-zero when the pinned rung is absent rather than grading a neighbour. This + is a script-level test, not an app test — nothing in the product can check it, + which is exactly why it is written down. - A curated entry with a `vision` capability is never preferred over a higher-ranked text model for the ★; capabilities do not enter the ordering. - Capabilities: `vision` requires **both** `IMAGE in inputs` and @@ -1035,6 +1318,42 @@ differs 0/2424.** Quality and memory are functions of *(model, quantization)*, not of hardware — hardware only enters by choosing the quantization. That is what makes a pinned manifest's numbers deterministic and build-time computable. +**The same result on a single model, swept.** The population statistic above says +hardware does not move quality. This says *why*, and it is the observation the +whole authoring design rests on. Qwen3 8B, `--ram 64G` held fixed, sweeping the +device budget: + +| `--memory` | `best_quant` | `quality` | `estimated_tps` | `score` | +|---|---|---|---|---| +| 6G | Q3_K_M | 75 | 17.9 | 65.1 | +| 7G | Q4_K_M | 78 | 13.4 | 63.8 | +| 8G | Q5_K_M | 81 | 10.7 | 63.6 | +| 10G | Q6_K | 82 | 9.0 | 66.2 | +| 12G | Q8_0 | **83** | 6.7 | 63.6 | +| 16G | Q8_0 | **83** | 6.7 | 67.4 | +| 24G | Q8_0 | **83** | 6.7 | 67.4 | +| 48G | Q8_0 | **83** | 6.7 | 67.4 | + +Read the last four rows: 12 GB to 48 GB, four times the memory, and `quality` +does not move. **It stops moving the instant the quantization stops moving.** +Hardware reaches quality through exactly one channel, `best_quant`, and that +channel is one we close by pinning the file. + +Three decisions follow from that, and this table is the reason for all three: + +1. **`rank` is authored once and shipped frozen.** A number that does not vary + with the machine is data, not a per-machine computation. There is nothing for + a scan on the user's device to discover. +2. **Only `score_components.quality` is read; `score` is discarded.** In the same + sweep `score` runs 65.1, 63.8, 63.6, 66.2, 63.6, 67.4 — non-monotonic, and + moving with hardware because it blends fit and speed into the quality term. + The composite is a statement about a laptop; the component is a statement + about a file. +3. **The declared budget must land llmfit on the pinned quantization.** Since + quality tracks `best_quant` exactly, scoring at the wrong budget scores the + wrong file. See **Authoring**, where this is a hard assertion rather than a + convention. + Device query versus advisor, same machine: | | llmfit | `ggml_backend_dev_memory()` | diff --git a/plans/community-local/frontend/01-shell.md b/plans/community-local/frontend/01-shell.md index d619f66a67..2c072ffbf0 100644 --- a/plans/community-local/frontend/01-shell.md +++ b/plans/community-local/frontend/01-shell.md @@ -21,7 +21,7 @@ dashboard are specified in [`01-dashboard.md`](01-dashboard.md). not a broken API. - For each healthy provider, load `GET /llm/providers/{provider}/models` and show only installed models with - the `completion` capability. Do not hardcode Ollama or model names. + the `completion` capability. Do not hardcode a runtime or model name. - A manual Refresh checks provider health and reloads installed models, so a model pulled outside SurfSense appears without restarting the app. - Refresh preserves the draft choice while it remains available. If the @@ -95,8 +95,8 @@ installed-state and validated-selection path. - Pull a compatible model outside the app, press Refresh, and the new model appears without restart. - An embedding-only model is not offered for generation. -- Stop Ollama: the screen reports the provider unavailable and Retry/Refresh - recovers after Ollama starts. +- Stop `llama-server`: the screen reports the runtime unavailable and + Retry/Refresh recovers once it is back. - Remove the current model: Refresh reports the stale selection and requires an explicit replacement. - Keyboard-only selection and save work in light and dark themes and at a diff --git a/plans/community-local/frontend/05-install-ux.md b/plans/community-local/frontend/05-install-ux.md index 1b26c228c1..598d059451 100644 --- a/plans/community-local/frontend/05-install-ux.md +++ b/plans/community-local/frontend/05-install-ux.md @@ -1,25 +1,38 @@ # Frontend — Phase 5: Install UX -> Owns: the hardware-ranked generation catalog and one-click install flow. +> Owns: the generation catalog and one-click install flow. > API contract: > [`../api/05a-model-recommendations.md`](../api/05a-model-recommendations.md). +> +> **Phase 7 supersedes the catalog half of this document** — +> [`../api/07-llamacpp-runtime.md`](../api/07-llamacpp-runtime.md) replaces +> Ollama with llama.cpp, deletes the hardware-scan gate, and changes the fit +> vocabulary. The sections below are updated to match it; anything describing +> `Scan hardware`, a `scanned` flag, five-level fit badges, `ollama_name`, or an +> llmfit-derived Explore list was true of phase 5 and is not true now. The +> OpenAI-compatible connection sections are unaffected. ## Goal Make choosing a local generation model understandable without asking the user -to know RAM budgets, quantization, Ollama tags, or GGUF files. Show exact -SurfSense-tested configurations first, then other installable catalog entries, -both ranked for the current computer. +to know RAM budgets, quantization, or GGUF files. Show the configurations +SurfSense has tested first, then a search over every GGUF model llama.cpp can +run, with an honest fit badge on every row. ## Onboarding The **Choose your AI model** page loads `GET /llm/catalog` and renders: -1. **Best for this computer** — exact configurations from the curated-model manifest - that fit this computer. -2. **More models** — remaining installable llmfit catalog entries. -3. **Installed** — already available local models, including models that are no - longer a good fit under the current policy. +1. **Recommended for this computer** — one entry, the highest-ranked curated + model whose fit state is `FITS`. +2. **Tested by SurfSense** — the curated manifest, every row badged. +3. **All models** — a live Hugging Face search over 204,797 GGUF repos, ordered + by downloads, via the separate search route. Absent when egress for + `model_search` is off. +4. **Installed** — already available local models, plus **Add a .gguf file**. + +`GET /llm/catalog` needs no network and returns rows 1, 2 and 4. Search is its +own request so the page never waits on `huggingface.co`. This full-page flow appears only until durable model onboarding is complete. Later model changes and recovery from a missing selection happen in @@ -56,44 +69,47 @@ installed models. Users can search that list and switch models directly. A separate **Manage models** action opens Settings → Models for downloads, provider configuration, and hardware rescans. -**The section is called Curated, not Recommended** — the bucket was renamed in -both the API response and the UI. Curated entries are grouped visually by family -(for example Qwen or Gemma), but each card remains an exact -parameter/quantization/runtime configuration. Do not render every parameter size -merely because its family is curated. An entry appears once only: curated models -are removed from Explore. - -**The page no longer waits for a hardware scan.** As built, the order is -Installed, then local image models, then Curated, and then either Explore or — -when `scanned` is false — a "Scan hardware" call to action in its place. Curated -and Installed come from the packaged manifest and the runtime's own inventory, so -they render on a fresh install with no probe, and a curated card is installable -straight from that state because the manifest already pins its `ollama_name`. -Opening the page does not trigger a scan; only the explicit button does. +**There is no hardware scan and no scan button.** The hardware line is present +on first paint: the budget comes from the runtime's own allocator in about +180 ms, so there is nothing to wait for and nothing to trigger. `Scan hardware`, +`Rescan hardware` and the `scanned` flag are all deleted. + +**Every row carries a fit badge, in both lists**, because fit is arithmetic +rather than a judgement. Three states only, with no `Unknown`: every row has a +file size, so every row can be priced. Wording is per platform — see **Fit +states** in [`../api/07-llamacpp-runtime.md`](../api/07-llamacpp-runtime.md), +which owns the copy. + +```text +● Full speed Runs entirely on the GPU +◐ Reduced speed Too big for the GPU, so part runs on the CPU +○ Won't fit Needs about 21 GB. This Mac has 13.6 GB +``` + +A search row is badged from its file size alone and rendered as approximate +until its header is read, which happens when the row is opened. Each card shows: - model and family; -- parameter size and quantization when known; -- fit badge (`Perfect`, `Good`, `Marginal`, or `Too tight`); -- estimated memory, disk size, tokens/second, and usable context when present; -- runtime that will install it; +- parameter size and quantization; +- fit badge, plus one plain line of why; +- download size, context length, architecture; - installed and selected state; -- license and estimate-confidence details in disclosure text. +- `vision` when the model can read images. No other capability surfaces. -Missing estimates display `Unknown`; they are never rendered as zero — **except -before any scan has run**, where the fit badge is hidden entirely rather than -showing "Unknown" to someone who was never offered an estimate. The badge -appears once a real scan stands behind the row. Use plain-language fit labels, -not raw llmfit scores, as the primary signal. A short -explanation says that estimates leave room for SurfSense itself and may differ -from real workloads. +Search rows additionally show provenance (*quantized from `Qwen/Qwen3-8B`*), +download count, licence, and an **other quantizations** disclosure. They carry +**no rank and no quality claim** — we describe them, we do not judge them. + +**Rank is never displayed.** It orders the curated rows within a fit state and +selects the ★; no number, score or star rating appears anywhere else. ## Interaction - **Download & Use** sends only `POST /llm/install {"catalog_id": "...", "select": true}`. The renderer does - not send an Ollama tag, artifact URL, local path, or quantization. + not send a repo, file, artifact URL, local path, or quantization. - Show bytes, percent, current phase, and cancellation while installing. Ignore duplicate clicks and keep other model actions disabled for a runtime that supports one pull at a time. @@ -101,26 +117,31 @@ from real workloads. available only after the API reports installed and selected state. - **Use** on an already installed compatible model calls the existing validated `PUT /llm/selection/generation` route. -- Manual **Rescan hardware** loads `GET /llm/catalog?refresh=true`. Ordinary - navigation uses the cached scan. -- `Marginal` requires a confirmation that responses may be slow or fail at long - context. `Too tight` cannot be newly installed but remains visible when - already installed. +- There is no rescan. `GET /llm/catalog` is cheap and needs no network; the + hardware budget is re-read on each load. +- **Reduced speed installs like Full speed** — no confirmation dialog. It runs, + slower, and llama.cpp places the layers. Only **Won't fit** blocks install, + and it states required and available bytes and names a smaller model rather + than greying out a control. +- Search shows a progress indicator while a row's header is fetched (2 to 3 + seconds) and resolves the approximate badge to a firm one. - A failed or cancelled install remains retryable and is never shown as selected. -- **Delete** calls the backend's provider model endpoint. The renderer never - edits Ollama storage. Deletion is unavailable during installs or active +- **Delete** calls the backend's model endpoint. The renderer never edits the + models directory. Deletion is unavailable during installs or active generation, and deleting the selected model never silently chooses another. ## States and degradation -- Skeleton only the catalog region during a scan; retain the page heading and - hardware explanation. There is no "first scan" to wait on any more: the page - renders Curated and Installed before one has ever run. -- If llmfit is unavailable or malformed, show a recommendation warning and the - installed-model controls returned by the API. Do not label unscored models as - curated on the strength of a score they do not have — curation comes from the - manifest, which needs no scan, so the two are independent. +- Nothing to skeleton on load: the hardware line, the curated rows and their + badges all render without a network call or a probe the user has to start. +- llmfit is not shipped and cannot fail at runtime. Its numbers were baked into + the manifest before release. +- If `model_search` egress is off or `huggingface.co` is unreachable, the search + section reports the destination unavailable. Recommended, Tested and Installed + are unaffected — that is the airgapped product. +- If a curated download 404s because the upstream repo moved, say the model is + no longer available from this source and fall through to the next-best entry. - If a runtime is unavailable, show its status and disable its install actions; other runtimes remain usable. - If no model fits, explain the limitation and keep remote providers such as @@ -182,11 +203,13 @@ confirmation names any generation or image roles that will be cleared. ## Airgap -Keep **Import local model** as an advanced secondary action. Electron supplies -the selected file/folder through a typed preload method; the renderer never -receives arbitrary filesystem powers. The API/runtime adapter validates the -artifact and imports atomically. Importing parser packs remains a separate -packaging flow and does not pass through llmfit. +**Add a .gguf file** sits beside Installed, not buried as an advanced action — +with search unavailable offline, it is one of only two ways an airgapped user +gets a model. Electron supplies the selected file through a typed preload +method; the renderer never receives arbitrary filesystem powers. The adapter +validates the header, rejects an unsupported architecture, and links the file +into the models directory. Importing parser packs remains a separate packaging +flow. ## Frontend structure @@ -215,19 +238,23 @@ src/features/onboarding/ ``` TanStack Query owns catalog, installed inventory, and selection invalidation. -The response shape is runtime-neutral; do not branch rendering on `ollama` -except for runtime-specific explanatory copy. +Search is a separate query with its own cache key and a 300 s stale time, +matching the API-side cache. The response shape is runtime-neutral; do not +branch rendering on a provider name. Fit-badge copy branches on the budget's +`uma` and `has_gpu` flags, not on the runtime. ## Acceptance -- A clean supported machine sees Curated models above Explore, and sees them - before it has scanned anything. -- Gemma/Qwen family grouping does not create unsupported parameter-size cards. -- Download & Use is one user action and ends with the exact installed runtime - model selected. -- Unknown estimates, llmfit failure, runtime failure, insufficient disk, - cancellation, and interrupted streams are represented without false success. -- Installed models remain selectable when recommendation scanning is down. +- A clean machine sees a hardware line, one ★ recommendation and every curated + row badged, on first paint, with no button and no network. +- No screen anywhere displays a rank, score or star rating. +- Search returns results ordered by downloads, each badged, none ranked. +- Download & Use is one user action and ends with the exact installed model + selected. +- Runtime failure, insufficient disk, a dead upstream pin, cancellation, and + interrupted streams are represented without false success. +- With `model_search` egress off, Recommended, Tested and Installed still work + and a `.gguf` can still be added from disk. - Keyboard-only operation, visible focus, live progress, light/dark themes, and a narrow desktop window all work. - Path B: install → model download → chat works after completion. From 2b49c04828b0d7a04bf48e0b7b29191306ce367d Mon Sep 17 00:00:00 2001 From: Anish Sarkar <104695310+AnishSarkar22@users.noreply.github.com> Date: Sat, 19 Sep 2026 20:28:59 +0530 Subject: [PATCH 04/60] feat(spec): ship Vulkan everywhere, defer CUDA to phase 8 --- plans/community-local/00-umbrella-plan.md | 3 +- plans/community-local/api/05c-packaging.md | 8 +- .../api/07-llamacpp-runtime.md | 219 +++++++- plans/community-local/api/08-cuda-backend.md | 493 ++++++++++++++++++ 4 files changed, 700 insertions(+), 23 deletions(-) create mode 100644 plans/community-local/api/08-cuda-backend.md diff --git a/plans/community-local/00-umbrella-plan.md b/plans/community-local/00-umbrella-plan.md index ddb78e66c9..f38445b609 100644 --- a/plans/community-local/00-umbrella-plan.md +++ b/plans/community-local/00-umbrella-plan.md @@ -30,6 +30,7 @@ Same phase number = integrate together. | **5** | [`05-install-ux.md`](frontend/05-install-ux.md) ◐ | [`05a-model-recommendations.md`](api/05a-model-recommendations.md) ✓ + [`05b-openai-compatible-connections.md`](api/05b-openai-compatible-connections.md) ✓ + [`05c-packaging.md`](api/05c-packaging.md) ◐ | [`05-packaging.md`](worker/05-packaging.md) ✓ | | **6** | import, license, egress settings ([`00d-pivot-plan.md`](00d-pivot-plan.md)) ✓ | `modules/migration/`, `modules/license/`, keychain, auto-update ([`00d-pivot-plan.md`](00d-pivot-plan.md)) ◐ | — (import reuses `ingest_document`) | | **7** | catalog rework, fit badges, HF search (in [`api/07-llamacpp-runtime.md`](api/07-llamacpp-runtime.md)) | [`07-llamacpp-runtime.md`](api/07-llamacpp-runtime.md) — replace Ollama with llama.cpp | — | +| **8** | — | [`08-cuda-backend.md`](api/08-cuda-backend.md) — CUDA as an optional Windows backend. **Not scheduled**: measured at 0.66 s per turn for 685 MB | — | **Demo:** phase 3 all streams. **Ship:** phase 6 = SurfSense 2.0.0 (1.0.x tags are taken by the old project versioning; see [`00d-pivot-plan.md`](00d-pivot-plan.md)). @@ -105,7 +106,7 @@ Docker Compose, Postgres, Zero, Redis, Celery, LangGraph, git KB, scrapers, MCP, | **Embed provider** | Bundled bge-small-en-v1.5 int8, in-process on onnxruntime | 384-dim, ~66MB, runs offline on CPU with no model server. Docling parses, Chonkie chunks. Remote embedding is a later opt-in, not a launch dependency. | | **Generation architecture** *(Phase 7 supersedes the local half — [`api/07-llamacpp-runtime.md`](api/07-llamacpp-runtime.md): llama.cpp is the one local runtime, llmfit moves to build time, the catalog opens to every GGUF on Hugging Face)* | llmfit catalog/advisor + curated-model policy + runtime adapters; Ollama local, multiple OpenAI-compatible connections remote | [`api/05a-model-recommendations.md`](api/05a-model-recommendations.md) and [`api/05b-openai-compatible-connections.md`](api/05b-openai-compatible-connections.md). llmfit supplies broad model metadata, hardware detection, and fit estimates; it is not a `Generator`, installer, or inference runtime. Local adapters resolve trusted artifacts and Ollama is first. Remote endpoints are named connection instances with their own URL and optional key; models are discovered live, not synchronized into SQLite. `SelectedModel(role)` stores the provider, connection identity when remote, and exact model id. SurfSense selects endpoints but does not load-balance their replicas. | | **llmfit integration** *(Phase 7: the binary stops shipping. llmfit runs in CI to rank the curated manifest; it is never a gate, a hardware source, or on a request path)* | **Pinned official binary, short-lived JSON CLI, normalized behind `ModelAdvisor`** | The API runs `llmfit --json system` and `llmfit --max-context 8192 --json fit`, caches one scan, and exposes only SurfSense DTOs. No fork, patch, Python import, or permanent llmfit server. CLI label/machine-code differences are normalized at one seam and fixture-tested before a version bump. Failure removes ranking, not installed-model selection or chat. | -| **Ollama runtime** *(**superseded by Phase 7** — [`api/07-llamacpp-runtime.md`](api/07-llamacpp-runtime.md) replaces it with `llama-server` in router mode: 11–31 MB instead of 501 MB, CUDA inside the Windows installer, and no MLX until after launch)* | **Bundled as a supervised sidecar (packaged); dev uses the developer's own `ollama serve`** | Chat can't depend on a daemon the user may not have installed. The packaged app ships the standalone Ollama archive (`electron/scripts/fetch-ollama.mjs` stages it, electron-builder carries it in `resources/ollama`) and Electron runs it as a third sidecar on a chosen port, passing `SURFSENSE_LOCAL_OLLAMA_BASE_URL` to the API. Models aren't shipped — the user pulls into the writable data dir after install. The Python sidecars and Ollama share one supervisor (`electron/src/main/sidecars/`): the supervisor spawns/reaps, one spec file per sidecar carries its identity. Boot now starts four — `api`, `worker-ingest`, `worker-studio`, and Ollama when packaged — and `watchImageModel` adds a fifth, sd-server, later, because only the API knows which image model was chosen. Ollama is best-effort at boot (only the API gates the window; its state surfaces via `/llm/providers`). | +| **Ollama runtime** *(**superseded by Phase 7** — [`api/07-llamacpp-runtime.md`](api/07-llamacpp-runtime.md) replaces it with `llama-server` in router mode: 11–31 MB instead of 501 MB, Vulkan on every platform off Apple Silicon, and no MLX until after launch)* | **Bundled as a supervised sidecar (packaged); dev uses the developer's own `ollama serve`** | Chat can't depend on a daemon the user may not have installed. The packaged app ships the standalone Ollama archive (`electron/scripts/fetch-ollama.mjs` stages it, electron-builder carries it in `resources/ollama`) and Electron runs it as a third sidecar on a chosen port, passing `SURFSENSE_LOCAL_OLLAMA_BASE_URL` to the API. Models aren't shipped — the user pulls into the writable data dir after install. The Python sidecars and Ollama share one supervisor (`electron/src/main/sidecars/`): the supervisor spawns/reaps, one spec file per sidecar carries its identity. Boot now starts four — `api`, `worker-ingest`, `worker-studio`, and Ollama when packaged — and `watchImageModel` adds a fifth, sd-server, later, because only the API knows which image model was chosen. Ollama is best-effort at boot (only the API gates the window; its state surfaces via `/llm/providers`). | | **Persistence** | SQLAlchemy 2.0 + Alembic, same as cloud | Models are the source of truth. `versions/` ships as PyInstaller data, resolved from the package's own `__file__` — de-risked in [`api/00-spike.md`](api/00-spike.md). | | **Migrations** | **Hand-written; autogenerate is off** | Autogenerate cannot see a rename — it emits drop + add, which deletes a column's data silently. The target database is one user's laptop, unbacked and uninspectable, so every revision is written and read by a person. `env.py` carries no `target_metadata`, so `--autogenerate` cannot be used by accident. Mature SQLite-backed apps make the same call — hand-written revisions throughout. | | **Schema owner** | Alembic only; **never** `create_all` | Cloud's `create_all`-on-startup races its own migrations and breaks releases. Local has one path to a schema, and a test fails if models and migrations drift. | diff --git a/plans/community-local/api/05c-packaging.md b/plans/community-local/api/05c-packaging.md index f741636f5d..58ef00f5a4 100644 --- a/plans/community-local/api/05c-packaging.md +++ b/plans/community-local/api/05c-packaging.md @@ -125,9 +125,11 @@ is measured on that machine by the runtime that will do the work.** licence and notices. Prune to `llama-server` plus the libraries it links: the macOS tarball ships 24 executables and only one is needed, which also shrinks the notarization surface. -- Sizes and per-OS details, including CUDA inside the Windows installer, are in - [`07-llamacpp-runtime.md`](07-llamacpp-runtime.md) under **Packaging, three - targets**. +- Sizes and per-OS details are in [`07-llamacpp-runtime.md`](07-llamacpp-runtime.md) + under **Packaging, three targets**. The GPU backend is **Vulkan on every + platform off Apple Silicon**; no CUDA payload ships. Adding one later is + specified in [`08-cuda-backend.md`](08-cuda-backend.md) and needs no code + change. - Code signing, notarization and antivirus smoke tests cover `llama-server` and its libraries. Note it spawns **child processes** in router mode, so the hardened runtime needs testing against a grandchild, not just the sidecar. diff --git a/plans/community-local/api/07-llamacpp-runtime.md b/plans/community-local/api/07-llamacpp-runtime.md index 72ee954224..d89bc14d37 100644 --- a/plans/community-local/api/07-llamacpp-runtime.md +++ b/plans/community-local/api/07-llamacpp-runtime.md @@ -53,8 +53,8 @@ Ollama's native API does not. | **Downloads** | **SurfSense fetches the GGUF**, not `POST /models` | `llama-server` is a second process we do not proxy; an in-process fetch is the only place `egress.require()` actually holds. Also buys resume, checksums, and the header as the file lands. | | **Quantization** | **One pinned file per entry in v1** | A pinned file is what "tested by SurfSense" can honestly claim. Per-machine selection is possible at no extra cost (one header read prices every quant), so a second variant on the largest entries is a cheap follow-on, not v1. See **The quantization ladder**. | | **Context** | **Fixed at load. Floor 16K, capped at the model's own `context_length`** | llama.cpp fixes context at load, so `num_ctx()`'s per-request sizing has no equivalent. The floor is 3K history plus ~8K grounding plus a reply. Growing on occupancy is additive later; it needs a mid-conversation reload and two constants with nothing measured behind them. | -| **Windows CUDA** | **In the installer.** No post-install download | Airgapped. Dropping Ollama frees ~500 MB; CUDA 13.4 + cudart costs 573 MB. Net ≈ +100 MB, under the 2 GB `makensis` ceiling that forced `pruneCudaRunners()`. | -| **CUDA version** | 13.4 only | CUDA 13 requires Turing (7.5)+. Shipping 12.4 as well for Pascal costs another 645 MB — that is what made Ollama's archive 1.8 GB. Pascal falls through to Vulkan automatically via backend scoring. | +| **GPU backend** | **Vulkan on every platform off Apple Silicon. No CUDA** | Measured on an RTX 3050 at `b11050`: CUDA leads Vulkan **9.1%** on `pp512`, **10.2%** on `pp8192`, **2.1%** on decode — 0.66 s on an 11 s turn, for 685 MB. Vulkan covers NVIDIA, AMD and Intel from one 31 MB archive, its loader ships with Windows, and it is what the app already does today (`pruneCudaRunners()`). CUDA is specified as an optional later addition in [`08-cuda-backend.md`](08-cuda-backend.md), which needs **no code change** — ggml selects it by the files present. | +| **Device selection** | First device with `type == GPU`. **Never sum** | The same physical card appears once per loaded backend, and an integrated GPU can advertise more memory than a discrete one (16198 MiB against 6002 MiB, measured). ggml's ordering already expresses backend preference, so this **is** backend selection. Specified in **7.2**. | | **Backend selection** | Runtime, by ggml | Releases are built `GGML_BACKEND_DL=ON`. A failed `dlopen` is skipped silently, so a missing Vulkan loader degrades to CPU rather than failing. | | **Prompt tier fallback** | Keys on **loopback**, not provider name | `provider == "ollama" → COMPACT` breaks the moment a Mac user runs a 4B through LM Studio. `host_destination()` already computes loopback. | @@ -138,7 +138,7 @@ Every machine gets a GPU-resident pick, so six rungs strand nobody. Three gaps: - **A gap at 16 GB.** 14B misses by 69 MiB (0.6%), so a 16 GB Mac drops to 8B with 3.9 GB spare. - **The 4070 row is decided by an uncalibrated constant.** 31 MiB of margin at - 1,024 MiB overhead; at Hermes' measured 1.5 GiB it does not fit and that + 1,024 MiB overhead; at Hermes' 2 GiB margin floor it does not fit and that machine drops a rung. **Manifest growth is follow-on work, not phase 7.** v1 ships these six re-pinned @@ -335,10 +335,12 @@ is derived from the codebase rather than chosen. | **Runtime overhead** | prediction | **unmeasured.** See below. | > **ponytail: runtime overhead.** The `overhead` term in -> `need = weights + KV(window) + overhead` is a placeholder. Hermes measured -> 1.5 GiB, but on a 32 GiB discrete card; today's `recommendation_reserve_gb: 2.0` -> is self-flagged as uncalibrated. On a 5.4 GB budget the difference between -> 1.0 and 1.5 GiB moves rows across the `FITS`/`PARTIAL` line, and on a 12 GB +> `need = weights + KV(window) + overhead` is a placeholder. Hermes uses +> `max(2 GiB, 9% of total)` (`_MARGIN_FLOOR = 2 << 30`, `_MARGIN_FRACTION = 0.09` +> in `local_runtime/hardware.py`), which on a 6 GB card holds back a third of it; +> today's `recommendation_reserve_gb: 2.0` is self-flagged as uncalibrated. On a +> 5.4 GB budget the difference between 1.0 and 2.0 GiB moves rows across the +> `FITS`/`PARTIAL` line, and on a 12 GB > NVIDIA card it decides whether Qwen3 14B is recommended at all (31 MiB of > margin at 1,024 MiB overhead). **Measure real RSS after loading three or four > models on each target before 7.4 ships.** It is the only unmeasured number in @@ -424,6 +426,91 @@ header and a truncated-response retry. OS APIs. First hit wins, cached; warm the probe in the background at first launch, not on the path of the first render (a cold `ggml_backend_load_all()` on macOS compiles 20 Metal shader libraries, measured at ~20 s once, 180 ms after). +That cold penalty is **Metal-only**: measured off Apple Silicon the same call is +77 ms with one backend and 205 ms with two. + +#### The probe must run from the library directory + +**This is the single most likely way this phase ships broken.** +`ggml_backend_load_all()` discovers backends by scanning **the directory of the +running executable**, not the directory the libraries were loaded from. The +backend probe runs inside the frozen `surfsense-api.exe`, which lives in +`resources/backend/`; the ggml libraries live in `resources/llamacpp/`. Different +directories, so the scan finds nothing. + +Verified on a Windows machine with a working RTX 3050, same minute, same process +otherwise: + +```text +python wprobe.py → device count 0 ← card invisible +cd && python wprobe.py → device count 2 ← CUDA0 + CPU +``` + +**Loading the libraries by absolute path is not sufficient.** The backend scan is +a separate step keyed on the host executable. Set the working directory around +`ggml_backend_load_all()`, or spawn the probe from the library directory. + +`GGML_BACKEND_PATH` is **not** the escape hatch — it expects a *file*, not a +directory. Passing one logs `load_backend: failed to load ` and leaves the +count at zero. + +The failure is silent and indistinguishable from a genuinely GPU-less machine: +every row badges *"Works here, runs on your processor"* and nothing reports an +error. See **Failure behavior**. + +#### Two library handles on Windows, one on Linux + +The exported symbols are split, and the intuitive single handle fails: + +| Library | Exports | +|---|---| +| `ggml.dll` / `libggml.so` | `ggml_backend_load_all`, `dev_count`, `dev_get` | +| `ggml-base.dll` / `libggml-base.so` | `dev_name`, `dev_description`, `dev_type`, `dev_memory` | + +Linux resolves the second through the ELF dependency, so one +`CDLL("libggml.so")` works — but `CDLL("libggml-base.so")` raises `undefined +symbol: ggml_backend_load_all`. **Windows needs both handles**; PE exports do not +chain, and `ggml.dll` alone raises `AttributeError: function +'ggml_backend_dev_name' not found`. + +#### Device selection: first `type == GPU`, never a sum + +A machine reports one entry per *(backend, device)* pair, so the same physical +card can appear more than once and an integrated GPU can advertise more memory +than a discrete one. Measured: + +```text +[0] Vulkan0 type=GPU 6002.0 MiB total, 5234.0 MiB free NVIDIA GeForce RTX 3050 +[1] Vulkan1 type=ACCEL 16198.3 MiB total AMD Radeon(TM) Graphics +[2] CPU type=CPU 31884.6 MiB total, 22750.2 MiB free AMD Ryzen 5 9600X +``` + +```python +def select_device(devices): + """First GPU wins. ggml orders backends by preference, so this is also + backend selection.""" + return next((d for d in devices if d.type == DeviceType.GPU), None) +``` + +- **First, not largest.** Sorting by memory picks the 16 GB integrated chip over + the 6 GB discrete card and places every layer on the slower device. +- **Never aggregate.** No `sum()` across devices. One device becomes + `HardwareBudget.usable_vram_bytes`. +- **`ACCEL` is not a GPU** for budgeting — an integrated part carving from system + RAM has no memory of its own to place layers in. + +This rule is required here, under Vulkan alone. It becomes load-bearing if +[`08-cuda-backend.md`](08-cuda-backend.md) ever ships, because two loaded +backends list the same card twice, **both typed `GPU`**, so type filtering does +not deduplicate them. + +#### `ram_available_bytes` does not come from ggml reliably + +ggml's CPU device reports real available memory on native Windows (31884.6 MiB +total against 22750.2 MiB free, measured) but **`total == free` under WSL2**, +where the figure is virtualised. Keep the OS API in the chain for the RAM half +rather than trusting the CPU device, since `ram_available_bytes` is what +separates `PARTIAL` from `TOO_BIG`. Two budget modes. **Capacity** (total − margin) for catalog pricing; **live** (free now) for launch decisions. Pricing against live-free while a model is @@ -434,9 +521,19 @@ Per-layer KV from the header. Split today's single `recommendation_reserve_gb: 2 (flagged `ponytail` as uncalibrated) into a measured runtime overhead, a device margin, and a UMA headroom. -> **ponytail:** the runtime overhead constant is a placeholder until measured -> against real RSS on all three targets. On a 5.4 GB budget it moves rows across -> the fits/spills line. +> **Do not double-count.** ggml's `free` **already excludes** the desktop's +> allocation and the backend context. Measured on a 6144 MiB RTX 3050: 986 MiB +> was gone before a single weight loaded — ~445 MiB to the desktop, ~541 MiB to +> the CUDA context (`nvidia-smi` idle free 5699 MiB against ggml's 5158 MiB). +> Budget from `free` **and** add a ~1 GiB overhead term and the same memory is +> charged twice, demoting rows that fit. State explicitly which number `margin` +> and `overhead` are each subtracted from: against `free`, `margin` is ~0 and +> `overhead` covers per-model compute buffers only. + +> **ponytail:** the *per-model* half of the overhead constant is still a +> placeholder until measured against real RSS with a model loaded. The +> pre-model half is now measured at 986 MiB on a discrete card. On a 5.4 GB +> budget it moves rows across the fits/spills line. **Tests:** decision-table over constructed profiles — no GGUF parsing in the fit tests, so the estimator is testable independently of the reader. @@ -464,6 +561,19 @@ own `context_length`. No ladder, no reload mid-conversation. Growing on occupancy is additive later; it would mean reloading the model between turns and two constants copied from Hermes with no evidence they suit RAG. +**Router mode runs with an empty models directory**, so the sidecar lifecycle is +testable before any model exists. Verified on Windows and Linux at `b11050`: +`GET /health` → `{"status":"ok"}`, `GET /models` → `{"data":[],"object":"list"}`, +and `GET /props` → `"role":"router"` — which is the reliable check that the +sidecar came up in router mode rather than single-model mode. SIGTERM shuts it +down cleanly (`cleaning up before exit`), and on Windows `taskkill /PID /T +/F` walks the tree and reports each child terminated. + +> **Partially verified only.** The `taskkill /T` check above ran with no model +> loaded, so it reaped `conhost.exe` rather than a model worker. The case 7.5 +> actually cares about — reaping the **grandchild** the router spawns per loaded +> model — still needs a run with a model resident. + **Tests:** a fake llama-server (aiohttp) covering health, models, props, load, chat stream, delete; install against a fake HF serving a real small GGUF. @@ -1009,14 +1119,13 @@ The gate is additive later. What it needs first is a prefill model: a handful of > **Why prefill, when the gate lands.** `HISTORY_BUDGET_TOKENS = 3000` plus ~24k > characters of grounding is roughly 8,000 prefill tokens against ~300 decoded, > the inverse of an agent's ratio. Decode is memory-bound; prefill is -> compute-bound, which is why CUDA leads Vulkan ~36-40% on `pp512` and ~10% on -> `tg128`. A decode-only prediction mis-ranks for this app. +> compute-bound. A decode-only prediction mis-ranks for this app. > **RAG is prefill-dominated.** `HISTORY_BUDGET_TOKENS = 3000` plus ~24k > characters of grounding is roughly 8,000 prefill tokens against ~300 decoded — > the inverse of an agent's ratio. Decode is memory-bound; prefill is -> compute-bound, which is why CUDA leads Vulkan ~36–40% on `pp512` and ~10% on -> `tg128`. A decode-only prediction mis-ranks for this app. Use llmfit's +> compute-bound. (An earlier draft cited a 36–40% CUDA lead here; measured, it +> is **9.1%** — see [`08-cuda-backend.md`](08-cuda-backend.md).) Use llmfit's > `prefill_tps`/`ttft_ms` from the manifest until measured class constants exist. **Tests:** policy decision-table across the six hardware profiles above. @@ -1030,12 +1139,13 @@ llama.cpp has no stable channel). | Target | Asset | Size | |---|---|---| | darwin-arm64 | `bin-macos-arm64.tar.gz` | 11 MB | -| win32-x64 | `bin-win-vulkan-x64.zip` + `cuda-13.4` + `cudart` | 31 + 150 + 423 MB | +| win32-x64 | `bin-win-vulkan-x64.zip` | 31 MB | | linux-x64 | `bin-ubuntu-vulkan-x64.tar.gz` | 30 MB | Verified: the Vulkan archive is the CPU archive plus exactly one file (`libggml-vulkan.so` / `ggml-vulkan.dll`), and all CPU micro-architecture -variants ship inside — 15 on Windows, 10 on Linux. `ggml_backend_load_best()` +variants ship inside — 15 on Windows, 10 on Linux (the CUDA archive carries 14 +on Linux, so the figure is per archive, not per platform). `ggml_backend_load_best()` searches the executable's own directory and probes `cuda` before `vulkan`, so the CUDA DLLs sit in the same flat folder and win automatically. @@ -1209,7 +1319,28 @@ compliance. That plausibly collapses three tiers to two. - Runtime missing or crashed: catalog stays visible, installs disabled, an already-selected remote connection still answers. -- No Vulkan loader: `dlopen` fails, backend skipped silently, CPU runs. +- No Vulkan loader: `dlopen` fails, backend skipped silently, CPU runs. (Not + observed on Windows — `vulkan-1.dll` is present in `C:\Windows\System32` on a + stock install, so nothing needs bundling there. Linux supplies it through + `libvulkan1`; see 7.5.) +- **A GPU exists and ggml cannot see it: say so.** The silent skip above is + correct when there is no GPU and wrong when there is one, and the two are + indistinguishable from ggml alone — both print `(none)` and exit **0**, with no + warning even under `GGML_BACKEND_DEBUG=1`. Two real causes: the probe ran + outside the library directory (7.2), or a backend library's dependencies are + missing from the package. Cross-check against the OS: + + | OS reports a GPU | ggml reports a GPU | Meaning | + |---|---|---| + | yes | yes | normal | + | no | no | genuine CPU-only machine, badge accordingly | + | **yes** | **no** | **broken install.** Say so; never badge CPU-only. | + + Sources: `Win32_VideoController` on Windows, `/sys/class/drm` on Linux. Filter + virtual adapters — the test machine carried a `Parsec Virtual Display Adapter` + beside two real GPUs. Odysseus codes around the same failure explicitly + (*"nvcc found but CUDA runtime is not visible"*), which is independent evidence + it is common enough to handle rather than assume away. - HF unreachable: curated and installed render from the manifest and disk; search reports the destination is unavailable, not an error. - Header read truncated: retry wider once, then fall back to a file-size estimate @@ -1263,7 +1394,24 @@ compliance. That plausibly collapses three tiers to two. the first user turn rather than dropping it. Only the user-facing set reaches the serialized response — `system_role` and `typed_content` never appear in an API payload. -- Runtime: fake llama-server for health/models/props/load/chat/delete. +- **Device selection**: `select_device()` over fixtures — a discrete 6 GB card + beside a 16 GB integrated one selects the **discrete** card; `ACCEL` alone + yields no GPU; `usable_vram_bytes` never exceeds any single device's memory + (asserted against a listing where the same card appears twice, which a naive + sum turns into 12 GB on a 6 GB card); a CPU-only listing returns `None`. +- **Library loading**: two `ctypes` handles on Windows, one on Linux. A + regression is an `AttributeError` at startup, not a wrong answer, so it is + worth an explicit test rather than leaving it to the first packaged run. +- **Probe working directory**: devices are found when the probe runs from the + library directory and **not** found when it does not. The second assertion is + what stops the search-path bug from returning; without it the test passes in a + dev shell and the bug ships. +- **Broken-install detection**: the three OS-versus-ggml rows in **Failure + behavior**; the mismatch row produces a distinct error state, never a + `has_gpu: false` budget. +- Runtime: fake llama-server for health/models/props/load/chat/delete; router + mode against an empty models directory, asserting `/props` reports + `role: router`. - Migration: an Ollama selection and an `ollama_pull` row, upgraded. - Packaging: staged checksum; `--list-devices` from the packaged path per OS. @@ -1277,7 +1425,8 @@ nor `huggingface.co` appears in it. - The recommended model installs and answers with citations. - A model found through search — one Ollama's library never carried — installs and answers. -- A Windows machine with an RTX card runs on CUDA, from the installer alone. +- A Windows machine with an RTX card runs on Vulkan, from the installer alone, + and the catalog prices against that card rather than an integrated GPU. - A machine with no usable GPU runs on CPU with no error. - Airgapped: curated install from a local `.gguf`, chat, and Studio all work. - `grep -i ollama` returns only the intended survivors. @@ -1354,6 +1503,38 @@ Three decisions follow from that, and this table is the reason for all three: wrong file. See **Authoring**, where this is a hard assertion rather than a convention. +**Windows and Linux, measured on an RTX 3050 / Ryzen 5 9600X at `b11050`.** +Both platforms were previously reasoned from documentation only. + +| Measurement | Value | +|---|---| +| `ggml_backend_load_all()`, one backend | 77 ms (Win) · 194 ms (Linux) | +| `ggml_backend_load_all()`, two backends | 205 ms | +| `ggml_backend_dev_memory()` first call, CUDA | 47.89 ms | +| `ggml_backend_dev_memory()` first call, Vulkan | 3.50 ms | +| `ggml_backend_dev_memory()`, CPU | 0.01 ms | +| RTX 3050, `nvidia-smi` idle | 6144 MiB total, 5699 MiB free | +| RTX 3050, ggml after context init | 6143.5 MiB total, 5158 MiB free | +| Consumed before any weight loads | **986 MiB** (~445 desktop, ~541 context) | +| CPU device, native Windows | 31884.6 MiB total, 22750.2 MiB free | +| CPU device, WSL2 | 26048.6 MiB total, **26048.6 MiB free** (virtualised) | +| `vulkan-1.dll` on stock Windows | present in `System32`, nothing to bundle | + +GPU backend comparison, same card and model (Qwen3 4B Q4_K_M, `-ngl 99`): + +| test | CUDA 13.4 | Vulkan | CUDA advantage | +|---|---|---|---| +| `pp512` | 1929.18 ± 8.84 | 1767.62 ± 1.43 | +9.1% | +| `pp8192` | 1537.12 ± 5.84 | 1394.77 ± 5.04 | +10.2% | +| `tg300` | 52.19 ± 0.22 | 51.13 ± 0.11 | +2.1% | + +0.66 s on an 11 s turn, for 685 MB. Full working in +[`08-cuda-backend.md`](08-cuda-backend.md) — **Appendix**, including why the +earlier 36–40% figure was retired. + +Asset sizes at `b11050`, verified against the live release: macos-arm64 11.2 MB, +win-vulkan 31.8 MB, ubuntu-vulkan 30.4 MB — the figures in **7.5** are accurate. + Device query versus advisor, same machine: | | llmfit | `ggml_backend_dev_memory()` | diff --git a/plans/community-local/api/08-cuda-backend.md b/plans/community-local/api/08-cuda-backend.md new file mode 100644 index 0000000000..be915f2cb4 --- /dev/null +++ b/plans/community-local/api/08-cuda-backend.md @@ -0,0 +1,493 @@ +# API — Phase 8: CUDA as an optional second backend + +> Owns: the Windows GPU backend payload and multi-device selection. Extends +> [`07-llamacpp-runtime.md`](07-llamacpp-runtime.md), which ships **Vulkan on +> every platform**. Nothing here is required for phase 7 to be complete, and +> nothing here changes application code. +> +> **Status: not scheduled.** This phase exists so the decision is written down +> with its evidence, not so it gets built. It ships only if the measurement in +> **8.0** comes back materially different from the one in the Appendix. + +## Goal + +Add CUDA alongside Vulkan on Windows, as a packaging change, when and only when +a measurement says it is worth 685 MB. + +## Why this is a separate phase + +Phase 7 ships Vulkan everywhere: + +```text +Mac Metal 11 MB no alternative exists +Windows Vulkan 86 MB on disk +Linux Vulkan 85 MB on disk +``` + +That is not a compromise position. It is what the app **already does today**: +`pruneCudaRunners()` in `electron/scripts/fetch-ollama.mjs` deletes the CUDA +runners out of the bundled Ollama at build time, because `cuda_v12 + cuda_v13` +was 1.7 GB of a 1.8 GB payload and blew the 2 GB `makensis` ceiling. Every +SurfSense user with an NVIDIA card is on Vulkan right now, and the `ponytail` +beside that function records the intended fix — an opt-in post-install pack, +which the airgapped rule later ruled out. + +So CUDA is not a regression to defend against. It is a **new capability**, newly +affordable because dropping Ollama frees ~500 MB. This document decides whether +to buy it. + +**Measured, it is worth 0.66 seconds per turn.** The full numbers are in the +Appendix. The one-line version: CUDA leads Vulkan by **9.1% on `pp512`, 10.2% on +`pp8192`, and 2.1% on decode** — not the 36–40% that earlier drafts of phase 7 +claimed, which is why that claim was removed rather than carried forward. + +## Decisions + +| Decision | Choice | Why | +|---|---|---| +| **Ship it?** | **No, not yet.** Vulkan only | 685 MB for a measured 0.66 s on an 11 s turn. The gap is real and small. Revisit on new evidence, not on a schedule. | +| **Platform** | **Windows only**, if ever | Linux is the weakest case: AppImage is a single downloadable file, Linux ships CUDA **13.3** against Windows' **13.4** so covering both pins two majors, and Linux desktop plus recent NVIDIA is the smallest population. Hermes reaches the same conclusion — its asset table has no `cuda` key for Ubuntu at all. | +| **Mac** | **Never** | Apple Silicon has no NVIDIA hardware. Metal is built into macOS. Not a decision. | +| **Coexistence** | Both backends, one flat folder | Verified: ggml loads every `ggml-*.dll` it finds and registers all of them. CUDA is probed first, so it wins automatically when present. | +| **Code changes** | **None** | The backend is a file in a folder, not a branch. `ggml_backend_load_best()` does the selection. Adding CUDA is `fetch-llamacpp.mjs` plus a rebuild. | +| **Device selection** | First device with `type == GPU`. **Never sum** | Specified in [`07-llamacpp-runtime.md`](07-llamacpp-runtime.md) — 7.2, and needed there anyway. What this phase adds is the duplicate case: with both backends loaded the same physical card appears twice, both typed `GPU`. | +| **CUDA version** | 13.4, Windows x64 | Requires Turing (7.5)+. Shipping 12.4 as well for Pascal costs another ~645 MB — that is what made Ollama's archive 1.8 GB. Pre-Turing cards fall through to Vulkan automatically. | +| **Reversibility** | Cheap in both directions | Adding CUDA is four files. Removing it is deleting four files. Neither touches a contract, a schema or a migration. | +| **Diagnostics** | Mandatory if shipped | A missing cudart is **silent**: the app reports no GPU, exit 0. See **Failure behavior**. | + +## Boundaries + +```text + packaging ──── fetch-llamacpp.mjs stages backend libraries + │ Vulkan always · CUDA optionally, Windows only + ▼ + ggml_backend_load_all() ── loads every ggml-*.dll in the executable's dir + │ probes cuda before vulkan · silent skip on failure + ▼ + device list ── may contain the SAME card once per loaded backend + │ + ▼ + select_device() ── first type == GPU wins. This IS backend selection. + │ + ▼ + HardwareBudget ── one device, never a sum +``` + +Everything above `select_device()` is packaging. Everything below is phase 7 and +does not change. That is the whole point of keeping this separate. + +## Contracts + +> **Provenance.** Every number and every listing in this document was measured on +> a Windows 11 (26200) machine with an RTX 3050, against llama.cpp `b11050`. The +> ggml calls were made live through `ctypes`. Nothing here is reasoned from +> documentation. + +### The device listing this phase exists to handle + +With both backends staged in one folder, this is what a real NVIDIA Windows +machine reports: + +```text +device count 4 + [0] CUDA0 type=GPU 6143.5 MiB total, 5166.0 MiB free NVIDIA GeForce RTX 3050 + [1] Vulkan0 type=GPU 6002.0 MiB total, 5166.0 MiB free NVIDIA GeForce RTX 3050 + [2] Vulkan1 type=ACCEL 16198.3 MiB total AMD Radeon(TM) Graphics + [3] CPU type=CPU 31884.6 MiB total AMD Ryzen 5 9600X +``` + +Four entries, **three** pieces of hardware. Read it carefully, because three +separate traps are visible in it: + +1. **`CUDA0` and `Vulkan0` are the same physical card.** Note the identical + `5166.0 MiB free`. Summing device memory reports 12 GB of GPU on a 6 GB card, + and every fit badge is then wrong in the dangerous direction — telling users a + model fits when it cannot. +2. **Both are typed `GPU`.** Filtering on device type does **not** deduplicate + them. Type separates the discrete card from the integrated one; it does not + separate backends. +3. **The integrated GPU advertises more memory than the real one** — 16198 MiB + against 6002 MiB, because it is carving from system RAM. Picking the device + with the most memory picks the slowest one. This trap exists under Vulkan + alone and is not specific to this phase. + +`total` also differs between backends for the same card (6143.5 vs 6002.0 MiB): +CUDA reports the device total, Vulkan the heap size. A 141 MiB disagreement, +small enough not to matter and large enough to confuse someone reading logs. + +### The selection rule + +**`select_device()` is specified in [`07-llamacpp-runtime.md`](07-llamacpp-runtime.md) +— 7.2**, and is required there regardless of whether this phase ever ships: the +integrated-GPU trap exists under Vulkan alone. It takes the first device with +`type == GPU`, never aggregates, and treats `ACCEL` as not a GPU. + +What this phase adds is the **duplicate** case. Under Vulkan alone a machine +lists each physical device once. With both backends loaded the same card is +listed once per backend, **both typed `GPU`** — so type filtering, which is +sufficient to separate discrete from integrated, is not sufficient to separate +backends. "First wins" already handles it, because ggml orders CUDA ahead of +Vulkan; the requirement is simply that nothing downstream aggregates. + +### Packaging shape + +Four files land beside `llama-server.exe`, in the same flat folder as everything +else. No subdirectory, no search path, no environment variable. + +| File | Size | +|---|---| +| `ggml-cuda.dll` | 138 MB | +| `cublasLt64_13.dll` | **492 MB** | +| `cublas64_13.dll` | 55 MB | +| `cudart64_13.dll` | 0.55 MB | + +From two pinned assets, both checksummed: + +```text +llama-b-bin-win-cuda-13.4-x64.zip 150.1 MB +cudart-llama-bin-win-cuda-13.4-x64.zip 423.5 MB +``` + +**The archives overlap almost entirely**, so the marginal costs are wildly +asymmetric and worth knowing before anyone estimates from the sum: + +| Starting from | Adding | Costs | +|---|---|---| +| Vulkan build | CUDA | **+685 MB** | +| CUDA build | Vulkan | **+42 MB** | + +Both archives are the same CPU base plus one backend library. All the weight is +`cudart`, and 72% of `cudart` is the single file `cublasLt64_13.dll`. + +On-disk totals, measured: + +```text +Vulkan only 86 MB +CUDA only 706 MB +both 748 MB (not 792 — the CPU base is shared) +``` + +### Out of scope + +Linux CUDA, ROCm, SYCL and OpenVINO. The assets exist +(`llama-b-bin-ubuntu-cuda-13.3-x64.tar.gz`, 149.1 MB, plus a 410.2 MB +cudart) and Hermes' comment that Linux CUDA is unavailable is **stale** — but +none of them is proposed here. Anyone adding one should read **8.2** first, +because the device-duplication rule generalises to every additional backend. + +Phase 7's runtime contract, catalog, fit states, manifest and recommendation +policy are untouched. + +## Phases + +### 8.0 — Re-measure, and only then decide + +**This phase gates every other phase.** Nothing below happens until this +produces a number materially better than the Appendix's. + +```bash +llama-bench -m -p 512,8192 -n 300 -ngl 99 +``` + +Run against both staged builds on the same card, same model, same build number. + +- `-p 8192` because that is the real prefill load. The default 512 understates a + RAG workload, and the gap varies with sequence length. +- `-ngl 99` so both runs are fully GPU-resident and the comparison is compute, + not offloading. +- Confirm from the log that Vulkan chose device 0. If decode collapses, it picked + the integrated GPU and the run is void. + +**Decision rule, fixed in advance so the result cannot be rationalised:** + +| Result | Action | +|---|---| +| CUDA leads by **< 15%** on `pp8192` | Vulkan stands. Close this phase. | +| **15–30%** | Judgement call. Weigh against 685 MB and the installer size. | +| **> 30%** | Ship it. Proceed to 8.1. | + +The Appendix measurement came back at **10.2%**. Re-measuring is worthwhile only +on a materially different NVIDIA generation — Ada or Blackwell, or a Turing card +at the CUDA 13 floor. Re-running on another Ampere card is not new evidence. + +**Tests:** none. This is a measurement, and its output is a decision. + +### 8.1 — Package + +`fetch-llamacpp.mjs` gains a second Windows asset, pinned by build number **and +SHA-256** like the first. + +- Stage both archives into the same flat `resources/llamacpp/`. +- Prune as phase 7 already does: `llama-server` plus libraries, nothing else. +- Retain upstream licence and notice files for the CUDA archives. +- Code signing, notarization and antivirus smoke tests now cover four more + binaries, one of which is 492 MB. Expect signing time to rise. +- Recheck the `makensis` ceiling. Phase 7 leaves ~470 MB of headroom against + today's installer; this consumes ~605 MB compressed of it. + +**Tests:** staged-checksum verification for both archives; the packaged folder +contains all four files; `llama-server --list-devices` from the final packaged +path reports a `CUDA` device on an NVIDIA runner. + +### 8.2 — Harden device selection + +`select_device()` ships in phase 7 (7.2). **No new code here** — this step is +only the extra test coverage the duplicate case needs, since phase 7 cannot +produce a listing containing the same card twice. + +**Tests:** + +- The four-device listing above, verbatim as a fixture, selects `CUDA0`. +- The same listing with `ggml-cuda.dll` absent selects `Vulkan0`. +- A Vulkan-only listing with a 6 GB discrete card and a 16 GB integrated one + selects the **discrete** card — the regression that would otherwise place + layers on the slow chip. +- `usable_vram_bytes` never exceeds any single device's memory. Asserted against + the duplicate listing, where a naive sum yields 12 GB on a 6 GB card. +- A CPU-only listing returns `None` and `has_gpu` is false. + +### 8.3 — Diagnose a broken install + +**Mandatory if this phase ships.** Without it, one packaging slip produces an app +that silently never uses the GPU. + +A missing cudart does not fail loudly. Measured on both Windows and Linux with +the card present and `ggml-cuda.dll` staged: + +```text +$ llama-server --list-devices +Available devices: + (none) +$ echo $? +0 +``` + +No dialog, no warning, no non-zero exit. `GGML_BACKEND_DEBUG=1` changes nothing. +`ldd` shows why — `libcudart.so.13 => not found` — but the program never says so. + +So cross-check against the operating system: + +| OS reports a GPU | ggml reports a GPU | Meaning | +|---|---|---| +| yes | yes | normal | +| no | no | genuine CPU-only machine, badge accordingly | +| **yes** | **no** | **broken install.** Say so. Never badge CPU-only. | + +Sources: `Win32_VideoController` on Windows, `/sys/class/drm` on Linux. Filter +virtual adapters — the measured machine carried a `Parsec Virtual Display +Adapter` alongside its two real GPUs. + +Odysseus codes around this same failure explicitly, warning *"nvcc found but CUDA +runtime (libcudart.so) is not visible — building llama-server for CPU only"*. +Independent confirmation that it is common enough to handle rather than assume +away. + +**Tests:** all three rows above as fixtures; the mismatch row produces a distinct +error state, not a `has_gpu: false` budget. + +### 8.4 — Verify on real hardware + +CI runners have no GPU, so `--list-devices` on a runner proves only that the +binary starts. A real NVIDIA machine must confirm, once per pinned build: + +- `--list-devices` reports a `CUDA` device from the packaged resource path. +- The probe reports a `CUDA0` device from inside the **frozen backend process**, + not from a shell in the llama.cpp folder. See **Failure behavior** — this is + the failure most likely to survive to release. +- A model loads and answers with all layers on the GPU. +- `taskkill /PID /T /F` reaps the model workers, not just the router. + +## Failure behavior + +- **cudart missing** — silent `(none)`, exit 0. Covered by 8.3. This is the + failure mode that ships if 8.3 is skipped. +- **Probe run from the wrong directory** — `ggml_backend_load_all()` searches + **the host executable's own directory**. From the frozen `surfsense-api.exe` + in `resources/backend/`, it finds nothing in `resources/llamacpp/` and reports + zero devices on a machine with a working card. Loading the libraries by + absolute path is **not sufficient**; the backend scan is separate. Verified: + `device count 0` from `python.exe`, `device count 2` after `chdir` to the + library folder, same machine, same second. +- **`GGML_BACKEND_PATH` is not the fix.** It expects a **file**, not a + directory. Passing a directory logs `load_backend: failed to load ` and + leaves the count at zero. +- **Pre-Turing NVIDIA card** — the CUDA backend is skipped silently and Vulkan + runs. Correct behavior, no message needed. +- **Duplicate devices** — not a failure, the normal state. Handled by + `select_device()`. +- **Vulkan loader missing on Windows** — not observed. `vulkan-1.dll` is present + in `C:\WINDOWS\System32` on a stock install. + +## Tests + +Collected from the phases above; all are unit tests over fixtures except the +last two. + +- `select_device()` across five listings: both backends, CUDA absent, + Vulkan-only with discrete plus integrated, CPU-only, and the duplicate case + asserting no aggregation. +- Broken-install detection across the three OS-versus-ggml rows. +- Two-handle `ctypes` loading on Windows (`ggml.dll` **and** `ggml-base.dll`); + single-handle on Linux (`libggml.so`). A regression here is an + `AttributeError` at startup, not a wrong answer. +- Probe succeeds when the working directory is the library folder and reports + zero when it is not — the second assertion is what stops the search-path bug + from returning. +- Packaging: checksums for both archives; all four files present in the packaged + folder. +- **Manual, on real NVIDIA hardware, once per pinned build:** 8.4's four checks. + +## Acceptance + +Only meaningful if 8.0 clears the bar. In order: + +- A measurement exists, at `pp8192`, on a named GPU, recorded in the Appendix + with its build number. +- A Windows machine with an RTX card runs on CUDA from the installer alone, no + post-install download. +- A pre-Turing NVIDIA machine runs on Vulkan with no error shown. +- A machine with no usable GPU runs on CPU with no error shown. +- A machine with a card but a broken CUDA payload says **broken install**, and + does not badge itself CPU-only. +- Fit badges on a dual-GPU machine price against the discrete card's memory + alone, never a sum. +- The installer stays under the `makensis` ceiling. + +## Appendix — measurements + +All taken 19 Sep 2026 on one machine, against llama.cpp `b11050` (`60b06ab9a`). + +**Hardware** + +```text +Windows 11 26200 · WSL2 Fedora 44 (glibc 2.43) on the same box +AMD Ryzen 5 9600X, 6C/12T, 31884.6 MiB RAM +NVIDIA GeForce RTX 3050, 6144 MiB, driver 616.92, compute capability 8.6 +AMD Radeon(TM) Graphics (integrated) · Parsec Virtual Display Adapter +``` + +Compute capability **8.6** clears CUDA 13's Turing 7.5 floor, so this card +validates the version decision rather than testing its edge. + +**The benchmark.** Qwen3 4B Q4_K_M (2.32 GiB, 4.02 B params), `-ngl 99`, fully +GPU-resident on both runs. + +| test | CUDA 13.4 | Vulkan | CUDA advantage | +|---|---|---|---| +| `pp512` | 1929.18 ± 8.84 | 1767.62 ± 1.43 | **+9.1%** | +| `pp8192` | 1537.12 ± 5.84 | 1394.77 ± 5.04 | **+10.2%** | +| `tg300` | 52.19 ± 0.22 | 51.13 ± 0.11 | **+2.1%** | + +Error bars are 0.1–0.9%, so the gap is real and the size of the gap is not in +doubt on this card. + +**What it is worth**, at phase 7's real turn shape — ~8,000 prefill tokens +(`HISTORY_BUDGET_TOKENS = 3000` plus ~8K of grounding) against ~300 decoded: + +```text + prefill decode total turn +CUDA 5.33 s 5.75 s 11.1 s +Vulkan 5.87 s 5.87 s 11.7 s + ───────── +difference 0.66 s (5.7%) +``` + +**685 MB for 0.66 seconds.** That is the trade, stated plainly. + +**The 36–40% claim is retired.** Earlier drafts of phase 7 cited *"CUDA leads +Vulkan ~36–40% on `pp512` and ~10% on `tg128`."* Measured here: **9.1%** and +**2.1%** — off by roughly 4× and 5×. No public head-to-head benchmark on +identical NVIDIA hardware was found to corroborate either figure, so this +measurement is the only direct evidence either way. + +A likely cause is that the older figure predates Vulkan reaching the tensor +cores. This card reports `matrix cores: NV_coopmat2v` under Vulkan, against +`matrix cores: none` for the integrated AMD chip. **Treat that as a hypothesis, +not a finding:** llama.cpp discussion #12548 notes the `matrix cores:` line is +not a reliable indicator of the fast path, since current builds report +`KHR_coopmat` on hardware that previously reported `NV_coopmat2` at identical +throughput. The throughput above was measured directly and does not depend on +that explanation. + +**Timings** + +| Call | Value | +|---|---| +| `ggml_backend_load_all()`, CUDA only | 77 ms | +| `ggml_backend_load_all()`, Vulkan only | 136 ms | +| `ggml_backend_load_all()`, both | 205 ms | +| `ggml_backend_dev_memory()` first call, CUDA | 47.89 ms | +| `ggml_backend_dev_memory()` first call, Vulkan | 3.50 ms | +| `ggml_backend_dev_memory()`, CPU | 0.01 ms | + +No Metal-style cold-compile penalty off Apple Silicon; phase 7's advice to warm +the probe in the background is a Mac concern. + +**VRAM accounting**, same card, three sources: + +| Source | Total | Free | +|---|---|---| +| `nvidia-smi`, idle | 6144 MiB | 5699 MiB | +| ggml, after CUDA context init | 6143.5 MiB | 5158–5166 MiB | +| ggml, Vulkan | 6002.0 MiB | 5166.0 MiB | + +**986 MiB of 6144 is gone before a single weight loads** — ~445 MiB to the +desktop, ~541 MiB to the CUDA context. Phase 7's 1,024 MiB overhead placeholder +is within 4% of that, but note the double-counting trap: ggml's `free` has +**already** subtracted both. Budget from `free` and add ~1 GB again and the same +memory is charged twice, demoting rows that fit. + +**Symbols are split across two libraries**, and the intuitive choice fails: + +| Library | Exports | +|---|---| +| `ggml.dll` / `libggml.so` | `ggml_backend_load_all`, `dev_count`, `dev_get` | +| `ggml-base.dll` / `libggml-base.so` | `dev_name`, `dev_description`, `dev_type`, `dev_memory` | + +Linux resolves through the ELF dependency, so one `CDLL(libggml.so)` handle +works. **Windows needs both handles** — PE exports do not chain, and loading +`ggml.dll` alone raises `AttributeError: function 'ggml_backend_dev_name' not +found`. + +**Assets and sizes at `b11050`**, verified against the live release. Phase 7's +figures were all confirmed to the megabyte. + +| Asset | Size | +|---|---| +| `llama-b11050-bin-macos-arm64.tar.gz` | 11.2 MB | +| `llama-b11050-bin-win-vulkan-x64.zip` | 31.8 MB | +| `llama-b11050-bin-win-cuda-13.4-x64.zip` | 150.1 MB | +| `cudart-llama-bin-win-cuda-13.4-x64.zip` | 423.5 MB | +| `llama-b11050-bin-ubuntu-vulkan-x64.tar.gz` | 30.4 MB | +| `llama-b11050-bin-ubuntu-cuda-13.3-x64.tar.gz` | 149.1 MB | +| `cudart-llama-b11050-bin-ubuntu-cuda-13.3-x64.tar.gz` | 410.2 MB | + +**Linux ships CUDA 13.3, Windows 13.4.** Covering both means pinning two CUDA +majors with separate cadences. + +**What other projects do**, and why none of it decides this: + +| Project | Backend policy | Bundles? | +|---|---|---| +| Hermes (`hermes_cli/local_runtime/`) | CUDA → Vulkan → CPU; **no Linux CUDA** | No — downloads per machine | +| Odysseus | CUDA → ROCm → Vulkan → CPU | No — **compiles** on the user's machine | +| Ollama | ships both CUDA majors | **Yes** — 1.8 GB archive | +| SurfSense today | Vulkan | Yes, with CUDA pruned at build time | + +Hermes and Odysseus both prefer CUDA, and **neither pays for it** — an NVIDIA +user fetches or builds CUDA, everyone else gets Vulkan, nobody carries unused +weight. Their preference is evidence the gap is real; it is not evidence it is +worth 685 MB bundled, because they never face that question. + +Ollama is the only true bundling precedent, and its answer produced the 1.8 GB +archive that `pruneCudaRunners()` exists to cut down. + +## References + +- [`ggml-backend-reg.cpp`](https://github.com/ggml-org/llama.cpp/blob/master/ggml/src/ggml-backend-reg.cpp) — `ggml_backend_load_best()`, executable-directory search, silent skip on `dlopen` failure +- [llama.cpp discussion #12548](https://github.com/ggml-org/llama.cpp/discussions/12548) — coopmat2 on Vulkan, and why the `matrix cores:` line is not a reliable indicator +- [llama.cpp discussion #10879](https://github.com/ggml-org/llama.cpp/discussions/10879) — Vulkan backend performance +- [CUDA / ROCm / Vulkan scoreboard](https://knightli.com/en/2026/04/23/llama-cpp-gpu-benchmark-cuda-rocm-vulkan-scoreboard/) — CUDA-only; notes the absence of same-hardware backend comparisons +- [NousResearch/hermes-agent](https://github.com/NousResearch/hermes-agent) `hermes_cli/local_runtime/binaries.py` — `select_backend()`, the asset table, the `cuda → vulkan → cpu` ladder +- [odysseus-dev/odysseus](https://github.com/odysseus-dev/odysseus) `routes/cookbook_helpers.py` — the `_odysseus_has_cudart()` check and its warning +- `electron/scripts/fetch-ollama.mjs` — `pruneCudaRunners()`, the 2 GB `makensis` ceiling, and the ponytail this phase answers From 53985064f2b13f4713bc17d9067b1f15ead332ae Mon Sep 17 00:00:00 2001 From: Anish Sarkar <104695310+AnishSarkar22@users.noreply.github.com> Date: Sat, 19 Sep 2026 20:36:41 +0530 Subject: [PATCH 05/60] feat(spec): move rank into the variant, self-calibrate the overhead --- .../api/05a-model-recommendations.md | 7 + .../api/07-llamacpp-runtime.md | 204 ++++++++++++++---- 2 files changed, 170 insertions(+), 41 deletions(-) diff --git a/plans/community-local/api/05a-model-recommendations.md b/plans/community-local/api/05a-model-recommendations.md index 68cdede83b..83192bd01b 100644 --- a/plans/community-local/api/05a-model-recommendations.md +++ b/plans/community-local/api/05a-model-recommendations.md @@ -1,5 +1,12 @@ # API — Phase 5: Hardware-aware model recommendations +> **The runtime and manifest halves are superseded by Phase 7.** +> [`07-llamacpp-runtime.md`](07-llamacpp-runtime.md) replaces Ollama with +> `llama-server`, moves llmfit out of the app, and takes the manifest to +> `schema_version` **3** with a `variants` list. The `schema_version: 2` shape +> below describes what ships **today**, not what phase 7 writes. The catalog +> boundary, the normalized shapes and the frontend contract still govern. +> > Owns: `backend/modules/llm/recommendations/`, the llmfit adapter, the > runtime-install boundary, and the normalized catalog API. Packaging: > [`05c-packaging.md`](05c-packaging.md). Frontend: diff --git a/plans/community-local/api/07-llamacpp-runtime.md b/plans/community-local/api/07-llamacpp-runtime.md index d89bc14d37..c89804497d 100644 --- a/plans/community-local/api/07-llamacpp-runtime.md +++ b/plans/community-local/api/07-llamacpp-runtime.md @@ -214,6 +214,9 @@ class GgufArtifact: quantization: str # "Q4_K_M" size_bytes: int mmproj: str | None = None # vision projector, fetched alongside + # A curated manifest variant is this plus judgement (`rank`, `rank_basis`, + # `validated`); a searched build is this alone. The runtime never sees the + # judgement half — see 7.4. @dataclass(frozen=True) @@ -332,7 +335,7 @@ is derived from the codebase rather than chosen. | Constant | Kind | Basis | |---|---|---| | **Context floor, 16K** | **runtime** | `HISTORY_BUDGET_TOKENS = 3000` plus ~8,000 tokens of grounding (~24k characters from `build_context`) plus a reply. Grounded, not chosen. | -| **Runtime overhead** | prediction | **unmeasured.** See below. | +| **Runtime overhead** | prediction | Pre-model half **measured at 986 MiB** on a discrete card; the per-model half is a bootstrap value, replaced by the machine's own measurement after its first load. See below and **7.2**. | > **ponytail: runtime overhead.** The `overhead` term in > `need = weights + KV(window) + overhead` is a placeholder. Hermes uses @@ -343,9 +346,9 @@ is derived from the codebase rather than chosen. > `FITS`/`PARTIAL` line, and on a 12 GB > NVIDIA card it decides whether Qwen3 14B is recommended at all (31 MiB of > margin at 1,024 MiB overhead). **Measure real RSS after loading three or four -> models on each target before 7.4 ships.** It is the only unmeasured number in -> the fit path, and the only one in this phase that a person still has to go and -> find out. +> models on each target before 7.4 ships**, and ship that as the bootstrap +> value — but it is wrong at most once per machine, because 7.2 records what the +> load actually cost and prefers that number afterwards. #### Not in v1, and why @@ -535,6 +538,45 @@ margin, and a UMA headroom. > pre-model half is now measured at 986 MiB on a discrete card. On a 5.4 GB > budget it moves rows across the fits/spills line. +#### The overhead constant is a bootstrap value, not a permanent guess + +The shipped constant is a prediction for a machine nobody tested. **After one +successful load, that machine's real figure is known**, and the same rule the +rest of this phase runs on applies: anything that differs per machine is measured +on that machine by the runtime that will do the work. + +The instrument is already in use — `ggml_backend_dev_memory()`, sampled either +side of `POST /models/load`: + +```python +real_overhead = (free_before - free_after) - weights_bytes - kv_bytes(n_ctx) +``` + +```text +first run predict with the shipped constant +after first load record the machine's real overhead, keyed by device name +every run after prefer the recorded value +``` + +Three reasons this is worth ~15 lines in the load path: + +- **The constant is wrong at most once per machine**, on a screen shown before + anything is installed, instead of wrong forever on every machine that was not + one of the three targets. +- **It self-corrects per backend and per driver.** A CUDA context alone costs + ~541 MiB (measured); Metal and Vulkan differ; a driver update changes it. A + committed table cannot track that and no longer has to. +- **It retires the last `ponytail` in the fit path.** Nobody repeats the + measurement session after the first release. + +Storage is a small JSON file in the app data dir keyed by device name — no +migration, and no settings table needs inventing for it. Invalidate on a +llama.cpp build bump or when the device name changes. + +**Tests:** a recorded value is preferred over the constant; a recorded value from +a different device name or build is ignored; a corrupt or missing file falls back +to the constant rather than raising, since this sits on the first-render path. + **Tests:** decision-table over constructed profiles — no GGUF parsing in the fit tests, so the estimator is testable independently of the reader. @@ -607,17 +649,9 @@ Manifest `schema_version` **3**. One entry, before and after: "label": "Qwen3 8B", "parameter_count": "8B", - // ── DERIVED. The script writes all of this. No human input, ever. ── - "artifacts": { - "llamacpp": { - "repo": "unsloth/Qwen3-8B-GGUF", - "file": "Qwen3-8B-Q4_K_M.gguf", - "quantization": "Q4_K_M", - "size_bytes": 5027784512, - "mmproj": null - } - }, - // Estimator inputs, read from the real GGUF header at authoring time. + // ── DERIVED, per model. The script writes all of this, no human input ever. + // Quantization-independent: measured, the architecture fields are + // identical across Q4_K_M, Q8_0 and f16 of the same model. "shape": { "architecture": "qwen3", "block_count": 36, @@ -628,21 +662,57 @@ Manifest `schema_version` **3**. One entry, before and after: "n_vocab": 151936 }, "capabilities": [], // user-facing only. Today: "vision" or nothing. + "decode_fraction": 1.0, // 1.0 dense; the active slice for MoE, computed + // from the header (see below) - // ── JUDGEMENT. Two fields, typed by a person. ── - "rank": 78, // our preference order for document Q&A. - // Not a benchmark. Never displayed. - "validated": true, // someone downloaded this exact file and ran it + // ── One entry per shipped build. v1 ships exactly one. + "variants": [ + { + // DERIVED + "repo": "unsloth/Qwen3-8B-GGUF", + "file": "Qwen3-8B-Q4_K_M.gguf", + "quantization": "Q4_K_M", + "size_bytes": 5027784512, + "mmproj": null, - "decode_fraction": 1.0 // 1.0 dense; the active slice for MoE, computed -} // from the header (see below) + // JUDGEMENT. Three fields, typed by a person, beside the build they judge. + "rank": 78, // preference order for document Q&A. + // Not a benchmark. Never displayed. + "rank_basis": "llmfit-1.1.11", // what produced it + "validated": true // someone ran THIS file + } + ] +} ``` -**The split is the point.** Everything above the line comes free from the GGUF -header and the Hugging Face listing; everything below it is three small decisions -a person makes in a minute. That is what keeps the manifest maintainable at -twenty entries instead of six — the tedious half scales automatically, the half -that needs a brain stays tiny. +**The split is the point.** Everything marked DERIVED comes free from the GGUF +header and the Hugging Face listing; everything marked JUDGEMENT is three small +decisions a person makes in a minute. That is what keeps the manifest +maintainable at twenty entries instead of six — the tedious half scales +automatically, the half that needs a brain stays tiny. + +**Why `rank` lives inside the variant.** Quality is a function of *(model, +quantization)*, not of the model alone: llmfit returns 75 · 78 · 81 · 82 · 83 for +Qwen3 8B at Q3_K_M · Q4_K_M · Q5_K_M · Q6_K · Q8_0 (see **Appendix**). A rank at +the top level therefore describes a build without naming it, and that is not +hypothetical — an earlier draft of this document carried five ranks taken at +Q8_0 while the manifest pinned Q4_K_M, and nothing in the schema could catch it. +Nested, the mismatch is **impossible to express**: the rank sits beside the +quantization it was measured at, and `test_curated_models.py` asserts the pair +mechanically instead of a person remembering a rule. + +`rank_basis` records what produced the number — `llmfit-` today, an +internal eval identifier later. It exists so a manifest part-way through that +migration is detectable, and so ranks from different bases are never compared. + +**`variants` is a list from the start, and v1 puts one thing in it.** Authoring +is unchanged: one pin, one rank, the same twenty minutes. The list costs nothing +now and is the only part of this schema that is expensive to add later, because +changing it means `schema_version: 4` and re-authoring every entry. It also +closes the gap named first under **The quantization ladder**: with two variants a +24 GB card and a 64 GB machine stop receiving the identical file. The ladder +sweep in **Authoring** already computes a rank for every rung and discards four +of five; this is where the rest would go. **`shape` is what makes the curated tier work offline.** Pricing a model means reading its GGUF header, and a curated entry is not downloaded yet — so without @@ -669,14 +739,20 @@ formula should land near 0.1, which is the range Hermes hand-authored (0.08 to prediction is wrong by an order of magnitude, which matters the moment the 80B-A3B-class rung lands. -`validated: true` means a person downloaded that exact file and ran it. False is -allowed and honest; it is not a gate. +`decode_fraction` stays **per model, not per variant**: it is an expert-to-total +byte ratio, so it barely moves with quantization. `shape` and `capabilities` are +per-model for the same reason — the architecture fields are identical across +quantizations, measured. -Dropped from v2: `minimum_fit` (a gate — only physics gates now), -`allowed_quantizations` (redundant once `artifacts.llamacpp` names the file), -`minimum_context` (the context ladder and the model's own `context_length` -replace it), and top-level `advisor_providers` (scoped an llmfit scan that no -longer runs). +`validated: true` means a person downloaded **that variant's** file and ran it. +False is allowed and honest; it is not a gate. Per-variant is the only place it +means anything precise once an entry ships more than one build. + +Dropped from v2: `artifacts` (replaced by `variants`, which can hold more than +one), `minimum_fit` (a gate — only physics gates now), `allowed_quantizations` +(redundant once a variant names its file), `minimum_context` (the model's own +`context_length` replaces it), and top-level `advisor_providers` (scoped an +llmfit scan that no longer runs). #### Authoring @@ -693,6 +769,7 @@ $ uv run scripts/refresh_curated_models.py \ Q3_K_M 88 · Q4_K_M 94 · Q5_K_M 96 · Q6_K 97 · Q8_0 97 pinned quant Q4_K_M found in ladder ✓ llmfit proposes rank 94 (would be highest in the list — currently 92) + writing variants[0]: Q4_K_M, rank 94, basis llmfit-1.1.11 MoE detected: 128 experts, 8 used → decode_fraction 0.09 ⚠ validated=false until you run this file. @@ -701,7 +778,7 @@ $ uv run scripts/refresh_curated_models.py \ ``` The person then downloads it, chats with it, confirms citations resolve, sets -`decode_fraction` and `validated: true`, and commits. Twenty minutes, most of it +`variants[0].validated: true`, and commits. Twenty minutes, most of it waiting for the download. ```text @@ -745,9 +822,16 @@ for budget in multiplicative_steps(lower, upper, ratio=1.01): --memory {budget} --ram 128G -n 20000 ladder[row.best_quant] = row.score_components.quality -rank_proposal = ladder[pinned_quant] # KeyError is a hard failure +variant.rank = ladder[pinned_quant] # KeyError is a hard failure +variant.rank_basis = f"llmfit-{llmfit_version}" ``` +Because the rank is written **into the variant**, it is stored beside the +quantization it was read at and cannot drift from it. The `best_quant` assertion +below is what catches a bad *read*; the schema is what prevents a bad *write*. +Adding a second variant later means indexing the same ladder at a second rung — +the sweep already computed it. + Three properties this needs, each learned from a measured failure: - **Multiplicative steps at 1 % or finer.** The window in which a given @@ -782,6 +866,12 @@ when someone adds a model, not when someone cuts a release. nothing else. It is never displayed, never compared outside the app, and never applied to a searched model. +It is a property of a **variant**, not of a model: the same model at two +quantizations is two builds that answer differently, and the schema stores the +number beside the build. Where this section says "a model's rank", read "the rank +of the variant under consideration". With one variant per entry, as v1 ships, the +two readings coincide. + Because it is only ever a sort key, it does not need to be a measurement — it needs to be an **order**. Hence `rank`, not `quality`: calling it quality would imply we measured something we did not. @@ -1084,6 +1174,11 @@ Curated sorts by `(fit state, -rank, model_id)` — fit coarsely, rank finely which is what today's `_sort_key` already does. Sorting by rank alone would put a `TOO_BIG` 32B at the top of an 8 GB machine's screen. +**One row per model, not per variant.** A model with two builds is one row; the +row takes the best state and rank among its variants, and the row's install +action uses that variant. Listing builds separately would show the same model +twice on a screen whose job is choosing a model. + The recommendation is the top row of the `FITS` bucket, subject to the speed floor. Search sorts by downloads. **Neither list displays a rank**; rank only orders curated rows and selects the @@ -1101,12 +1196,28 @@ Deleted: the collision-resolution block, `_placeholder_row()`, **Recommendation policy**, its own module and its own test file: ```text -resident = curated entries whose state is FITS -resident → max(rank, -size) reason: best-rank-resident -else → no recommendation; PARTIAL entries stay installable, +resident = (entry, variant) pairs whose state is FITS +resident → max(variant.rank, -variant.size_bytes) reason: best-rank-resident +else → no recommendation; PARTIAL pairs stay installable, just never starred ``` +```python +candidates = [(e, v) for e in curated for v in e.variants + if fit(e.shape, v.size_bytes).state is FitState.FITS] +pick = max(candidates, key=lambda ev: (ev[1].rank, -ev[1].size_bytes), + default=None) +``` + +**The policy ranges over builds, not models**, because that is what the user +installs and what `rank` describes. With v1's single variant per entry the +cross-product is the entry list and the behavior is identical — but writing it +this way now is the difference between adding a `Q6_K` build later as a manifest +edit and rewriting the policy, its test file and every fixture. A model whose +larger variant is `TOO_BIG` and whose smaller one `FITS` is still recommendable +on its smaller build; that falls out of the cross-product rather than needing a +special case. + **No speed gate in v1.** An earlier draft gated on predicted latency, which required two constants with nothing measured behind them. The policy is now "highest rank among the entries that fit entirely on the device", and the tie @@ -1372,9 +1483,20 @@ compliance. That plausibly collapses three tiers to two. all — asserted on the serialized response, so it cannot be reintroduced silently. - Manifest (extends `test_curated_models.py`, which already covers unique ids, - display metadata, and each rejection path): every entry has a `shape`, a - `rank` and a pinned `quantization`; every `shape.architecture` is - in the 152-name list; a curated row prices and badges with no network at all. + display metadata, and each rejection path): every entry has a `shape` and at + least one variant; every variant has a `quantization`, a `rank` and a + `rank_basis`; every `shape.architecture` is in the 152-name list; a curated + row prices and badges with no network at all. Note + `test_packaged_manifest_has_eight_unique_curated_models` asserts `len == 8` + and `advisor_providers == ["Alibaba"]` — both become wrong, and both are + updated in the same commit as the manifest, never before. +- **A model with two variants is one row**, taking the best state and rank among + them, and recommending the variant that produced them. Asserted with a fixture + carrying a `TOO_BIG` large build and a `FITS` small one: the row is + recommendable on the smaller build. +- **Ranks are never compared across `rank_basis` values.** A fixture mixing an + `llmfit-*` rank with an eval-derived one is rejected rather than silently + sorted. - **`rank` rises with parameter count within a family.** Today this passes trivially — the shipped six are one family and score `33 · 48 · 63 · 78 · 85 · 92` at the pinned Q4_K_M. The first failure is the signal to look, and it is expected: a From d8f7947265e42fb36724510f574118115d162abb Mon Sep 17 00:00:00 2001 From: Anish Sarkar <104695310+AnishSarkar22@users.noreply.github.com> Date: Sat, 19 Sep 2026 20:37:05 +0530 Subject: [PATCH 06/60] docs(spec): strip the superseded recommendation half out of 05a --- plans/community-local/00-umbrella-plan.md | 2 +- .../api/05a-model-recommendations.md | 538 ++++++------------ plans/community-local/api/05c-packaging.md | 6 +- .../api/07-llamacpp-runtime.md | 4 +- plans/community-local/frontend/01-shell.md | 4 +- .../community-local/frontend/05-install-ux.md | 6 +- 6 files changed, 170 insertions(+), 390 deletions(-) diff --git a/plans/community-local/00-umbrella-plan.md b/plans/community-local/00-umbrella-plan.md index f38445b609..bf37ed7549 100644 --- a/plans/community-local/00-umbrella-plan.md +++ b/plans/community-local/00-umbrella-plan.md @@ -105,7 +105,7 @@ Docker Compose, Postgres, Zero, Redis, Celery, LangGraph, git KB, scrapers, MCP, | **Retrieval** | **FTS5 + sqlite-vec hybrid, cosine rescore** | Semantic + keyword from day one: both legs widen recall, cosine orders. Embeddings on ingest must be queried properly — not keyword-only, not in-memory scan over BLOBs. | | **Embed provider** | Bundled bge-small-en-v1.5 int8, in-process on onnxruntime | 384-dim, ~66MB, runs offline on CPU with no model server. Docling parses, Chonkie chunks. Remote embedding is a later opt-in, not a launch dependency. | | **Generation architecture** *(Phase 7 supersedes the local half — [`api/07-llamacpp-runtime.md`](api/07-llamacpp-runtime.md): llama.cpp is the one local runtime, llmfit moves to build time, the catalog opens to every GGUF on Hugging Face)* | llmfit catalog/advisor + curated-model policy + runtime adapters; Ollama local, multiple OpenAI-compatible connections remote | [`api/05a-model-recommendations.md`](api/05a-model-recommendations.md) and [`api/05b-openai-compatible-connections.md`](api/05b-openai-compatible-connections.md). llmfit supplies broad model metadata, hardware detection, and fit estimates; it is not a `Generator`, installer, or inference runtime. Local adapters resolve trusted artifacts and Ollama is first. Remote endpoints are named connection instances with their own URL and optional key; models are discovered live, not synchronized into SQLite. `SelectedModel(role)` stores the provider, connection identity when remote, and exact model id. SurfSense selects endpoints but does not load-balance their replicas. | -| **llmfit integration** *(Phase 7: the binary stops shipping. llmfit runs in CI to rank the curated manifest; it is never a gate, a hardware source, or on a request path)* | **Pinned official binary, short-lived JSON CLI, normalized behind `ModelAdvisor`** | The API runs `llmfit --json system` and `llmfit --max-context 8192 --json fit`, caches one scan, and exposes only SurfSense DTOs. No fork, patch, Python import, or permanent llmfit server. CLI label/machine-code differences are normalized at one seam and fixture-tested before a version bump. Failure removes ranking, not installed-model selection or chat. | +| **llmfit integration** *(Phase 7: the binary stops shipping and **never runs in CI**. A person runs `scripts/refresh_curated_models.py` by hand a few times a year, llmfit proposes a rank, and the number is committed as source. It is never a gate, a hardware source, or on a request path)* | **Pinned official binary, short-lived JSON CLI, normalized behind `ModelAdvisor`** | The API runs `llmfit --json system` and `llmfit --max-context 8192 --json fit`, caches one scan, and exposes only SurfSense DTOs. No fork, patch, Python import, or permanent llmfit server. CLI label/machine-code differences are normalized at one seam and fixture-tested before a version bump. Failure removes ranking, not installed-model selection or chat. | | **Ollama runtime** *(**superseded by Phase 7** — [`api/07-llamacpp-runtime.md`](api/07-llamacpp-runtime.md) replaces it with `llama-server` in router mode: 11–31 MB instead of 501 MB, Vulkan on every platform off Apple Silicon, and no MLX until after launch)* | **Bundled as a supervised sidecar (packaged); dev uses the developer's own `ollama serve`** | Chat can't depend on a daemon the user may not have installed. The packaged app ships the standalone Ollama archive (`electron/scripts/fetch-ollama.mjs` stages it, electron-builder carries it in `resources/ollama`) and Electron runs it as a third sidecar on a chosen port, passing `SURFSENSE_LOCAL_OLLAMA_BASE_URL` to the API. Models aren't shipped — the user pulls into the writable data dir after install. The Python sidecars and Ollama share one supervisor (`electron/src/main/sidecars/`): the supervisor spawns/reaps, one spec file per sidecar carries its identity. Boot now starts four — `api`, `worker-ingest`, `worker-studio`, and Ollama when packaged — and `watchImageModel` adds a fifth, sd-server, later, because only the API knows which image model was chosen. Ollama is best-effort at boot (only the API gates the window; its state surfaces via `/llm/providers`). | | **Persistence** | SQLAlchemy 2.0 + Alembic, same as cloud | Models are the source of truth. `versions/` ships as PyInstaller data, resolved from the package's own `__file__` — de-risked in [`api/00-spike.md`](api/00-spike.md). | | **Migrations** | **Hand-written; autogenerate is off** | Autogenerate cannot see a rename — it emits drop + add, which deletes a column's data silently. The target database is one user's laptop, unbacked and uninspectable, so every revision is written and read by a person. `env.py` carries no `target_metadata`, so `--autogenerate` cannot be used by accident. Mature SQLite-backed apps make the same call — hand-written revisions throughout. | diff --git a/plans/community-local/api/05a-model-recommendations.md b/plans/community-local/api/05a-model-recommendations.md index 83192bd01b..a9721890e5 100644 --- a/plans/community-local/api/05a-model-recommendations.md +++ b/plans/community-local/api/05a-model-recommendations.md @@ -1,414 +1,196 @@ -# API — Phase 5: Hardware-aware model recommendations - -> **The runtime and manifest halves are superseded by Phase 7.** -> [`07-llamacpp-runtime.md`](07-llamacpp-runtime.md) replaces Ollama with -> `llama-server`, moves llmfit out of the app, and takes the manifest to -> `schema_version` **3** with a `variants` list. The `schema_version: 2` shape -> below describes what ships **today**, not what phase 7 writes. The catalog -> boundary, the normalized shapes and the frontend contract still govern. +# API — Phase 5: Model selection and prompt tiers + +> **The recommendation half of this phase is gone.** llmfit-in-the-app, the +> Ollama runtime adapter, the Curated/Explore partitioning and the catalog HTTP +> surface were all replaced by +> [`07-llamacpp-runtime.md`](07-llamacpp-runtime.md). What remains here still +> governs and is not superseded: **prompt tiers**, **model fingerprinting at +> selection time**, and **onboarding**. See **What moved to Phase 7** for the +> mapping, so nothing is lost by deletion. > -> Owns: `backend/modules/llm/recommendations/`, the llmfit adapter, the -> runtime-install boundary, and the normalized catalog API. Packaging: -> [`05c-packaging.md`](05c-packaging.md). Frontend: -> [`../frontend/05-install-ux.md`](../frontend/05-install-ux.md). Remote -> connections: [`05b-openai-compatible-connections.md`](05b-openai-compatible-connections.md). +> Owns: `modules/llm/profile/`, `modules/llm/prompting/`, the `selected_models` +> fingerprint columns, and the onboarding marker. Remote connections: +> [`05b-openai-compatible-connections.md`](05b-openai-compatible-connections.md). +> Packaging: [`05c-packaging.md`](05c-packaging.md). Frontend: +> [`../frontend/05-install-ux.md`](../frontend/05-install-ux.md). ## Goal -Show generation models the current computer can run, put exact configurations -tested by SurfSense first, and install the selected configuration with one -action. llmfit supplies catalog metadata, hardware detection, and fit estimates; -SurfSense owns support policy, artifact trust, installation, and inference. - -This phase does not change the bundled bge-small embedding model, its -384-dimensional index, ingestion, retrieval, or chat SSE. - -## Boundaries - -```text -llmfit catalog + hardware fit - │ - ▼ - LlmfitAdvisor adapter - │ normalized ScoredModel[] - ▼ - CatalogService + curated models JSON - │ - ▼ - runtime artifact resolution - ├── Ollama (v1) - └── llama.cpp (future) - │ - ▼ - Curated + Explore API -``` - -- **llmfit** answers which generation configurations fit this hardware. It is - not an inference runtime and is never forked or modified by SurfSense. -- **Curated models** name exact team-tested configurations. Family membership - alone never grants support. -- **Runtime adapters** answer whether a scored model has an artifact SurfSense - can install, then own download, storage, process lifecycle, and inference. -- **The frontend** renders the normalized SurfSense contract. It never parses - llmfit output or chooses an artifact URL. - -## llmfit integration contract - -Ship a pinned official llmfit binary and invoke it as a short-lived subprocess. -Do not add a permanent `llmfit serve` sidecar for a scan performed once per API -process. Keep the adapter boundary so it can move to llmfit's REST API later -without changing the domain service or frontend. - -Commands: - -```bash -llmfit --json system -llmfit --max-context 8192 --json fit -``` - -As built there is **no `-n` result cap** — the scan takes llmfit's whole -compatible set, which is what "without a result count limit" under Catalog -assembly asks for. `_fit()` appends `--providers ` when a provider filter -is passed, which is how the curated manifest's `advisor_providers` narrows a -targeted scan. - -The adapter accepts only the fields SurfSense uses: - -- `name` — canonical model id; -- `fit_level` and `score`; -- `runtime`, `run_mode`, and `best_quant`; -- `memory_required_gb`, `memory_available_gb`, and `utilization_pct`; -- `disk_size_gb`, `estimated_tps`, `prefill_tps`, and `ttft_ms`; -- `estimate_confidence` and `estimate_basis`; -- `effective_context_length`; -- `capability_ids` and `license`; -- `ollama_name` and `gguf_sources`. - -llmfit's CLI currently overloads some machine-code fields with human labels -while its REST API returns lowercase machine codes plus `*_label`. Normalize -both forms at the adapter seam (`"Good"` and `"good"` become `good`) and retain -unknown notes instead of failing the whole catalog. Pin the binary version and -keep representative JSON fixtures so an upgrade that changes this contract -fails tests. - -`provider` in llmfit output means the model publisher, not the SurfSense -inference provider. Name it `publisher` in SurfSense DTOs. `runtime` is an -llmfit execution estimate, not proof that SurfSense has an installer. - -Cache the successful system profile and scored catalog for the API process. -A manual refresh invalidates both and performs one new scan. Concurrent callers -share one in-flight scan. - -**Built one layer deeper than that.** The cache is on disk, at -`{data_dir}/llmfit-scan.json`, written atomically through a `.tmp` rename and -invalidated when either its `cache_version` or the recorded `llmfit_version` -stops matching — so a scan survives an app restart, not just the process. The -in-memory copy and its lock sit in front of it. The consequence that matters: -**`scan()` never spawns llmfit unless `refresh=True`.** Loading the catalog -page costs no hardware probe; only the explicit "Scan hardware" action does. - -## Domain contracts - -```python -class ModelAdvisor(Protocol): - async def scan(self, max_context: int) -> AdvisorCatalog: ... - - -class LocalRuntime(Generator, Protocol): - async def resolve(self, model: ScoredModel) -> InstallPlan | None: ... - async def installed_models(self) -> list[InstalledModel]: ... - def install(self, plan: InstallPlan) -> AsyncIterator[DownloadProgress]: ... -``` - -`Generator` remains the chat seam used by `modules/chat`. OpenAI-compatible -remote connections do not implement `LocalRuntime`: they expose already-served -models and cannot install or delete them. Do not add llmfit to the provider -registry: it advises; it never answers a chat request. - -The normalized `ScoredModel` carries a canonical id and fit metadata only. -`InstallPlan` carries the selected runtime, runtime-specific model id, expected -bytes, quantization, and trusted artifact identity. The plan is constructed on -the server and is never accepted from the renderer. - -### Ollama runtime, first implementation - -- Resolve only llmfit rows with a non-empty, known-good `ollama_name`. -- Reuse the existing `/api/tags`, `/api/show`, `/api/pull`, and `/api/chat` - integration. -- Score the artifact SurfSense will actually pull. If an Ollama tag does not - pin llmfit's `best_quant`, do not claim that quantization was installed. -- The packaged private Ollama remains authoritative for installed state and - model storage; llmfit's own installed flag is informational only. - -### llama.cpp, later - -> **Now.** [`07-llamacpp-runtime.md`](07-llamacpp-runtime.md) is this section, -> specified. It goes further than "adding a runtime": llama.cpp becomes the -> *only* local runtime, llmfit leaves the installer for CI, and the Explore -> bucket becomes a Hugging Face search over every GGUF rather than the subset -> llmfit happens to know. - -A future adapter resolves a canonical model and quantization to a verified GGUF, -downloads atomically, starts `llama-server`, and implements the existing -`Generator` stream. Adding it changes the runtime registry and packaging, not - the llmfit adapter, curated-model policy, catalog service, or frontend response shape. - -## Curated models manifest - -Package: - -```text -backend/modules/llm/recommendations/curated-models.json -``` - -Shape: - -```json -{ - "schema_version": 2, - "advisor_providers": ["Alibaba"], - "models": [ - { - "model_id": "Qwen/Qwen3-8B", - "family": "Qwen3", - "minimum_fit": "good", - "minimum_context": 8192, - "allowed_quantizations": ["Q4_K_M"], - "artifacts": { - "ollama": { - "name": "qwen3:8b", - "quantization": "Q4_K_M" - } - }, - "label": "Qwen3 8B", - "parameter_count": "8B", - "size_bytes": 5225374496 - } - ] -} -``` - -`schema_version` is **2**. Version 2 added the three display fields — `label`, -`parameter_count`, `size_bytes` — so a curated row can render before any -hardware scan has produced an estimate for it, and `advisor_providers` at the -top level to scope a targeted scan. The shipped manifest carries eight Qwen -entries. - -Entries are exact model configurations, even though the frontend groups them by -`family`. Prefer an immutable runtime artifact or digest when the runtime -supports one; a mutable family name is not a verification boundary. The -manifest contains no hardware score and is not copied into SQLite. - -## Catalog assembly - -SurfSense requests llmfit's complete compatible model set without a result -count limit. For each llmfit generation model: - -1. Normalize its identity and score. -2. Exclude embedding-only entries. -3. Ask each enabled local runtime to resolve an install plan. -4. Drop uninstalled entries with no resolvable trusted artifact. -5. Overlay the curated-model manifest. -6. Apply SurfSense's real context target and reserve memory for the OS, - Electron, API, worker, parser, and fixed embedding model. -7. Enforce one catalog row per exact `(runtime, runtime_model)` target. An - explicit curated mapping wins over an external mapping. When external names - collide, show one row named with the exact runtime target and use the most - conservative colliding hardware estimate instead of guessing a catalog - identity. Record collisions once per scan in developer logs; they are not a - user-actionable recommendation warning. -8. Partition and rank: - - **Curated** (called *Recommended* when this was written, renamed in the - code): every exact manifest entry; - - **Explore:** remaining installable `perfect`, `good`, or `marginal` - entries; - - **Installed:** always visible, including a warning when marginal or too - tight. - -Ranking is deterministic: fit class, llmfit score, then canonical id. Curated -rows are excluded from Explore, so nothing appears twice. - -**Two rules here were built the other way round, deliberately.** - -*A curated model that does not fit stays curated.* This spec had it move to -Explore with its fit warning. The code keeps it in its own section at whatever -fit the scan found, on the reasoning that a poor fit is better shown as an -honest badge than hidden among thousands of Explore rows — and that acting on -one is already gated by the marginal/too-tight install confirmation. The promise -this section was protecting still holds: team-tested is not a claim about every -computer, it is just made in the badge instead of in the bucket. - -*Neither Curated nor Installed needs a scan.* Steps 1-8 read as a pipeline that -begins with llmfit output, which would leave a fresh install looking at an empty -page until it ran a hardware probe. As built, Curated is populated straight from -the manifest — the display fields added in `schema_version` 2 are exactly what -makes a row renderable with no estimate — and Installed from the runtime's own -inventory. A manifest row that no scan has matched is still installable, because -resolving it needs only the `ollama_name` the manifest already pins, so a real -install plan is registered against the placeholder and `/install` behaves -identically. Rows with no estimate carry `fit: unknown` and null resource -fields, and the frontend suppresses the badge entirely until a scan has -happened rather than showing "unknown" to someone who never asked. Only Explore -and the fit badges require step 1. - -## HTTP contract - -### `GET /llm/system` - -Returns the normalized hardware profile, scan warnings, llmfit version, and -whether estimates are available. - -### `GET /llm/catalog` - -Returns: - -```json -{ - "hardware": {}, - "curated": [], - "explore": [], - "installed": [], - "warnings": [], - "scanned": false, - "llmfit_version": null, - "runtime_status": {} -} -``` - -The bucket is `curated`, not `recommended`. `scanned` tells the frontend whether -a hardware scan stands behind these rows, which is what lets it render curated -and installed models immediately and offer "Scan hardware" in place of Explore. - -Each catalog row includes an opaque `catalog_id`, canonical model id, family, -display label, fit, resource estimates, license, installed/selected state, -the chosen runtime name, and a server-derived `can_delete` capability. It does -not expose artifact URLs or local paths. -`?refresh=true` invalidates the cached hardware/catalog scan. - -### `POST /llm/install` - -Body: - -```json -{"catalog_id": "", "select": true} -``` - -The API re-runs server-side resolution against the current catalog, checks disk -space, and streams the existing newline-delimited progress shape. It rejects a -stale, unknown, unresolvable, or untrusted id before streaming. On success, -`select=true` updates `SelectedModel(GENERATION)` to the installed runtime id. -Partial artifacts never become selectable. - -Keep `GET /llm/providers`, local provider inventory, and -`GET/PUT /llm/selection/generation`. Remote inventory belongs to -`GET /llm/connections/{id}/models`; see the connection spec. The -provider-specific pull route may remain during migration, but the final -frontend calls the normalized install route. +Decide, once, at the moment a model is selected, **how that model should be +prompted** — and keep that decision durable, cheap to re-tune, and independent of +which runtime serves the model. -### `GET /llm/onboarding` - -Returns whether the user has completed model onboarding. +This phase does not choose models, score hardware, install anything, or render a +catalog. Phase 7 owns all of that. -### `POST /llm/onboarding` +It does not change the bundled bge-small embedding model, its 384-dimensional +index, ingestion, retrieval, or chat SSE. -Writes the durable completion marker. Requires a persisted generation -selection. Image is optional. Selecting or clearing a model never writes or -resets this marker. Settings Use actions must not call this route. +## What moved to Phase 7 -### `DELETE /llm/providers/{provider}/models/{model_name}` +Kept as a map rather than deleted silently, because the code implementing the +left column still exists until 7.6 removes it. -Deletes an installed local generation model through the runtime API. For -Ollama, the adapter calls `DELETE /api/delete` with the exact model name and -tag. The endpoint rejects remote providers, embedding-only models, active -downloads, active chat streams, and deletion while the separate Studio worker -is generating. If the deleted model is -selected, the same transaction clears `SelectedModel` and reports -`selection_cleared: true`. +| Was specified here | Now | +|---|---| +| llmfit as a packaged subprocess (`--json system`, `--json fit`) | **Not shipped.** Authoring-time only, on a maintainer's machine — [`07`](07-llamacpp-runtime.md) **Decisions**, and [`05c-packaging.md`](05c-packaging.md) | +| `LlmfitAdvisor`, the `ModelAdvisor` protocol, `AdvisorCatalog` | Deleted. Hardware comes from `ggml_backend_dev_memory()`, fit from the GGUF header — [`07`](07-llamacpp-runtime.md) 7.1–7.2 | +| `{data_dir}/llmfit-scan.json`, `CACHE_VERSION`, the `scanned` flag | Deleted. There is no scan and no scan button | +| Ollama runtime adapter, `ollama_name` resolution, `/api/pull` | `llama-server` router mode — [`07`](07-llamacpp-runtime.md) **Runtime contract**, 7.3 | +| Curated / Explore / Installed partitioning, the five-way `fit_level` | Two tiers (curated manifest, Hugging Face search) and three fit states — [`07`](07-llamacpp-runtime.md) 7.4 | +| `curated-models.json` `schema_version: 2`, `artifacts.ollama` | `schema_version: 3` with a `variants` list — [`07`](07-llamacpp-runtime.md) 7.4 | +| `GET /llm/system`, `GET /llm/catalog`, `POST /llm/install`, `DELETE /llm/providers/{provider}/models/{name}` | Respecified in [`07`](07-llamacpp-runtime.md) **HTTP routes** | +| Catalog failure behavior, catalog tests, catalog acceptance | [`07`](07-llamacpp-runtime.md) **Failure behavior**, **Tests**, **Acceptance** | + +Two pieces of this phase's reasoning survived the rewrite, and are worth naming +because phase 7 kept them rather than reinvented them: + +- **A model no scan has scored is still installable.** This phase argued a + curated row must render and install with no estimate behind it. Phase 7 goes + further: `shape` in the manifest prices every curated row offline, on first + paint, with no network at all. +- **Team-tested is not a claim about every computer.** This phase moved that + claim out of the bucket and into the badge. Phase 7 keeps the split exactly — + `rank` is a preference order over models we tested, the fit badge is + subtraction, and the two are never rendered as the same thing. ## Prompt tiers -Not in the original spec, and now depended on by chat and by every Studio -format: a selected model is **fingerprinted at selection time**, and its -fingerprint decides which of three prompts it is given. +The live core of this phase, depended on by [`03-chat.md`](03-chat.md) and +[`../worker/04-studio.md`](../worker/04-studio.md). + +A selected model is **fingerprinted at selection time**, and its fingerprint +decides which of three prompts it is given. + +### Classification -`modules/llm/profile/` owns the classification. A `Fingerprint` carries -`provider`, `name`, and three nullable facts — `params_b`, `vendor`, and `line` -(`flagship` or `small`) — collected from Ollama's `/api/tags` and `/api/show`, -from a remote connection's `/models`, or, failing both, from heuristics on the -name. `classify()` turns it into a `Tier`: +`modules/llm/profile/` owns it. A `Fingerprint` carries `provider`, `name`, and +three nullable facts — `params_b`, `vendor`, and `line` (`flagship` or `small`) — +collected from the runtime's own inventory, from a remote connection's +`/models`, or, failing both, from heuristics on the name. `classify()` turns it +into a `Tier`: | Known | Tier | |---|---| -| `params_b` < 7.0 | `compact` | -| `params_b` < 100.0 | `capable` | +| `params_b` < `COMPACT_MAX_B` (7.0) | `compact` | +| `params_b` < `CAPABLE_MAX_B` (100.0) | `capable` | | `params_b` ≥ 100.0 | `frontier` | | no count, but a `vendor` | `frontier` | | no count, `line` is flagship / small | `frontier` / `capable` | -| nothing, provider is Ollama | `compact` | -| nothing, remote | `capable` | +| nothing, and the endpoint is **local** | `compact` | +| nothing, and the endpoint is **remote** | `capable` | The thresholds encode a claim about scaffolding, not about quality: below the -first a model loses accuracy when asked to follow a structure, between the two -it gains from one, and above the second it writes better from judgement than -from steps. The last two rows are the same bet — a hosted endpoint runs models -too big for a laptop, Ollama runs the laptop. +first a model loses accuracy when asked to follow a structure, between the two it +gains from one, and above the second it writes better from judgement than from +steps. The last two rows are the same bet — a hosted endpoint runs models too big +for a laptop, a local one runs the laptop. + +> **The last two rows must key on loopback, not on a provider name.** As built, +> `classify()` reads `provider == "ollama" → COMPACT`, which breaks the moment a +> Mac user runs a 4B through LM Studio and gets `capable` prompts for a compact +> model. `host_destination()` already computes loopback and returns `None` for +> it. Phase 7 makes this change alongside the provider rename — +> [`07`](07-llamacpp-runtime.md) **Decisions**, *Prompt tier fallback*. Revision `0010` adds `params_b`, `vendor` and `line` to `selected_models`. **The tier itself is not stored**: `SelectedModel.tier` calls `classify()` on read, so retuning a threshold changes behaviour without a migration or a re-selection. +That is the property to preserve — it is what lets the two constants above move +on evidence rather than on a release. + +### Loading `modules/llm/prompting/` is the other half. `load(package, tier, case=None, -**slots)` reads `{package}/prompts/{tier}.md`, or `{package}/prompts/{case}/{tier}.md` -when a package has more than one prompt, and fills `$slot` placeholders through -`string.Template`; `focus()` appends the user's steer as one line. Every prompt -therefore lives as markdown beside the code that uses it — eleven cases at three -tiers, so 33 files: chat, the four `content/` formats, `office/`, `web/html/`, -the two `media/visual/` formats, and podcast outline and draft separately. A -unit test asserts every case ships all three. No `import` statement names any of -them, so the PyInstaller specs collect them by path, split along what each binary -runs: `api.spec` takes only `modules.chat`'s three, and `worker.spec` takes those -plus every `*.md` under `worker.studio`. +**slots)` reads `{package}/prompts/{tier}.md`, or +`{package}/prompts/{case}/{tier}.md` when a package has more than one prompt, and +fills `$slot` placeholders through `string.Template`; `focus()` appends the +user's steer as one line. + +Every prompt lives as markdown beside the code that uses it: **23 cases × 3 +tiers = 69 files**, covering chat, the `content/` formats, `office/`, +`web/html/`, the `media/visual/` formats, and podcast outline and draft +separately. A unit test asserts every case ships all three. + +No `import` statement names any of them, so the PyInstaller specs collect them by +path, split along what each binary runs: `api.spec` takes only `modules.chat`'s +three, and `worker.spec` takes those plus every `*.md` under `worker.studio`. + +> **What a tier carries will narrow.** [`07`](07-llamacpp-runtime.md) 7.8 +> introduces constrained decoding — 48 of the 69 prompts demand JSON, and +> `response_format: {"type": "json_schema"}` makes malformed output mechanically +> impossible. Once format compliance no longer depends on persuasion, a tier +> carries **reasoning depth only**, which plausibly collapses three tiers to two. +> Nothing measures that today, so the split stands. + +## Onboarding + +Unchanged by phase 7, which lists `GET|POST /llm/onboarding` among the routes it +does not touch. + +### `GET /llm/onboarding` + +Returns whether the user has completed model onboarding. + +### `POST /llm/onboarding` + +Writes the durable completion marker. Requires a persisted generation selection. +Image is optional. + +Two invariants worth restating, because both are easy to break from the +frontend: selecting or clearing a model **never** writes or resets this marker, +and Settings *Use* actions must not call this route. + +## Selection storage + +`SelectedModel(role)` stores the provider, the connection identity when remote, +the exact model id, and the three fingerprint facts. One row per role, so the +role is the primary key and choosing again updates in place. + +- `provider` is the SurfSense inference provider, never the model publisher. + llmfit's output overloaded that word; SurfSense DTOs say `publisher` for the + other meaning. The distinction outlives llmfit and is worth keeping. +- Remote inventory is discovered live and never synchronized into SQLite — see + [`05b-openai-compatible-connections.md`](05b-openai-compatible-connections.md). +- `"ollama"` becomes `"llamacpp"` in revision `0012`, which also clears + selections pointing at weights the app no longer manages — + [`07`](07-llamacpp-runtime.md) 7.6. ## Failure behavior -- llmfit missing, timed out, or malformed: return installed runtime models and - a recommendation warning; do not prevent app startup or chat with an already - selected model. -- Incomplete hardware detection: return the profile plus warnings and avoid a - false `perfect` claim. -- Runtime unavailable: keep catalog information visible and disable installs - for that runtime. -- No trusted runtime artifact: omit Download; never pass through a guessed URL. -- Insufficient disk: reject before download with required and available bytes. -- Cancelled/interrupted pull: clean temporary state where the runtime permits - it and never auto-select the incomplete model. -- Deletion never edits runtime storage directly and never silently selects a - replacement model. +Narrowed to what this phase still owns. + +- **No fingerprint facts at all**: `classify()` falls through to the loopback + rule rather than raising. A wrong tier degrades prompt quality; it never fails + a request. +- **A prompt file missing for a tier**: a packaging failure, caught by the unit + test asserting all three exist per case, not handled at runtime. +- **Onboarding marker written without a selection**: rejected. The marker means + the user finished choosing, and it is the one thing that must not become true + early. ## Tests -- Adapter fixtures cover system JSON, fit JSON, labels versus machine codes, - nullable estimates, unknown fields, timeout, nonzero exit, and malformed JSON. -- Catalog policy covers Curated/Explore partitioning, family grouping metadata, - deterministic ranking, deduplication, memory reserve, and installed models - that no longer fit. Also: a catalog served with no scan at all, curated - placeholders staying installable, and the scan cache surviving a restart. -- Runtime resolution covers valid/absent Ollama mappings and prevents an - arbitrary renderer-supplied tag or URL. -- Integration covers catalog → install progress → installed inventory → - selection, with fake llmfit output and a stub runtime. -- Deletion coverage includes exact Ollama payloads, remote/capability rejection, - active-operation conflicts, and selected-model cleanup. -- Packaging smoke runs `llmfit --json system` on each clean target OS. +- `classify()` across every row of the table above, including both endpoint + rows, asserting a small local model served through a non-Ollama local endpoint + is still `compact` once the loopback keying lands. +- `SelectedModel.tier` recomputes on read: changing `COMPACT_MAX_B` in a test + changes the tier of an already-persisted row, with no migration. +- Every one of the 23 cases ships all three tier files. +- `load()` fills `$slot` placeholders; `focus()` appends exactly one line. +- Onboarding: the marker requires a persisted generation selection, and + selecting or clearing a model leaves it untouched. +- Packaging: `api.spec` carries `modules.chat`'s three prompts and `worker.spec` + carries every `*.md` under `worker.studio`, asserted on the frozen binary + rather than the source tree. ## Acceptance -- A clean supported machine sees Curated models followed by Explore, with Curated - present before any hardware scan has run. -- Every enabled Download action resolves to a runtime artifact SurfSense can - install. -- One Download & Use action installs and selects a model. -- llmfit being unavailable degrades recommendations without breaking installed - model selection or chat. -- Adding a fake llama.cpp runtime in tests requires no change to llmfit parsing, - Curated partitioning, or the frontend catalog schema. +- Selecting a model records its fingerprint, and chat and every Studio format + receive the tier that fingerprint implies. +- Retuning a tier threshold changes behaviour on the next request, with no + migration and no re-selection. +- A local endpoint that is not Ollama receives `compact` prompts for a small + model. +- Onboarding completes only with a generation selection persisted, and never + resets when the selection changes. diff --git a/plans/community-local/api/05c-packaging.md b/plans/community-local/api/05c-packaging.md index 58ef00f5a4..788698e910 100644 --- a/plans/community-local/api/05c-packaging.md +++ b/plans/community-local/api/05c-packaging.md @@ -21,8 +21,7 @@ Installers for Paths A / B / C. pointed at them; `scripts/fetch_embedding_model.py` fetches the same files for CI. - A pinned `llama-server` build plus the versioned curated-model manifest; catalog and install routes are specified in - [`07-llamacpp-runtime.md`](07-llamacpp-runtime.md) (which supersedes the - route table in [`05a-model-recommendations.md`](05a-model-recommendations.md)). + [`07-llamacpp-runtime.md`](07-llamacpp-runtime.md). ## Files no import statement names @@ -143,8 +142,7 @@ user finds by search. ### First-run model download The generation model arrives in first-run setup with progress through the -normalized catalog boundary -([`05a-model-recommendations.md`](05a-model-recommendations.md)): +catalog boundary ([`07-llamacpp-runtime.md`](07-llamacpp-runtime.md)): - `GET /llm/system` — the hardware budget, from the runtime's allocator. No scan, no status flag, no button. diff --git a/plans/community-local/api/07-llamacpp-runtime.md b/plans/community-local/api/07-llamacpp-runtime.md index c89804497d..9d5ad2eeef 100644 --- a/plans/community-local/api/07-llamacpp-runtime.md +++ b/plans/community-local/api/07-llamacpp-runtime.md @@ -4,8 +4,8 @@ > packaging for all three targets. Supersedes the Ollama runtime decision in > [`../00-umbrella-plan.md`](../00-umbrella-plan.md) and the `hf.co/` fallback in > [`05d-llmfit-catalog-expansion.md`](05d-llmfit-catalog-expansion.md). Extends -> [`05a-model-recommendations.md`](05a-model-recommendations.md) — its -> "llama.cpp, later" section is now this document. +> [`05a-model-recommendations.md`](05a-model-recommendations.md), which is +> reduced to what survives here: prompt tiers, fingerprinting and onboarding. ## Goal diff --git a/plans/community-local/frontend/01-shell.md b/plans/community-local/frontend/01-shell.md index 2c072ffbf0..1f051eb6e6 100644 --- a/plans/community-local/frontend/01-shell.md +++ b/plans/community-local/frontend/01-shell.md @@ -111,5 +111,5 @@ installed-state and validated-selection path. Workspace routes — [`../api/01-skeleton.md`](../api/01-skeleton.md). Model provider inventory and validated generation selection — -[`../api/03-chat.md`](../api/03-chat.md). Ranked catalog and installation — -[`../api/05a-model-recommendations.md`](../api/05a-model-recommendations.md). +[`../api/03-chat.md`](../api/03-chat.md). Catalog and installation — +[`../api/07-llamacpp-runtime.md`](../api/07-llamacpp-runtime.md). diff --git a/plans/community-local/frontend/05-install-ux.md b/plans/community-local/frontend/05-install-ux.md index 598d059451..921719df92 100644 --- a/plans/community-local/frontend/05-install-ux.md +++ b/plans/community-local/frontend/05-install-ux.md @@ -2,7 +2,7 @@ > Owns: the generation catalog and one-click install flow. > API contract: -> [`../api/05a-model-recommendations.md`](../api/05a-model-recommendations.md). +> [`../api/07-llamacpp-runtime.md`](../api/07-llamacpp-runtime.md). > > **Phase 7 supersedes the catalog half of this document** — > [`../api/07-llamacpp-runtime.md`](../api/07-llamacpp-runtime.md) replaces @@ -269,7 +269,7 @@ branch rendering on a provider name. Fit-badge copy branches on the budget's ## Needs from API -Normalized catalog and install stream — -[`../api/05a-model-recommendations.md`](../api/05a-model-recommendations.md). +Catalog, search and install stream — +[`../api/07-llamacpp-runtime.md`](../api/07-llamacpp-runtime.md). Binary/model packaging and airgap imports — [`../api/05c-packaging.md`](../api/05c-packaging.md). From 7cac4f351db51b3d9ea87cbbcf1fed3ce6d0a0cb Mon Sep 17 00:00:00 2001 From: Anish Sarkar <104695310+AnishSarkar22@users.noreply.github.com> Date: Sat, 19 Sep 2026 22:03:41 +0530 Subject: [PATCH 07/60] feat(spec): gate recommendations on speed, pick KV precision by fit --- .../api/07-llamacpp-runtime.md | 521 ++++++++++++++++-- plans/community-local/api/08-cuda-backend.md | 22 + .../community-local/frontend/05-install-ux.md | 14 +- 3 files changed, 503 insertions(+), 54 deletions(-) diff --git a/plans/community-local/api/07-llamacpp-runtime.md b/plans/community-local/api/07-llamacpp-runtime.md index 9d5ad2eeef..85abdbc3fb 100644 --- a/plans/community-local/api/07-llamacpp-runtime.md +++ b/plans/community-local/api/07-llamacpp-runtime.md @@ -37,13 +37,13 @@ Ollama's native API does not. | **Runtime** | `llama-server` router mode, single sidecar | `--models-dir`, `--models-max 1`, LRU eviction, one PID for the supervisor to reap. Maps 1:1 onto every Ollama call the adapter makes. | | **MLX** | **Not shipped.** Revisit after launch | Ollama switched its Apple Silicon engine to MLX in Mar 2026, so the swap costs Mac speed on models under ~14B. MLX's format covers 23,985 HF repos against GGUF's 204,797 — one format and the full catalog wins. Mac users who want MLX point a connection at LM Studio; see **The Mac path**. | | **llmfit** | **Authoring-time only. Not shipped, and not in CI.** | A person runs `scripts/refresh_curated_models.py` when adding or changing a curated entry — a few times a year — and commits the numbers. llmfit is not a gate, not a hardware source, not in any request path, and not in the build. Removes 27 MB, a pinned version, and five failure modes per installer. | -| **Gating** | Only physics refuses | `can_install = state != TOO_BIG`, and `TOO_BIG` *is* the physics refusal — weights + KV at the 16K floor exceeding VRAM **plus** RAM. No other fit level gates anything; `PARTIAL` installs like `FITS`. Eligibility (architecture, chat template, gated repo) blocks separately and is not a fit state. See **Fit states**. | +| **Gating** | Only physics refuses | `can_install = state != TOO_BIG`, and `TOO_BIG` *is* the physics refusal — `need` at the 16K floor exceeding usable VRAM **plus** RAM. No other fit level gates anything; `PARTIAL` installs like `FITS`. Eligibility (architecture, chat template, gated repo) blocks separately and is not a fit state. See **Fit states**. | | **Catalog** | Two tiers — curated manifest, and Hugging Face search | Curated is the offline product and ships frozen. Search is a network feature that is simply absent airgapped. | | **Fit estimate** | Our own, from the GGUF header | llmfit cannot score a model outside its database (`llmfit plan` states the precondition). Search needs an estimate anyway; once it exists, llmfit's is redundant *and* less accurate here. | | **`rank`** | **Curated entries only. A searched model never carries one, from llmfit or anywhere else** | A preference order over models we tested, **for this app's job** — answering from the user's documents with citations that resolve — not general capability. An integer typed by a person, with llmfit proposing. Called `rank`, not `quality`, because it is only ever a sort key: nothing reads its magnitude, so it must not imply a measurement we do not have. Attaching one to an arbitrary repo attaches a base model's score to a derivative that behaves differently: of the top 100 GGUF repos by downloads, 38 match an llmfit entry, and those matches include `Huihui-Qwen3.8-27B-abliterated-GGUF` and `Qwen3.8-27B-Uncensored-GGUF` resolving to the base model's score. A wrong number people trust is worse than a blank they investigate. Search rows are **described, not judged**. | | **Search order** | `sort=downloads`, descending | The only HF sort that behaves as a default. Presented as a popularity fact, never as an endorsement. See **Search ordering**. | | **Fit badge** | **Every row, both tiers** | Fit is subtraction, not judgement — the opposite of `rank`. Coarse from file size in the search list, exact from the header on open, exact from `shape` offline for curated. Three states, no `unknown`. | -| **Recommendation** | Highest `rank` among `FITS` entries. **No speed gate in v1** | A latency gate needs a prefill model, and llmfit's `prefill_tps` is an estimate for a generic machine. Rather than invent a threshold, v1 starves the gate: the ★ is the best-ranked entry that fits entirely on the device, tie-broken toward the smaller build. Additive once `llama-bench pp512` numbers exist per device class. | +| **Recommendation** | Highest `rank` among entries predicted **fast enough**, not among entries that fit entirely | Residency is a mechanism; speed is the goal, and on small cards they disagree. Measured on an RTX 3050: Qwen3 4B spills ~20% and Qwen3 8B ~28%, and a user runs the 8B without noticing. A `FITS`-only rule stars a 1.7B on that machine while the owner is happily using the 8B. v1 approximates the gate with an offload-fraction ceiling; the roofline and its self-calibration follow. See **Recommendation policy**. | | **Curated manifest** | **Stays, schema 3, authored by hand** | It is the whole product airgapped, the first-run default, and the only tier the recommendation reads. `scripts/refresh_curated_models.py` writes it; a person commits it; **llmfit never runs in CI**. | | **Why `rank` can be authored at all** | Quality is a function of *(model, quantization)*, not of hardware | Measured twice: across 2,424 models at a fixed quant, quality differs 0/2424; and on one model swept 12G to 48G, quality does not move once `best_quant` stops moving. Hardware reaches quality through exactly one channel, the quantization, and pinning the file closes it. So the number is a property of a file, computable once and shipped as data, and a scan on the user's machine has nothing to discover. This is the load-bearing fact under the whole authoring design. See **Appendix**. | | **Which llmfit field** | `score_components.quality` only. **`score` is discarded** | `score` blends fit, speed and context into quality, so three of its four terms describe a machine. In the same sweep it runs 65.1, 63.8, 63.6, 66.2, 63.6, 67.4 — non-monotonic and hardware-dependent. The composite is a statement about a laptop; the component is a statement about a file. Same reason `fit_level`, `memory_required_gb`, `estimated_tps` and `best_quant` are all read and thrown away. | @@ -52,6 +52,7 @@ Ollama's native API does not. | **Hardware budget** | `ggml_backend_dev_memory()` via `ctypes`, from the shipped libs | The allocator's own view. Falls back to `llama-server --list-devices`, then OS APIs. | | **Downloads** | **SurfSense fetches the GGUF**, not `POST /models` | `llama-server` is a second process we do not proxy; an in-process fetch is the only place `egress.require()` actually holds. Also buys resume, checksums, and the header as the file lands. | | **Quantization** | **One pinned file per entry in v1** | A pinned file is what "tested by SurfSense" can honestly claim. Per-machine selection is possible at no extra cost (one header read prices every quant), so a second variant on the largest entries is a cheap follow-on, not v1. See **The quantization ladder**. | +| **KV precision** | **`f16` when it fits, `q8_0` when it buys residency.** Symmetric `-ctk`/`-ctv` always | Measured: `f16` KV at a 16K window **cannot allocate** on a 6 GB card (`ErrorOutOfDeviceMemory`), while `q8_0` at the same depth runs. Quality is lossless either way. So the choice is not a global default but an output of the fit calculation — pay the precision cost only where it converts a spill into residency. Requires flash attention; see **Failure behavior**. | | **Context** | **Fixed at load. Floor 16K, capped at the model's own `context_length`** | llama.cpp fixes context at load, so `num_ctx()`'s per-request sizing has no equivalent. The floor is 3K history plus ~8K grounding plus a reply. Growing on occupancy is additive later; it needs a mid-conversation reload and two constants with nothing measured behind them. | | **GPU backend** | **Vulkan on every platform off Apple Silicon. No CUDA** | Measured on an RTX 3050 at `b11050`: CUDA leads Vulkan **9.1%** on `pp512`, **10.2%** on `pp8192`, **2.1%** on decode — 0.66 s on an 11 s turn, for 685 MB. Vulkan covers NVIDIA, AMD and Intel from one 31 MB archive, its loader ships with Windows, and it is what the app already does today (`pruneCudaRunners()`). CUDA is specified as an optional later addition in [`08-cuda-backend.md`](08-cuda-backend.md), which needs **no code change** — ggml selects it by the files present. | | **Device selection** | First device with `type == GPU`. **Never sum** | The same physical card appears once per loaded backend, and an integrated GPU can advertise more memory than a discrete one (16198 MiB against 6002 MiB, measured). ggml's ordering already expresses backend preference, so this **is** backend selection. Specified in **7.2**. | @@ -67,7 +68,8 @@ Ollama's native API does not. │ ▼ FitEstimator ── advisory only - │ weights + KV(window) + overhead vs VRAM + RAM + │ need = weights + KV(window) + compute_buffers + │ usable = device_free − fit_reserve │ → FITS | PARTIAL | TOO_BIG (TOO_BIG = the physics refusal) ▼ CatalogService @@ -118,12 +120,31 @@ The shipped six, measured at Q4_K_M and a 16K window: | Qwen3 14B | 9.00 | 40 | 8 | 10,969 | | Qwen3 32B | 19.76 | 64 | 8 | 22,047 | +> **This column excludes `compute_buffers` and predates the fit reserve**, so it +> under-states `need` by roughly 170 MiB at 16K and prices against raw free +> memory. `GPU` / `RAM` in the map below is therefore a simplification — the real +> output is an offload fraction, and the map shows it where measured. +> +> **The one row since measured came out a state worse than predicted, and it is +> the row that rewrote the recommendation policy.** Qwen3 4B on an RTX 3050 needs +> 4,858 MiB against `5,234 − 1,028 = 4,206 MiB` usable: `PARTIAL` at `f ≈ 0.20`, +> not `FITS` with 490 MiB spare. llama.cpp spilled 602 MiB of weights and 320 MiB +> of KV. Under a `FITS`-only rule that machine's ★ fell to **Qwen3 1.7B** — while +> its owner runs **Qwen3 8B** (`f ≈ 0.28`, computed) without noticeable lag. That +> contradiction is what moved the gate from residency to speed; see +> **Recommendation policy**. +> +> The other five rows are unverified predictions and are **not** re-derived here: +> applying a reserve measured once on a 6 GB Windows card to an M4 Max would be +> inventing numbers, which is the failure this phase exists to avoid. Re-derive +> each row when its hardware is measured. + What each machine gets: ```text machine 0.6B 1.7B 4B 8B 14B 32B → recommended M2 8 GB GPU GPU GPU RAM RAM — → Qwen3 4B (831 MiB spare) -RTX 3050 6 GB GPU GPU GPU RAM RAM RAM → Qwen3 4B (490 MiB spare) +RTX 3050 6 GB GPU GPU ~20% ~28% RAM RAM → Qwen3 8B (f ≈ 0.28) M4 16 GB GPU GPU GPU GPU RAM RAM → Qwen3 8B (3,857 MiB spare) RTX 4070 12 GB GPU GPU GPU GPU GPU RAM → Qwen3 14B (31 MiB spare) RTX 4090 24 GB GPU GPU GPU GPU GPU GPU → Qwen3 32B (953 MiB spare) @@ -187,6 +208,31 @@ path now, not dead code. Sidecar flags: `--models-dir /models --port --models-max 1 --sleep-idle-seconds 300 --no-ui --jinja --reasoning-format deepseek`. +Per-model at load, decided by the fit calculation rather than fixed here: +`-fa on` and, when `q8_0` is chosen, `-ctk q8_0 -ctv q8_0`. **Set both cache +types or neither** — symmetric quantization enables the fused flash-attention +kernel, while a mismatched pair falls back to an unoptimised path that exists +for correctness only. + +`--sleep-idle-seconds` is not optional: measured at `b11050`, a model without it +self-evicted after roughly 30 s idle, which turns the second question of a +conversation into a reload. + +> **Never pass an explicit `-ngl`.** `--fit` owns layer placement, and setting +> `n_gpu_layers` by hand **disables it**: +> +> ``` +> common_fit_params: failed to fit params to free device memory: +> n_gpu_layers already set by user to 99, abort +> ``` +> +> Measured at `b11050`: with `-ngl 99` the model then loaded **entirely on the +> CPU** — 478 MiB of VRAM touched on a machine with a working RTX 3050, no error, +> exit 0. It looks like it worked. This is the same silent-GPU-blindness class as +> the probe search path (7.2) and the missing-backend case in **Failure +> behavior**, and it is the easiest of the three to introduce by accident, since +> `-ngl 99` reads as "use the GPU harder". + `--no-ui` because llama-server ships its own web UI. `--reasoning-format deepseek` routes `` blocks to `message.reasoning_content`; without it a thinking model's trace enters `parts[]` and `resolve_citations()` rewrites `[n]` tokens @@ -236,9 +282,13 @@ class ModelShape: @dataclass(frozen=True) class HardwareBudget: - usable_vram_bytes: int # capacity (total − margin) or live (free now) + """One device, never a sum across devices. See 7.2 — device selection.""" + usable_vram_bytes: int # device_free − fit_reserve. NOT raw free: llama.cpp's + # own fitter refuses to use the last slice, measured + # at 1,028 MiB of 5,234 free on an RTX 3050. total_device_bytes: int - ram_available_bytes: int + fit_reserve_bytes: int # what was subtracted, carried so the badge can explain + ram_available_bytes: int # from OS APIs, not the ggml CPU device (see 7.2) uma: bool # unified memory: selects badge copy and headroom has_gpu: bool # False when --list-devices prints "(none)" @@ -252,8 +302,13 @@ class FitState(StrEnum): @dataclass(frozen=True) class FitVerdict: state: FitState - need_bytes: int # weights + KV(window) + overhead - budget_bytes: int # vram, or vram + ram for the TOO_BIG comparison + need_bytes: int # weights + KV(window) + compute_buffers + budget_bytes: int # usable_vram, or usable_vram + ram for TOO_BIG + offload_fraction: float # 0.0 fully resident … 1.0 all on the CPU. + # The recommendation gates on this, and the + # badge's reason line is graded by it. Do not + # collapse it into `state` — three buckets lose + # the difference between 5% and 70% spilled. approximate: bool = False # priced from file size alone, header not read yet @@ -335,20 +390,23 @@ is derived from the codebase rather than chosen. | Constant | Kind | Basis | |---|---|---| | **Context floor, 16K** | **runtime** | `HISTORY_BUDGET_TOKENS = 3000` plus ~8,000 tokens of grounding (~24k characters from `build_context`) plus a reply. Grounded, not chosen. | -| **Runtime overhead** | prediction | Pre-model half **measured at 986 MiB** on a discrete card; the per-model half is a bootstrap value, replaced by the machine's own measurement after its first load. See below and **7.2**. | - -> **ponytail: runtime overhead.** The `overhead` term in -> `need = weights + KV(window) + overhead` is a placeholder. Hermes uses -> `max(2 GiB, 9% of total)` (`_MARGIN_FLOOR = 2 << 30`, `_MARGIN_FRACTION = 0.09` -> in `local_runtime/hardware.py`), which on a 6 GB card holds back a third of it; -> today's `recommendation_reserve_gb: 2.0` is self-flagged as uncalibrated. On a -> 5.4 GB budget the difference between 1.0 and 2.0 GiB moves rows across the -> `FITS`/`PARTIAL` line, and on a 12 GB -> NVIDIA card it decides whether Qwen3 14B is recommended at all (31 MiB of -> margin at 1,024 MiB overhead). **Measure real RSS after loading three or four -> models on each target before 7.4 ships**, and ship that as the bootstrap -> value — but it is wrong at most once per machine, because 7.2 records what the -> load actually cost and prefers that number afterwards. +| **Fit reserve** | prediction | **Measured: 1,028 MiB** on a Windows / Vulkan / discrete 6 GB card. A per-platform bootstrap, replaced by the machine's own figure after its first load. Two rows still blank. See **7.2**. | +| **Compute buffers** | prediction | **Measured: ~170 MiB** for a 4B at 16K. Part of `need`, scales with context and batch. | + +> **No longer a ponytail, with one caveat.** The reserve was an unmeasured +> placeholder at ~1,024 MiB. Measured on an RTX 3050, `--fit` holds back +> **1,028.34 MiB** of 5,234 free — the guess was within 4.3 MiB. Hermes' +> equivalent is `max(2 GiB, 9% of total)` (`_MARGIN_FLOOR = 2 << 30`, +> `_MARGIN_FRACTION = 0.09` in `local_runtime/hardware.py`), roughly twice as +> conservative. +> +> **The caveat is generality, not accuracy.** 1,028 MiB is 17% of a 6 GB card, so +> that single point cannot distinguish a constant reserve from a proportional +> one, and the two diverge badly at 24 GB. Metal and Linux are unmeasured and the +> numbers certainly do not carry — Metal has no separate VRAM at all. This is why +> 7.2 stores the reserve as a `(platform, backend)` table with blanks rather than +> a scalar, and why an unmeasured row rounds **up**: over-estimating costs a +> pessimistic badge on one screen, under-estimating ships the llmfit failure. #### Not in v1, and why @@ -360,11 +418,11 @@ from a memory figure that was not true on that machine. | Constant | What it would enable | Why not now | What it needs first | |---|---|---|---| -| `felt_cost` | a formula predicting how slow a model *feels* on this machine | the recommendation is "highest `rank` that fits"; no latency judgement is made | a prefill model — `llama-bench pp512` per device class. llmfit's `prefill_tps` is an estimate for a generic machine. | -| **Speed floor** | refusing to star a model that fits but crawls | same | a `felt_cost` that means something, plus a threshold from real sessions rather than copied from an agent app | +| `felt_cost` | a time estimate replacing v1's crude `MAX_OFFLOAD` ceiling | the roofline is specified under **Recommendation policy**; v1 ships the ceiling because it fixes the observed failure with one number | its constants, which come from completed turns rather than a benchmark — see **Where the constants come from** | +| **Speed floor** | a threshold in seconds rather than in spilled fraction | same | a `felt_cost` that means something, plus a threshold from real sessions rather than copied from an agent app | | **Ladder rungs** | starting narrow so a **bigger model stays GPU-resident**, widening only when a conversation needs it. The benefit is memory, not long chats: Qwen3 4B on an 8 GB M2 is `FITS` at 16K and `PARTIAL` at 40K, purely from KV cache. | context is fixed at load | evidence that mid-conversation reloads are worth the complexity for document Q&A | | **Growth threshold** | when to step up a rung | same | measured occupancy over real sessions | -| `decode_fraction` | correct speed prediction for mixture-of-experts models | **nothing to apply it to** — no MoE entry ships in v1. The value itself is computable today, from `expert_used_count` and the tensor table. | only the 80B-A3B-class rung | +| `decode_fraction` | correct speed prediction, and the only way an MoE is priced sanely | it is an **input to the speed model**, which v1 approximates with `MAX_OFFLOAD`. Compute and commit it now regardless: it is free from the header, and `1.0` for every dense entry shipped. | nothing — this one is ready | #### When they come back @@ -519,10 +577,75 @@ Two budget modes. **Capacity** (total − margin) for catalog pricing; **live** (free now) for launch decisions. Pricing against live-free while a model is loaded makes every row read as too large. -`modules/llm/fit.py` — `weights + KV(window) + overhead` vs `VRAM + RAM`. -Per-layer KV from the header. Split today's single `recommendation_reserve_gb: 2.0` -(flagged `ponytail` as uncalibrated) into a measured runtime overhead, a device -margin, and a UMA headroom. +#### Two subtractions, on opposite sides of the comparison + +`modules/llm/fit.py`. The single `overhead` term earlier drafts carried is **two +different quantities**, and collapsing them is what made the first prediction of +this wrong by 922 MiB: + +```text +need = weights + KV(window) + compute_buffers what the model allocates +usable = device_free − fit_reserve what --fit will actually use +FITS iff need ≤ usable +``` + +Measured, Qwen3 4B Q4_K_M at 16K on an RTX 3050 with 5,234 MiB free: + +| | MiB | +|---|---| +| `Vulkan0` model buffer | 2078.04 | +| `Vulkan0` KV buffer | 1984.00 | +| `Vulkan0` compute buffer | 143.62 | +| **placed on device** | **4205.66** | +| **left deliberately unused** | **1028.34** | + +Treating that 1,028 MiB as part of `need` rather than as a subtraction from +available memory predicts `FITS` with 378 MiB spare. llama.cpp instead spilled +602 MiB of weights and 320 MiB of KV to the CPU — a `PARTIAL`. Same numbers, +opposite verdict, purely from which side of the comparison the term sits on. + +`compute_buffers` was absent from the formula entirely. It is ~170 MiB for a 4B +at 16K (143.62 device + 26.01 host) and scales with context and batch, so it is +not a rounding error on a 6 GB card. + +This replaces today's single `recommendation_reserve_gb: 2.0`, which was flagged +`ponytail` as uncalibrated. + +#### The fit reserve is a table, not a number + +Keyed by `(platform, backend, memory model)`, because none of it transfers: Metal +has no separate VRAM at all, and a CUDA context alone costs ~541 MiB against +Vulkan's much lighter footprint. + +| platform / backend | reserve | basis | +|---|---|---| +| windows / vulkan / discrete | **1028 MiB** | measured, `b11050`, RTX 3050 6 GB | +| linux / vulkan / discrete | — | unmeasured | +| macos / metal / uma | — | unmeasured | +| no GPU device | n/a | `PARTIAL` is unreachable; see **Fit states** | + +> **6 GB cannot distinguish a constant from a proportion.** 1,028 MiB is 17% of +> that card; Hermes uses `max(2 GiB, 9%)`. One measurement cannot tell the two +> models apart, and they diverge badly on a 24 GB card. Record the figure with +> its device, and do not generalise from it. + +**Why shipping with two blank rows is safe.** An unmeasured row falls back to a +deliberately generous default, because the errors are asymmetric: + +- **reserve too large** → predicts `PARTIAL` where reality is `FITS`. The user + sees *Reduced speed*, installs anyway (`PARTIAL` installs exactly like + `FITS`), and the self-calibration below replaces the guess after the first + load. Under-promise, self-correcting. +- **reserve too small** → predicts `FITS` where reality is `PARTIAL`. That is + precisely the llmfit failure this phase exists to delete: a confident badge + about a model that spills. + +So the bootstrap does not have to be right. It has to **not under-estimate** — +the same rule as **Boundaries**: *unknown shapes round up; never underestimate +memory*. + +Per-layer KV comes from the header; the formula is validated exactly in the +**Appendix**. > **Do not double-count.** ggml's `free` **already excludes** the desktop's > allocation and the backend context. Measured on a 6144 MiB RTX 3050: 986 MiB @@ -552,6 +675,10 @@ side of `POST /models/load`: real_overhead = (free_before - free_after) - weights_bytes - kv_bytes(n_ctx) ``` +This residual **already contains the compute buffers**, which is why they need no +separate per-model table: the first load measures everything that is neither +weights nor KV, on the machine that will run it. + ```text first run predict with the shipped constant after first load record the machine's real overhead, keyed by device name @@ -592,6 +719,17 @@ streaming. Install fetches the GGUF into `--models-dir` with real filenames, not llama.cpp's content-addressed HF cache layout (`LLAMA_CACHE` points at the app data dir so nothing writes to `~/.cache`). +> **ponytail: the mechanism below is unverified.** `POST /models/load` with +> `{"model": …, "args": ["-c", "16384"]}` was sent at `b11050` and the router +> **ignored the args** — the worker came up at the model's own default +> (`n_ctx_slot` 34304 / 28160 / 12288 for the 0.6B / 1.7B / 4B, not 16384). The +> same `-c 16384` passed directly to `llama-server` on the command line **is** +> honoured, so the flag works and the router's plumbing for it does not, at least +> not in this shape. Fixed-context-at-load is a Decisions-table entry and the +> reason the 16K floor exists, so **resolve this first in 7.3**: either the field +> name differs, the args need another shape, or context must be set per model at +> router startup. Do not build the context ladder on top of an unconfirmed call. + **Context is fixed at load, not grown.** `POST /models/load {"args": ["-c", N]}` with `N` the largest window that keeps the verdict at **`FITS`** — not the largest that fits at all. KV cache is allocated upfront and competes with the @@ -603,6 +741,14 @@ own `context_length`. No ladder, no reload mid-conversation. Growing on occupancy is additive later; it would mean reloading the model between turns and two constants copied from Hermes with no evidence they suit RAG. +**The router costs no device memory until a model loads** (measured: VRAM +unchanged at the idle baseline with the router up and three models discovered), +and it **auto-discovers** everything in `--models-dir` as `status: "unloaded"`, +spawning one worker per loaded model with `--port 0 --model `. Pass +`--sleep-idle-seconds` deliberately: at `b11050` a model self-evicted after +roughly 30 s idle without it, which would turn the second question of a +conversation into a reload. + **Router mode runs with an empty models directory**, so the sidecar lifecycle is testable before any model exists. Verified on Windows and Linux at `b11050`: `GET /health` → `{"status":"ok"}`, `GET /models` → `{"data":[],"object":"list"}`, @@ -611,10 +757,21 @@ sidecar came up in router mode rather than single-model mode. SIGTERM shuts it down cleanly (`cleaning up before exit`), and on Windows `taskkill /PID /T /F` walks the tree and reports each child terminated. -> **Partially verified only.** The `taskkill /T` check above ran with no model -> loaded, so it reaped `conhost.exe` rather than a model worker. The case 7.5 -> actually cares about — reaping the **grandchild** the router spawns per loaded -> model — still needs a run with a model resident. +**Grandchild reaping verified**, with a model resident, on Windows at `b11050`: + +```text +router 132 llama-server.exe + child 10828 conhost.exe + child 8820 llama-server.exe ← the model worker + grandchild 18688 conhost.exe + +taskkill /PID 132 /T /F → all four terminated, depth first +SURVIVORS: none +``` + +> **Windows only.** macOS and Linux reap through process groups and signals, not +> `taskkill /T`, and the macOS hardened runtime case (7.5) is still untested +> against a grandchild. **Tests:** a fake llama-server (aiohttp) covering health, models, props, load, chat stream, delete; install against a fake HF serving a real small GGUF. @@ -718,7 +875,7 @@ of five; this is where the rest would go. reading its GGUF header, and a curated entry is not downloaded yet — so without these fields an airgapped machine has no fit badge on the one tier it can use. Committing what the header said at authoring time is what lets `weights + -KV(16K) + overhead` run against the manifest alone, on first paint, with no +KV(16K) + compute_buffers` run against the manifest alone, on first paint, with no network. It is the same reason Hermes carries estimator inputs inline. `decode_fraction` is the fraction of the build's bytes read per decoded token: @@ -1064,8 +1221,10 @@ five-level score (`perfect`/`good`/`marginal`/`too_tight`/`unknown`) and becomes a three-state statement about **where the weights will live**. There is no `unknown`: every row has a file size, so every row has a badge. -Let `need = weights + KV(window) + overhead`, `vram` = usable device memory from -the budget probe, `ram` = host memory available to the budget. +Let `need = weights + KV(window) + compute_buffers`, `vram` = +`HardwareBudget.usable_vram_bytes` (device free **minus** the fit reserve), and +`ram` = host memory available to the budget. The two subtractions sit on opposite +sides of the comparison and must not be collapsed into one term — see 7.2. | State | Condition | What happens at load | |---|---|---| @@ -1077,6 +1236,12 @@ the budget probe, `ram` = host memory available to the budget. will not fit at 16K cannot be rescued by a smaller context, and the remedy to offer is a smaller quantization or a smaller model. +**Keep `offload_fraction`, do not collapse it into the state.** `PARTIAL` spans +everything from barely-noticeable to unusable, and the number is already known +from the same subtraction that produced the state. Two consumers need it: the +recommendation gate, and the graded reason line below. Discarding it is what +made an earlier draft star a 1.7B on a machine happily running an 8B. + **Wording is per platform. The state set is not.** One label set is wrong on two of the three targets, so the copy branches on two facts the budget already carries: `uma`, and whether any non-CPU device exists. @@ -1139,12 +1304,18 @@ transfer that does not happen. With no GPU at all, `FITS` means the processor an > hyphens.** Use commas, full stops or parentheses. Applies to badges, empty > states, error copy and the recommendation line. -> **Optional refinement, not a design change.** `PARTIAL` spans a wide range: 5% -> spilled is barely noticeable, 70% crawls. The number is already known -> (`need - vram`), so the reason line can be graded without adding a state — -> *"A little too big for the GPU. Most of it still fits."* against *"Well over -> your GPU's memory. Expect it to be slow."* Same badge, sharper sentence. Ships -> after the first version if wanted. +**Grade the `PARTIAL` reason line by `offload_fraction`.** Not optional: the +range is wide enough that one sentence is wrong at both ends, and the number is +already on the verdict. + +```text +f ≲ 0.25 ◐ Reduced speed A little too big for the graphics card. Most of it still fits. +f ≳ 0.5 ◐ Reduced speed Well over your graphics card's memory. Expect it to be slow. +``` + +The verdict word does not change, because the state has not changed. Only the +explanation sharpens. An earlier draft filed this as a copy refinement; it is +really the same information the recommendation gate needs, surfaced. > **Decision: `PARTIAL` stays available on unified memory.** Hermes' > `_uma_budget()` sets `ram_available_bytes = 0`, so on Apple Silicon their @@ -1196,19 +1367,36 @@ Deleted: the collision-resolution block, `_placeholder_row()`, **Recommendation policy**, its own module and its own test file: ```text -resident = (entry, variant) pairs whose state is FITS -resident → max(variant.rank, -variant.size_bytes) reason: best-rank-resident -else → no recommendation; PARTIAL pairs stay installable, +eligible = (entry, variant) pairs that are not TOO_BIG + and whose offload_fraction ≤ MAX_OFFLOAD +eligible → max(variant.rank, -variant.size_bytes) reason: best-rank-eligible +else → no recommendation; every non-TOO_BIG pair stays installable, just never starred ``` ```python candidates = [(e, v) for e in curated for v in e.variants - if fit(e.shape, v.size_bytes).state is FitState.FITS] + for verdict in [fit(e.shape, v.size_bytes)] + if verdict.state is not FitState.TOO_BIG + and verdict.offload_fraction <= MAX_OFFLOAD] pick = max(candidates, key=lambda ev: (ev[1].rank, -ev[1].size_bytes), default=None) ``` +> **`MAX_OFFLOAD` is provisional, and may not be needed at all.** On the one +> machine measured, *every* offload fraction stayed usable: even fully on the CPU +> this model decoded at 13 t/s, above reading pace, and prefill held at half +> device speed. No sensible threshold would have fired anywhere in the curated +> range. Before shipping a constant, check whether the gate ever triggers — a +> mechanism that never fires is worse than no mechanism, because it reads as +> protection that was never tested. The reason the `FITS`-only rule had to go is +> unchanged and does not depend on this number: it refused configurations that +> demonstrably work. + +`MAX_OFFLOAD` is a v1 placeholder for the speed gate below, not a latency +judgement. Set it generously: the failure it exists to prevent is starring a +model that crawls, and the failure it replaced was refusing models that work. + **The policy ranges over builds, not models**, because that is what the user installs and what `rank` describes. With v1's single variant per entry the cross-product is the entry list and the behavior is identical — but writing it @@ -1218,14 +1406,130 @@ larger variant is `TOO_BIG` and whose smaller one `FITS` is still recommendable on its smaller build; that falls out of the cross-product rather than needing a special case. -**No speed gate in v1.** An earlier draft gated on predicted latency, which -required two constants with nothing measured behind them. The policy is now -"highest rank among the entries that fit entirely on the device", and the tie -break prefers the smaller build, which leaves more room for context. +#### Why the gate is on speed, not on residency + +An earlier draft gated on `FITS` alone, on the reasoning that a latency gate +needed constants nobody had measured. That is true and it still produced the +wrong answer, because **residency is a mechanism and speed is the goal.** They +coincide on a large card and diverge on a small one: + +```text +RTX 3050, 6 GB, measured + Qwen3 4B f ≈ 0.20 spilled → PARTIAL + Qwen3 8B f ≈ 0.28 spilled → PARTIAL, and runs without noticeable lag +``` + +A `FITS`-only rule stars **Qwen3 1.7B** on that machine while its owner is +running the 8B perfectly happily. The rule is not conservative, it is wrong: it +bans a configuration that works. + +The reason a fifth spilled is barely noticeable is that decode is +bandwidth-bound and the split is not proportional. Which is also why a single +threshold on `f` is a crude instrument, and why the real gate is a time estimate. -The gate is additive later. What it needs first is a prefill model: a handful of -`llama-bench pp512` runs per device class, since llmfit's `prefill_tps` and -`ttft_ms` are estimates for a generic machine and this app is prefill-dominated. +#### The speed model + +Keep the offload fraction the fit calculation already computes. Decode reads the +weights once per token, so: + +```text +f = bytes_on_cpu / total_bytes 0.0 resident … 1.0 all CPU +t_token = W × decode_fraction × ( f/BW_cpu + (1−f)/BW_gpu ) +decode_tps = efficiency / t_token +``` + +**Measured, and it holds.** Qwen3 4B on an RTX 3050, `-ngl` used to set `f` +directly so model size is held constant: + +| layers on GPU | `f` | decode t/s | vs resident | implied `r` | +|---|---|---|---|---| +| 36/36 | 0.00 | **52.45** | 1.00 | — | +| 28/36 | 0.22 | 33.17 | 0.63 | 3.7 | +| 18/36 | 0.50 | 19.85 | 0.38 | 4.3 | +| 9/36 | 0.75 | 12.93 | 0.25 | 5.1 | +| 0/36 | 1.00 | 13.13 | 0.25 | 4.0 | + +The slowdown reduces to `1/(f·r + 1 − f)` where `r = BW_gpu / BW_cpu`, and every +point fits **r ≈ 4** within ±20%. Cross-checked against absolutes: 131 GB/s +effective on the device (78% of the card's rating) against 33 GB/s on the host, +ratio 4.0. So `r` comes from published bandwidths and needs no per-card +measurement. + +> **`r` is what makes spill tolerable here, and it is hardware-specific.** On a +> high-bandwidth card `r` is far larger — roughly 12 on a 4090 — so the same +> `f = 0.3` would cost ~78% rather than ~31%. Small cards have modest bandwidth +> *and* are the only ones that spill, which limits the damage, but a threshold +> calibrated on a 3050 is **not** conservative for other hardware. On unified +> memory `r` approaches 1 and spilling barely means anything. + +**Prefill degrades roughly half as much as decode**, which matters more here than +anywhere because this app is prefill-dominated: + +| `f` | prefill ratio | decode ratio | +|---|---|---| +| 0.22 | 0.81 | 0.63 | +| 0.50 | 0.67 | 0.38 | +| 1.00 | **0.50** | **0.25** | + +Fully on the CPU, prefill still runs at half device speed while decode drops to a +quarter. A turn of ~8,000 prefill against ~300 decoded therefore loses +substantially less to spilling than any decode-focused benchmark implies — and +than Hermes' `PLEASANT_FLOOR_TOK_S = 20.0`, which was tuned for agentic bursts, +would suggest. **Do not import that threshold.** + +`decode_fraction` is the manifest field already defined and computed from the +header — **this is the thing it is for.** The spec previously filed it as +"nothing to apply it to"; the speed model is the application. + +Decode alone still mis-ranks for this app, because a turn is ~8,000 prefill +tokens against ~300 decoded. Prefill is compute-bound rather than +bandwidth-bound and degrades differently under offload, so it needs its own +term: + +```text +felt_time ≈ prefill_tokens / prefill_tps + decode_tokens / decode_tps +eligible = pairs whose felt_time ≤ budget +pick = max(eligible, key=rank) +``` + +#### Where the constants come from: every turn is a measurement + +`BW_gpu`, `BW_cpu`, `efficiency` and the prefill rate are per-machine. Both +reference implementations ship them as hardcoded tables — Hermes has a +`GPU_BANDWIDTH` dict and `_estimate_speed(..., offload_frac=0.0)`; Odysseus adds +`FALLBACK_K = {"cuda": 220, "rocm": 180, "metal": 150, …}`; llmfit's own output +admits `"method": "gpu_bandwidth_roofline"`, `"efficiency": 0.55`. + +**We do not copy that.** A table of bandwidths for cards nobody tested is the +llmfit failure in a new costume. Instead: + +> **A completed chat turn already carries everything the model needs** — prompt +> tokens, generated tokens, time to first token, total time — and the offload +> fraction of the loaded model is known. That is enough to solve for this +> machine's constants, with no benchmark, no extra binary and no user-visible +> step. + +```text +first run predict with a conservative shipped default +after turn 1 record (prompt_tokens, decode_tokens, ttft, total, f) +thereafter predict with this machine's own numbers +``` + +Identical in shape to the memory self-calibration in **7.2**, and it reuses the +same per-device store. It is also why no `llama-bench` run ever has to ship. + +#### Staging + +| | | +|---|---| +| **v1** | Retain `f` on the verdict. Gate on an offload-fraction ceiling instead of `FITS`. One number, conservative, and it fixes the 3050 case. | +| **next** | The roofline above with a shipped default, plus badge copy graded by `f`. | +| **then** | Turn-based self-calibration, which retires the shipped constants. | +| **never** | A hardcoded per-GPU bandwidth table. | + +The v1 step is deliberately crude: an `f` ceiling is not a latency judgement and +should not be described as one. It is a placeholder that is wrong in the safe +direction — it stars models that run, rather than refusing models that work. > **Why prefill, when the gate lands.** `HISTORY_BUDGET_TOKENS = 3000` plus ~24k > characters of grounding is roughly 8,000 prefill tokens against ~300 decoded, @@ -1434,6 +1738,19 @@ compliance. That plausibly collapses three tiers to two. observed on Windows — `vulkan-1.dll` is present in `C:\Windows\System32` on a stock install, so nothing needs bundling there. Linux supplies it through `libvulkan1`; see 7.5.) +- **Quantized KV without working flash attention: assert, do not assume.** + `q8_0` cache requires flash attention, and when the fused kernel is + unavailable llama.cpp falls back to CPU attention **silently** — no warning, + the device sits near 0% utilisation and throughput collapses. Two known + triggers: asymmetric `-ctk`/`-ctv`, and Vulkan on non-NVIDIA hardware, whose + vendor-neutral `GL_KHR_cooperative_matrix` path was only optimised in mid-2026 + and has an open report of extreme degradation on AMD. So: set both cache types + identically, verify flash attention engaged after load, and fall back to `f16` + — accepting the memory cost — rather than running at CPU-attention speed. + **This is the third silent-degradation case in this phase**, after the probe's + backend search path (7.2) and a missing backend dependency below. The pattern + is constant: llama.cpp degrades quietly and exits 0. Assume nothing worked + until something says it did. - **A GPU exists and ggml cannot see it: say so.** The silent skip above is correct when there is no GPU and wrong when there is one, and the two are indistinguishable from ggml alone — both print `(none)` and exit **0**, with no @@ -1470,6 +1787,21 @@ compliance. That plausibly collapses three tiers to two. - GGUF reader: truncated file, oversized vocab, split parts, Range retry. - Estimator: decision-table over constructed profiles; f16 vs q8_0 KV (a 1 GB swing on an 8 GB machine, measured). +- **KV precision is chosen, not defaulted**: a profile where `f16` fits selects + `f16`; one where only `q8_0` fits selects `q8_0` and records why; one where + neither fits is `PARTIAL` or `TOO_BIG` as the sizes dictate. +- **`-ctk` and `-ctv` are always equal** in any launch argument set the provider + builds — asserted on the spawned command line, since a mismatch silently + disables the fused kernel. +- **The two subtractions stay on their own sides.** A profile where + `weights + KV + compute ≤ device_free` but `> device_free − fit_reserve` + asserts **`PARTIAL`**, not `FITS` — the real RTX 3050 case, and the one a + single collapsed `overhead` term gets backwards. +- **An unmeasured `(platform, backend)` row rounds up.** A budget for a platform + with no measured reserve never yields a *more* optimistic verdict than the + measured Windows row on equivalent memory. +- `compute_buffers` is part of `need`: removing the term flips at least one + fixture from `PARTIAL` to `FITS`, which is the regression to catch. - Fit states: each of `FITS` / `PARTIAL` / `TOO_BIG` at its boundary, ±1 byte; `TOO_BIG` evaluated at the 16K floor, not the requested window; `PARTIAL` reachable on a `uma` budget (the Hermes divergence — a regression here silently @@ -1478,6 +1810,15 @@ compliance. That plausibly collapses three tiers to two. the right platform wording for all three budget shapes. - Recommendation policy: the six hardware profiles above, asserting both pick and reason key; and that a searched row is never eligible to be the pick. +- **A `PARTIAL` pair below `MAX_OFFLOAD` is starrable.** The regression fixture + is the measured RTX 3050: a higher-ranked model at `f ≈ 0.28` beats a fully + resident lower-ranked one. A `FITS`-only policy fails this test, which is the + point of having it. +- **`offload_fraction` survives to the verdict** and is not recomputed by the + renderer; the graded reason line is selected from it, and the same fraction + drives both the gate and the copy. +- `TOO_BIG` is still refused regardless of rank: the ceiling relaxes residency, + never physics. - Catalog: curated renders with no network; search gated on egress; arch, template, and gated-repo rejections; a search row carries no `rank` field at all — asserted on the serialized response, so it cannot be reintroduced @@ -1642,6 +1983,80 @@ Both platforms were previously reasoned from documentation only. | CPU device, WSL2 | 26048.6 MiB total, **26048.6 MiB free** (virtualised) | | `vulkan-1.dll` on stock Windows | present in `System32`, nothing to bundle | +**Offload sweep**, Qwen3 4B Q4_K_M, `-ngl` controlling `f` directly, f16 KV, +`-p 512 -n 300`, Vulkan, RTX 3050: + +| ngl | `f` | pp512 | tg300 | +|---|---|---|---| +| 99 | 0.00 | 1805.50 ± 1.75 | 52.45 ± 0.17 | +| 28 | 0.22 | 1454.68 ± 0.93 | 33.17 ± 0.25 | +| 18 | 0.50 | 1206.29 ± 9.86 | 19.85 ± 1.06 | +| 9 | 0.75 | 1027.97 ± 26.10 | 12.93 ± 0.43 | +| 0 | 1.00 | 907.43 ± 26.49 | 13.13 ± 0.67 | + +`ngl 0` edging out `ngl 9` is real: splitting across devices costs transfers that +outweigh nine layers of device work. + +**KV precision.** Resident, depth 0: `f16` pp8192 1437.66 / tg300 52.47 against +`q8_0` 1300.44 / 51.34. + +> **That comparison is at the wrong depth and should not be quoted as q8_0's +> cost.** At depth 0 the cache is nearly empty, so its precision cannot matter; +> published CUDA figures showing ~18% at 8K depth were measured under different +> conditions. The valid comparison — `f16` at 16K depth — **could not be taken**, +> because it fails to allocate on this card. What is known: `q8_0` at depth 16384 +> decodes at 27.46 t/s against 51.34 at depth 0, and that halving is the cost of +> attending over a long context, not of quantizing it. **q8_0's real cost at +> depth on Vulkan is unmeasured**; it needs a card with enough VRAM to run `f16` +> at 16K. + +**The decisive result.** `f16` KV at a 16K window on a 6 GB card: + +```text +ggml_vulkan: Device memory allocation of size 316407808 failed. +ggml_vulkan: vk::Device::allocateMemory: ErrorOutOfDeviceMemory +llama_bench: error: failed to create context +``` + +`q8_0` at the same depth runs. On this class of card the planned configuration +does not fit, which is what makes KV precision a fit-calculation output rather +than a default. + +**Fit terms, measured.** Qwen3 4B Q4_K_M at `-c 16384`, `--fit` on, RTX 3050 +with 5,234 MiB free, from llama.cpp's own allocation log: + +| buffer | MiB | +|---|---| +| `Vulkan0` model | 2078.04 | +| `CPU_Mapped` model (spilled) | 602.16 | +| `Vulkan0` KV | 1984.00 | +| `CPU` KV (spilled) | 320.00 | +| `Vulkan0` compute | 143.62 | +| `Vulkan_Host` compute | 26.01 | +| `Vulkan_Host` output | 2.32 | +| **placed on device** | **4205.66** | +| **fit reserve (unused)** | **1028.34** | + +**The KV formula is exact, not approximate.** +`2 × 36 layers × 8 kv_heads × 128 × 2 bytes × 16384` = **2304 MiB**, against +`1984 + 320` measured. Qwen3 0.6B independently: predicted `378 + 1792 + ~60` = +2,234 MiB, measured VRAM delta **2,234 MiB**. That is the evidence that a curated +row can be priced offline from committed `shape` fields alone. + +**Router lifecycle.** Router up with three models discovered: VRAM unchanged from +idle. Worker spawn: `--port 0 --model `. Idle eviction ~30 s without +`--sleep-idle-seconds`. Grandchild reaping via `taskkill /T /F`: four processes, +depth first, no survivors. + +**`-ngl` disables `--fit`.** With `-ngl 99`: `common_fit_params: failed to fit +params to free device memory: n_gpu_layers already set by user to 99, abort`, +after which the model loaded entirely on CPU with 478 MiB of VRAM touched, no +error, exit 0. + +**`POST /models/load` ignored `{"args": ["-c","16384"]}`** — workers came up at +`n_ctx_slot` 34304 / 28160 / 12288 (model defaults). The same flag on the +`llama-server` command line **is** honoured (`n_ctx_slot = 16384`). See 7.3. + GPU backend comparison, same card and model (Qwen3 4B Q4_K_M, `-ngl 99`): | test | CUDA 13.4 | Vulkan | CUDA advantage | diff --git a/plans/community-local/api/08-cuda-backend.md b/plans/community-local/api/08-cuda-backend.md index be915f2cb4..f523be4f52 100644 --- a/plans/community-local/api/08-cuda-backend.md +++ b/plans/community-local/api/08-cuda-backend.md @@ -54,6 +54,7 @@ claimed, which is why that claim was removed rather than carried forward. | **CUDA version** | 13.4, Windows x64 | Requires Turing (7.5)+. Shipping 12.4 as well for Pascal costs another ~645 MB — that is what made Ollama's archive 1.8 GB. Pre-Turing cards fall through to Vulkan automatically. | | **Reversibility** | Cheap in both directions | Adding CUDA is four files. Removing it is deleting four files. Neither touches a contract, a schema or a migration. | | **Diagnostics** | Mandatory if shipped | A missing cudart is **silent**: the app reports no GPU, exit 0. See **Failure behavior**. | +| **Quantized KV** | A real argument **for** CUDA, and the only one that strengthened | `q8_0` KV requires flash attention. CUDA's FA path is the mature one; Vulkan has two, and the vendor-neutral `GL_KHR_cooperative_matrix` path used by AMD and Intel was only optimised in mid-2026 and carries an open report of extreme degradation on AMD. Since [`07`](07-llamacpp-runtime.md) now selects `q8_0` whenever it converts a spill into residency, the backend's FA quality stops being cosmetic. Note CUDA's `GGML_CUDA_FA_ALL_QUANTS` build flag decides which K/V combinations get fused kernels. | ## Boundaries @@ -198,6 +199,21 @@ Run against both staged builds on the same card, same model, same build number. - Confirm from the log that Vulkan chose device 0. If decode collapses, it picked the integrated GPU and the run is void. +**What the follow-up measurements changed.** Since this document was written, +the offload sweep in [`07`](07-llamacpp-runtime.md) established two things that +bear on the decision: + +- **The roofline holds with `r ≈ 4` on a 6 GB card**, so the speed cost of + spilling is now predictable rather than guessed. CUDA's advantage over Vulkan + on the same card was 2.1% on decode, which moves `r` to ~4.1 — immaterial. +- **Prefill degrades half as much as decode under offload** (0.50 against 0.25 + fully on the CPU). Because this app is prefill-dominated, the workload is + *less* sensitive to backend decode speed than a chat or agent app would be — + which weakens the case for CUDA further, since its measured lead was almost + entirely in prefill (10.2%) and nearly absent in decode (2.1%). + +Together these make the 0.66 s per turn figure below more, not less, reliable. + **Decision rule, fixed in advance so the result cannot be rationalised:** | Result | Action | @@ -313,6 +329,12 @@ binary starts. A real NVIDIA machine must confirm, once per pinned build: leaves the count at zero. - **Pre-Turing NVIDIA card** — the CUDA backend is skipped silently and Vulkan runs. Correct behavior, no message needed. +- **Flash attention not engaged** — with CUDA present this is less likely than on + Vulkan, but the failure shape is identical: quantized KV falls back to CPU + attention with no warning and near-zero device utilisation. The assertion + specified in [`07`](07-llamacpp-runtime.md) — **Failure behavior** applies + unchanged; shipping CUDA does not remove the need for it, it only makes the + check more likely to pass. - **Duplicate devices** — not a failure, the normal state. Handled by `select_device()`. - **Vulkan loader missing on Windows** — not observed. `vulkan-1.dll` is present diff --git a/plans/community-local/frontend/05-install-ux.md b/plans/community-local/frontend/05-install-ux.md index 921719df92..f65b82affc 100644 --- a/plans/community-local/frontend/05-install-ux.md +++ b/plans/community-local/frontend/05-install-ux.md @@ -82,10 +82,16 @@ which owns the copy. ```text ● Full speed Runs entirely on the GPU -◐ Reduced speed Too big for the GPU, so part runs on the CPU +◐ Reduced speed A little too big for the GPU. Most of it still fits. +◐ Reduced speed Well over the GPU's memory. Expect it to be slow. ○ Won't fit Needs about 21 GB. This Mac has 13.6 GB ``` +**Reduced speed has two reason lines, chosen by `offload_fraction`**, which the +verdict carries. The verdict word does not change; only the explanation +sharpens, because one sentence is wrong at both ends of a range running from +barely noticeable to unusable. + A search row is badged from its file size alone and rendered as approximate until its header is read, which happens when the row is opened. @@ -123,6 +129,12 @@ selects the ★; no number, score or star rating appears anywhere else. slower, and llama.cpp places the layers. Only **Won't fit** blocks install, and it states required and available bytes and names a smaller model rather than greying out a control. +- **A Reduced speed model can carry the ★.** The recommendation gates on + predicted speed, not on full residency — on a 6 GB card the best model to use + is routinely one that spills a little. Measured: an RTX 3050 runs Qwen3 8B at + roughly 28% on the CPU without noticeable lag, while a residency-only rule + would have starred a 1.7B. Do not assume the ★ is always a **Full speed** row, + and do not style it as though it were. - Search shows a progress indicator while a row's header is fetched (2 to 3 seconds) and resolves the approximate badge to a firm one. - A failed or cancelled install remains retryable and is never shown as From 9c5935ceef2400208a1aa6dce222a08df7a57e7c Mon Sep 17 00:00:00 2001 From: Anish Sarkar <104695310+AnishSarkar22@users.noreply.github.com> Date: Sat, 19 Sep 2026 22:20:29 +0530 Subject: [PATCH 08/60] docs(spec): close the gaps a build order found in phase 7 --- .../api/07-llamacpp-runtime.md | 200 ++++++++++++++---- 1 file changed, 164 insertions(+), 36 deletions(-) diff --git a/plans/community-local/api/07-llamacpp-runtime.md b/plans/community-local/api/07-llamacpp-runtime.md index 85abdbc3fb..76734c056c 100644 --- a/plans/community-local/api/07-llamacpp-runtime.md +++ b/plans/community-local/api/07-llamacpp-runtime.md @@ -52,7 +52,7 @@ Ollama's native API does not. | **Hardware budget** | `ggml_backend_dev_memory()` via `ctypes`, from the shipped libs | The allocator's own view. Falls back to `llama-server --list-devices`, then OS APIs. | | **Downloads** | **SurfSense fetches the GGUF**, not `POST /models` | `llama-server` is a second process we do not proxy; an in-process fetch is the only place `egress.require()` actually holds. Also buys resume, checksums, and the header as the file lands. | | **Quantization** | **One pinned file per entry in v1** | A pinned file is what "tested by SurfSense" can honestly claim. Per-machine selection is possible at no extra cost (one header read prices every quant), so a second variant on the largest entries is a cheap follow-on, not v1. See **The quantization ladder**. | -| **KV precision** | **`f16` when it fits, `q8_0` when it buys residency.** Symmetric `-ctk`/`-ctv` always | Measured: `f16` KV at a 16K window **cannot allocate** on a 6 GB card (`ErrorOutOfDeviceMemory`), while `q8_0` at the same depth runs. Quality is lossless either way. So the choice is not a global default but an output of the fit calculation — pay the precision cost only where it converts a spill into residency. Requires flash attention; see **Failure behavior**. | +| **KV precision** | **`f16` when it fits, `q8_0` when it buys residency.** Symmetric `-ctk`/`-ctv` always | Measured: `f16` KV at a 16K window **cannot allocate** on a 6 GB card (`ErrorOutOfDeviceMemory`), while `q8_0` at the same depth runs. Quality is lossless either way. So the choice is not a global default but an output of the fit calculation — pay the precision cost only where it converts a spill into residency. Requires flash attention; see **Failure behavior**. **Conditional on per-model load args working** — see 7.3; if they do not, this decision degrades to one global choice. | | **Context** | **Fixed at load. Floor 16K, capped at the model's own `context_length`** | llama.cpp fixes context at load, so `num_ctx()`'s per-request sizing has no equivalent. The floor is 3K history plus ~8K grounding plus a reply. Growing on occupancy is additive later; it needs a mid-conversation reload and two constants with nothing measured behind them. | | **GPU backend** | **Vulkan on every platform off Apple Silicon. No CUDA** | Measured on an RTX 3050 at `b11050`: CUDA leads Vulkan **9.1%** on `pp512`, **10.2%** on `pp8192`, **2.1%** on decode — 0.66 s on an 11 s turn, for 685 MB. Vulkan covers NVIDIA, AMD and Intel from one 31 MB archive, its loader ships with Windows, and it is what the app already does today (`pruneCudaRunners()`). CUDA is specified as an optional later addition in [`08-cuda-backend.md`](08-cuda-backend.md), which needs **no code change** — ggml selects it by the files present. | | **Device selection** | First device with `type == GPU`. **Never sum** | The same physical card appears once per loaded backend, and an integrated GPU can advertise more memory than a discrete one (16198 MiB against 6002 MiB, measured). ggml's ordering already expresses backend preference, so this **is** backend selection. Specified in **7.2**. | @@ -209,10 +209,21 @@ Sidecar flags: `--models-dir /models --port --models-max 1 --sleep-idle-seconds 300 --no-ui --jinja --reasoning-format deepseek`. Per-model at load, decided by the fit calculation rather than fixed here: -`-fa on` and, when `q8_0` is chosen, `-ctk q8_0 -ctv q8_0`. **Set both cache -types or neither** — symmetric quantization enables the fused flash-attention -kernel, while a mismatched pair falls back to an unoptimised path that exists -for correctness only. +`-c `, `-fa on` and, when `q8_0` is chosen, `-ctk q8_0 -ctv q8_0`. +**Set both cache types or neither** — symmetric quantization enables the fused +flash-attention kernel, while a mismatched pair falls back to an unoptimised +path that exists for correctness only. + +> **All four of those flags ride the same unverified mechanism.** The ponytail +> in 7.3 records that `POST /models/load` ignored `{"args": ["-c", "16384"]}`. +> It is written up as a context problem because context is what was measured, +> but nothing distinguishes `-c` from `-fa` or `-ctk`/`-ctv`: they are the same +> field on the same call. If the router turns out to honour flags only at +> startup, then **KV precision cannot be per model either**, and the Decisions +> row above degrades from "an output of the fit calculation" to one global +> choice made for whichever model is largest. Resolve the mechanism once and +> both decisions follow; do not resolve it for context alone and assume the +> rest. `--sleep-idle-seconds` is not optional: measured at `b11050`, a model without it self-evicted after roughly 30 s idle, which turns the second question of a @@ -243,10 +254,20 @@ that appeared inside the reasoning. > **Provenance.** The shapes and runtime calls below are grounded: the > llama-server README at `b11043`, `nm` on the shipped libraries plus a live > `ctypes` call, real GGUF headers read over HTTP Range, and live Hugging Face -> responses. **The HTTP routes are proposed, not reported** — no such surface -> exists yet. They follow this codebase's conventions: the `/llm` prefix, `*Read` -> Pydantic response models in `modules/llm/schemas.py`, and the NDJSON -> `{"type": …}` stream frame that `_event()` already emits. +> responses. **The HTTP routes below are proposed, not reported.** They follow +> this codebase's conventions: the `/llm` prefix, `*Read` Pydantic response +> models in `modules/llm/schemas.py`, and the NDJSON `{"type": …}` stream frame +> that `_event()` already emits. +> +> **Three of the six are not new paths, they are replacements at the same path.** +> `GET /llm/system`, `GET /llm/catalog` and `POST /llm/install` exist today, in +> `modules/llm/recommendations/router.py`, serving `RecommendationSystemRead`, +> `RecommendationCatalogRead` (with the `scanned` flag the renderer reads) and +> the same NDJSON install stream. So this is an **in-place breaking reshape, not +> an addition**: route, response schema and frontend have to move in one commit, +> because there is no additive path that does not serve two incompatible shapes +> at once. Only the two `/llm/search` routes and `DELETE /llm/models/…` are +> genuinely new surface. ### Domain shapes @@ -342,6 +363,22 @@ which is the `scanned` flag this phase deletes. is **opening a search result**, not hovering or typing. The list-level badge stays `approximate` until then. +**Installing a searched build needs an id the manifest cannot supply.** +`InstallRequest` keys on `catalog_id`, and [`../frontend/05-install-ux.md`](../frontend/05-install-ux.md) +requires that the renderer send *only* that — no repo, file, artifact URL, local +path or quantization. A curated row's id comes from the manifest; a search hit +has no manifest entry, so there is nothing to key on. Closing that by letting the +renderer post a repo and file would hand the frontend the ability to name an +arbitrary download, which is the exact capability the install contract exists to +withhold. + +So `GET /llm/search/{repo}` **mints a server-side install ticket per quant** and +returns it as that row's `catalog_id`, held with the same 300 s TTL as the search +cache. The install route then resolves tickets and manifest ids through one +lookup and cannot tell them apart. An expired ticket reuses the existing +`422 catalog id is stale or unknown; refresh the catalog` rather than inventing a +second staleness error, since it is the same failure and already has copy. + Deleted with the Ollama adapter: `GET /llm/providers/{provider}/catalog` and `POST /llm/providers/{provider}/pull`. Both are Ollama-shaped, and their replacements are the rows above. @@ -357,8 +394,9 @@ Unchanged: `GET /llm/providers`, `GET|PUT /llm/selection/{role}`, | Alembic revision for 7.6 | `0012` (head is `0011_document_cancelled_status`) | | Egress destinations | `model_download`, `model_search` (both `huggingface.co`) | -`"ollama"` appears at **18 non-test Python sites**; each becomes `"llamacpp"` or -is deleted with the adapter. +`"ollama"` appears at **115 lines across 24 non-test Python files**; each becomes +`"llamacpp"` or is deleted with the adapter. Counted across the whole tree it is +**421 lines in 66 files**, which is the figure 7.6 works from. ### Out of scope @@ -444,9 +482,26 @@ feature that tells the truth. ## Phases -Phases 7.0–7.5 keep Ollama registered and working. `REGISTRY` carries both -adapters, so each ships independently and is reversible. **7.6 is the only -irreversible phase and the only one that touches user data.** +Phases 7.0–7.5 keep Ollama registered. `REGISTRY` carries both adapters, so each +ships independently and is reversible. **7.6 is the only irreversible phase and +the only one that touches user data.** + +> **"Registered and working" is true of chat, not of catalog or install, and the +> difference is worth stating before someone plans around the stronger claim.** +> Two seams are llmfit-shaped and cannot serve both runtimes at once: +> `LocalRuntime.resolve(ScoredModel) -> InstallPlan` asks a runtime to map a +> *scored* model onto an install plan, and `LlamaCppProvider` has nothing to +> resolve because it installs a `GgufArtifact`; and `CatalogService.__init__` +> requires an advisor that 7.4 removes. So from 7.4 onward **Ollama's catalog and +> install path is dead even though its adapter is still registered**, and what +> survives to 7.6 is that an existing Ollama selection still answers, through +> `resolve_generation()` and the chat path. +> +> That is enough for the property the staging is for — no user loses chat +> mid-swap, and every phase before 7.6 reverts with one commit — but it is not +> "both runtimes fully working side by side", and trying to preserve that would +> mean keeping two catalog services alive for the sake of a runtime being +> deleted three phases later. ### 7.0 — De-Ollama the domain @@ -464,6 +519,16 @@ llmfit adapter switches to `recommend --force-runtime llamacpp --output-llamacpp **Tests:** `llmfit.py` fixtures. No UI change. +> **Most of this phase cancels itself out, so do not ship it on its own.** The +> `CACHE_VERSION` bump is pointless because 7.4 deletes `CACHE_VERSION` and the +> scan cache; retargeting the llmfit adapter is pointless because 7.4 moves +> llmfit out of the app, leaving `recommendations/llmfit.py` with no consumer. +> What genuinely survives is the **domain change** at the top of this section — +> `ollama_name` becoming `artifacts`, and the five deletions — plus the llmfit +> *parsing*, which becomes the authoring script's. Fold the rest into 7.4 and +> land them together rather than writing code whose only reader is deleted two +> phases later. + ### 7.1 — GGUF header reader `modules/llm/gguf/header.py`, stdlib only. Local file or `Range: bytes=0-8388607` @@ -483,13 +548,23 @@ header and a truncated-response retry. ### 7.2 — Hardware budget and estimator -`modules/llm/hardware.py` — `ctypes` into `libggml`, then `--list-devices`, then +`modules/llm/hardware/` — `ctypes` into `libggml`, then `--list-devices`, then OS APIs. First hit wins, cached; warm the probe in the background at first launch, not on the path of the first render (a cold `ggml_backend_load_all()` on macOS compiles 20 Metal shader libraries, measured at ~20 s once, 180 ms after). That cold penalty is **Metal-only**: measured off Apple Silicon the same call is 77 ms with one backend and 205 ms with two. +> **Folders, not two files.** Earlier drafts named `hardware.py` and `fit.py`. +> Between them they carry library loading, the working-directory dance, device +> enumeration, device selection, the `--list-devices` fallback parser, OS memory +> APIs, the OS-versus-ggml cross-check, budget assembly, calibration storage, the +> KV term, compute buffers, the reserve table, the need/usable comparison, KV +> precision selection, context sizing and badge copy. That is sixteen +> responsibilities in two files, against the repo's one-responsibility-per-file +> rule in `AGENTS.md`. Split them along those lines; the names above are the +> package, not the module. + #### The probe must run from the library directory **This is the single most likely way this phase ships broken.** @@ -579,7 +654,7 @@ loaded makes every row read as too large. #### Two subtractions, on opposite sides of the comparison -`modules/llm/fit.py`. The single `overhead` term earlier drafts carried is **two +`modules/llm/fit/`. The single `overhead` term earlier drafts carried is **two different quantities**, and collapsing them is what made the first prediction of this wrong by 922 MiB: @@ -1065,7 +1140,7 @@ and sanely spread — which is exactly the population the curated tier draws fro > but the numbers described files the manifest does not ship. **Any rank quoted > anywhere must name the quantization it was taken at.** -Three checks before committing a rank: +Four checks before committing a rank: 1. within a family, rank rises with parameter count — the invariant in **Tests** 2. anything at or near 100 is suspect; nothing worth curating maxes a scale @@ -1531,17 +1606,19 @@ The v1 step is deliberately crude: an `f` ceiling is not a latency judgement and should not be described as one. It is a placeholder that is wrong in the safe direction — it stars models that run, rather than refusing models that work. -> **Why prefill, when the gate lands.** `HISTORY_BUDGET_TOKENS = 3000` plus ~24k -> characters of grounding is roughly 8,000 prefill tokens against ~300 decoded, -> the inverse of an agent's ratio. Decode is memory-bound; prefill is -> compute-bound. A decode-only prediction mis-ranks for this app. - -> **RAG is prefill-dominated.** `HISTORY_BUDGET_TOKENS = 3000` plus ~24k -> characters of grounding is roughly 8,000 prefill tokens against ~300 decoded — -> the inverse of an agent's ratio. Decode is memory-bound; prefill is -> compute-bound. (An earlier draft cited a 36–40% CUDA lead here; measured, it -> is **9.1%** — see [`08-cuda-backend.md`](08-cuda-backend.md).) Use llmfit's -> `prefill_tps`/`ttft_ms` from the manifest until measured class constants exist. +> **RAG is prefill-dominated, which is why the gate must measure prefill.** +> `HISTORY_BUDGET_TOKENS = 3000` plus ~24k characters of grounding is roughly +> 8,000 prefill tokens against ~300 decoded — the inverse of an agent's ratio. +> Decode is memory-bound; prefill is compute-bound. A decode-only prediction +> mis-ranks for this app. (An earlier draft cited a 36–40% CUDA lead here; +> measured, it is **9.1%** — see [`08-cuda-backend.md`](08-cuda-backend.md).) +> +> **There is no manifest field to fall back on.** An earlier draft ended this +> block with *"use llmfit's `prefill_tps`/`ttft_ms` from the manifest until +> measured class constants exist"*. Those were `ScoredModel` fields, produced by +> a scan that no longer runs, and schema 3 carries `shape`, `capabilities`, +> `decode_fraction` and `variants` and nothing else. Until the turn-based +> calibration below lands there is a shipped default and no second source. **Tests:** policy decision-table across the six hardware profiles above. @@ -1560,12 +1637,15 @@ llama.cpp has no stable channel). Verified: the Vulkan archive is the CPU archive plus exactly one file (`libggml-vulkan.so` / `ggml-vulkan.dll`), and all CPU micro-architecture variants ship inside — 15 on Windows, 10 on Linux (the CUDA archive carries 14 -on Linux, so the figure is per archive, not per platform). `ggml_backend_load_best()` -searches the executable's own directory and probes `cuda` before `vulkan`, so -the CUDA DLLs sit in the same flat folder and win automatically. +on Linux, so the figure is per archive, not per platform). -cudart carries `cudart64_*.dll`, `cublas64_*.dll`, `cublasLt64_*.dll`; all must -be beside `llama-server.exe` or it fails at launch with a missing-DLL error. +> **No CUDA payload ships, and the CUDA staging notes have moved.** Earlier +> drafts described the flat-folder probe order and the `cudart64_*` / +> `cublas64_*` / `cublasLt64_*` DLLs here, before the backend decision was made. +> This phase stages Vulkan on every platform off Apple Silicon and nothing else, +> so those notes belong to [`08-cuda-backend.md`](08-cuda-backend.md), which +> already carries them. Leaving them in a Vulkan-only packaging section reads as +> work to do. Prune to `llama-server` plus its libraries: 24 executables → 1, which also shrinks the macOS notarization surface. @@ -1586,7 +1666,7 @@ from the final packaged resource path. ### 7.6 — Remove Ollama completely -~440 references across 66 files. The only irreversible phase. +421 references across 66 files. The only irreversible phase. **Delete:** @@ -1599,6 +1679,24 @@ electron/scripts/fetch-ollama.mjs electron/ollama/ 501 MB staged ``` +**llmfit's packaging goes in the same phase**, and is easy to miss because none +of it has "ollama" in the name, so the definition-of-done grep below will not +catch a single one: + +```text +electron/scripts/fetch-llmfit.mjs +electron/scripts/check-llmfit.mjs +backend/tests/fixtures/llmfit/ fit.json, system.json +``` + +plus `SidecarContext.llmfitPath`, `python.ts`'s `SURFSENSE_LOCAL_LLMFIT_PATH`, +the four `llmfit_*` settings in `shared/config.py`, `recommendation_reserve_gb` +(replaced by the reserve table in 7.2), and the llmfit staging and smoke steps +in `release-local.yml`. **Decisions** says llmfit is not shipped and not in the +build; this is what that costs in files. `modules/llm/recommendations/llmfit.py` +itself is deleted earlier, in 7.4, when its parsing moves into the authoring +script. + **Rewrite:** `shared/config.py` (`ollama_base_url`/`ollama_models_dir` → `llamacpp_*`), `sidecars/types.ts` (`SidecarContext.ollama*`), `sidecars/python.ts` (stops passing `OLLAMA_*`), `electron-builder.yml`, @@ -1606,6 +1704,14 @@ electron/ollama/ 501 MB staged `.github/workflows/release-local.yml`, `modules/llm/router.py`, `modules/llm/resolution.py`, `modules/llm/selection.py`, `modules/chat/errors.py`. +**Two files in `modules/llm/profile/` are Ollama-shaped and neither appears in +any phase's file list.** `classify.py` is covered by the **Decisions** table +(*Prompt tier fallback* — key on loopback, not on a provider name) but is not +listed anywhere as work. `fingerprint.py` is not covered at all: +`from_ollama(name, tag, show)` is built from `/api/tags` and `/api/show`, and it +needs a `from_llamacpp()` reading `GET /models` and `GET /props` instead. Both +land here. + **Frontend:** `selected-roles.tsx` (`llamacpp` → "Local"; a loopback connection is arguably also "Local"), `chat-error-notice.tsx`. **Keep** `connection-form.tsx`'s Ollama preset. @@ -1613,7 +1719,18 @@ is arguably also "Local"), `chat-error-notice.tsx`. **Keep** **Egress:** `ollama_pull → registry.ollama.ai` becomes two destinations against `huggingface.co` — `model_download` (a repo you named) and `model_search` (text you typed). Same host, different consent. `ollama_pull_host()`'s `hf.co/` -special-case disappears. +special-case disappears, and with both destinations now static the dynamic +host-resolution path goes with it. + +> **The migration maps an existing grant to `model_download` only.** +> `model_search` starts unset. This is the one place "same host, different +> consent" has to be honoured rather than restated: a user who allowed model +> downloads allowed *fetching a file they named*, and search sends text they are +> typing to the same host for a different reason. Splitting one destination into +> two and carrying the grant across both would manufacture consent nobody gave, +> which is precisely the distinction the split exists to draw. The asymmetry is +> safe in the other direction: an unset `model_search` degrades to the airgapped +> product, which **Failure behavior** already specifies. **Three things that will bite:** @@ -1639,7 +1756,18 @@ returns only the `connection-form.tsx` preset and its test, revisions `0004` and `0009`, and the new migration's `WHERE destination = 'ollama_pull'`. **Tests:** a migration test over a database seeded with an Ollama selection and -an `ollama_pull` egress row. +an `ollama_pull` egress row, asserting the grant lands on `model_download` and +that `model_search` is **not** created. + +> **`test_ollama_num_ctx.py` is deleted here and its subject is not.** Those two +> tests assert that a full Studio grounding gets a window large enough to hold +> it, which is a real regression guard: Ollama's 4096 default silently dropped a +> 24k-character grounding to a third of its length, measured. llama.cpp fixes +> context at load, so `num_ctx()` has no successor, but the behaviour does. The +> replacement belongs in 7.2, against the context floor: assert the 16K floor +> covers `HISTORY_BUDGET_TOKENS` plus `build_context`'s grounding budget plus a +> reply, so the floor is checked against its stated derivation rather than +> asserted as a constant. Write it before deleting the old one. ### 7.7 — Capabilities and multimodal From 750ba30d70426e787fa5bdc18633822c7aa811f9 Mon Sep 17 00:00:00 2001 From: Anish Sarkar <104695310+AnishSarkar22@users.noreply.github.com> Date: Sun, 20 Sep 2026 20:07:42 +0530 Subject: [PATCH 09/60] build(electron): stage a pinned llama.cpp build in place of Ollama and llmfit Co-Authored-By: Claude Sonnet 5 --- surfsense_local/electron/.gitignore | 2 +- .../electron/build/entitlements.mac.plist | 4 +- surfsense_local/electron/electron-builder.yml | 15 +- surfsense_local/electron/package.json | 12 +- .../electron/scripts/check-llmfit.mjs | 31 --- .../electron/scripts/fetch-llamacpp.mjs | 211 ++++++++++++++++++ .../electron/scripts/fetch-llmfit.mjs | 190 ---------------- .../electron/scripts/fetch-ollama.mjs | 91 -------- .../electron/scripts/fetch-sdcpp.mjs | 2 +- 9 files changed, 227 insertions(+), 331 deletions(-) delete mode 100644 surfsense_local/electron/scripts/check-llmfit.mjs create mode 100644 surfsense_local/electron/scripts/fetch-llamacpp.mjs delete mode 100644 surfsense_local/electron/scripts/fetch-llmfit.mjs delete mode 100644 surfsense_local/electron/scripts/fetch-ollama.mjs diff --git a/surfsense_local/electron/.gitignore b/surfsense_local/electron/.gitignore index 1c2e654fac..b6b363f406 100644 --- a/surfsense_local/electron/.gitignore +++ b/surfsense_local/electron/.gitignore @@ -2,6 +2,6 @@ node_modules/ out/ dist/ release/ -ollama/ +llamacpp/ llmfit/ sdcpp/ diff --git a/surfsense_local/electron/build/entitlements.mac.plist b/surfsense_local/electron/build/entitlements.mac.plist index 2bf82d49a0..b894999518 100644 --- a/surfsense_local/electron/build/entitlements.mac.plist +++ b/surfsense_local/electron/build/entitlements.mac.plist @@ -8,14 +8,14 @@ com.apple.security.cs.allow-unsigned-executable-memory - com.apple.security.cs.disable-library-validation com.apple.security.cs.allow-dyld-environment-variables - + com.apple.security.network.client com.apple.security.network.server diff --git a/surfsense_local/electron/electron-builder.yml b/surfsense_local/electron/electron-builder.yml index 45ad240a66..ca648175b8 100644 --- a/surfsense_local/electron/electron-builder.yml +++ b/surfsense_local/electron/electron-builder.yml @@ -5,7 +5,7 @@ directories: buildResources: build # The asar carries only the main + preload bundles. The renderer SPA, the two -# frozen Python binaries, and the Ollama server ride alongside in resources/, +# frozen Python binaries, and llama-server ride alongside in resources/, # where the main process spawns them by path (sidecars/) and loads the SPA from # disk (index.ts). files: @@ -25,13 +25,12 @@ extraResources: # build:parser). First PDF stays offline; the worker is not blocked on Hub. - from: ../backend/models to: models - # The Ollama server (fetch-ollama.mjs stages it here). Models are pulled by the - # user post-install into the writable data dir, not shipped. - - from: ollama - to: ollama - # Pinned hardware advisor (fetch-llmfit.mjs verifies and stages it atomically). - - from: llmfit - to: llmfit + # llama-server in router mode (fetch-llamacpp.mjs verifies a pinned build and + # SHA-256, then prunes 24 executables to one plus its libraries). The GPU + # backend is Vulkan everywhere off Apple Silicon; no CUDA payload ships. + # Weights are downloaded post-install into the writable data dir, never bundled. + - from: llamacpp + to: llamacpp # stable-diffusion.cpp's sd-server for local image generation (fetch-sdcpp.mjs). # Empty on hosts with no prebuilt binary; the sidecar then simply does not start. - from: sdcpp diff --git a/surfsense_local/electron/package.json b/surfsense_local/electron/package.json index 9275b38778..cf60052199 100644 --- a/surfsense_local/electron/package.json +++ b/surfsense_local/electron/package.json @@ -11,7 +11,7 @@ "main": "./out/main/index.js", "scripts": { "postinstall": "install-electron", - "predev": "pnpm build:model && pnpm build:parser && node scripts/fetch-llmfit.mjs --binary-only", + "predev": "pnpm build:model && pnpm build:parser && pnpm build:llamacpp", "dev": "concurrently -k -n vite,electron -c cyan,magenta \"pnpm --dir ../frontend dev\" \"wait-on tcp:127.0.0.1:5173 && node scripts/electron-dev.mjs\"", "build": "electron-vite build", "build:frontend": "pnpm --dir ../frontend install && pnpm --dir ../frontend build", @@ -19,14 +19,12 @@ "build:model": "uv --directory ../backend run scripts/fetch_embedding_model.py models", "build:voice": "uv --directory ../backend run scripts/fetch_kokoro_model.py models", "build:parser": "uv --directory ../backend run scripts/fetch_docling_models.py models", - "build:ollama": "node scripts/fetch-ollama.mjs", - "build:llmfit": "node scripts/fetch-llmfit.mjs", + "build:llamacpp": "node scripts/fetch-llamacpp.mjs", "build:sdcpp": "node scripts/fetch-sdcpp.mjs", - "dist": "pnpm build:frontend && pnpm build:binaries && pnpm build:model && pnpm build:voice && pnpm build:parser && pnpm build:ollama && pnpm build:llmfit && pnpm build:sdcpp && pnpm build && electron-builder", - "test": "node --disable-warning=MODULE_TYPELESS_PACKAGE_JSON --test src/main/*.test.ts", + "dist": "pnpm build:frontend && pnpm build:binaries && pnpm build:model && pnpm build:voice && pnpm build:parser && pnpm build:llamacpp && pnpm build:sdcpp && pnpm build && electron-builder", + "test": "node --disable-warning=MODULE_TYPELESS_PACKAGE_JSON --test src/main/*.test.ts src/main/**/*.test.ts", "typecheck": "tsc --noEmit -p tsconfig.json", - "check:sidecars": "node scripts/check-sidecars.mjs", - "check:llmfit": "node scripts/check-llmfit.mjs" + "check:sidecars": "node scripts/check-sidecars.mjs" }, "devDependencies": { "@types/node": "^24", diff --git a/surfsense_local/electron/scripts/check-llmfit.mjs b/surfsense_local/electron/scripts/check-llmfit.mjs deleted file mode 100644 index e81b89e92e..0000000000 --- a/surfsense_local/electron/scripts/check-llmfit.mjs +++ /dev/null @@ -1,31 +0,0 @@ -// Runnable staging/final-resource smoke: version pin plus real hardware JSON. -import assert from "node:assert/strict" -import { execFileSync } from "node:child_process" -import { existsSync } from "node:fs" -import { join } from "node:path" -import { fileURLToPath } from "node:url" - -import { VERSION, checkConfiguration } from "./fetch-llmfit.mjs" - -checkConfiguration() - -const here = fileURLToPath(new URL(".", import.meta.url)) -const binary = - process.argv[2] ?? - join(here, "..", "llmfit", process.platform === "win32" ? "llmfit.exe" : "llmfit") - -assert.ok(existsSync(binary), `llmfit binary not found: ${binary}`) - -const run = (args) => - execFileSync(binary, args, { - encoding: "utf8", - timeout: 30_000, - windowsHide: true, - }).trim() - -const version = run(["--version"]) -assert.match(version, new RegExp(`\\b${VERSION.replaceAll(".", "\\.")}\\b`)) - -const system = JSON.parse(run(["--json", "system"])) -assert.ok(system && typeof system === "object", "llmfit system output must be a JSON object") -console.log(`llmfit ${VERSION} smoke OK: ${binary}`) diff --git a/surfsense_local/electron/scripts/fetch-llamacpp.mjs b/surfsense_local/electron/scripts/fetch-llamacpp.mjs new file mode 100644 index 0000000000..b99a751835 --- /dev/null +++ b/surfsense_local/electron/scripts/fetch-llamacpp.mjs @@ -0,0 +1,211 @@ +// Download and atomically stage the pinned llama.cpp build for this host. +// +// llama.cpp publishes about ten builds a day and has no stable channel, so the +// build number and its SHA-256 are both pinned here. The runtime this replaced +// verified no checksum at all; this does, because an unpinned generation runtime +// is the one dependency that can change under us between two identical builds. +// +// The GPU backend is Vulkan on every platform off Apple Silicon. It covers +// NVIDIA, AMD and Intel from one archive, its loader ships with Windows, and it +// is already what this app shipped: the previous runtime's CUDA runners were +// stripped at build time for exactly the same reason. +// Measured at b11050 on an RTX 3050, CUDA leads Vulkan 9.1% on prefill and 2.1% +// on decode, which is 0.66 s of an 11 s turn for 685 MB. See +// plans/community-local/api/08-cuda-backend.md. +import { execFileSync } from "node:child_process" +import { createHash } from "node:crypto" +import { + chmodSync, + copyFileSync, + existsSync, + mkdirSync, + mkdtempSync, + lstatSync, + readdirSync, + readFileSync, + readlinkSync, + renameSync, + rmSync, + symlinkSync, + writeFileSync, +} from "node:fs" +import { tmpdir } from "node:os" +import { basename, dirname, join } from "node:path" +import { fileURLToPath } from "node:url" + +export const BUILD = "b11050" +const BASE = `https://github.com/ggml-org/llama.cpp/releases/download/${BUILD}` +const HERE = fileURLToPath(new URL(".", import.meta.url)) +const OUT = join(HERE, "..", "llamacpp") + +export const TARGETS = { + "darwin-arm64": { + asset: `llama-${BUILD}-bin-macos-arm64.tar.gz`, + sha256: "e64c549a443d1353f440f436039d449a431fbcc1528b03838dda3d9a26530062", + }, + "win32-x64": { + asset: `llama-${BUILD}-bin-win-vulkan-x64.zip`, + sha256: "f1844840ad85b54405b6d5d0d9e8130a6d89c7094186b520fe4c1d9143290e7f", + }, + "linux-x64": { + asset: `llama-${BUILD}-bin-ubuntu-vulkan-x64.tar.gz`, + sha256: "eba230a6c76dee7422d1851651d7c4634fb6a39aed57e524a3612e9ed2bd1d89", + }, +} + +const key = `${process.platform}-${process.arch}` +const SERVER = process.platform === "win32" ? "llama-server.exe" : "llama-server" +const TAR = + process.platform === "win32" + ? join(process.env.SystemRoot ?? "C:\\Windows", "System32", "tar.exe") + : "tar" + +const LIBRARY = /\.(dylib|so)(\.\d+)*$|\.dll$/i +// The archives carry 24 executables and we run one. Their per-tool support +// libraries go with them, which also shrinks the macOS notarization surface. +const OTHER_TOOL_LIBRARY = /^(lib)?llama-(?!server)[a-z0-9-]+-impl\./i + +function isKept(name) { + if (name === SERVER) return true + if (OTHER_TOOL_LIBRARY.test(name)) return false + return LIBRARY.test(name) +} + +function find(root, predicate) { + for (const entry of readdirSync(root, { withFileTypes: true })) { + const path = join(root, entry.name) + if (entry.isDirectory()) { + const nested = find(path, predicate) + if (nested) return nested + } else if (predicate(entry.name)) { + return path + } + } +} + +async function download(url, destination) { + const response = await fetch(url, { redirect: "follow" }) + if (!response.ok) throw new Error(`download failed (${response.status}): ${url}`) + writeFileSync(destination, Buffer.from(await response.arrayBuffer())) +} + +/** Prove the pruned stage still runs, and that it can see its backends. */ +function verifyStage(stage) { + const output = execFileSync(join(stage, SERVER), ["--list-devices"], { + cwd: stage, + encoding: "utf8", + timeout: 120_000, + windowsHide: true, + }) + if (!/Available devices|no devices found/i.test(output)) { + throw new Error(`llama-server --list-devices said nothing useful: ${output}`) + } + return output +} + +export function checkConfiguration() { + for (const target of ["darwin-arm64", "win32-x64", "linux-x64"]) { + const config = TARGETS[target] + if (!config || !config.asset.includes(`-${BUILD}-`)) { + throw new Error(`missing pinned ${BUILD} asset for ${target}`) + } + if (!/^[0-9a-f]{64}$/.test(config.sha256)) { + throw new Error(`invalid SHA-256 for ${target}`) + } + if (target !== "darwin-arm64" && !config.asset.includes("vulkan")) { + throw new Error(`${target} must ship the Vulkan build, got ${config.asset}`) + } + } +} + +async function main() { + checkConfiguration() + if (process.argv.includes("--check")) { + console.log(`llama.cpp ${BUILD} target matrix OK`) + return + } + + const target = TARGETS[key] + if (!target) throw new Error(`no bundled llama.cpp for ${key}`) + + if (existsSync(join(OUT, SERVER))) { + console.log(`llama.cpp ${BUILD} already staged in ${OUT}`) + return + } + + const work = mkdtempSync(join(tmpdir(), "surfsense-llamacpp-")) + const unpacked = join(work, "unpacked") + const stage = `${OUT}.stage-${process.pid}` + const backup = `${OUT}.old-${process.pid}` + try { + mkdirSync(stage, { recursive: true }) + mkdirSync(unpacked, { recursive: true }) + + const archive = join(work, target.asset) + console.log(`downloading ${BASE}/${target.asset}`) + await download(`${BASE}/${target.asset}`, archive) + + const actual = createHash("sha256").update(readFileSync(archive)).digest("hex") + if (actual !== target.sha256) { + throw new Error( + `SHA-256 mismatch for ${target.asset}: expected ${target.sha256}, got ${actual}`, + ) + } + + execFileSync(TAR, ["-xf", archive, "-C", unpacked], { stdio: "inherit" }) + const server = find(unpacked, (entry) => entry === SERVER) + if (!server) throw new Error(`${target.asset} did not contain ${SERVER}`) + + // Windows archives are flat; macOS and Linux nest under one directory. + const source = dirname(server) + // macOS and Linux version their libraries through symlink chains + // (libggml.dylib -> libggml.0.dylib -> libggml.0.24.0.dylib), and the + // runtime loads by the name in the link. Copying only regular files leaves + // llama-server unable to resolve its own @rpath entries, so recreate the + // links rather than dereferencing them into duplicate copies. + let kept = 0 + let bytes = 0 + for (const entry of readdirSync(source, { withFileTypes: true })) { + if (!isKept(entry.name)) continue + const from = join(source, entry.name) + const to = join(stage, entry.name) + if (entry.isSymbolicLink()) { + symlinkSync(readlinkSync(from), to) + kept += 1 + continue + } + if (!entry.isFile()) continue + copyFileSync(from, to) + if (process.platform !== "win32") chmodSync(to, 0o755) + kept += 1 + bytes += lstatSync(from).size + } + + const licence = find(unpacked, (entry) => entry.toUpperCase().startsWith("LICENSE")) + if (licence) copyFileSync(licence, join(stage, "LICENSE")) + + verifyStage(stage) + + rmSync(backup, { recursive: true, force: true }) + if (existsSync(OUT)) renameSync(OUT, backup) + try { + renameSync(stage, OUT) + } catch (error) { + if (existsSync(backup)) renameSync(backup, OUT) + throw error + } + rmSync(backup, { recursive: true, force: true }) + const mb = (bytes / 1024 / 1024).toFixed(1) + console.log(`llama.cpp ${BUILD} staged in ${OUT}: ${kept} files, ${mb} MB (${basename(target.asset)})`) + } finally { + rmSync(work, { recursive: true, force: true }) + rmSync(stage, { recursive: true, force: true }) + } +} + +if (process.argv[1] === fileURLToPath(import.meta.url)) { + main().catch((error) => { + console.error(error) + process.exitCode = 1 + }) +} diff --git a/surfsense_local/electron/scripts/fetch-llmfit.mjs b/surfsense_local/electron/scripts/fetch-llmfit.mjs deleted file mode 100644 index 6b2d5b863b..0000000000 --- a/surfsense_local/electron/scripts/fetch-llmfit.mjs +++ /dev/null @@ -1,190 +0,0 @@ -// Download and atomically stage the pinned official llmfit binary for this host. -// SHA-256 values are from the matching .sha256 assets on the v1.1.11 release. -import { execFileSync } from "node:child_process" -import { createHash } from "node:crypto" -import { - chmodSync, - copyFileSync, - existsSync, - mkdirSync, - mkdtempSync, - readdirSync, - readFileSync, - renameSync, - rmSync, - writeFileSync, -} from "node:fs" -import { tmpdir } from "node:os" -import { basename, join } from "node:path" -import { fileURLToPath } from "node:url" - -export const VERSION = "1.1.11" -const TAG = `v${VERSION}` -const BASE = `https://github.com/AlexsJones/llmfit/releases/download/${TAG}` -const HERE = fileURLToPath(new URL(".", import.meta.url)) -const OUT = join(HERE, "..", "llmfit") - -export const TARGETS = { - "linux-x64": { - asset: `llmfit-${TAG}-x86_64-unknown-linux-musl.tar.gz`, - sha256: "ec5e2ac6438ba7bc1d6452a581087f2a6533d504136c1f2a70500b0f558896c5", - }, - "darwin-arm64": { - asset: `llmfit-${TAG}-aarch64-apple-darwin.tar.gz`, - sha256: "f42249d8af58067dd47ef25f997be24ed2a7c6659499bfd16e7c8c431c6e8baf", - }, - "darwin-x64": { - asset: `llmfit-${TAG}-x86_64-apple-darwin.tar.gz`, - sha256: "9c9f56ccf64fee455c5a38b14aa3dd92d95755778e926f986de1c427b6ced136", - }, - "win32-x64": { - asset: `llmfit-${TAG}-x86_64-pc-windows-msvc.zip`, - sha256: "3255ed3ab916a0d1034cf11288f53c2790be8c7194b3df7325d627614548a65a", - }, -} - -const key = `${process.platform}-${process.arch}` -const binaryName = process.platform === "win32" ? "llmfit.exe" : "llmfit" -// Under bash on Windows, PATH resolves to Git's GNU tar, which reads "C:\..." as a host and can't open zip. -const TAR = - process.platform === "win32" ? join(process.env.SystemRoot ?? "C:\\Windows", "System32", "tar.exe") : "tar" - -function find(root, predicate) { - for (const entry of readdirSync(root, { withFileTypes: true })) { - const path = join(root, entry.name) - if (entry.isDirectory()) { - const nested = find(path, predicate) - if (nested) return nested - } else if (predicate(entry.name)) { - return path - } - } -} - -function verifyVersion(binary) { - const output = execFileSync(binary, ["--version"], { - encoding: "utf8", - timeout: 10_000, - windowsHide: true, - }).trim() - if (!new RegExp(`\\b${VERSION.replaceAll(".", "\\.")}\\b`).test(output)) { - throw new Error(`expected llmfit ${VERSION}, got: ${output || ""}`) - } -} - -async function download(url, destination) { - const response = await fetch(url, { redirect: "follow" }) - if (!response.ok) throw new Error(`download failed (${response.status}): ${url}`) - writeFileSync(destination, Buffer.from(await response.arrayBuffer())) -} - -async function copyReleaseDocument(unpacked, stage, name, required = false) { - const archived = find(unpacked, (entry) => entry.toUpperCase() === name) - if (archived) { - copyFileSync(archived, join(stage, name)) - return - } - - try { - const response = await fetch( - `https://raw.githubusercontent.com/AlexsJones/llmfit/${TAG}/${name}`, - ) - if (response.ok) { - writeFileSync(join(stage, name), Buffer.from(await response.arrayBuffer())) - } else if (required) { - throw new Error(`upstream ${name} unavailable (${response.status})`) - } - } catch (error) { - if (required) throw error - } -} - -export function checkConfiguration() { - const expected = ["linux-x64", "darwin-arm64", "darwin-x64", "win32-x64"] - for (const target of expected) { - const config = TARGETS[target] - if (!config || !config.asset.startsWith(`llmfit-${TAG}-`)) { - throw new Error(`missing official ${TAG} asset for ${target}`) - } - if (!/^[0-9a-f]{64}$/.test(config.sha256)) { - throw new Error(`invalid SHA-256 for ${target}`) - } - } -} - -async function main() { - checkConfiguration() - if (process.argv.includes("--check")) { - console.log(`llmfit ${VERSION} target matrix OK`) - return - } - - const target = TARGETS[key] - if (!target) throw new Error(`no bundled llmfit for ${key}`) - const binaryOnly = process.argv.includes("--binary-only") - - const existing = join(OUT, binaryName) - if (existsSync(existing)) { - try { - verifyVersion(existing) - if (binaryOnly || existsSync(join(OUT, "LICENSE"))) { - console.log(`llmfit ${VERSION} already staged in ${OUT}`) - return - } - } catch (error) { - console.warn(`replacing invalid llmfit stage: ${error.message}`) - } - } - - const work = mkdtempSync(join(tmpdir(), "surfsense-llmfit-")) - const unpacked = join(work, "unpacked") - const stage = `${OUT}.stage-${process.pid}` - const backup = `${OUT}.old-${process.pid}` - try { - rmSync(stage, { recursive: true, force: true }) - mkdirSync(stage, { recursive: true }) - mkdirSync(unpacked, { recursive: true }) - - const archive = join(work, target.asset) - const url = `${BASE}/${target.asset}` - console.log(`downloading ${url}`) - await download(url, archive) - - const actual = createHash("sha256").update(readFileSync(archive)).digest("hex") - if (actual !== target.sha256) { - throw new Error(`SHA-256 mismatch for ${target.asset}: expected ${target.sha256}, got ${actual}`) - } - - // bsdtar is available on all supported packaging runners and reads zip too. - execFileSync(TAR, ["-xf", archive, "-C", unpacked], { stdio: "inherit" }) - const extracted = find(unpacked, (entry) => entry === binaryName) - if (!extracted) throw new Error(`${target.asset} did not contain ${binaryName}`) - copyFileSync(extracted, join(stage, binaryName)) - if (process.platform !== "win32") chmodSync(join(stage, binaryName), 0o755) - - await copyReleaseDocument(unpacked, stage, "LICENSE", !binaryOnly) - await copyReleaseDocument(unpacked, stage, "NOTICE") - verifyVersion(join(stage, binaryName)) - - rmSync(backup, { recursive: true, force: true }) - if (existsSync(OUT)) renameSync(OUT, backup) - try { - renameSync(stage, OUT) - } catch (error) { - if (existsSync(backup)) renameSync(backup, OUT) - throw error - } - rmSync(backup, { recursive: true, force: true }) - console.log(`llmfit ${VERSION} staged in ${OUT} (${basename(target.asset)})`) - } finally { - rmSync(work, { recursive: true, force: true }) - rmSync(stage, { recursive: true, force: true }) - } -} - -if (process.argv[1] === fileURLToPath(import.meta.url)) { - main().catch((error) => { - console.error(error) - process.exitCode = 1 - }) -} diff --git a/surfsense_local/electron/scripts/fetch-ollama.mjs b/surfsense_local/electron/scripts/fetch-ollama.mjs deleted file mode 100644 index 2b46065b86..0000000000 --- a/surfsense_local/electron/scripts/fetch-ollama.mjs +++ /dev/null @@ -1,91 +0,0 @@ -// Stage the Ollama server into electron/ollama so electron-builder can ship it -// as a sidecar (see electron-builder.yml extraResources, sidecars/ollama.ts). -// Downloads the standalone archive for the host OS and extracts it, native -// layout preserved (bin/ollama + lib/ollama, or ollama.exe + lib on Windows). -// Idempotent: a present binary is left alone. Run per-OS, on the same machine -// that packages the app (the runtime can't be cross-fetched). -import { execFileSync } from "node:child_process" -import { existsSync, mkdirSync, readdirSync, rmSync } from "node:fs" -import { tmpdir } from "node:os" -import { join } from "node:path" -import { fileURLToPath } from "node:url" - -const OUT = join(fileURLToPath(new URL(".", import.meta.url)), "..", "ollama") -// Pinned to a release: ollama.com/download stopped serving stable archive URLs, -// and a pinned runtime keeps the packaged build reproducible. Bump deliberately. -const VERSION = "v0.33.3" -const BASE = `https://github.com/ollama/ollama/releases/download/${VERSION}` - -// Full GPU-capable archives, one per host. Linux ships zstd tarballs now; macOS -// stays gzip, Windows zip. -const TARGETS = { - "linux-x64": { url: `${BASE}/ollama-linux-amd64.tar.zst`, kind: "zst" }, - "linux-arm64": { url: `${BASE}/ollama-linux-arm64.tar.zst`, kind: "zst" }, - "darwin-x64": { url: `${BASE}/ollama-darwin.tgz`, kind: "tgz" }, - "darwin-arm64": { url: `${BASE}/ollama-darwin.tgz`, kind: "tgz" }, - "win32-x64": { url: `${BASE}/ollama-windows-amd64.zip`, kind: "zip" }, -} - -// tar flags per archive kind (bsdtar reads zip on macOS/Windows). -const TAR_FLAGS = { - tgz: ["-xzf"], - zst: ["--zstd", "-xf"], - zip: ["-xf"], -} -// Under bash on Windows, PATH resolves to Git's GNU tar, which reads "C:\..." as a host and can't open zip. -const TAR = - process.platform === "win32" ? join(process.env.SystemRoot ?? "C:\\Windows", "System32", "tar.exe") : "tar" - -// The archive carries a whole CUDA toolkit per major version (cuda_v12 + -// cuda_v13 is 1.7 GB of the 1.8 GB on Windows), which on its own pushed the -// installer payload past the 2 GB that 32-bit makensis can mmap. Vulkan is on -// by default and resolves the loader from the host GPU driver, so the remaining -// runners still cover NVIDIA, AMD, and Intel. -// ponytail: NVIDIA falls back to Vulkan, which is slower than CUDA. The upgrade -// path is an opt-in post-install pack: Ollama discovers runners by path, so -// dropping cuda_* back into lib/ollama is all it takes. -function pruneCudaRunners() { - const runners = join(OUT, "lib", "ollama") - if (!existsSync(runners)) return - for (const entry of readdirSync(runners)) { - if (!entry.startsWith("cuda_")) continue - rmSync(join(runners, entry), { recursive: true, force: true }) - console.log(`pruned ${entry}`) - } -} - -const binary = process.platform === "win32" ? "ollama.exe" : "ollama" -const staged = [join(OUT, "bin", binary), join(OUT, binary)] -if (staged.some(existsSync)) { - // Prune here too: a tree staged by an earlier build still carries the runners. - pruneCudaRunners() - console.log(`ollama already staged in ${OUT}`) - process.exit(0) -} - -const key = `${process.platform}-${process.arch}` -const target = TARGETS[key] -if (!target) { - console.error(`no bundled ollama for ${key}`) - process.exit(1) -} - -rmSync(OUT, { recursive: true, force: true }) -mkdirSync(OUT, { recursive: true }) - -const archive = join(tmpdir(), `ollama.${target.kind}`) -console.log(`downloading ${target.url}`) -execFileSync("curl", ["-fSL", "--retry", "3", target.url, "-o", archive], { stdio: "inherit" }) - -console.log(`extracting into ${OUT}`) -execFileSync(TAR, [...TAR_FLAGS[target.kind], archive, "-C", OUT], { - stdio: "inherit", -}) -rmSync(archive, { force: true }) - -if (!staged.some(existsSync)) { - console.error(`extraction did not yield ${binary} under ${OUT}`) - process.exit(1) -} -pruneCudaRunners() -console.log("ollama staged") diff --git a/surfsense_local/electron/scripts/fetch-sdcpp.mjs b/surfsense_local/electron/scripts/fetch-sdcpp.mjs index d925273464..5db52921fa 100644 --- a/surfsense_local/electron/scripts/fetch-sdcpp.mjs +++ b/surfsense_local/electron/scripts/fetch-sdcpp.mjs @@ -5,7 +5,7 @@ // // Only the Vulkan and Metal builds are staged. The CUDA and ROCm archives are // 182-537 MB and Vulkan already reaches NVIDIA, AMD, and Intel through the host -// driver -- the same trade the bundled Ollama makes. +// driver -- the same trade the bundled llama.cpp build makes. // // Upstream publishes rolling master builds with no semver and no checksums, so // the tag and a locally computed SHA-256 are both pinned here. Bump deliberately. From 4fafae315d768d7d3821d5890e890acd0eaf6cb7 Mon Sep 17 00:00:00 2001 From: Anish Sarkar <104695310+AnishSarkar22@users.noreply.github.com> Date: Sun, 20 Sep 2026 20:07:42 +0530 Subject: [PATCH 10/60] ci(release-local): pin ubuntu-22.04 and smoke llama-server instead of Ollama Co-Authored-By: Claude Sonnet 5 --- .github/workflows/release-local.yml | 46 +++++++++++------------------ 1 file changed, 17 insertions(+), 29 deletions(-) diff --git a/.github/workflows/release-local.yml b/.github/workflows/release-local.yml index ce5a9863a5..1b964dcd84 100644 --- a/.github/workflows/release-local.yml +++ b/.github/workflows/release-local.yml @@ -36,19 +36,18 @@ jobs: fail-fast: false matrix: include: - - os: ubuntu-latest + # Pinned: `ubuntu-latest` migrates to 26.04 and would silently raise + # the AppImage's glibc floor. llama.cpp's Vulkan build needs 2.34. + - os: ubuntu-22.04 platform: --linux electron_cache: ~/.cache/electron - llmfit_resource: release/linux-unpacked/resources/llmfit/llmfit # Apple Silicon only: torch (2.3+) and onnxruntime (1.24+) no longer ship Intel Mac wheels. - os: macos-15 platform: --mac electron_cache: ~/Library/Caches/electron - llmfit_resource: release/mac-arm64/SurfSense.app/Contents/Resources/llmfit/llmfit - os: windows-latest platform: --win electron_cache: ~\AppData\Local\electron\Cache - llmfit_resource: release/win-unpacked/resources/llmfit/llmfit.exe defaults: run: @@ -193,15 +192,9 @@ jobs: pnpm install --frozen-lockfile pnpm build - - name: Stage Ollama + - name: Stage llama.cpp working-directory: surfsense_local/electron - run: node scripts/fetch-ollama.mjs - - - name: Stage and smoke llmfit - working-directory: surfsense_local/electron - run: | - node scripts/fetch-llmfit.mjs - node scripts/check-llmfit.mjs + run: node scripts/fetch-llamacpp.mjs - name: Package installer working-directory: surfsense_local/electron @@ -238,12 +231,8 @@ jobs: "${CMD[@]}" - - name: Smoke llmfit from final resources - working-directory: surfsense_local/electron - run: node scripts/check-llmfit.mjs "${{ matrix.llmfit_resource }}" - - name: Smoke the frozen API (Linux) - if: matrix.os == 'ubuntu-latest' + if: matrix.os == 'ubuntu-22.04' run: | set -e export SURFSENSE_LOCAL_DATA_DIR="$RUNNER_TEMP/ss-smoke" @@ -258,20 +247,19 @@ jobs: done echo "::error::packaged API never became healthy"; kill $API; exit 1 - - name: Smoke the bundled Ollama (Linux) - if: matrix.os == 'ubuntu-latest' + - name: Smoke the bundled llama.cpp (Linux) + if: matrix.os == 'ubuntu-22.04' run: | set -e - export OLLAMA_HOST=127.0.0.1:11565 - ./surfsense_local/electron/release/linux-unpacked/resources/ollama/bin/ollama serve & - OLL=$! - for i in $(seq 1 30); do - if curl -fsS "http://127.0.0.1:11565/" >/dev/null; then - echo "ollama OK"; kill $OLL; exit 0 - fi - sleep 1 - done - echo "::error::bundled ollama never responded"; kill $OLL; exit 1 + DIR=./surfsense_local/electron/release/linux-unpacked/resources/llamacpp + # Run from the library directory: ggml scans the running executable's + # own directory for backends and silently reports none anywhere else, + # which is indistinguishable from a machine with no GPU. + OUT=$(cd "$DIR" && ./llama-server --list-devices 2>&1) + echo "$OUT" + if ! grep -qiE "Available devices|no devices found" <<<"$OUT"; then + echo "::error::packaged llama-server could not list devices"; exit 1 + fi - name: Upload installers uses: actions/upload-artifact@v4 From ca26d10c6a2387320c2fb38b13ca18f2f2a72a27 Mon Sep 17 00:00:00 2001 From: Anish Sarkar <104695310+AnishSarkar22@users.noreply.github.com> Date: Sun, 20 Sep 2026 20:07:42 +0530 Subject: [PATCH 11/60] feat(electron): run llama-server in router mode as a sidecar Co-Authored-By: Claude Sonnet 5 --- surfsense_local/electron/src/main/index.ts | 81 ++++++++++-- .../src/main/sidecars/llamacpp.test.ts | 115 ++++++++++++++++++ .../electron/src/main/sidecars/llamacpp.ts | 91 ++++++++++++++ .../electron/src/main/sidecars/ollama.ts | 34 ------ .../electron/src/main/sidecars/platform.ts | 2 +- .../electron/src/main/sidecars/python.ts | 17 ++- .../electron/src/main/sidecars/sdcpp.ts | 2 +- .../electron/src/main/sidecars/types.ts | 24 +++- 8 files changed, 305 insertions(+), 61 deletions(-) create mode 100644 surfsense_local/electron/src/main/sidecars/llamacpp.test.ts create mode 100644 surfsense_local/electron/src/main/sidecars/llamacpp.ts delete mode 100644 surfsense_local/electron/src/main/sidecars/ollama.ts diff --git a/surfsense_local/electron/src/main/index.ts b/surfsense_local/electron/src/main/index.ts index df29d0dd5e..590f0d5bb0 100644 --- a/surfsense_local/electron/src/main/index.ts +++ b/surfsense_local/electron/src/main/index.ts @@ -1,3 +1,4 @@ +import { mkdirSync, statSync } from "node:fs" import { join } from "node:path" import { @@ -17,8 +18,11 @@ import { autoUpdater } from "electron-updater" import { managedOriginalPath } from "./document-files.ts" import { getFreePort, waitForHealth } from "./net.ts" import { loadSecret } from "./secret.ts" -import { ollamaSpec } from "./sidecars/ollama.ts" -import { exe } from "./sidecars/platform.ts" +import { + llamacppSpec, + LLAMACPP_SIDECAR, + PRESET_FILE, +} from "./sidecars/llamacpp.ts" import { apiSpec, workerSpec } from "./sidecars/python.ts" import { sdcppSpec, @@ -127,6 +131,46 @@ function watchImageModel(ctx: SidecarContext): void { timer.unref() } +// llama-server reads its per-model arguments from a preset INI **once, at +// startup**: appending a section while it runs does not surface the model, +// measured. The API rewrites that file whenever it installs a model or reprices +// one, so the router has to be restarted to see it. Same shape as +// watchImageModel, and for the same reason: the API is the authority, and a +// change is user-initiated and rare. +function watchGenerationPreset(ctx: SidecarContext): void { + if (ctx.llamacppModelsDir == null) return + const preset = join(ctx.llamacppModelsDir, PRESET_FILE) + let current = presetStamp(preset) + + const reconcile = async () => { + if (!sidecars || shuttingDown) return + const stamp = presetStamp(preset) + if (stamp === current) return + current = stamp + + if (sidecars.has(LLAMACPP_SIDECAR)) await stopNamed(sidecars, LLAMACPP_SIDECAR) + const spec = llamacppSpec(ctx) + if (spec) startOne(sidecars, spec, onSidecarCrash) + } + + const timer = setInterval(() => { + void reconcile().catch(() => { + // Mid-write or mid-restart; the next tick tries again. + }) + }, 5000) + timer.unref() +} + +/** Size and mtime, which is enough to notice a rewrite and costs no read. */ +function presetStamp(path: string): string { + try { + const stats = statSync(path) + return `${stats.size}:${stats.mtimeMs}` + } catch { + return "absent" + } +} + async function bootSidecars(): Promise<{ apiUrl: string; dataDir: string }> { const host = "127.0.0.1" const packaged = app.isPackaged @@ -149,34 +193,45 @@ async function bootSidecars(): Promise<{ apiUrl: string; dataDir: string }> { modelsDir: packaged ? join(process.resourcesPath, "models") : join(app.getAppPath(), "..", "backend", "models"), - llmfitPath: packaged - ? join(process.resourcesPath, "llmfit", exe("llmfit")) - : join(app.getAppPath(), "llmfit", exe("llmfit")), + // The staged llama.cpp build, same bytes either way: `fetch-llamacpp.mjs` + // writes electron/llamacpp/ and packaging copies that folder verbatim. + llamacppBinariesDir: packaged + ? join(process.resourcesPath, "llamacpp") + : join(app.getAppPath(), "llamacpp"), } + // Unconditional, because dev needs a runtime too and DATA_DIR already keeps + // dev models out of the real install (~/.surfsense-dev). + ctx.llamacppPort = await getFreePort(host) + ctx.llamacppModelsDir = join(dataDir, "models") + ctx.llamacppUrl = `http://${host}:${ctx.llamacppPort}` + // llama-server exits 1 when --models-dir does not exist, and on a clean + // install nothing has created it yet: only a download would, and a download + // needs the runtime. Measured: "failed to initialize router models: error: + // '' does not exist or is not a directory". + mkdirSync(ctx.llamacppModelsDir, { recursive: true }) + if (packaged) { - ctx.ollamaPort = await getFreePort(host) - ctx.ollamaModelsDir = join(dataDir, "ollama") - ctx.ollamaUrl = `http://${host}:${ctx.ollamaPort}` ctx.imagePort = await getFreePort(host) ctx.imageModelsDir = join(dataDir, "images") ctx.imageUrl = `http://${host}:${ctx.imagePort}` } - // ollamaSpec is null in dev (the developer runs their own `ollama serve`). - // sd-server is absent here on purpose: watchImageModel owns it, because only - // the API knows which model was chosen. + // llamacppSpec is null in dev, where no binary is staged. sd-server is absent + // here on purpose: watchImageModel owns it, because only the API knows which + // model was chosen. const specs = [ apiSpec(ctx), workerSpec(ctx, "ingest"), workerSpec(ctx, "studio"), - ollamaSpec(ctx), + llamacppSpec(ctx), ].filter( (s): s is SidecarSpec => s !== null ) sidecars = startAll(specs, onSidecarCrash) watchImageModel(ctx) + watchGenerationPreset(ctx) - // gate on the API only; fail fast if it dies during startup. Ollama is + // gate on the API only; fail fast if it dies during startup. llama-server is // best-effort (its state shows via /llm/providers; an early exit hits onSidecarCrash). await waitForHealth(host, apiPort, { child: sidecars.get("api") }) return { apiUrl: `http://${host}:${apiPort}`, dataDir } diff --git a/surfsense_local/electron/src/main/sidecars/llamacpp.test.ts b/surfsense_local/electron/src/main/sidecars/llamacpp.test.ts new file mode 100644 index 0000000000..a057b71d8a --- /dev/null +++ b/surfsense_local/electron/src/main/sidecars/llamacpp.test.ts @@ -0,0 +1,115 @@ +import assert from "node:assert/strict" +import { mkdtempSync, mkdirSync, writeFileSync } from "node:fs" +import { tmpdir } from "node:os" +import { join } from "node:path" +import test from "node:test" + +import { llamacppSpec, PRESET_FILE } from "./llamacpp.ts" +import { exe } from "./platform.ts" +import type { SidecarContext } from "./types.ts" + +function staged(): { ctx: SidecarContext; models: string } { + const root = mkdtempSync(join(tmpdir(), "llamacpp-")) + const binaries = join(root, "resources") + const runtime = join(binaries, "llamacpp") + const models = join(root, "models") + mkdirSync(runtime, { recursive: true }) + mkdirSync(models, { recursive: true }) + writeFileSync(join(runtime, exe("llama-server")), "") + return { + ctx: { + packaged: true, + backendDir: root, + binariesDir: binaries, + host: "127.0.0.1", + apiPort: 1, + dataDir: root, + secret: "s", + llamacppPort: 9999, + llamacppModelsDir: models, + llamacppBinariesDir: runtime, + }, + models, + } +} + +test("runs in dev too, because only the binary's path differs", () => { + // The previous runtime was a system daemon a developer already had running, + // so dev needed no wiring. A pinned build has to be started by the app in + // both modes, or dev tests a version the app never ships. + const { ctx } = staged() + assert.ok(llamacppSpec({ ...ctx, packaged: false })) +}) + +test("does not run when the pinned build is not staged", () => { + const { ctx } = staged() + assert.equal( + llamacppSpec({ ...ctx, llamacppBinariesDir: join(ctx.binariesDir, "absent") }), + null + ) +}) + +test("runs from the library directory so ggml can find its backends", () => { + // ggml scans the running executable's own directory. Anywhere else it reports + // no devices, silently, and every model lands on the CPU. + const { ctx } = staged() + const spec = llamacppSpec(ctx) + assert.ok(spec) + assert.equal(spec.cwd, ctx.llamacppBinariesDir) +}) + +test("starts with no model, because the router serves an empty directory", () => { + const { ctx } = staged() + const spec = llamacppSpec(ctx) + assert.ok(spec) + assert.ok(!spec.args.includes("--models-preset")) + assert.ok(spec.args.includes("--models-dir")) +}) + +test("passes the preset file once the API has written one", () => { + const { ctx, models } = staged() + writeFileSync(join(models, PRESET_FILE), "[m]\nctx-size = 16384\n") + + const spec = llamacppSpec(ctx) + + assert.ok(spec) + assert.ok(spec.args.includes("--models-preset")) + assert.equal(spec.args.at(-1), join(models, PRESET_FILE)) +}) + +test("keeps a loaded model resident between two questions", () => { + // Measured: without this a model self-evicted after roughly 30s idle, which + // turns the second question of a conversation into a reload. + const { ctx } = staged() + const spec = llamacppSpec(ctx) + assert.ok(spec) + assert.equal(spec.args[spec.args.indexOf("--sleep-idle-seconds") + 1], "300") +}) + +test("never sets a layer count, which would abort --fit", () => { + // With -ngl set by hand the fitter aborts and the model loads entirely on the + // CPU: no error, exit 0, and it looks like it worked. + const { ctx } = staged() + const spec = llamacppSpec(ctx) + assert.ok(spec) + assert.ok(!spec.args.some((a) => a === "-ngl" || a === "--n-gpu-layers")) +}) + +test("keeps llama.cpp's cache inside the app data dir", () => { + const { ctx, models } = staged() + const spec = llamacppSpec(ctx) + assert.ok(spec) + assert.equal(spec.env.LLAMA_CACHE, models) +}) + +test("never sets a router wide reasoning budget", () => { + // Two reasons, and the second is the one that bites. A budget here applies to + // every answer, not just the short calls that need it. And the server only + // reads the per request `thinking_budget_tokens` while this flag sits at its + // default, so passing it would silently undo the fix that keeps a thinking + // model from returning an empty title. + const { ctx } = staged() + const spec = llamacppSpec(ctx) + assert.ok(spec) + assert.ok(!spec.args.some((arg) => arg.startsWith("--reasoning-budget"))) +}) diff --git a/surfsense_local/electron/src/main/sidecars/llamacpp.ts b/surfsense_local/electron/src/main/sidecars/llamacpp.ts new file mode 100644 index 0000000000..8e807a874a --- /dev/null +++ b/surfsense_local/electron/src/main/sidecars/llamacpp.ts @@ -0,0 +1,91 @@ +/** + * llama-server in router mode: the local generation runtime. + * + * One process owns the models directory and spawns a worker per loaded model, + * which is why the supervisor's existing tree kill matters here. Unlike + * sd-server it starts with no model at all, so it boots with the others: the + * router runs happily against an empty directory and reports `role: router`. + * + * Runs in dev as well as packaged, and for the same reason the Python sidecars + * do: only the path to the binary differs. The previous runtime was a system + * daemon a developer happened to have running, so dev needed no wiring; this + * one is a pinned build, and testing against a different one tests something + * the app never ships. + * + * Per-model arguments do not come from the API at runtime. `POST /models/load` + * accepts an `args` field and ignores it, measured, so the window and cache + * precision the fit calculation chose are written to a preset INI that the + * router reads **once at startup**. Changing a model's plan, or installing one, + * therefore means rewriting that file and restarting this sidecar. + */ +import { existsSync } from "node:fs" +import { join } from "node:path" + +import { exe } from "./platform.ts" +import type { SidecarContext, SidecarSpec } from "./types.ts" + +export const LLAMACPP_SIDECAR = "llamacpp" + +/** The preset file the API writes; the router reads it at startup. */ +export const PRESET_FILE = "models.ini" + +export function binaryPath(ctx: SidecarContext): string { + return join(ctx.llamacppBinariesDir ?? "", exe("llama-server")) +} + +/** + * Null when the pinned build is not staged, which is the only condition that + * matters: `pnpm build:llamacpp` fetches it, and `predev` runs that too. + */ +export function llamacppSpec(ctx: SidecarContext): SidecarSpec | null { + if ( + ctx.llamacppPort == null || + ctx.llamacppModelsDir == null || + ctx.llamacppBinariesDir == null + ) { + return null + } + + const binary = binaryPath(ctx) + if (!existsSync(binary)) return null + + const directory = ctx.llamacppBinariesDir + const preset = join(ctx.llamacppModelsDir, PRESET_FILE) + + const args = [ + "--models-dir", + ctx.llamacppModelsDir, + "--host", + ctx.host, + "--port", + String(ctx.llamacppPort), + // One model resident at a time. This app asks one question at a time, and + // a second resident model is memory taken from the one being used. + "--models-max", + "1", + // Without this a model self-evicted after roughly 30s idle, measured, which + // turns the second question of a conversation into a reload. + "--sleep-idle-seconds", + "300", + // llama-server ships its own web UI, which we neither need nor want exposed. + "--no-ui", + "--jinja", + // Routes blocks to message.reasoning_content. Without it a thinking + // model's trace enters the answer and citation rewriting corrupts it. + "--reasoning-format", + "deepseek", + ] + // Absent until the API has priced a model, and the router rejects a missing file. + if (existsSync(preset)) args.push("--models-preset", preset) + + return { + name: LLAMACPP_SIDECAR, + cmd: binary, + args, + // ggml scans the running executable's own directory for backend libraries, + // so this cwd is what lets it find them. Elsewhere it reports no devices, + // silently, and every model would run on the CPU. + cwd: directory, + env: { LLAMA_CACHE: ctx.llamacppModelsDir }, + } +} diff --git a/surfsense_local/electron/src/main/sidecars/ollama.ts b/surfsense_local/electron/src/main/sidecars/ollama.ts deleted file mode 100644 index f55d847a50..0000000000 --- a/surfsense_local/electron/src/main/sidecars/ollama.ts +++ /dev/null @@ -1,34 +0,0 @@ -/** - * The bundled Ollama server. Packaged only: dev uses the developer's own - * `ollama serve`, so this returns null and nothing is spawned. Its runners live - * beside the binary (lib/ollama), which Ollama discovers on its own. - */ -import { existsSync } from "node:fs" -import { join } from "node:path" - -import { exe } from "./platform.ts" -import type { SidecarContext, SidecarSpec } from "./types.ts" - -// The archive layout differs per OS (bin/ollama vs a root-level ollama.exe), so -// resolve the binary rather than assume one shape. -function resolveBinary(dir: string): string { - const name = exe("ollama") - const candidates = [join(dir, "bin", name), join(dir, name)] - return candidates.find(existsSync) ?? candidates[0] -} - -export function ollamaSpec(ctx: SidecarContext): SidecarSpec | null { - if (!ctx.packaged || ctx.ollamaPort == null || !ctx.ollamaModelsDir) return null - - const dir = join(ctx.binariesDir, "ollama") - return { - name: "ollama", - cmd: resolveBinary(dir), - args: ["serve"], - cwd: dir, - env: { - OLLAMA_HOST: `${ctx.host}:${ctx.ollamaPort}`, - OLLAMA_MODELS: ctx.ollamaModelsDir, - }, - } -} diff --git a/surfsense_local/electron/src/main/sidecars/platform.ts b/surfsense_local/electron/src/main/sidecars/platform.ts index 2bd7b6884c..136589c6af 100644 --- a/surfsense_local/electron/src/main/sidecars/platform.ts +++ b/surfsense_local/electron/src/main/sidecars/platform.ts @@ -1,5 +1,5 @@ export const isWindows = process.platform === "win32" -// onedir binaries and the Ollama archive name the executable after the tool; +// onedir binaries and the llama.cpp archive name the executable after the tool; // Windows adds .exe. export const exe = (name: string): string => (isWindows ? `${name}.exe` : name) diff --git a/surfsense_local/electron/src/main/sidecars/python.ts b/surfsense_local/electron/src/main/sidecars/python.ts index 09641b0f5b..b2a195acbd 100644 --- a/surfsense_local/electron/src/main/sidecars/python.ts +++ b/surfsense_local/electron/src/main/sidecars/python.ts @@ -1,7 +1,7 @@ /** * The Python sidecars: the API, and the worker once per queue. Same shape: a * frozen onedir binary when packaged, `uv run` in dev, same SURFSENSE_LOCAL_* - * env. Both reach Ollama, so both need the bundled address. + * env. Both reach llama-server, so both need the bundled address. */ import { join } from "node:path" @@ -17,9 +17,15 @@ function pythonEnv(ctx: SidecarContext): Record { SURFSENSE_LOCAL_SECRET: ctx.secret, ...(ctx.modelsDir && { SURFSENSE_LOCAL_MODELS_DIR: ctx.modelsDir }), ...(ctx.packaged && { HF_HUB_OFFLINE: "1" }), - ...(ctx.ollamaUrl && { SURFSENSE_LOCAL_OLLAMA_BASE_URL: ctx.ollamaUrl }), - ...(ctx.ollamaModelsDir && { - SURFSENSE_LOCAL_OLLAMA_MODELS_DIR: ctx.ollamaModelsDir, + ...(ctx.llamacppUrl && { SURFSENSE_LOCAL_LLAMACPP_BASE_URL: ctx.llamacppUrl }), + ...(ctx.llamacppModelsDir && { + SURFSENSE_LOCAL_LLAMACPP_MODELS_DIR: ctx.llamacppModelsDir, + }), + // The API probes hardware by loading ggml from here. Without it the probe + // runs from the wrong directory, finds no backends, and reports a CPU-only + // machine with no error at all. + ...(ctx.llamacppBinariesDir && { + SURFSENSE_LOCAL_LLAMACPP_LIBRARY_DIR: ctx.llamacppBinariesDir, }), ...(ctx.imageUrl && { SURFSENSE_LOCAL_IMAGE_BASE_URL: ctx.imageUrl }), ...(ctx.imageModelsDir && { @@ -45,8 +51,7 @@ export function apiSpec(ctx: SidecarContext): SidecarSpec { ...pythonCmd(ctx, "api", "main.py"), env: { ...pythonEnv(ctx), - ...(ctx.llmfitPath && { SURFSENSE_LOCAL_LLMFIT_PATH: ctx.llmfitPath }), - }, + }, } } diff --git a/surfsense_local/electron/src/main/sidecars/sdcpp.ts b/surfsense_local/electron/src/main/sidecars/sdcpp.ts index 5c9f676bf1..0ae5985e8c 100644 --- a/surfsense_local/electron/src/main/sidecars/sdcpp.ts +++ b/surfsense_local/electron/src/main/sidecars/sdcpp.ts @@ -1,6 +1,6 @@ /** * stable-diffusion.cpp's sd-server: local image generation. Packaged only, like - * Ollama, and unlike Ollama it holds exactly one model, named by -m at startup + * llama-server, and unlike it this holds exactly one model, named by -m at startup * and never switchable at runtime. So the process follows the chosen model: the * API says which weights and which flags, and index.ts restarts it on a change. * diff --git a/surfsense_local/electron/src/main/sidecars/types.ts b/surfsense_local/electron/src/main/sidecars/types.ts index 7938b3e230..5e808561cd 100644 --- a/surfsense_local/electron/src/main/sidecars/types.ts +++ b/surfsense_local/electron/src/main/sidecars/types.ts @@ -25,12 +25,24 @@ export interface SidecarContext { secret: string /** Packaged: read-only bundled embedding, voice, and Docling parser packs. */ modelsDir?: string - /** Packaged: absolute path to the pinned llmfit executable. */ - llmfitPath?: string - /** Packaged: the bundled Ollama's port, model dir, and URL for API + worker. */ - ollamaPort?: number - ollamaModelsDir?: string - ollamaUrl?: string + /** + * llama-server's router port, models dir, and URL for API + worker. + * + * Set in dev and packaged both: only `llamacppBinariesDir` differs between + * them, the same way `modelsDir` and the Python sidecars already work. + */ + llamacppPort?: number + llamacppModelsDir?: string + llamacppUrl?: string + /** + * Where the staged llama.cpp build lives. + * + * The backend needs this as well as Electron: its hardware probe loads ggml + * by ctypes and must run from this directory, because ggml scans the running + * executable's own directory for backends and silently finds none anywhere + * else. Unset, every machine badges as having no GPU and nothing says why. + */ + llamacppBinariesDir?: string /** Packaged: the bundled sd-server's port, model dir, and URL for API + worker. */ imagePort?: number imageModelsDir?: string From 64dafcb507219a20559f9e7f3a585490dec22ab6 Mon Sep 17 00:00:00 2001 From: Anish Sarkar <104695310+AnishSarkar22@users.noreply.github.com> Date: Sun, 20 Sep 2026 20:07:42 +0530 Subject: [PATCH 12/60] feat(config): point the backend at llama-server and drop the llmfit settings Co-Authored-By: Claude Sonnet 5 --- surfsense_local/backend/shared/config.py | 21 ++++++++------------- 1 file changed, 8 insertions(+), 13 deletions(-) diff --git a/surfsense_local/backend/shared/config.py b/surfsense_local/backend/shared/config.py index a7570a74f4..b26e8ff63c 100644 --- a/surfsense_local/backend/shared/config.py +++ b/surfsense_local/backend/shared/config.py @@ -60,14 +60,17 @@ class SearchSettings(BaseSettings): class LLMSettings(BaseSettings): - """The generation runtime. Electron starts Ollama and passes its address.""" + """The generation runtime. Electron starts llama-server and passes its address.""" model_config = SettingsConfigDict(env_prefix="SURFSENSE_LOCAL_") - # Dev fallback for a bare `ollama serve`. The packaged app never hits this: - # Electron runs Ollama on a port it chose and sets the env var. - ollama_base_url: str = "http://127.0.0.1:11434" - ollama_models_dir: Path | None = None + # llama-server in router mode. Electron picks the port, points the router at + # the models directory, and passes the staged library directory: the probe + # has to run from there, because ggml scans the running executable's own + # directory for backends and silently finds none anywhere else. + llamacpp_base_url: str = "http://127.0.0.1:8080" + llamacpp_models_dir: Path | None = None + llamacpp_library_dir: Path | None = None # The bundled sd-server, same arrangement: Electron picks the port and only # runs it once its model is downloaded. Absent models_dir means the host has @@ -75,14 +78,6 @@ class LLMSettings(BaseSettings): image_base_url: str = "http://127.0.0.1:1234" image_models_dir: Path | None = None - # The packaged app passes an absolute resource path. Development resolves - # the command from PATH and degrades recommendations when it is absent. - llmfit_path: Path = Path("llmfit") - llmfit_expected_version: str = "1.1.11" - llmfit_timeout_seconds: float = 30.0 - llmfit_max_context: int = 8192 - # ponytail: calibration constant; replace with measured peak app overhead. - recommendation_reserve_gb: float = 2.0 @lru_cache From efb89c8735b786fe151b506a43f1ca013aeb516d Mon Sep 17 00:00:00 2001 From: Anish Sarkar <104695310+AnishSarkar22@users.noreply.github.com> Date: Sun, 20 Sep 2026 20:07:42 +0530 Subject: [PATCH 13/60] feat(llm/gguf): read GGUF headers and derive model shape from local or remote files Co-Authored-By: Claude Sonnet 5 --- .../backend/modules/llm/gguf/__init__.py | 13 +++ .../backend/modules/llm/gguf/reader.py | 107 ++++++++++++++++++ .../backend/modules/llm/gguf/shape.py | 57 ++++++++++ .../backend/modules/llm/gguf/source.py | 52 +++++++++ .../backend/tests/unit/llm/gguf/__init__.py | 0 .../backend/tests/unit/llm/gguf/build.py | 69 +++++++++++ .../tests/unit/llm/gguf/test_header.py | 81 +++++++++++++ .../tests/unit/llm/gguf/test_source.py | 73 ++++++++++++ 8 files changed, 452 insertions(+) create mode 100644 surfsense_local/backend/modules/llm/gguf/__init__.py create mode 100644 surfsense_local/backend/modules/llm/gguf/reader.py create mode 100644 surfsense_local/backend/modules/llm/gguf/shape.py create mode 100644 surfsense_local/backend/modules/llm/gguf/source.py create mode 100644 surfsense_local/backend/tests/unit/llm/gguf/__init__.py create mode 100644 surfsense_local/backend/tests/unit/llm/gguf/build.py create mode 100644 surfsense_local/backend/tests/unit/llm/gguf/test_header.py create mode 100644 surfsense_local/backend/tests/unit/llm/gguf/test_source.py diff --git a/surfsense_local/backend/modules/llm/gguf/__init__.py b/surfsense_local/backend/modules/llm/gguf/__init__.py new file mode 100644 index 0000000000..35ef89d8ff --- /dev/null +++ b/surfsense_local/backend/modules/llm/gguf/__init__.py @@ -0,0 +1,13 @@ +from modules.llm.gguf.reader import GgufHeader, TruncatedHeaderError, read_gguf_header +from modules.llm.gguf.shape import read_header, to_shape +from modules.llm.gguf.source import shape_from_file, shape_from_url + +__all__ = [ + "GgufHeader", + "TruncatedHeaderError", + "read_gguf_header", + "read_header", + "shape_from_file", + "shape_from_url", + "to_shape", +] diff --git a/surfsense_local/backend/modules/llm/gguf/reader.py b/surfsense_local/backend/modules/llm/gguf/reader.py new file mode 100644 index 0000000000..79990f991c --- /dev/null +++ b/surfsense_local/backend/modules/llm/gguf/reader.py @@ -0,0 +1,107 @@ +"""Parse a GGUF metadata block. Standard library only. + +The header is the only thing a fit estimate needs, and it sits at the front of +the file, so it can be read over an HTTP range without downloading weights. +""" + +import struct +from dataclasses import dataclass +from typing import Any + +MAGIC = b"GGUF" + +_UINT8, _INT8, _UINT16, _INT16, _UINT32, _INT32 = 0, 1, 2, 3, 4, 5 +_FLOAT32, _BOOL, _STRING, _ARRAY, _UINT64, _INT64, _FLOAT64 = 6, 7, 8, 9, 10, 11, 12 + +_SCALARS: dict[int, tuple[str, int]] = { + _UINT8: (" None: + self._data = data + self._at = 0 + + def take(self, n: int) -> bytes: + end = self._at + n + if end > len(self._data): + raise TruncatedHeaderError( + f"header needs {end} bytes, got {len(self._data)}" + ) + chunk = self._data[self._at : end] + self._at = end + return chunk + + def scalar(self, kind: int) -> Any: + fmt, size = _SCALARS[kind] + return struct.unpack(fmt, self.take(size))[0] + + def string(self) -> str: + return self.take(self.scalar(_UINT64)).decode("utf-8", errors="replace") + + def value(self, kind: int) -> Any: + if kind == _STRING: + return self.string() + if kind == _ARRAY: + elem = self.scalar(_UINT32) + count = self.scalar(_UINT64) + return [self.value(elem) for _ in range(count)] + if kind in _SCALARS: + return self.scalar(kind) + raise ValueError(f"unknown GGUF value type {kind}") + + +def read_gguf_header(data: bytes) -> GgufHeader: + """Metadata and tensor table, or TruncatedHeaderError if `data` is short.""" + cursor = _Cursor(data) + if cursor.take(4) != MAGIC: + raise ValueError("not a GGUF file") + cursor.scalar(_UINT32) # format version + tensor_count = cursor.scalar(_UINT64) + kv_count = cursor.scalar(_UINT64) + + metadata: dict[str, Any] = {} + for _ in range(kv_count): + key = cursor.string() + metadata[key] = cursor.value(cursor.scalar(_UINT32)) + + tensors: list[tuple[str, tuple[int, ...]]] = [] + for _ in range(tensor_count): + name = cursor.string() + dims = tuple(cursor.scalar(_UINT64) for _ in range(cursor.scalar(_UINT32))) + cursor.scalar(_UINT32) # element type + cursor.scalar(_UINT64) # offset + tensors.append((name, dims)) + + return GgufHeader(metadata=metadata, tensors=tuple(tensors)) diff --git a/surfsense_local/backend/modules/llm/gguf/shape.py b/surfsense_local/backend/modules/llm/gguf/shape.py new file mode 100644 index 0000000000..e2cc040dc3 --- /dev/null +++ b/surfsense_local/backend/modules/llm/gguf/shape.py @@ -0,0 +1,57 @@ +"""Turn a parsed header into the estimator's inputs. + +Architecture-scoped keys are named after the architecture itself, so every +lookup goes through the `general.architecture` value rather than a fixed prefix. +""" + +from typing import Any + +from modules.llm.fit import ModelShape +from modules.llm.gguf.reader import GgufHeader, read_gguf_header + + +def _widest(value: Any, default: int = 0) -> int: + """A field that is scalar in most models and per-layer in some. + + The widest layer is the honest reduction: it is what the cache must be sized + for, and taking the first element instead under-prices a hybrid model. + """ + if isinstance(value, list): + return max((int(v) for v in value), default=default) + if value is None: + return default + return int(value) + + +def to_shape(header: GgufHeader) -> ModelShape: + meta = header.metadata + arch = str(meta.get("general.architecture", "unknown")) + + def field(name: str, default: int = 0) -> int: + return _widest(meta.get(f"{arch}.{name}"), default) + + key_length = field("attention.key_length") + value_length = field("attention.value_length") + embedding = field("embedding_length") + head_count = field("attention.head_count") + # Older headers omit the per-head lengths and imply them from the embedding. + if not key_length and embedding and head_count: + key_length = value_length = embedding // head_count + + tokens = meta.get("tokenizer.ggml.tokens") + return ModelShape( + architecture=arch, + block_count=field("block_count"), + head_count_kv=field("attention.head_count_kv", 1), + key_length=key_length, + value_length=value_length, + context_length=field("context_length"), + n_vocab=len(tokens) if isinstance(tokens, list) else field("vocab_size"), + sliding_window=field("attention.sliding_window"), + expert_count=field("expert_count"), + ) + + +def read_header(data: bytes) -> ModelShape: + """Bytes from the front of a GGUF file to the shape a fit estimate needs.""" + return to_shape(read_gguf_header(data)) diff --git a/surfsense_local/backend/modules/llm/gguf/source.py b/surfsense_local/backend/modules/llm/gguf/source.py new file mode 100644 index 0000000000..b4be310fb4 --- /dev/null +++ b/surfsense_local/backend/modules/llm/gguf/source.py @@ -0,0 +1,52 @@ +"""Where header bytes come from: a file on disk, or the front of a remote file. + +Hugging Face serves `Range` requests on model files, so a model can be priced +before a single weight is downloaded. That is what puts a real fit badge on a +search result. +""" + +from pathlib import Path + +import httpx + +from modules.llm.fit import ModelShape +from modules.llm.gguf.reader import TruncatedHeaderError +from modules.llm.gguf.shape import read_header + +# A 135M model's metadata ends at 1.77 MB, but Qwen3-Coder-30B-A3B's runs to +# 5.94 MB with 579 tensor entries after it. Header size scales with vocabulary, +# not with model size, so start wide and widen once more before giving up. +INITIAL_BYTES = 8 * 1024 * 1024 +WIDENED_BYTES = 24 * 1024 * 1024 + + +def shape_from_file(path: Path) -> ModelShape: + with path.open("rb") as handle: + return _read_widening(lambda n: _read_prefix(handle, n)) + + +async def shape_from_url(client: httpx.AsyncClient, url: str) -> ModelShape: + """Read only the header of a remote GGUF, over HTTP range requests.""" + + async def fetch(size: int) -> bytes: + reply = await client.get(url, headers={"Range": f"bytes=0-{size - 1}"}) + reply.raise_for_status() + return reply.content + + data = await fetch(INITIAL_BYTES) + try: + return read_header(data) + except TruncatedHeaderError: + return read_header(await fetch(WIDENED_BYTES)) + + +def _read_prefix(handle, size: int) -> bytes: + handle.seek(0) + return handle.read(size) + + +def _read_widening(fetch) -> ModelShape: + try: + return read_header(fetch(INITIAL_BYTES)) + except TruncatedHeaderError: + return read_header(fetch(WIDENED_BYTES)) diff --git a/surfsense_local/backend/tests/unit/llm/gguf/__init__.py b/surfsense_local/backend/tests/unit/llm/gguf/__init__.py new file mode 100644 index 0000000000..e69de29bb2 diff --git a/surfsense_local/backend/tests/unit/llm/gguf/build.py b/surfsense_local/backend/tests/unit/llm/gguf/build.py new file mode 100644 index 0000000000..1f20bd10e0 --- /dev/null +++ b/surfsense_local/backend/tests/unit/llm/gguf/build.py @@ -0,0 +1,69 @@ +"""A minimal GGUF writer, so header tests need no checked-in binary. + +Built from the format spec rather than from the reader under test, so the two +can disagree. +""" + +import struct + +UINT32, FLOAT32, BOOL, STRING, ARRAY, UINT64 = 4, 6, 7, 8, 9, 10 + + +def _string(value: str) -> bytes: + raw = value.encode() + return struct.pack(" bytes: + if kind == STRING: + return _string(str(value)) + if kind == UINT32: + return struct.pack(" bytes: + """One scalar metadata entry.""" + return _string(key) + struct.pack(" bytes: + """One metadata entry holding a list, as vocabularies and per-layer fields are.""" + body = b"".join(_value(elem_kind, v) for v in values) + return ( + _string(key) + + struct.pack(" bytes: + """One tensor-table row.""" + return ( + _string(name) + + struct.pack(" bytes: + """A whole file: magic, counts, metadata, then the tensor table.""" + tensors = tensors or [] + return ( + b"GGUF" + + struct.pack(" list[bytes]: + """The fields a fit estimate needs, named as Qwen3 headers name them.""" + return [ + kv("general.architecture", STRING, "qwen3"), + kv("qwen3.block_count", UINT32, 28), + kv("qwen3.attention.head_count_kv", UINT32, 8), + kv("qwen3.attention.key_length", UINT32, 128), + kv("qwen3.attention.value_length", UINT32, 128), + kv("qwen3.context_length", UINT32, 40960), + array("tokenizer.ggml.tokens", STRING, ["a", "b", "c"]), + ] + + +def test_a_header_yields_the_shape_a_fit_estimate_needs() -> None: + """Everything `need` is computed from, and nothing else.""" + shape = read_header(gguf(qwen3_entries())) + + assert shape.architecture == "qwen3" + assert shape.block_count == 28 + assert shape.head_count_kv == 8 + assert shape.context_length == 40960 + assert shape.n_vocab == 3 + + +def test_a_model_without_a_sliding_window_reports_none() -> None: + """Zero means full attention on every layer, which is the common case.""" + assert read_header(gguf(qwen3_entries())).sliding_window == 0 + + +def test_per_layer_kv_heads_are_read_as_the_widest_layer() -> None: + """`head_count_kv` is scalar or array. An array means per-layer heads, and + collapsing it to its first element under-prices a hybrid model's cache.""" + entries = [e for e in qwen3_entries() if b"head_count_kv" not in e] + entries.append(array("qwen3.attention.head_count_kv", UINT32, [2, 8, 8, 2])) + + assert read_header(gguf(entries)).head_count_kv == 8 + + +def test_a_truncated_header_says_so_rather_than_guessing() -> None: + """The caller retries with a wider range; a partial parse would ship a + confident number derived from half a file.""" + whole = gguf(qwen3_entries()) + + with pytest.raises(TruncatedHeaderError): + read_header(whole[: len(whole) // 2]) + + +def test_something_that_is_not_a_gguf_is_rejected() -> None: + """A wrong file is a rejection, unlike a short one, which is a wider read.""" + with pytest.raises(ValueError): + read_header(b"not a gguf file at all") + + +def test_a_mixture_of_experts_reports_its_expert_counts() -> None: + """Without these an MoE entry's speed prediction is wrong by roughly 10x.""" + entries = [ + *qwen3_entries(), + kv("qwen3.expert_count", UINT32, 128), + kv("qwen3.expert_used_count", UINT32, 8), + ] + shape = read_header(gguf(entries)) + + assert shape.expert_count == 128 + + +def test_unknown_value_types_do_not_stop_the_parse() -> None: + """Uploaders put arbitrary keys in headers, and one we cannot read is not a + reason to refuse a model we can otherwise price.""" + entries = [*qwen3_entries(), kv("some.vendor.metric", FLOAT32, 0.5)] + + assert read_header(gguf(entries)).block_count == 28 diff --git a/surfsense_local/backend/tests/unit/llm/gguf/test_source.py b/surfsense_local/backend/tests/unit/llm/gguf/test_source.py new file mode 100644 index 0000000000..446aa3fbca --- /dev/null +++ b/surfsense_local/backend/tests/unit/llm/gguf/test_source.py @@ -0,0 +1,73 @@ +"""Reading a header without downloading the model it belongs to.""" + +import httpx +import pytest + +from modules.llm.gguf import shape_from_file, shape_from_url +from modules.llm.gguf.source import INITIAL_BYTES +from tests.unit.llm.gguf.build import STRING, UINT32, array, gguf, kv + +pytestmark = pytest.mark.unit + + +def a_gguf(vocab: int = 3) -> bytes: + """A header whose size is driven by its vocabulary, as real ones are.""" + return gguf( + [ + kv("general.architecture", STRING, "qwen3"), + kv("qwen3.block_count", UINT32, 28), + kv("qwen3.attention.head_count_kv", UINT32, 8), + kv("qwen3.attention.key_length", UINT32, 128), + kv("qwen3.attention.value_length", UINT32, 128), + kv("qwen3.context_length", UINT32, 40960), + array("tokenizer.ggml.tokens", STRING, [f"t{i}" for i in range(vocab)]), + ] + ) + + +@pytest.mark.asyncio +async def test_only_the_front_of_the_file_is_requested() -> None: + """The point of the whole exercise: price a 40 GB model without fetching it.""" + seen: list[str] = [] + + def handler(request: httpx.Request) -> httpx.Response: + seen.append(request.headers["Range"]) + return httpx.Response(206, content=a_gguf()[: INITIAL_BYTES]) + + transport = httpx.MockTransport(handler) + async with httpx.AsyncClient(transport=transport) as client: + shape = await shape_from_url(client, "https://example.invalid/m.gguf") + + assert seen == [f"bytes=0-{INITIAL_BYTES - 1}"] + assert shape.block_count == 28 + + +@pytest.mark.asyncio +async def test_a_short_first_read_is_retried_wider_before_giving_up() -> None: + """Header size scales with vocabulary, so one budget cannot suit every model.""" + whole = a_gguf(vocab=4000) + sizes: list[int] = [] + + def handler(request: httpx.Request) -> httpx.Response: + start, end = request.headers["Range"].removeprefix("bytes=").split("-") + size = int(end) - int(start) + 1 + sizes.append(size) + # Starve the first read, satisfy the second. + served = whole[: len(whole) // 3] if len(sizes) == 1 else whole + return httpx.Response(206, content=served) + + transport = httpx.MockTransport(handler) + async with httpx.AsyncClient(transport=transport) as client: + shape = await shape_from_url(client, "https://example.invalid/m.gguf") + + assert len(sizes) == 2 + assert sizes[1] > sizes[0] + assert shape.n_vocab == 4000 + + +def test_a_local_file_reads_the_same_way(tmp_path) -> None: + """An imported .gguf is the airgapped path, and it uses one code path.""" + path = tmp_path / "m.gguf" + path.write_bytes(a_gguf()) + + assert shape_from_file(path).architecture == "qwen3" From 8ca33e8d3b66085d06c5134f989b2166b32ea709 Mon Sep 17 00:00:00 2001 From: Anish Sarkar <104695310+AnishSarkar22@users.noreply.github.com> Date: Sun, 20 Sep 2026 20:07:42 +0530 Subject: [PATCH 14/60] feat(llm/hardware): probe devices, libraries, and system memory through ggml Co-Authored-By: Claude Sonnet 5 --- .../backend/modules/llm/hardware/__init__.py | 23 +++ .../backend/modules/llm/hardware/budget.py | 130 ++++++++++++++ .../backend/modules/llm/hardware/devices.py | 48 ++++++ .../backend/modules/llm/hardware/libraries.py | 65 +++++++ .../backend/modules/llm/hardware/probe.py | 69 ++++++++ .../backend/modules/llm/hardware/selection.py | 22 +++ .../modules/llm/hardware/system_memory.py | 62 +++++++ .../tests/unit/llm/hardware/__init__.py | 0 .../tests/unit/llm/hardware/test_budget.py | 161 ++++++++++++++++++ .../tests/unit/llm/hardware/test_probe.py | 65 +++++++ .../tests/unit/llm/hardware/test_selection.py | 74 ++++++++ 11 files changed, 719 insertions(+) create mode 100644 surfsense_local/backend/modules/llm/hardware/__init__.py create mode 100644 surfsense_local/backend/modules/llm/hardware/budget.py create mode 100644 surfsense_local/backend/modules/llm/hardware/devices.py create mode 100644 surfsense_local/backend/modules/llm/hardware/libraries.py create mode 100644 surfsense_local/backend/modules/llm/hardware/probe.py create mode 100644 surfsense_local/backend/modules/llm/hardware/selection.py create mode 100644 surfsense_local/backend/modules/llm/hardware/system_memory.py create mode 100644 surfsense_local/backend/tests/unit/llm/hardware/__init__.py create mode 100644 surfsense_local/backend/tests/unit/llm/hardware/test_budget.py create mode 100644 surfsense_local/backend/tests/unit/llm/hardware/test_probe.py create mode 100644 surfsense_local/backend/tests/unit/llm/hardware/test_selection.py diff --git a/surfsense_local/backend/modules/llm/hardware/__init__.py b/surfsense_local/backend/modules/llm/hardware/__init__.py new file mode 100644 index 0000000000..8516c72408 --- /dev/null +++ b/surfsense_local/backend/modules/llm/hardware/__init__.py @@ -0,0 +1,23 @@ +from modules.llm.hardware.budget import ( + LLAMA_CPP_FIT_MARGIN_BYTES, + BudgetMode, + build_budget, +) +from modules.llm.hardware.devices import Device, DeviceType +from modules.llm.hardware.libraries import GgmlLibraries, load +from modules.llm.hardware.probe import probe_devices +from modules.llm.hardware.selection import select_device +from modules.llm.hardware.system_memory import available_bytes + +__all__ = [ + "LLAMA_CPP_FIT_MARGIN_BYTES", + "BudgetMode", + "Device", + "DeviceType", + "GgmlLibraries", + "available_bytes", + "build_budget", + "load", + "probe_devices", + "select_device", +] diff --git a/surfsense_local/backend/modules/llm/hardware/budget.py b/surfsense_local/backend/modules/llm/hardware/budget.py new file mode 100644 index 0000000000..5c5f0b8dfb --- /dev/null +++ b/surfsense_local/backend/modules/llm/hardware/budget.py @@ -0,0 +1,130 @@ +"""Assemble one device plus host memory into what the estimator subtracts. + +Two modes, because the same subtraction answers two different questions. +**Capacity** prices a catalog: a shelf of models you might install later, which +must not move with whatever the machine happens to be doing right now. **Live** +decides a launch: this model, into this memory, in a moment. Pricing the catalog +against live free memory makes every large row read TOO_BIG whenever a browser +is open, which is a badge that lies in the direction users notice. +""" + +from collections.abc import Sequence +from enum import StrEnum + +from modules.llm.fit import HardwareBudget +from modules.llm.hardware.devices import Device, DeviceType +from modules.llm.hardware.selection import select_device + +# llama.cpp's own `fit_params_target`, one GiB per device, read from +# common/common.h at b11050 and observed on both Vulkan and Metal. Not an +# estimate of the margin: the value the fitter will actually apply, and one we +# can pin through --fit-target rather than predict. +LLAMA_CPP_FIT_MARGIN_BYTES = 1024 * 1024 * 1024 + + +class BudgetMode(StrEnum): + """Which question the budget is answering.""" + + CAPACITY = "capacity" # pricing a catalog + LIVE = "live" # deciding a load + + +# What a host keeps for itself once a model is resident. Capacity mode assumes +# the rest is reclaimable, which is what makes a catalog stable between renders. +_HOST_RESERVE_BYTES = 2 * 1024 * 1024 * 1024 + + +def build_budget( + devices: Sequence[Device], + ram_available_bytes: int, + *, + mode: BudgetMode = BudgetMode.LIVE, + ram_total_bytes: int | None = None, +) -> HardwareBudget: + """One device's memory plus the host's, never a sum across devices.""" + device = select_device(devices) + ram = _host_memory(devices, ram_available_bytes, mode, ram_total_bytes) + uma = any(d.type is DeviceType.GPU and _is_unified(d, devices) for d in devices) + + if device is None: + return HardwareBudget( + device_free_bytes=0, + device_total_bytes=0, + fit_reserve_bytes=LLAMA_CPP_FIT_MARGIN_BYTES, + ram_available_bytes=ram, + uma=uma, + has_gpu=False, + ) + + return HardwareBudget( + device_free_bytes=_device_free(device, uma, ram), + device_total_bytes=device.total_bytes, + fit_reserve_bytes=LLAMA_CPP_FIT_MARGIN_BYTES, + ram_available_bytes=ram, + uma=uma, + has_gpu=True, + ) + + +def _is_unified(gpu: Device, devices: Sequence[Device]) -> bool: + """A GPU sharing the CPU's description is sharing its memory. + + Apple Silicon reports the same part name for both, which is the cheap and + reliable signal. It decides badge copy, not arithmetic. + """ + return any( + d.type is DeviceType.CPU and d.description == gpu.description for d in devices + ) + + +def _device_free(device: Device, uma: bool, host_memory: int) -> int: + """How much the device can actually be given. + + On a discrete card this is the card's own figure: dedicated VRAM is a + separate pool and its `free` really is live. + + On unified memory it is neither. Metal reports + `recommendedMaxWorkingSetSize` as both total and free, a static number that + does not move however busy the machine is, and the memory it describes is + the same memory the OS and every other app are using. So **both** limits + apply and the smaller governs: the working set is a ceiling Metal will not + allocate past, and host memory is what is actually there to give. + + Spending the working set as though it were live is what sized a 28,672 token + window on an 8 GB Mac with 2.3 GB reclaimable. The load paged and took 43 + seconds, which is long enough for title generation to time out at 30. + """ + if not uma: + return device.free_bytes + return min(device.free_bytes, host_memory) + + +def _host_memory( + devices: Sequence[Device], + ram_available_bytes: int, + mode: BudgetMode, + ram_total_bytes: int | None, +) -> int: + """Live memory as given, or capacity as total minus what the host keeps. + + The reserve is only meaningful against physical RAM. When nothing states + that, the live reading is a **floor** rather than another figure to deduct + from: subtracting twice reported 0 GB on a busy 8 GB machine and badged + every model `Won't fit`, which is the confident-and-wrong failure this + estimator exists to avoid. + """ + if mode is BudgetMode.LIVE: + return ram_available_bytes + total = ram_total_bytes or _cpu_total(devices) + if not total: + # Nothing states physical RAM, so the live reading is the best floor + # available. Deducting the reserve from it as well is what reported + # 0 GB and is never right: the reserve assumes a total, not a remainder. + return ram_available_bytes + return max(0, total - _HOST_RESERVE_BYTES) + + +def _cpu_total(devices: Sequence[Device]) -> int: + """Physical RAM. ggml restates it as `free` on several platforms, and that + is exactly the number capacity mode wants.""" + return next((d.total_bytes for d in devices if d.type is DeviceType.CPU), 0) diff --git a/surfsense_local/backend/modules/llm/hardware/devices.py b/surfsense_local/backend/modules/llm/hardware/devices.py new file mode 100644 index 0000000000..25a3513fc2 --- /dev/null +++ b/surfsense_local/backend/modules/llm/hardware/devices.py @@ -0,0 +1,48 @@ +"""What ggml reports about one device. + +The type enum is ggml's, read from `ggml-backend.h`. It has grown twice, so an +integer we do not recognise is treated as "not a GPU" rather than as an error: +this runs on the first render, and every Mac already lists a device type that a +three-member enum would reject. +""" + +from dataclasses import dataclass +from enum import IntEnum + + +class DeviceType(IntEnum): + """GGML_BACKEND_DEVICE_TYPE_*, in declaration order.""" + + CPU = 0 + GPU = 1 + IGPU = 2 # integrated, carving from host memory + ACCEL = 3 # BLAS, AMX. Accelerate appears on every Mac. + META = 4 # wraps several devices for tensor parallelism + UNKNOWN = -1 + + @classmethod + def parse(cls, raw: int) -> "DeviceType": + try: + return cls(raw) + except ValueError: + return cls.UNKNOWN + + +@dataclass(frozen=True) +class Device: + name: str + description: str + type: DeviceType + total_bytes: int + free_bytes: int + + @property + def reports_live_memory(self) -> bool: + """False when `free` merely restates `total`. + + Measured: native Windows reports real free memory, but WSL2 virtualises + it and macOS restates the total on both the Metal and CPU devices. So a + live figure is the exception, and the RAM half of a budget has to come + from the OS instead. + """ + return self.total_bytes > 0 and self.free_bytes < self.total_bytes diff --git a/surfsense_local/backend/modules/llm/hardware/libraries.py b/surfsense_local/backend/modules/llm/hardware/libraries.py new file mode 100644 index 0000000000..b668191cd0 --- /dev/null +++ b/surfsense_local/backend/modules/llm/hardware/libraries.py @@ -0,0 +1,65 @@ +"""Loading ggml through ctypes. + +Two handles on Windows, one everywhere else. The exported symbols are split +across `ggml` and `ggml-base`, and PE exports do not chain, so a single +`CDLL("ggml.dll")` raises AttributeError on the first device query. ELF and +Mach-O resolve the second library through their own dependency records, where +the opposite is true: `CDLL("libggml-base.so")` cannot see `load_all`. +""" + +import ctypes +import sys +from dataclasses import dataclass +from pathlib import Path + + +@dataclass(frozen=True) +class GgmlLibraries: + """`core` registers backends and counts devices; `base` describes them.""" + + core: ctypes.CDLL + base: ctypes.CDLL + + +def _candidates(stem: str) -> tuple[str, ...]: + if sys.platform == "win32": + return (f"{stem}.dll",) + if sys.platform == "darwin": + return (f"lib{stem}.dylib",) + return (f"lib{stem}.so",) + + +def _load(directory: Path, stem: str) -> ctypes.CDLL: + errors = [] + for name in _candidates(stem): + try: + return ctypes.CDLL(str(directory / name)) + except OSError as error: + errors.append(f"{name}: {error}") + raise OSError(f"could not load {stem} from {directory}: {'; '.join(errors)}") + + +def load(directory: Path) -> GgmlLibraries: + """Open ggml from a staged directory and declare the signatures we call.""" + core = _load(directory, "ggml") + base = core if sys.platform != "win32" else _load(directory, "ggml-base") + + core.ggml_backend_load_all.restype = None + core.ggml_backend_dev_count.restype = ctypes.c_size_t + core.ggml_backend_dev_get.restype = ctypes.c_void_p + core.ggml_backend_dev_get.argtypes = [ctypes.c_size_t] + + for name in ("name", "description"): + fn = getattr(base, f"ggml_backend_dev_{name}") + fn.restype = ctypes.c_char_p + fn.argtypes = [ctypes.c_void_p] + + base.ggml_backend_dev_type.restype = ctypes.c_int + base.ggml_backend_dev_type.argtypes = [ctypes.c_void_p] + base.ggml_backend_dev_memory.restype = None + base.ggml_backend_dev_memory.argtypes = [ + ctypes.c_void_p, + ctypes.POINTER(ctypes.c_size_t), + ctypes.POINTER(ctypes.c_size_t), + ] + return GgmlLibraries(core=core, base=base) diff --git a/surfsense_local/backend/modules/llm/hardware/probe.py b/surfsense_local/backend/modules/llm/hardware/probe.py new file mode 100644 index 0000000000..bd2e2634cc --- /dev/null +++ b/surfsense_local/backend/modules/llm/hardware/probe.py @@ -0,0 +1,69 @@ +"""Ask ggml what this machine has. + +**The scan runs from the directory of the running executable, not the directory +the libraries were loaded from.** Verified on a machine with a working RTX 3050, +same process otherwise: run from elsewhere it reports zero devices, run from the +library directory it reports two. Loading by absolute path does not help, and +`GGML_BACKEND_PATH` is not an escape hatch because it expects a file. + +The failure is silent and indistinguishable from a genuinely GPU-less machine, +so the working directory is set around the call rather than assumed. +""" + +import ctypes +import os +import threading +from contextlib import contextmanager +from pathlib import Path + +from modules.llm.hardware.devices import Device, DeviceType +from modules.llm.hardware.libraries import load + +# chdir is process-global. The probe runs once at startup and its result is +# cached, so the window is short, but it still has to be exclusive. +_CWD_LOCK = threading.Lock() + + +@contextmanager +def _working_directory(directory: Path): + with _CWD_LOCK: + previous = Path.cwd() + os.chdir(directory) + try: + yield + finally: + os.chdir(previous) + + +def probe_devices(library_dir: Path) -> list[Device]: + """Every device ggml can see, in ggml's own preference order. + + An empty list means ggml found nothing, which is not the same as the machine + having nothing. `gpu_crosscheck` is what tells those apart. + """ + libraries = load(library_dir) + with _working_directory(library_dir): + libraries.core.ggml_backend_load_all() + # The Metal shader compile happens here, not in load_all: measured at + # 19 s cold and about 45 ms warm on an M2. Warm this call, not that one. + count = libraries.core.ggml_backend_dev_count() + + devices = [] + for index in range(count): + handle = libraries.core.ggml_backend_dev_get(index) + free, total = ctypes.c_size_t(), ctypes.c_size_t() + libraries.base.ggml_backend_dev_memory( + handle, ctypes.byref(free), ctypes.byref(total) + ) + devices.append( + Device( + name=libraries.base.ggml_backend_dev_name(handle).decode(), + description=libraries.base.ggml_backend_dev_description( + handle + ).decode(), + type=DeviceType.parse(libraries.base.ggml_backend_dev_type(handle)), + total_bytes=total.value, + free_bytes=free.value, + ) + ) + return devices diff --git a/surfsense_local/backend/modules/llm/hardware/selection.py b/surfsense_local/backend/modules/llm/hardware/selection.py new file mode 100644 index 0000000000..1310511b6a --- /dev/null +++ b/surfsense_local/backend/modules/llm/hardware/selection.py @@ -0,0 +1,22 @@ +"""Which device the catalog prices against. + +One device, never a sum. The same physical card appears once per loaded backend +and an integrated GPU can advertise more memory than a discrete one, so both +"largest" and "total" are wrong in the direction that tells users a model fits +when it cannot. +""" + +from collections.abc import Sequence + +from modules.llm.hardware.devices import Device, DeviceType + + +def select_device(devices: Sequence[Device]) -> Device | None: + """The first device ggml types as a discrete GPU, or None. + + First rather than largest: ggml already orders backends by preference, so + taking the first is also backend selection. Sorting by memory picks a 16 GB + integrated chip over a 6 GB discrete card and places every layer on the + slower of the two. + """ + return next((d for d in devices if d.type is DeviceType.GPU), None) diff --git a/surfsense_local/backend/modules/llm/hardware/system_memory.py b/surfsense_local/backend/modules/llm/hardware/system_memory.py new file mode 100644 index 0000000000..c374215727 --- /dev/null +++ b/surfsense_local/backend/modules/llm/hardware/system_memory.py @@ -0,0 +1,62 @@ +"""Host memory actually available, from the OS rather than from ggml. + +ggml's CPU device reports a live figure on native Windows and, so far, nowhere +else: under WSL2 it is virtualised, and on macOS it restates physical RAM on +both the Metal and CPU devices. That number decides PARTIAL against TOO_BIG, so +it is worth asking the OS directly. +""" + +import os +import subprocess +import sys + + +def available_bytes() -> int: + """Best available reading, falling back to total rather than to zero.""" + if sys.platform == "darwin": + return _darwin() + if sys.platform.startswith("linux"): + return _linux() + return _total() + + +def _total() -> int: + try: + return os.sysconf("SC_PAGE_SIZE") * os.sysconf("SC_PHYS_PAGES") + except (ValueError, OSError, AttributeError): + return 0 + + +def _linux() -> int: + try: + with open("/proc/meminfo") as handle: + for line in handle: + if line.startswith("MemAvailable:"): + return int(line.split()[1]) * 1024 + except OSError: + pass + return _total() + + +def _darwin() -> int: + """Free plus inactive pages: what a large allocation can actually claim.""" + try: + output = subprocess.run( + ["vm_stat"], capture_output=True, text=True, timeout=5, check=True + ).stdout + except (OSError, subprocess.SubprocessError): + return _total() + + page_size = 4096 + counts: dict[str, int] = {} + for line in output.splitlines(): + if "page size of" in line: + page_size = int(line.split("page size of")[1].split()[0]) + elif ":" in line: + key, _, value = line.partition(":") + digits = value.strip().rstrip(".") + if digits.isdigit(): + counts[key.strip()] = int(digits) + + reclaimable = counts.get("Pages free", 0) + counts.get("Pages inactive", 0) + return reclaimable * page_size if reclaimable else _total() diff --git a/surfsense_local/backend/tests/unit/llm/hardware/__init__.py b/surfsense_local/backend/tests/unit/llm/hardware/__init__.py new file mode 100644 index 0000000000..e69de29bb2 diff --git a/surfsense_local/backend/tests/unit/llm/hardware/test_budget.py b/surfsense_local/backend/tests/unit/llm/hardware/test_budget.py new file mode 100644 index 0000000000..fcf6bbb6cd --- /dev/null +++ b/surfsense_local/backend/tests/unit/llm/hardware/test_budget.py @@ -0,0 +1,161 @@ +"""Turning a device listing into the numbers the estimator subtracts.""" + +import pytest + +from modules.llm.hardware import BudgetMode, Device, DeviceType, build_budget + +pytestmark = pytest.mark.unit + +MIB = 1024**2 + +DISCRETE = Device("Vulkan0", "RTX 3050", DeviceType.GPU, 6002 * MIB, 5234 * MIB) +INTEGRATED = Device("Vulkan1", "Radeon", DeviceType.IGPU, 16198 * MIB, 16198 * MIB) +CPU_LIVE = Device("CPU", "Ryzen 5", DeviceType.CPU, 31884 * MIB, 22750 * MIB) +CPU_RESTATED = Device("CPU", "Apple M2", DeviceType.CPU, 8192 * MIB, 8192 * MIB) +METAL = Device("MTL0", "Apple M2", DeviceType.GPU, 5461 * MIB, 5461 * MIB) + + +def test_the_reserve_is_llama_cpps_own_margin_not_an_estimate_of_it() -> None: + """fit_params_target defaults to one GiB per device, so usable is exact.""" + budget = build_budget([DISCRETE, INTEGRATED, CPU_LIVE], ram_available_bytes=22750 * MIB) + + assert budget.fit_reserve_bytes == 1024 * MIB + assert budget.usable_vram_bytes == (5234 - 1024) * MIB + + +def test_the_budget_never_exceeds_the_one_device_it_chose() -> None: + """The integrated part advertises three times the memory and holds no layers.""" + budget = build_budget([DISCRETE, INTEGRATED, CPU_LIVE], ram_available_bytes=22750 * MIB) + + assert budget.device_total_bytes == 6002 * MIB + + +def test_unified_memory_is_flagged_so_the_copy_can_branch() -> None: + """On Apple Silicon there is no separate pool to spill into, so the badge says + CPU rather than system RAM, which would describe a transfer that never happens.""" + assert build_budget([METAL, CPU_RESTATED], ram_available_bytes=2000 * MIB).uma + + +def test_a_machine_with_no_gpu_still_produces_a_budget() -> None: + """It badges as running on the processor. PARTIAL is unreachable, not an error.""" + budget = build_budget([CPU_LIVE], ram_available_bytes=22750 * MIB) + + assert not budget.has_gpu + assert budget.usable_vram_bytes == 0 + + +def test_the_ram_half_comes_from_the_caller_not_from_ggml() -> None: + """ggml's CPU device restates the total on macOS and under WSL2, and that + number is what separates PARTIAL from TOO_BIG.""" + budget = build_budget([METAL, CPU_RESTATED], ram_available_bytes=1500 * MIB) + + assert budget.ram_available_bytes == 1500 * MIB + + +def test_catalog_pricing_uses_capacity_not_whatever_is_free_right_now() -> None: + """Two modes, and using the wrong one makes the catalog lie. + + Pricing against live free memory means the badges move with whatever else the + machine happens to be doing: an 8 GB Mac with a browser open reports a couple + of GB free and every large row reads TOO_BIG, though the model would run once + the memory is reclaimed. Capacity is the honest basis for a shelf of models + you might install later. + """ + busy = build_budget( + [METAL, CPU_RESTATED], ram_available_bytes=2000 * MIB, mode=BudgetMode.CAPACITY + ) + idle = build_budget( + [METAL, CPU_RESTATED], ram_available_bytes=7000 * MIB, mode=BudgetMode.CAPACITY + ) + + assert busy.ram_available_bytes == idle.ram_available_bytes + + +def test_launch_decisions_use_what_is_actually_free() -> None: + """The opposite case: a model is about to be loaded now, into memory that + exists now, so the live reading is the one that governs.""" + budget = build_budget( + [METAL, CPU_RESTATED], ram_available_bytes=2000 * MIB, mode=BudgetMode.LIVE + ) + + assert budget.ram_available_bytes == 2000 * MIB + + +def test_capacity_never_reports_less_memory_than_is_free_right_now() -> None: + """Found in dev: with no CPU device to state physical RAM, capacity fell back + to the live reading and then subtracted the host reserve from it anyway. + + On a busy 8 GB machine that is 1.9 GB minus 2 GB, which floors at zero and + badges every model `Won't fit ... This PC has 0 GB`. A fallback is a floor, + not another thing to deduct from. + """ + live = 1_900_000_000 + + budget = build_budget([], ram_available_bytes=live, mode=BudgetMode.CAPACITY) + + assert budget.ram_available_bytes >= live + + +def test_capacity_still_holds_a_reserve_back_when_it_knows_the_total() -> None: + """The reserve is what the host keeps once a model is resident, and it is + only meaningful against physical RAM.""" + budget = build_budget( + [METAL, CPU_RESTATED], ram_available_bytes=1_900_000_000, + mode=BudgetMode.CAPACITY, + ) + + assert budget.ram_available_bytes == 8192 * MIB - 2 * 1024**3 + + +def test_unified_memory_is_bounded_by_what_is_actually_free() -> None: + """Metal reports `recommendedMaxWorkingSetSize` as both total and free, a + static figure that never moves however busy the machine is. + + Spending it as though it were live is what sized a 28,672 token window on a + Mac with 2.3 GB reclaimable: 4,355 MiB of weights and cache came out of the + same 8 GB the OS was using, the load paged, and it took 43 seconds. + """ + budget = build_budget( + [METAL, CPU_RESTATED], + ram_available_bytes=2300 * MIB, + mode=BudgetMode.LIVE, + ) + + assert budget.device_free_bytes <= 2300 * MIB + + +def test_unified_memory_still_respects_the_graphics_working_set() -> None: + """Both limits apply. Metal refuses allocations past the working set however + much system memory is free, so the smaller of the two governs.""" + budget = build_budget( + [METAL, CPU_RESTATED], + ram_available_bytes=7000 * MIB, + mode=BudgetMode.LIVE, + ) + + assert budget.device_free_bytes <= METAL.free_bytes + + +def test_a_unified_catalog_prices_against_capacity_not_this_moment() -> None: + """Rows must not move because a browser is open, so the shelf is priced + against what the machine could give a model, minus what the host keeps.""" + busy = build_budget( + [METAL, CPU_RESTATED], ram_available_bytes=1000 * MIB, + mode=BudgetMode.CAPACITY, + ) + idle = build_budget( + [METAL, CPU_RESTATED], ram_available_bytes=7000 * MIB, + mode=BudgetMode.CAPACITY, + ) + + assert busy.device_free_bytes == idle.device_free_bytes + + +def test_a_discrete_card_is_untouched_by_the_mode() -> None: + """Dedicated VRAM is a separate pool and its `free` really is live, so + nothing about host memory should narrow it.""" + live = build_budget([DISCRETE, CPU_LIVE], 22750 * MIB, mode=BudgetMode.LIVE) + capacity = build_budget([DISCRETE, CPU_LIVE], 22750 * MIB, mode=BudgetMode.CAPACITY) + + assert live.device_free_bytes == DISCRETE.free_bytes + assert capacity.device_free_bytes == DISCRETE.free_bytes diff --git a/surfsense_local/backend/tests/unit/llm/hardware/test_probe.py b/surfsense_local/backend/tests/unit/llm/hardware/test_probe.py new file mode 100644 index 0000000000..6038d40322 --- /dev/null +++ b/surfsense_local/backend/tests/unit/llm/hardware/test_probe.py @@ -0,0 +1,65 @@ +"""The ctypes probe, against the libraries that actually ship. + +Skipped when llama.cpp has not been staged, because there is nothing honest to +assert without it: a mocked ggml would only prove the mock works. +""" + +from pathlib import Path + +import pytest + +from modules.llm.hardware import DeviceType, load, probe_devices, select_device + +pytestmark = pytest.mark.unit + +LIBRARY_DIR = Path(__file__).resolve().parents[5] / "electron" / "llamacpp" +staged = pytest.mark.skipif( + not LIBRARY_DIR.exists(), reason="run `pnpm build:llamacpp` to stage llama.cpp" +) + + +@staged +def test_the_probe_finds_the_machines_devices() -> None: + """Every supported host has at least a CPU device, so an empty list is a bug.""" + devices = probe_devices(LIBRARY_DIR) + + assert devices + assert any(d.type is DeviceType.CPU for d in devices) + + +@staged +def test_the_probe_leaves_the_working_directory_where_it_found_it() -> None: + """It chdirs to satisfy ggml's backend scan, and this process has other work.""" + before = Path.cwd() + + probe_devices(LIBRARY_DIR) + + assert Path.cwd() == before + + +@staged +def test_a_device_type_outside_the_enum_never_reaches_a_caller() -> None: + """Accelerate lists on every Mac at a value a three-member enum would reject.""" + for device in probe_devices(LIBRARY_DIR): + assert isinstance(device.type, DeviceType) + + +@staged +def test_the_selected_device_is_never_larger_than_a_real_one() -> None: + """The guard against summing devices, asserted on whatever this host reports.""" + devices = probe_devices(LIBRARY_DIR) + chosen = select_device(devices) + + if chosen is not None: + assert chosen.total_bytes <= max(d.total_bytes for d in devices) + + +@staged +def test_one_handle_resolves_every_symbol_off_windows() -> None: + """Windows needs two handles because PE exports do not chain. Everywhere else + the dependency record does it, and a regression here is an AttributeError at + startup rather than a wrong answer, so it is worth pinning.""" + libraries = load(LIBRARY_DIR) + + assert libraries.core.ggml_backend_load_all + assert libraries.base.ggml_backend_dev_memory diff --git a/surfsense_local/backend/tests/unit/llm/hardware/test_selection.py b/surfsense_local/backend/tests/unit/llm/hardware/test_selection.py new file mode 100644 index 0000000000..5f801a9529 --- /dev/null +++ b/surfsense_local/backend/tests/unit/llm/hardware/test_selection.py @@ -0,0 +1,74 @@ +"""Choosing the device to budget against. + +Both listings below are real: the Windows one from an RTX 3050 machine, the +macOS one from an M2. Each contains a trap that a plausible rule falls into. +""" + +import pytest + +from modules.llm.hardware import Device, DeviceType, select_device + +pytestmark = pytest.mark.unit + +MIB = 1024**2 + +# Measured on Windows. The integrated Radeon advertises 16 GB of system RAM +# against the discrete card's 6 GB. +WINDOWS = [ + Device("Vulkan0", "NVIDIA GeForce RTX 3050", DeviceType.GPU, 6002 * MIB, 5234 * MIB), + Device("Vulkan1", "AMD Radeon(TM) Graphics", DeviceType.IGPU, 16198 * MIB, 16198 * MIB), + Device("CPU", "AMD Ryzen 5 9600X", DeviceType.CPU, 31884 * MIB, 22750 * MIB), +] + +# Measured on an M2. Note the Accelerate BLAS device, which is neither GPU nor CPU. +MACOS = [ + Device("MTL0", "Apple M2", DeviceType.GPU, 5461 * MIB, 5461 * MIB), + Device("BLAS", "Accelerate", DeviceType.ACCEL, 0, 0), + Device("CPU", "Apple M2", DeviceType.CPU, 8192 * MIB, 8192 * MIB), +] + + +def test_the_discrete_card_wins_over_a_larger_integrated_one() -> None: + """Sorting by memory picks the 16 GB integrated chip and puts every layer on + the slower device. ggml already orders backends by preference, so first wins.""" + assert select_device(WINDOWS).name == "Vulkan0" + + +def test_an_accelerator_is_never_mistaken_for_a_gpu() -> None: + """Every Mac lists an Accelerate BLAS device, and it holds no weights.""" + assert select_device(MACOS).name == "MTL0" + + +def test_the_same_card_listed_once_per_backend_is_never_summed() -> None: + """With CUDA and Vulkan both loaded a card appears twice, both typed GPU, so + type filtering does not deduplicate. Summing reports 12 GB on a 6 GB card and + every fit badge is then wrong in the dangerous direction.""" + doubled = [ + Device("CUDA0", "NVIDIA GeForce RTX 3050", DeviceType.GPU, 6143 * MIB, 5166 * MIB), + Device("Vulkan0", "NVIDIA GeForce RTX 3050", DeviceType.GPU, 6002 * MIB, 5166 * MIB), + Device("CPU", "AMD Ryzen 5 9600X", DeviceType.CPU, 31884 * MIB, 22750 * MIB), + ] + + chosen = select_device(doubled) + + assert chosen.name == "CUDA0" + assert chosen.total_bytes <= 6143 * MIB + + +def test_a_machine_with_no_gpu_says_so() -> None: + """Not an error. It badges as running on the processor, and PARTIAL becomes + unreachable because there is nothing to spill from.""" + assert select_device([MACOS[2]]) is None + + +def test_an_integrated_gpu_alone_is_not_budgeted_as_a_gpu() -> None: + """A part carving from system RAM has no memory of its own to place layers in.""" + assert select_device([WINDOWS[1], WINDOWS[2]]) is None + + +def test_a_device_type_ggml_adds_later_is_not_a_gpu_and_is_not_a_crash() -> None: + """ggml has grown this enum twice. An unknown integer must degrade to "not a + GPU" rather than raise, because this runs on the first render.""" + listing = [Device("FUTURE0", "something new", DeviceType.parse(99), 8 * MIB, 8 * MIB)] + + assert select_device(listing) is None From a1170b28a6d01e32207def7728742a0dd90b463b Mon Sep 17 00:00:00 2001 From: Anish Sarkar <104695310+AnishSarkar22@users.noreply.github.com> Date: Sun, 20 Sep 2026 20:07:42 +0530 Subject: [PATCH 15/60] feat(llm/fit): estimate KV cache, compute buffers, and load plans per model Co-Authored-By: Claude Sonnet 5 --- .../backend/modules/llm/fit/__init__.py | 24 +++ .../backend/modules/llm/fit/budget.py | 25 ++++ .../modules/llm/fit/compute_buffers.py | 25 ++++ .../backend/modules/llm/fit/copy.py | 65 ++++++++ .../backend/modules/llm/fit/estimate.py | 97 ++++++++++++ .../backend/modules/llm/fit/kv_cache.py | 33 +++++ .../backend/modules/llm/fit/plan_load.py | 102 +++++++++++++ .../backend/modules/llm/fit/sliding_window.py | 29 ++++ .../backend/modules/llm/fit/states.py | 9 ++ .../backend/modules/llm/fit/types.py | 37 +++++ .../backend/tests/unit/llm/fit/__init__.py | 0 .../tests/unit/llm/fit/test_badge_copy.py | 75 ++++++++++ .../tests/unit/llm/fit/test_estimate.py | 139 ++++++++++++++++++ .../tests/unit/llm/fit/test_kv_cache.py | 44 ++++++ .../tests/unit/llm/fit/test_plan_load.py | 117 +++++++++++++++ .../tests/unit/llm/fit/test_sliding_window.py | 53 +++++++ 16 files changed, 874 insertions(+) create mode 100644 surfsense_local/backend/modules/llm/fit/__init__.py create mode 100644 surfsense_local/backend/modules/llm/fit/budget.py create mode 100644 surfsense_local/backend/modules/llm/fit/compute_buffers.py create mode 100644 surfsense_local/backend/modules/llm/fit/copy.py create mode 100644 surfsense_local/backend/modules/llm/fit/estimate.py create mode 100644 surfsense_local/backend/modules/llm/fit/kv_cache.py create mode 100644 surfsense_local/backend/modules/llm/fit/plan_load.py create mode 100644 surfsense_local/backend/modules/llm/fit/sliding_window.py create mode 100644 surfsense_local/backend/modules/llm/fit/states.py create mode 100644 surfsense_local/backend/modules/llm/fit/types.py create mode 100644 surfsense_local/backend/tests/unit/llm/fit/__init__.py create mode 100644 surfsense_local/backend/tests/unit/llm/fit/test_badge_copy.py create mode 100644 surfsense_local/backend/tests/unit/llm/fit/test_estimate.py create mode 100644 surfsense_local/backend/tests/unit/llm/fit/test_kv_cache.py create mode 100644 surfsense_local/backend/tests/unit/llm/fit/test_plan_load.py create mode 100644 surfsense_local/backend/tests/unit/llm/fit/test_sliding_window.py diff --git a/surfsense_local/backend/modules/llm/fit/__init__.py b/surfsense_local/backend/modules/llm/fit/__init__.py new file mode 100644 index 0000000000..fee0b839be --- /dev/null +++ b/surfsense_local/backend/modules/llm/fit/__init__.py @@ -0,0 +1,24 @@ +from modules.llm.fit.budget import HardwareBudget +from modules.llm.fit.compute_buffers import compute_buffer_bytes +from modules.llm.fit.copy import Badge, badge +from modules.llm.fit.estimate import CONTEXT_FLOOR_TOKENS, FitVerdict, estimate +from modules.llm.fit.kv_cache import kv_cache_bytes +from modules.llm.fit.plan_load import LoadPlan, plan_load +from modules.llm.fit.states import FitState +from modules.llm.fit.types import KvPrecision, ModelShape + +__all__ = [ + "CONTEXT_FLOOR_TOKENS", + "Badge", + "FitState", + "FitVerdict", + "HardwareBudget", + "KvPrecision", + "LoadPlan", + "ModelShape", + "badge", + "compute_buffer_bytes", + "estimate", + "kv_cache_bytes", + "plan_load", +] diff --git a/surfsense_local/backend/modules/llm/fit/budget.py b/surfsense_local/backend/modules/llm/fit/budget.py new file mode 100644 index 0000000000..4a6e666be8 --- /dev/null +++ b/surfsense_local/backend/modules/llm/fit/budget.py @@ -0,0 +1,25 @@ +"""One device's memory, never a sum across devices.""" + +from dataclasses import dataclass + + +@dataclass(frozen=True) +class HardwareBudget: + """What a single device offers, and what the runtime will decline to use. + + `fit_reserve_bytes` is llama.cpp's own margin rather than an estimate of it: + `fit_params_target` defaults to one GiB per device and is settable through + `--fit-target`, so this is a number we pass in, not one we predict. + """ + + device_free_bytes: int + device_total_bytes: int + fit_reserve_bytes: int + ram_available_bytes: int + uma: bool + has_gpu: bool + + @property + def usable_vram_bytes(self) -> int: + """What --fit will actually place layers in.""" + return max(0, self.device_free_bytes - self.fit_reserve_bytes) diff --git a/surfsense_local/backend/modules/llm/fit/compute_buffers.py b/surfsense_local/backend/modules/llm/fit/compute_buffers.py new file mode 100644 index 0000000000..0d906f987e --- /dev/null +++ b/surfsense_local/backend/modules/llm/fit/compute_buffers.py @@ -0,0 +1,25 @@ +"""The scratch memory a graph needs beyond weights and cache. + +Absent from earlier drafts of the estimator entirely, which under-stated `need` +by enough to flip a verdict on a 6 GB card. + +> ponytail: two measured points, one model, one backend. Qwen3 1.7B on Metal at +> b11050 reported 102.24 MiB at 16384 and 222.24 MiB at 40960, and a 4B on +> Vulkan reported 143.62 device plus 26.01 host at 16384. The line below passes +> through the Metal pair and is the best available answer, not a law. Replace it +> with the machine's own residual after the first load, which measures this term +> along with everything else that is neither weights nor cache. +""" + +_BASE_BYTES = 23 * 1024**2 +_PER_TOKEN_BYTES = 5_120 + + +def compute_buffer_bytes(n_ctx: int) -> int: + """Scratch bytes for a graph over an `n_ctx` window, rounded up. + + Rounding up is the rule the whole estimator runs on: over-stating costs a + pessimistic badge, under-stating ships a confident badge about a model that + spills. + """ + return _BASE_BYTES + _PER_TOKEN_BYTES * n_ctx diff --git a/surfsense_local/backend/modules/llm/fit/copy.py b/surfsense_local/backend/modules/llm/fit/copy.py new file mode 100644 index 0000000000..646f1ffd1c --- /dev/null +++ b/surfsense_local/backend/modules/llm/fit/copy.py @@ -0,0 +1,65 @@ +"""Badge wording. + +A badge is a **verdict plus one plain line of why**. The verdict is what someone +choosing a model needs (fast, slower, or impossible); the mechanism is the +explanation, not the headline. + +Wording branches on two facts the budget already carries, `uma` and `has_gpu`, +and never on the runtime's name. One label set is wrong on two of the three +targets. + +**No em dashes and no hyphens in anything here.** Commas, full stops or +parentheses. That is a standing rule for every user facing string in this phase. +""" + +from dataclasses import dataclass + +from modules.llm.fit.budget import HardwareBudget +from modules.llm.fit.estimate import FitVerdict +from modules.llm.fit.states import FitState + +# Above this the spill is no longer something a user can ignore. Below it, most +# of the model is still resident and the difference is barely noticeable. +_NOTICEABLE_SPILL = 0.35 + + +@dataclass(frozen=True) +class Badge: + verdict: str + reason: str + + +def badge(fit: FitVerdict, budget: HardwareBudget) -> Badge: + if fit.state is FitState.TOO_BIG: + return Badge("Won't fit", _refusal(fit, budget)) + if fit.state is FitState.PARTIAL: + return Badge("Reduced speed", _spill(fit, budget)) + if not budget.has_gpu: + return Badge("Works here", "Runs on your processor") + return Badge( + "Full speed", + "Runs entirely on the GPU" if budget.uma else "Runs entirely on your graphics card", + ) + + +def _spill(fit: FitVerdict, budget: HardwareBudget) -> str: + where = "the GPU" if budget.uma else "the graphics card" + if fit.offload_fraction <= _NOTICEABLE_SPILL: + return f"A little too big for {where}. Most of it still fits." + return f"Well over {where}'s memory. Expect it to be slow." + + +def _refusal(fit: FitVerdict, budget: HardwareBudget) -> str: + machine = "This Mac" if budget.uma else "This PC" + return f"Needs about {_gb(fit.need_bytes)}. {machine} has {_gb(fit.budget_bytes)}" + + +def _gb(value: int) -> str: + """Decimal GB, as storage and download sizes are quoted everywhere else. + + One decimal, with a trailing zero dropped, so a pair reads "21 GB" and + "13.6 GB" rather than "21.0 GB" and "14 GB". Rounding the second to a whole + number loses the half gigabyte that decided the answer. + """ + size = round(value / 1000**3, 1) + return f"{size:g} GB" diff --git a/surfsense_local/backend/modules/llm/fit/estimate.py b/surfsense_local/backend/modules/llm/fit/estimate.py new file mode 100644 index 0000000000..c404419c10 --- /dev/null +++ b/surfsense_local/backend/modules/llm/fit/estimate.py @@ -0,0 +1,97 @@ +"""Does this model fit in this machine, and if not, how much spills. + + need = weights + KV(window) + compute_buffers what the model allocates + usable = device_free - fit_reserve what --fit will use + +The two subtractions sit on opposite sides of the comparison. Collapsing them +into one `overhead` term predicts residency for a configuration that measurably +spills, which is the specific failure this estimator exists to avoid. +""" + +from dataclasses import dataclass + +from modules.llm.fit.budget import HardwareBudget +from modules.llm.fit.compute_buffers import compute_buffer_bytes +from modules.llm.fit.kv_cache import kv_cache_bytes +from modules.llm.fit.states import FitState +from modules.llm.fit.types import KvPrecision, ModelShape + +CONTEXT_FLOOR_TOKENS = 16384 + + +@dataclass(frozen=True) +class FitVerdict: + state: FitState + need_bytes: int + budget_bytes: int + offload_fraction: float + approximate: bool = False + + @property + def can_install(self) -> bool: + """Only physics refuses. Reduced speed runs, so it never blocks.""" + return self.state is not FitState.TOO_BIG + + +def estimate( + shape: ModelShape, + weights_bytes: int, + budget: HardwareBudget, + *, + n_ctx: int = CONTEXT_FLOOR_TOKENS, + precision: KvPrecision = KvPrecision.F16, + mmproj_bytes: int = 0, +) -> FitVerdict: + """Price one build against one device. + + `TOO_BIG` is evaluated at the context floor rather than the requested window: + a model that will not fit at 16K cannot be rescued by a smaller context, and + the remedy to offer is a smaller build. + """ + # The projector is counted here because `--fit` does not count it: + # llama.cpp issue #19980. Left out, a vision model our sum calls resident + # can still fail to allocate on a tight machine. + need = ( + weights_bytes + + mmproj_bytes + + kv_cache_bytes(shape, n_ctx, precision) + + compute_buffer_bytes(n_ctx) + ) + usable = budget.usable_vram_bytes + + floor_need = ( + weights_bytes + + mmproj_bytes + + kv_cache_bytes(shape, CONTEXT_FLOOR_TOKENS, precision) + + compute_buffer_bytes(CONTEXT_FLOOR_TOKENS) + ) + if floor_need > usable + budget.ram_available_bytes: + return FitVerdict( + state=FitState.TOO_BIG, + need_bytes=need, + budget_bytes=usable + budget.ram_available_bytes, + offload_fraction=1.0, + ) + + if need <= usable: + return FitVerdict( + state=FitState.FITS, + need_bytes=need, + budget_bytes=usable, + offload_fraction=0.0, + ) + + # > ponytail: this is the theoretical minimum spill, and llama.cpp spills in + # > whole layers, so reality rounds up. Measured on an RTX 3050, Qwen3 4B + # > spilled 602 MiB of weights plus 320 MiB of cache, f = 0.19, where this + # > arithmetic gives 0.12. The states agree and the ordering agrees; the + # > fraction does not. It matters because the badge copy is graded on it at + # > 0.25 and 0.5, so calibrate against layer-granular placement before those + # > thresholds are trusted. + spilled = need - usable + return FitVerdict( + state=FitState.PARTIAL, + need_bytes=need, + budget_bytes=usable, + offload_fraction=min(1.0, spilled / need), + ) diff --git a/surfsense_local/backend/modules/llm/fit/kv_cache.py b/surfsense_local/backend/modules/llm/fit/kv_cache.py new file mode 100644 index 0000000000..c77cb7d4cb --- /dev/null +++ b/surfsense_local/backend/modules/llm/fit/kv_cache.py @@ -0,0 +1,33 @@ +"""The conversation cache's memory cost, which is architecture-derived and +independent of how the weights were quantized.""" + +from modules.llm.fit.sliding_window import global_layer_count +from modules.llm.fit.types import KvPrecision, ModelShape + + +def kv_cache_bytes(shape: ModelShape, n_ctx: int, precision: KvPrecision) -> int: + """Bytes the KV cache occupies for a window of `n_ctx` tokens. + + One key and one value per attention head per layer per token. Verified exact + against llama.cpp's own allocation on two backends. + + A model with sliding-window attention holds the full context on only some of + its layers, but the header does not say which, so an architecture we cannot + name is priced as though every layer were global. That over-states memory, + which is the only direction it is safe to be wrong in. + """ + bytes_per_layer_token = int( + shape.head_count_kv + * (shape.key_length + shape.value_length) + * precision.bytes_per_element + ) + + if shape.sliding_window <= 0 or shape.sliding_window >= n_ctx: + return bytes_per_layer_token * shape.block_count * n_ctx + + globals_ = global_layer_count(shape.architecture, shape.block_count) + if globals_ is None: + return bytes_per_layer_token * shape.block_count * n_ctx + + locals_ = shape.block_count - globals_ + return bytes_per_layer_token * (globals_ * n_ctx + locals_ * shape.sliding_window) diff --git a/surfsense_local/backend/modules/llm/fit/plan_load.py b/surfsense_local/backend/modules/llm/fit/plan_load.py new file mode 100644 index 0000000000..c6f8ad4546 --- /dev/null +++ b/surfsense_local/backend/modules/llm/fit/plan_load.py @@ -0,0 +1,102 @@ +"""The window and cache precision to load a model with. + +Context is fixed at load, so this is decided once and cannot be renegotiated +mid-conversation. Two rules shape it: + +- **Prefer residency over window.** KV is allocated upfront and competes with the + weights, so maximising context silently demotes a model out of GPU residency. + Widen only while the verdict stays FITS. +- **Prefer f16 over q8_0.** Quality is identical, but quantized cache requires a + working flash-attention kernel, and when one is unavailable llama.cpp falls + back to CPU attention silently. Take that dependency only where it pays. +""" + +from dataclasses import dataclass + +from modules.llm.fit.budget import HardwareBudget +from modules.llm.fit.estimate import CONTEXT_FLOOR_TOKENS, FitVerdict, estimate +from modules.llm.fit.states import FitState +from modules.llm.fit.types import KvPrecision, ModelShape + + +@dataclass(frozen=True) +class LoadPlan: + n_ctx: int + precision: KvPrecision + verdict: FitVerdict + + +def plan_load( + shape: ModelShape, + weights_bytes: int, + budget: HardwareBudget, + live: HardwareBudget | None = None, +) -> LoadPlan: + """Choose the widest window that stays resident, at the cheapest precision. + + The cap beats the floor. A model trained to 8192 tokens cannot be asked for + 16384 just because a grounded turn would like one: the floor describes what + this app needs, not what a model can do. + + Two budgets, answering two questions. `budget` is capacity, and it decides + the **verdict**, so the plan agrees with the badge the catalog already + showed: a machine that is busy this second has not become one that cannot + run the model. `live` is what is free right now, and it caps **widening** + only, because context past the floor is opportunistic and on unified memory + it is spent out of the same pool the OS is using. + + Without that split, a 1.7B took a 28,672 token window on an 8 GB Mac with + 2.3 GB reclaimable, the load paged, and it took 43 seconds. + """ + ceiling = shape.context_length or CONTEXT_FLOOR_TOKENS + floor = min(CONTEXT_FLOOR_TOKENS, ceiling) + + for precision in (KvPrecision.F16, KvPrecision.Q8_0): + at_floor = estimate( + shape, weights_bytes, budget, n_ctx=floor, precision=precision + ) + if at_floor.state is not FitState.FITS: + continue + n_ctx = _widest_resident( + shape, weights_bytes, live or budget, precision, ceiling, floor + ) + return LoadPlan( + n_ctx=n_ctx, + precision=precision, + verdict=estimate( + shape, weights_bytes, budget, n_ctx=n_ctx, precision=precision + ), + ) + + # Nothing keeps it resident, so hold the floor and let --fit place the layers. + return LoadPlan( + n_ctx=floor, + precision=KvPrecision.F16, + verdict=estimate(shape, weights_bytes, budget, n_ctx=floor), + ) + + +def _widest_resident( + shape: ModelShape, + weights_bytes: int, + budget: HardwareBudget, + precision: KvPrecision, + ceiling: int, + floor: int, +) -> int: + """Binary search the window, in whole thousands of tokens.""" + low, high = floor, ceiling + while low < high: + mid = min(ceiling, (low + high + 1024) // 2) + mid -= mid % 1024 + if mid <= low: + break + fits = ( + estimate(shape, weights_bytes, budget, n_ctx=mid, precision=precision).state + is FitState.FITS + ) + if fits: + low = mid + else: + high = mid - 1024 + return max(floor, min(low, ceiling)) diff --git a/surfsense_local/backend/modules/llm/fit/sliding_window.py b/surfsense_local/backend/modules/llm/fit/sliding_window.py new file mode 100644 index 0000000000..5029ee6ffc --- /dev/null +++ b/surfsense_local/backend/modules/llm/fit/sliding_window.py @@ -0,0 +1,29 @@ +"""Which layers of a sliding-window model hold the full context. + +The GGUF header states the window size but never the pattern, so the pattern is +a per-architecture fact. llama.cpp carries the same knowledge in `src/models`; +this is the subset that matters to the catalog, and anything absent is priced as +full attention rather than guessed at. +""" + +# Global layers per cycle: a value of 6 means one layer in six attends to the +# whole context and the other five attend to `sliding_window` tokens. +_GLOBAL_LAYER_EVERY: dict[str, int] = { + "gemma2": 2, + "gemma3": 6, + "gemma3n": 6, + "cohere2": 4, + "phi3": 2, +} + + +def global_layer_count(architecture: str, block_count: int) -> int | None: + """How many layers attend to the whole window, or None when unknown. + + None is the honest answer for an architecture nobody has characterised, and + the caller must then round up rather than assume a ratio. + """ + cycle = _GLOBAL_LAYER_EVERY.get(architecture) + if cycle is None: + return None + return block_count // cycle diff --git a/surfsense_local/backend/modules/llm/fit/states.py b/surfsense_local/backend/modules/llm/fit/states.py new file mode 100644 index 0000000000..cb3c414d02 --- /dev/null +++ b/surfsense_local/backend/modules/llm/fit/states.py @@ -0,0 +1,9 @@ +"""Where a model's weights will live, which is the only thing a badge claims.""" + +from enum import StrEnum + + +class FitState(StrEnum): + FITS = "fits" + PARTIAL = "partial" + TOO_BIG = "too_big" diff --git a/surfsense_local/backend/modules/llm/fit/types.py b/surfsense_local/backend/modules/llm/fit/types.py new file mode 100644 index 0000000000..1096f6b233 --- /dev/null +++ b/surfsense_local/backend/modules/llm/fit/types.py @@ -0,0 +1,37 @@ +"""What the estimator reasons over: a model's shape and a machine's memory.""" + +from dataclasses import dataclass +from enum import StrEnum + + +class KvPrecision(StrEnum): + """How the conversation cache is stored. Lossless either way; f16 costs twice + the memory of q8_0 and needs no flash-attention kernel to be fast.""" + + F16 = "f16" + Q8_0 = "q8_0" + + @property + def bytes_per_element(self) -> float: + # q8_0 is one byte per weight plus a shared scale per 32-element block. + return 2.0 if self is KvPrecision.F16 else 1.0 + 2.0 / 32.0 + + +@dataclass(frozen=True) +class ModelShape: + """Estimator inputs, named after the GGUF metadata keys they come from. + + Quantization-independent: measured, the architecture fields are identical + across Q4_K_M, Q8_0 and f16 of the same model, so one header read prices + every build of it. + """ + + architecture: str + block_count: int + head_count_kv: int + key_length: int + value_length: int + context_length: int + n_vocab: int + sliding_window: int = 0 + expert_count: int = 0 diff --git a/surfsense_local/backend/tests/unit/llm/fit/__init__.py b/surfsense_local/backend/tests/unit/llm/fit/__init__.py new file mode 100644 index 0000000000..e69de29bb2 diff --git a/surfsense_local/backend/tests/unit/llm/fit/test_badge_copy.py b/surfsense_local/backend/tests/unit/llm/fit/test_badge_copy.py new file mode 100644 index 0000000000..27355e3938 --- /dev/null +++ b/surfsense_local/backend/tests/unit/llm/fit/test_badge_copy.py @@ -0,0 +1,75 @@ +"""What a fit verdict says to a person. + +A badge is a verdict plus one plain line of why. The verdict is what someone +choosing a model needs; the mechanism is the explanation, not the headline. +""" + +import pytest + +from modules.llm.fit import FitState, HardwareBudget, badge +from modules.llm.fit.estimate import FitVerdict + +pytestmark = pytest.mark.unit + +MIB = 1024**2 +DISCRETE = HardwareBudget(6000 * MIB, 8000 * MIB, 1024 * MIB, 16000 * MIB, False, True) +APPLE = HardwareBudget(5461 * MIB, 5461 * MIB, 1024 * MIB, 6144 * MIB, True, True) +NO_GPU = HardwareBudget(0, 0, 1024 * MIB, 16000 * MIB, False, False) + + +def verdict(state: FitState, f: float = 0.0) -> FitVerdict: + """A verdict needing 21 GB against 13.6 GB available.""" + return FitVerdict(state, 21 * 1000**3, 13_600_000_000, f) + + +def test_no_user_facing_string_uses_an_em_dash_or_a_hyphen() -> None: + """A standing copy rule for this phase. Commas, full stops, parentheses.""" + for budget in (DISCRETE, APPLE, NO_GPU): + for state in FitState: + if state is FitState.PARTIAL and not budget.has_gpu: + continue + text = badge(verdict(state), budget) + assert "—" not in text.verdict + text.reason + assert "-" not in text.verdict + text.reason + + +def test_full_speed_rather_than_fast() -> None: + """Fast is a promise the badge cannot keep: a 32B on a 4090 is still slower + than a 4B. Full speed is relative to the model, which is what the state means. + """ + assert badge(verdict(FitState.FITS), DISCRETE).verdict == "Full speed" + + +def test_a_machine_with_no_gpu_is_told_it_works_not_that_it_is_fast() -> None: + """Technically FITS, but "Full speed" reads as a boast about a slow situation + when there is no faster alternative to contrast with.""" + assert badge(verdict(FitState.FITS), NO_GPU).verdict == "Works here" + + +def test_apple_silicon_is_never_told_about_system_ram() -> None: + """There is no separate pool to spill into. Some layers run on the CPU + backend against the same physical memory, so "uses system RAM" would + describe a transfer that does not happen.""" + text = badge(verdict(FitState.PARTIAL, 0.3), APPLE) + + assert "system RAM" not in text.reason + assert "graphics card" not in text.reason + + +def test_a_small_spill_and_a_large_one_do_not_get_the_same_sentence() -> None: + """PARTIAL spans barely noticeable to unusable, and the fraction is already + on the verdict. One sentence is wrong at both ends.""" + slight = badge(verdict(FitState.PARTIAL, 0.1), DISCRETE) + severe = badge(verdict(FitState.PARTIAL, 0.7), DISCRETE) + + assert slight.verdict == severe.verdict == "Reduced speed" + assert slight.reason != severe.reason + + +def test_a_refusal_states_both_numbers() -> None: + """Never a bare disabled control: say what it needs and what there is.""" + text = badge(verdict(FitState.TOO_BIG), DISCRETE) + + assert text.verdict == "Won't fit" + assert "21 GB" in text.reason + assert "13.6 GB" in text.reason diff --git a/surfsense_local/backend/tests/unit/llm/fit/test_estimate.py b/surfsense_local/backend/tests/unit/llm/fit/test_estimate.py new file mode 100644 index 0000000000..58cd7be975 --- /dev/null +++ b/surfsense_local/backend/tests/unit/llm/fit/test_estimate.py @@ -0,0 +1,139 @@ +"""The fit verdict, against llama.cpp's own fitter. + +Reference numbers come from `common_params_fit_impl` under `-v` at b11050 on an +M2 / 8 GB, which prints what it projects and what margin it must leave. +""" + +import pytest + +from modules.llm.fit import ( + FitState, + HardwareBudget, + KvPrecision, + ModelShape, + estimate, +) + +pytestmark = pytest.mark.unit + +MIB = 1024**2 + +QWEN3_1_7B = ModelShape( + architecture="qwen3", + block_count=28, + head_count_kv=8, + key_length=128, + value_length=128, + context_length=40960, + n_vocab=151936, +) +WEIGHTS = 1050 * MIB # MTL0_Mapped model buffer, measured + +# MTL0: 5461.3 MiB total, 5460 MiB free as the fitter saw it. +M2 = HardwareBudget( + device_free_bytes=5460 * MIB, + device_total_bytes=5461 * MIB, + fit_reserve_bytes=1024 * MIB, + ram_available_bytes=2000 * MIB, + uma=True, + has_gpu=True, +) + + +def test_the_measured_resident_case_fits() -> None: + """`projected to use 2944 MiB vs 5460 MiB free`, and it loaded 29/29.""" + verdict = estimate(QWEN3_1_7B, WEIGHTS, M2, n_ctx=16384) + + assert verdict.state is FitState.FITS + assert verdict.offload_fraction == 0.0 + + +def test_a_model_that_fits_in_free_memory_but_not_past_the_reserve_is_partial() -> None: + """The regression a single collapsed `overhead` term gets backwards. + + `need` sits under raw free memory and over what --fit will actually use, so + the two subtractions have to stay on opposite sides of the comparison. + """ + shape = ModelShape( + architecture="qwen3", + block_count=28, + head_count_kv=8, + key_length=128, + value_length=128, + context_length=40960, + n_vocab=151936, + ) + # 5000 MiB of need: below 5460 free, above 5460 - 1024 usable. + weights = 5000 * MIB - estimate(shape, 0, M2, n_ctx=16384).need_bytes + + verdict = estimate(shape, weights, M2, n_ctx=16384) + + assert M2.device_free_bytes > verdict.need_bytes > M2.usable_vram_bytes + assert verdict.state is FitState.PARTIAL + + +def test_the_measured_spilling_case_reports_how_much_went_to_the_cpu() -> None: + """At -c 40960 the fitter projected 5752 MiB and settled on 23 of 29 layers.""" + verdict = estimate(QWEN3_1_7B, WEIGHTS, M2, n_ctx=40960) + + assert verdict.state is FitState.PARTIAL + assert 0.0 < verdict.offload_fraction < 1.0 + + +def test_a_model_over_vram_plus_ram_is_refused_by_physics() -> None: + """The only state that blocks install, evaluated at the 16K floor.""" + huge = ModelShape( + architecture="qwen3", + block_count=64, + head_count_kv=8, + key_length=128, + value_length=128, + context_length=40960, + n_vocab=151936, + ) + + verdict = estimate(huge, 60_000 * MIB, M2, n_ctx=16384) + + assert verdict.state is FitState.TOO_BIG + assert not verdict.can_install + + +def test_partial_installs_exactly_like_fits() -> None: + """Only physics refuses. Reduced speed is a label, never a gate.""" + partial = estimate(QWEN3_1_7B, WEIGHTS, M2, n_ctx=40960) + + assert partial.state is FitState.PARTIAL + assert partial.can_install + + +def test_quantizing_the_cache_can_turn_a_spill_into_residency() -> None: + """This is why precision is an output of the fit calculation, not a default.""" + at_f16 = estimate(QWEN3_1_7B, WEIGHTS, M2, n_ctx=40960, precision=KvPrecision.F16) + at_q8 = estimate(QWEN3_1_7B, WEIGHTS, M2, n_ctx=40960, precision=KvPrecision.Q8_0) + + assert at_f16.state is FitState.PARTIAL + assert at_q8.state is FitState.FITS + + +def test_a_vision_projector_is_counted_because_the_fitter_does_not() -> None: + """llama.cpp issue #19980: `--fit` does not account for mmproj memory, so a + vision model that our sum says fits can still fail to allocate. We add the + projector's bytes to `need` rather than trusting the fitter to. + """ + text_only = estimate(QWEN3_1_7B, WEIGHTS, M2, n_ctx=16384) + with_projector = estimate( + QWEN3_1_7B, WEIGHTS, M2, n_ctx=16384, mmproj_bytes=600 * MIB + ) + + assert with_projector.need_bytes == text_only.need_bytes + 600 * MIB + + +def test_a_projector_can_be_what_tips_a_model_over() -> None: + """Which is the whole point of counting it.""" + resident = estimate(QWEN3_1_7B, 2400 * MIB, M2, n_ctx=16384) + with_projector = estimate( + QWEN3_1_7B, 2400 * MIB, M2, n_ctx=16384, mmproj_bytes=900 * MIB + ) + + assert resident.state is FitState.FITS + assert with_projector.state is FitState.PARTIAL diff --git a/surfsense_local/backend/tests/unit/llm/fit/test_kv_cache.py b/surfsense_local/backend/tests/unit/llm/fit/test_kv_cache.py new file mode 100644 index 0000000000..4983ee156a --- /dev/null +++ b/surfsense_local/backend/tests/unit/llm/fit/test_kv_cache.py @@ -0,0 +1,44 @@ +"""The KV term, checked against llama.cpp's own allocation log. + +Expected values are what `llama_kv_cache: size = ...` printed at b11050, not a +re-derivation of the formula under test. +""" + +import pytest + +from modules.llm.fit import KvPrecision, ModelShape, kv_cache_bytes + +pytestmark = pytest.mark.unit + +MIB = 1024**2 + +# Qwen3 1.7B: 28 blocks, 8 kv heads, 128 key/value. Measured on an M2 at b11050. +QWEN3_1_7B = ModelShape( + architecture="qwen3", + block_count=28, + head_count_kv=8, + key_length=128, + value_length=128, + context_length=40960, + n_vocab=151936, +) + + +def test_the_window_is_priced_as_llama_cpp_allocates_it() -> None: + """`llama_kv_cache: size = 1792.00 MiB (16384 cells, 28 layers)`, measured.""" + assert kv_cache_bytes(QWEN3_1_7B, 16384, KvPrecision.F16) == 1792 * MIB + + +def test_a_wider_window_costs_proportionally_more() -> None: + """Same model at 40960 cells reported 4480.00 MiB, measured in the same run.""" + assert kv_cache_bytes(QWEN3_1_7B, 40960, KvPrecision.F16) == 4480 * MIB + + +def test_quantizing_the_cache_roughly_halves_it() -> None: + """GGUF's block_q8_0 is 32 int8 weights plus one fp16 scale, so 34 bytes buys + what f16 spends 64 on. 1792 MiB * 34/64 = 952 MiB. + + This is the swing that converts a spill into residency on a small card, which + is why precision is an output of the fit calculation rather than a default. + """ + assert kv_cache_bytes(QWEN3_1_7B, 16384, KvPrecision.Q8_0) == 952 * MIB diff --git a/surfsense_local/backend/tests/unit/llm/fit/test_plan_load.py b/surfsense_local/backend/tests/unit/llm/fit/test_plan_load.py new file mode 100644 index 0000000000..51d6bcc3c2 --- /dev/null +++ b/surfsense_local/backend/tests/unit/llm/fit/test_plan_load.py @@ -0,0 +1,117 @@ +"""What we hand the runtime at load: a window and a cache precision. + +Both are outputs of the fit calculation rather than defaults, because on a small +card the choice is what decides whether the model is resident at all. +""" + +import pytest + +from modules.llm.fit import ( + CONTEXT_FLOOR_TOKENS, + FitState, + HardwareBudget, + KvPrecision, + ModelShape, + plan_load, +) + +pytestmark = pytest.mark.unit + +MIB = 1024**2 +QWEN3_1_7B = ModelShape("qwen3", 28, 8, 128, 128, 40960, 151936) + + +def budget(free_mib: int) -> HardwareBudget: + """A unified-memory device with llama.cpp's own one GiB margin.""" + return HardwareBudget(free_mib * MIB, free_mib * MIB, 1024 * MIB, 2000 * MIB, True, True) + + +def test_a_roomy_machine_keeps_the_lossless_cache() -> None: + """f16 needs no flash-attention kernel, so it is preferred when it fits.""" + plan = plan_load(QWEN3_1_7B, 1050 * MIB, budget(5460)) + + assert plan.precision is KvPrecision.F16 + assert plan.verdict.state is FitState.FITS + + +def test_precision_drops_only_when_it_buys_residency() -> None: + """The measured case: f16 at 16K cannot allocate on a 6 GB card, q8_0 runs. + + Quality is lossless either way, so the cost is a flash-attention dependency + we take on only where it converts a spill into residency. + """ + plan = plan_load(QWEN3_1_7B, 2600 * MIB, budget(5460)) + + assert plan.precision is KvPrecision.Q8_0 + assert plan.verdict.state is FitState.FITS + + +def test_the_window_never_drops_below_the_floor() -> None: + """3000 history tokens plus roughly 8000 of grounding plus a reply. + + llama.cpp would reduce context to 4096 on its own if we left it unset, well + under what a grounded turn needs, so the floor is ours to hold. + """ + plan = plan_load(QWEN3_1_7B, 4200 * MIB, budget(5460)) + + assert plan.n_ctx == CONTEXT_FLOOR_TOKENS + assert plan.verdict.state is FitState.PARTIAL + + +def test_the_window_never_exceeds_what_the_model_was_trained_for() -> None: + """Capped at the model's own context_length, however much memory there is.""" + plan = plan_load(QWEN3_1_7B, 1050 * MIB, budget(64000)) + + assert plan.n_ctx == QWEN3_1_7B.context_length + + +def test_a_roomy_machine_widens_the_window_rather_than_leaving_it_at_the_floor() -> None: + """Headroom buys conversation memory, which is the thing a user notices.""" + plan = plan_load(QWEN3_1_7B, 1050 * MIB, budget(8000)) + + assert plan.n_ctx > CONTEXT_FLOOR_TOKENS + assert plan.verdict.state is FitState.FITS + + +def test_a_model_trained_narrower_than_the_floor_is_not_stretched_past_it() -> None: + """The floor and the cap conflict below 16K, and the cap has to win. + + Found end to end: SmolLM2 135M declares `context_length` 8192 and the plan + asked for 16384, which is a window the model was never trained for. The floor + says what a grounded turn needs; it cannot say what a model can do. + """ + smol = ModelShape("llama", 30, 3, 64, 64, 8192, 49152) + + plan = plan_load(smol, 100 * MIB, budget(5460)) + + assert plan.n_ctx == 8192 + + +def test_the_floor_is_attempted_whatever_is_free_right_now() -> None: + """The verdict must match the badge the catalog showed, which is priced + against capacity. A machine that is busy this second has not become one that + cannot run the model, and saying so after the user installed it would be the + catalog contradicting itself. + """ + capacity = HardwareBudget(5460 * MIB, 5460 * MIB, 1024 * MIB, 6144 * MIB, True, True) + nearly_nothing = HardwareBudget(600 * MIB, 5460 * MIB, 1024 * MIB, 600 * MIB, True, True) + + plan = plan_load(QWEN3_1_7B, 1050 * MIB, capacity, live=nearly_nothing) + + assert plan.n_ctx == CONTEXT_FLOOR_TOKENS + assert plan.verdict.state is FitState.FITS + + +def test_widening_past_the_floor_respects_what_is_actually_free() -> None: + """Extra context is opportunistic, and on unified memory it is spent out of + the same pool the OS is using. Taking 3.1 GB of cache to hold 28k tokens on + a machine with 2.3 GB free is what made a load take 43 seconds. + """ + capacity = HardwareBudget(5460 * MIB, 5460 * MIB, 1024 * MIB, 6144 * MIB, True, True) + live = HardwareBudget(1900 * MIB, 5460 * MIB, 1024 * MIB, 1900 * MIB, True, True) + + roomy = plan_load(QWEN3_1_7B, 1050 * MIB, capacity) + constrained = plan_load(QWEN3_1_7B, 1050 * MIB, capacity, live=live) + + assert roomy.n_ctx > constrained.n_ctx + assert constrained.n_ctx == CONTEXT_FLOOR_TOKENS diff --git a/surfsense_local/backend/tests/unit/llm/fit/test_sliding_window.py b/surfsense_local/backend/tests/unit/llm/fit/test_sliding_window.py new file mode 100644 index 0000000000..1fe042003d --- /dev/null +++ b/surfsense_local/backend/tests/unit/llm/fit/test_sliding_window.py @@ -0,0 +1,53 @@ +"""Sliding-window attention: most layers hold a short window, a few hold it all. + +The header declares the window size but never the pattern of which layers use +it, so the pattern is a per-architecture fact we either know or refuse to guess. +""" + +import pytest + +from modules.llm.fit import KvPrecision, ModelShape, kv_cache_bytes + +pytestmark = pytest.mark.unit + +MIB = 1024**2 + + +def gemma_like(architecture: str) -> ModelShape: + """48 blocks declaring a 1024-token window, as a real Gemma header reports.""" + return ModelShape( + architecture=architecture, + block_count=48, + head_count_kv=8, + key_length=128, + value_length=128, + context_length=131072, + n_vocab=262144, + sliding_window=1024, + ) + + +def test_a_known_pattern_prices_only_the_global_layers_at_full_width() -> None: + """Gemma 3 is 5 local to 1 global, so 8 of 48 layers hold the whole window. + + Worked by hand: per layer per token is 8 heads * 256 = 2048 elements at + 2 bytes = 4096 bytes. Eight global layers * 16384 tokens = 512 MiB. Forty + local layers * 1024 tokens = 160 MiB. Total 672 MiB, against 3072 MiB if + every layer were priced at full width. That ratio is 4.57x, which is the + Gemma over-estimate the spec quotes at 4.6x from a separate measurement. + """ + cost = kv_cache_bytes(gemma_like("gemma3"), 16384, KvPrecision.F16) + + assert cost == 672 * MIB + + +def test_an_unknown_architecture_declining_to_guess_rounds_up() -> None: + """A window with no known pattern is priced as though every layer were global. + + Over-estimating costs a pessimistic badge on one screen. Under-estimating + ships a confident badge about a model that spills, which is the failure this + whole phase exists to delete. + """ + cost = kv_cache_bytes(gemma_like("something-nobody-has-seen"), 16384, KvPrecision.F16) + + assert cost == 3072 * MIB From d25b3aa6c6c1b992320ff7bc4de8ce1faa65ffc7 Mon Sep 17 00:00:00 2001 From: Anish Sarkar <104695310+AnishSarkar22@users.noreply.github.com> Date: Sun, 20 Sep 2026 20:07:42 +0530 Subject: [PATCH 16/60] feat(llm/providers): accept a transport and json_schema in the OpenAI-compatible provider Co-Authored-By: Claude Sonnet 5 --- .../llm/providers/openai_compatible/chat.py | 34 +++++++++++++++++-- .../modules/llm/providers/protocols.py | 2 +- 2 files changed, 32 insertions(+), 4 deletions(-) diff --git a/surfsense_local/backend/modules/llm/providers/openai_compatible/chat.py b/surfsense_local/backend/modules/llm/providers/openai_compatible/chat.py index 187b32f111..fcd29a8d0b 100644 --- a/surfsense_local/backend/modules/llm/providers/openai_compatible/chat.py +++ b/surfsense_local/backend/modules/llm/providers/openai_compatible/chat.py @@ -14,13 +14,29 @@ class OpenAICompatibleChatProvider: name = "openai_compatible" - def __init__(self, base_url: str, api_key: str | None = None) -> None: + def __init__( + self, + base_url: str, + api_key: str | None = None, + *, + transport: httpx.BaseTransport | None = None, + thinking_off: dict[str, object] | None = None, + ) -> None: self._base_url = base_url.rstrip("/") self._api_key = api_key + # Local runtimes compose this provider and need a seam to test against. + self._transport = transport + # What `reasoning=False` means on this endpoint, supplied by whoever + # knows what it is. There is no portable OpenAI shape for it, and a + # strict endpoint rejects a field it does not recognise, so a caller + # that cannot name one gets today's behaviour: the flag is ignored. + self._thinking_off = thinking_off def _client(self) -> httpx.AsyncClient: headers = {"Authorization": f"Bearer {self._api_key}"} if self._api_key else {} - return httpx.AsyncClient(timeout=TIMEOUT, headers=headers) + return httpx.AsyncClient( + timeout=TIMEOUT, headers=headers, transport=self._transport + ) async def health(self) -> bool: try: @@ -61,6 +77,7 @@ async def chat( max_tokens: int | None = None, temperature: float | None = None, reasoning: bool | None = None, + json_schema: dict | None = None, ) -> AsyncIterator[str]: body: dict[str, object] = { "model": model, @@ -74,7 +91,18 @@ async def chat( body["max_tokens"] = max_tokens if temperature is not None: body["temperature"] = temperature - # `reasoning` is intentionally ignored: it has no portable OpenAI shape. + if json_schema is not None: + # Masks every token that would produce invalid JSON, so malformed + # output stops being something to repair afterwards. + body["response_format"] = { + "type": "json_schema", + "json_schema": {"name": "response", "schema": json_schema}, + } + if reasoning is False and self._thinking_off is not None: + # A caller asking for no reasoning is usually also capping tokens, + # and a thinking model spends that cap before its first answer + # token, so this is what keeps a short request from returning "". + body.update(self._thinking_off) async with ( self._client() as client, client.stream( diff --git a/surfsense_local/backend/modules/llm/providers/protocols.py b/surfsense_local/backend/modules/llm/providers/protocols.py index f355809b45..343afcd618 100644 --- a/surfsense_local/backend/modules/llm/providers/protocols.py +++ b/surfsense_local/backend/modules/llm/providers/protocols.py @@ -6,7 +6,7 @@ class Generator(Protocol): - """Anything that can answer. Ollama today, other backends later.""" + """Anything that can answer. llama.cpp locally, or a remote endpoint.""" name: str From 0a86d4db11a68ff8c2871ac887aeef78ebafdb50 Mon Sep 17 00:00:00 2001 From: Anish Sarkar <104695310+AnishSarkar22@users.noreply.github.com> Date: Sun, 20 Sep 2026 20:07:43 +0530 Subject: [PATCH 17/60] feat(llm/providers): add the llamacpp provider with presets, downloads, and thinking control Co-Authored-By: Claude Sonnet 5 --- .../backend/modules/llm/providers/__init__.py | 14 +- .../llm/providers/llamacpp/__init__.py | 31 ++++ .../llm/providers/llamacpp/capabilities.py | 77 +++++++++ .../llm/providers/llamacpp/download.py | 76 +++++++++ .../llm/providers/llamacpp/messages.py | 35 ++++ .../modules/llm/providers/llamacpp/preset.py | 69 ++++++++ .../llm/providers/llamacpp/provider.py | 150 +++++++++++++++++ .../llm/providers/llamacpp/router_client.py | 115 +++++++++++++ .../llm/providers/llamacpp/thinking.py | 34 ++++ .../tests/unit/llm/providers/__init__.py | 0 .../unit/llm/providers/llamacpp/__init__.py | 0 .../llm/providers/llamacpp/fake_router.py | 125 ++++++++++++++ .../providers/llamacpp/test_capabilities.py | 81 +++++++++ .../llamacpp/test_constrained_output.py | 91 ++++++++++ .../llm/providers/llamacpp/test_download.py | 135 +++++++++++++++ .../llm/providers/llamacpp/test_preset.py | 85 ++++++++++ .../llm/providers/llamacpp/test_provider.py | 157 ++++++++++++++++++ .../providers/llamacpp/test_router_client.py | 88 ++++++++++ .../llm/providers/llamacpp/test_thinking.py | 72 ++++++++ .../tests/unit/llm/test_system_role.py | 75 +++++++++ 20 files changed, 1503 insertions(+), 7 deletions(-) create mode 100644 surfsense_local/backend/modules/llm/providers/llamacpp/__init__.py create mode 100644 surfsense_local/backend/modules/llm/providers/llamacpp/capabilities.py create mode 100644 surfsense_local/backend/modules/llm/providers/llamacpp/download.py create mode 100644 surfsense_local/backend/modules/llm/providers/llamacpp/messages.py create mode 100644 surfsense_local/backend/modules/llm/providers/llamacpp/preset.py create mode 100644 surfsense_local/backend/modules/llm/providers/llamacpp/provider.py create mode 100644 surfsense_local/backend/modules/llm/providers/llamacpp/router_client.py create mode 100644 surfsense_local/backend/modules/llm/providers/llamacpp/thinking.py create mode 100644 surfsense_local/backend/tests/unit/llm/providers/__init__.py create mode 100644 surfsense_local/backend/tests/unit/llm/providers/llamacpp/__init__.py create mode 100644 surfsense_local/backend/tests/unit/llm/providers/llamacpp/fake_router.py create mode 100644 surfsense_local/backend/tests/unit/llm/providers/llamacpp/test_capabilities.py create mode 100644 surfsense_local/backend/tests/unit/llm/providers/llamacpp/test_constrained_output.py create mode 100644 surfsense_local/backend/tests/unit/llm/providers/llamacpp/test_download.py create mode 100644 surfsense_local/backend/tests/unit/llm/providers/llamacpp/test_preset.py create mode 100644 surfsense_local/backend/tests/unit/llm/providers/llamacpp/test_provider.py create mode 100644 surfsense_local/backend/tests/unit/llm/providers/llamacpp/test_router_client.py create mode 100644 surfsense_local/backend/tests/unit/llm/providers/llamacpp/test_thinking.py create mode 100644 surfsense_local/backend/tests/unit/llm/test_system_role.py diff --git a/surfsense_local/backend/modules/llm/providers/__init__.py b/surfsense_local/backend/modules/llm/providers/__init__.py index 35301a2b33..eb63fd0624 100644 --- a/surfsense_local/backend/modules/llm/providers/__init__.py +++ b/surfsense_local/backend/modules/llm/providers/__init__.py @@ -1,21 +1,21 @@ from collections.abc import Callable -from modules.llm.providers.ollama.provider import OllamaProvider +from modules.llm.providers.llamacpp import PROVIDER, LlamaCppProvider from modules.llm.providers.protocols import Generator from shared.config import get_llm_settings -# Name to provider. Adding one is a new folder and one line here, never a change -# to a consumer: the router resolves everything through get_provider(). REGISTRY: dict[str, Callable[[], Generator]] = { - "ollama": lambda: OllamaProvider(get_llm_settings().ollama_base_url), + PROVIDER: lambda: LlamaCppProvider( + get_llm_settings().llamacpp_base_url, + get_llm_settings().llamacpp_models_dir, + ), } def provider_names() -> list[str]: - return list(REGISTRY) + return sorted(REGISTRY) def get_provider(name: str) -> Generator | None: - """Build one local runtime provider by name.""" factory = REGISTRY.get(name) - return factory() if factory is not None else None + return factory() if factory else None diff --git a/surfsense_local/backend/modules/llm/providers/llamacpp/__init__.py b/surfsense_local/backend/modules/llm/providers/llamacpp/__init__.py new file mode 100644 index 0000000000..10982cd37e --- /dev/null +++ b/surfsense_local/backend/modules/llm/providers/llamacpp/__init__.py @@ -0,0 +1,31 @@ +from modules.llm.providers.llamacpp.capabilities import ( + Capabilities, + Modality, + read_capabilities, +) +from modules.llm.providers.llamacpp.download import download_gguf +from modules.llm.providers.llamacpp.messages import for_template +from modules.llm.providers.llamacpp.preset import ( + PRESET_FILE, + ModelPreset, + render_presets, + write_presets, +) +from modules.llm.providers.llamacpp.provider import PROVIDER, LlamaCppProvider +from modules.llm.providers.llamacpp.router_client import RouterClient, RouterModel + +__all__ = [ + "PRESET_FILE", + "PROVIDER", + "Capabilities", + "LlamaCppProvider", + "Modality", + "ModelPreset", + "RouterClient", + "RouterModel", + "download_gguf", + "for_template", + "read_capabilities", + "render_presets", + "write_presets", +] diff --git a/surfsense_local/backend/modules/llm/providers/llamacpp/capabilities.py b/surfsense_local/backend/modules/llm/providers/llamacpp/capabilities.py new file mode 100644 index 0000000000..07cabfbec8 --- /dev/null +++ b/surfsense_local/backend/modules/llm/providers/llamacpp/capabilities.py @@ -0,0 +1,77 @@ +"""What a model can accept, and what its chat template can express. + +Two sources, two kinds of fact, and only one of them reaches a person: + + GET /models -> architecture.input_modalities what the model can ACCEPT + GET /props -> chat_template_caps what the TEMPLATE supports + +Template derived rather than guessed from a name. A regex over model names is +how you end up telling someone a model reads images when nothing can hand it +one. +""" + +from dataclasses import dataclass +from enum import StrEnum + + +class Modality(StrEnum): + TEXT = "text" + IMAGE = "image" + + +@dataclass(frozen=True) +class Capabilities: + """One struct, but not one list that reaches the renderer. + + `system_role` and `typed_content` change how a request is built and mean + nothing to a person; `vision` is the only thing here anyone chooses a model + for. + """ + + inputs: tuple[Modality, ...] + system_role: bool + typed_content: bool + tools: bool + context_tokens: int | None + + @property + def can_see(self) -> bool: + """Both halves, because either alone is a lie. + + A model can accept images architecturally while its template takes only + string content, which leaves no way to send it one. + """ + return Modality.IMAGE in self.inputs and self.typed_content + + @property + def user_facing(self) -> tuple[str, ...]: + """The only capability that is a badge. Everything else is a constraint.""" + return ("vision",) if self.can_see else () + + +def read_capabilities(model_id: str, models: dict, props: dict) -> Capabilities: + """Combine the two reports into one answer about one model. + + Absent fields are read generously, except where being wrong is silent: a + runtime that does not report `supports_system_role` is assumed to support + one, because dropping the system message takes the grounding and the + citation instructions with it and nothing says so. + """ + row = next( + (entry for entry in models.get("data", []) if entry.get("id") == model_id), + {}, + ) + declared = (row.get("architecture") or {}).get("input_modalities") or ["text"] + inputs = tuple( + Modality(name) for name in declared if name in set(Modality) + ) or (Modality.TEXT,) + + template = props.get("chat_template_caps") or {} + settings = props.get("default_generation_settings") or {} + return Capabilities( + inputs=inputs, + system_role=bool(template.get("supports_system_role", True)), + typed_content=bool(template.get("supports_typed_content", False)), + tools=bool(template.get("supports_tools", False)), + context_tokens=settings.get("n_ctx"), + ) diff --git a/surfsense_local/backend/modules/llm/providers/llamacpp/download.py b/surfsense_local/backend/modules/llm/providers/llamacpp/download.py new file mode 100644 index 0000000000..798806fb2c --- /dev/null +++ b/surfsense_local/backend/modules/llm/providers/llamacpp/download.py @@ -0,0 +1,76 @@ +"""Fetch a GGUF into the models directory. + +SurfSense downloads rather than delegating to `POST /models`, for one reason +that decides it and two that pay for it. llama-server is a second process we do +not proxy, so an in-process fetch is **the only place `egress.require()` can +hold**. It also buys resume, which matters because a model is gigabytes, and a +checksum, which matters because a half written file is indistinguishable from a +whole one to a runtime that discovers a directory. +""" + +import hashlib +from collections.abc import AsyncIterator +from pathlib import Path + +import httpx + +from modules.llm.providers.types import DownloadProgress + +TIMEOUT = httpx.Timeout(600.0, connect=10.0) +_CHUNK = 1024 * 1024 + + +async def download_gguf( + url: str, + destination: Path, + *, + sha256: str | None = None, + transport: httpx.BaseTransport | None = None, +) -> AsyncIterator[DownloadProgress]: + """Stream a model to `destination`, yielding progress as it lands. + + Writes to a sibling `.part` file and renames only once the whole file is + present and verified, so the router never discovers a partial model. A + cancelled download keeps the `.part` for a later resume. + """ + partial = destination.with_suffix(destination.suffix + ".part") + destination.parent.mkdir(parents=True, exist_ok=True) + already = partial.stat().st_size if partial.exists() else 0 + + headers = {"Range": f"bytes={already}-"} if already else {} + async with httpx.AsyncClient( + timeout=TIMEOUT, transport=transport, follow_redirects=True + ) as client, client.stream("GET", url, headers=headers) as reply: + reply.raise_for_status() + resumed = reply.status_code == 206 + total = already + int(reply.headers.get("content-length", 0)) + + # The server ignored our Range, so the bytes we have are worthless. + mode = "ab" if resumed and already else "wb" + done = already if resumed else 0 + + with partial.open(mode) as handle: + async for chunk in reply.aiter_bytes(_CHUNK): + handle.write(chunk) + done += len(chunk) + yield DownloadProgress("downloading", completed=done, total=total) + + if sha256 is not None: + actual = _digest(partial) + if actual != sha256: + partial.unlink(missing_ok=True) + raise ValueError( + f"checksum mismatch for {destination.name}: " + f"expected {sha256}, got {actual}" + ) + + partial.replace(destination) + yield DownloadProgress("complete", completed=done, total=total) + + +def _digest(path: Path) -> str: + digest = hashlib.sha256() + with path.open("rb") as handle: + while chunk := handle.read(_CHUNK): + digest.update(chunk) + return digest.hexdigest() diff --git a/surfsense_local/backend/modules/llm/providers/llamacpp/messages.py b/surfsense_local/backend/modules/llm/providers/llamacpp/messages.py new file mode 100644 index 0000000000..0b313cb922 --- /dev/null +++ b/surfsense_local/backend/modules/llm/providers/llamacpp/messages.py @@ -0,0 +1,35 @@ +"""Shaping a conversation for the template that will render it. + +The adapter downgrades at this seam so `modules/chat` never learns that +templates differ. It assembles one conversation; what a particular model can be +told is a property of the model, and belongs here. +""" + +from collections.abc import Sequence + +from modules.llm.providers.llamacpp.capabilities import Capabilities +from modules.llm.providers.types import Message + + +def for_template( + messages: Sequence[Message], capabilities: Capabilities +) -> list[Message]: + """The same conversation, expressed in what this template can carry. + + A template with no system role gets the instructions folded into the first + user turn rather than losing them. Dropping them is the silent failure: the + model answers without the sources or the citation format and sounds exactly + as confident as it would have otherwise. + """ + if capabilities.system_role: + return list(messages) + + system = [m for m in messages if m.role == "system"] + rest = [m for m in messages if m.role != "system"] + if not system or not rest: + return list(messages) + + preamble = "\n\n".join(m.content for m in system) + first, *later = rest + folded = Message(first.role, f"{preamble}\n\n{first.content}") + return [folded, *later] diff --git a/surfsense_local/backend/modules/llm/providers/llamacpp/preset.py b/surfsense_local/backend/modules/llm/providers/llamacpp/preset.py new file mode 100644 index 0000000000..3054a60692 --- /dev/null +++ b/surfsense_local/backend/modules/llm/providers/llamacpp/preset.py @@ -0,0 +1,69 @@ +"""Per-model launch arguments, written where the router will read them. + +`POST /models/load` takes an `args` field that does nothing. Measured at b11050 +on macOS: `{"args": ["-c","4096"]}`, `{"args": ["--ctx-size","4096","-fa","on"]}` +and `{"preset": "..."}` all return `200 {"success":true}` while the spawned +worker's argv stays byte identical. `--models-preset ` is the mechanism +that works, so this module is the seam between the fit calculation and the +runtime. + +The file is read once, at router startup. Adding a section while the router runs +does not surface the model, so installing one means rewriting this and +restarting the sidecar. +""" + +from dataclasses import dataclass +from pathlib import Path + +from modules.llm.fit import KvPrecision + +# The name Electron watches for changes and passes as `--models-preset`. Both +# sides have to agree on it, so it is stated once here. +PRESET_FILE = "models.ini" + + +@dataclass(frozen=True) +class ModelPreset: + """What the fit calculation decided for one model.""" + + model_id: str + path: str + n_ctx: int + precision: KvPrecision + + +def render_presets(presets: list[ModelPreset]) -> str: + """The INI text. One section per model, keyed by the id the router reports. + + Deliberately absent: any layer count. Setting `n_gpu_layers` by hand aborts + `--fit`, after which llama.cpp loads the model entirely on the CPU with no + error and exit 0. `--fit` owns placement; we only tell it how much to place. + """ + return "\n".join(_section(p) for p in presets) + + +def _section(preset: ModelPreset) -> str: + lines = [ + f"[{preset.model_id}]", + f"model = {preset.path}", + f"ctx-size = {preset.n_ctx}", + # One user, one question. The default of four sizes the KV cache for + # concurrency this app never uses. + "parallel = 1", + ] + if preset.precision is KvPrecision.Q8_0: + # Both halves together, or the fused flash-attention kernel is skipped + # and attention silently falls back to the CPU. + lines += [ + "cache-type-k = q8_0", + "cache-type-v = q8_0", + "flash-attn = on", + ] + return "\n".join(lines) + "\n" + + +def write_presets(path: Path, presets: list[ModelPreset]) -> None: + """Replace the preset file atomically, so a half-written INI never loads.""" + temporary = path.with_suffix(f"{path.suffix}.tmp") + temporary.write_text(render_presets(presets)) + temporary.replace(path) diff --git a/surfsense_local/backend/modules/llm/providers/llamacpp/provider.py b/surfsense_local/backend/modules/llm/providers/llamacpp/provider.py new file mode 100644 index 0000000000..38e7be94e9 --- /dev/null +++ b/surfsense_local/backend/modules/llm/providers/llamacpp/provider.py @@ -0,0 +1,150 @@ +"""The llama.cpp adapter: answers questions and holds models on disk. + +Satisfies the same `Generator` protocol the previous runtime did, so +resolution, selection and the catalog do not learn that the runtime changed. + +Chat is **composed, not reimplemented**. llama-server speaks OpenAI on +`/v1/chat/completions`, so the streaming, error handling and message shaping in +`OpenAICompatibleChatProvider` already work against it. + +Deliberately **not** a `ModelStore`. That protocol requires `pull()`, which made +sense when the runtime fetched its own weights from a name. Here SurfSense fetches the +GGUF itself, because that is the only place `egress.require()` can hold, and +because it buys resume, checksums and the header as the file lands. Downloading +is a catalog concern; this adapter answers questions and reports what is on disk. +""" + +import logging +from collections.abc import AsyncIterator +from pathlib import Path + +import httpx + +from modules.llm.providers.llamacpp.capabilities import Capabilities, read_capabilities +from modules.llm.providers.llamacpp.messages import for_template +from modules.llm.providers.llamacpp.router_client import RouterClient +from modules.llm.providers.llamacpp.thinking import THINKING_OFF +from modules.llm.providers.openai_compatible.chat import OpenAICompatibleChatProvider +from modules.llm.providers.types import Message, Model + +PROVIDER = "llamacpp" + +logger = logging.getLogger(__name__) + + +class LlamaCppProvider: + """One sidecar, any GGUF.""" + + name = PROVIDER + requires_key = False + + def __init__( + self, + base_url: str, + models_dir: Path | None = None, + *, + transport: httpx.BaseTransport | None = None, + ) -> None: + self._base_url = base_url.rstrip("/") + self._models_dir = models_dir + self._router = RouterClient(self._base_url, transport=transport) + self._chat = OpenAICompatibleChatProvider( + f"{self._base_url}/v1", transport=transport, thinking_off=THINKING_OFF + ) + + async def health(self) -> bool: + return await self._router.health() + + async def capabilities(self, model: str) -> Capabilities: + """What this model accepts and what its template can express. + + Read per model rather than cached across them: two files in the same + directory can carry entirely different templates. + """ + return read_capabilities( + model, await self._router.raw_models(), await self._router.props(model) + ) + + async def models(self) -> list[Model]: + """Everything in the models directory, resident or not. + + The router auto-discovers the directory, so installed state is a fact + about disk rather than something we track separately. + """ + return [ + Model(model.id, installed=True, capabilities=("completion",)) + for model in await self._router.models() + ] + + async def delete(self, name: str) -> None: + """Remove the weights from disk. + + **Not** `DELETE /models`. The router only removes what it downloaded + into its own cache and refuses everything else: measured, `model + name=... is not removable (not from cache)`, a 500, with the file left + on disk. Everything SurfSense installs lands in `--models-dir`, so that + call can never succeed for us. + + This is the same argument as install. We own the models directory; the + runtime reads it. The router forgets the model on its next restart, + which the preset rewrite triggers. + """ + if self._models_dir is None: + raise RuntimeError("no models directory configured") + path = self._models_dir / f"{name}.gguf" + if not path.exists(): + raise FileNotFoundError(name) + path.unlink() + + async def chat( + self, + model: str, + messages: list[Message], + *, + max_tokens: int | None = None, + temperature: float | None = None, + reasoning: bool | None = None, + json_schema: dict | None = None, + ) -> AsyncIterator[str]: + await self._ensure_loaded(model) + # Downgrade at the seam: `modules/chat` assembles one conversation and + # never learns that templates differ. + shaped = for_template(messages, await self.capabilities(model)) + try: + async for chunk in self._chat.chat( + model, + shaped, + max_tokens=max_tokens, + temperature=temperature, + reasoning=reasoning, + json_schema=json_schema, + ): + yield chunk + except httpx.HTTPStatusError as error: + # llama.cpp issue #29006: json_schema on the chat endpoint returns + # 400 for some templates, though the model itself is fine. Losing a + # whole Studio format to a template quirk is worse than falling back + # to an unconstrained answer, which the parser can still repair. + if json_schema is None or error.response.status_code != 400: + raise + logger.warning( + "%s rejected a json_schema request; retrying unconstrained", model + ) + async for chunk in self._chat.chat( + model, + shaped, + max_tokens=max_tokens, + temperature=temperature, + reasoning=reasoning, + ): + yield chunk + + async def _ensure_loaded(self, model: str) -> None: + """Models are discovered `unloaded`, so the first turn has to ask. + + Reloading a resident model would evict and re-read gigabytes between two + turns, so this checks before it acts. + """ + resident = {m.id for m in await self._router.models() if m.loaded} + if model not in resident: + await self._router.load(model) diff --git a/surfsense_local/backend/modules/llm/providers/llamacpp/router_client.py b/surfsense_local/backend/modules/llm/providers/llamacpp/router_client.py new file mode 100644 index 0000000000..a358e6f3e2 --- /dev/null +++ b/surfsense_local/backend/modules/llm/providers/llamacpp/router_client.py @@ -0,0 +1,115 @@ +"""The HTTP surface llama-server exposes in router mode. + +One sidecar process holds the models directory and spawns a worker per loaded +model. The router itself costs no device memory until something loads, and it +runs happily against an empty directory, so the lifecycle is testable before any +model exists. +""" + +from dataclasses import dataclass + +import httpx + +# A load can take minutes on a large model; a health check is instant. +TIMEOUT = httpx.Timeout(600.0, connect=5.0) + + +@dataclass(frozen=True) +class RouterModel: + """One model the router discovered, and whether it is resident.""" + + id: str + loaded: bool + args: tuple[str, ...] = () + + +class RouterClient: + """Calls the router makes available. Chat is not here: it speaks OpenAI on + `/v1/chat/completions`, so `OpenAICompatibleChatProvider` serves it unchanged + rather than being reimplemented.""" + + def __init__( + self, base_url: str, *, transport: httpx.BaseTransport | None = None + ) -> None: + self._base_url = base_url.rstrip("/") + self._transport = transport + + def _client(self) -> httpx.AsyncClient: + return httpx.AsyncClient( + base_url=self._base_url, timeout=TIMEOUT, transport=self._transport + ) + + async def health(self) -> bool: + try: + async with self._client() as client: + return (await client.get("/health")).status_code == 200 + except httpx.HTTPError: + return False + + async def is_router(self) -> bool: + """Router mode, not single-model mode. + + Both answer /health, so this is the check that the sidecar came up the + way we asked it to. + """ + try: + async with self._client() as client: + reply = await client.get("/props") + return reply.status_code == 200 and reply.json().get("role") == "router" + except (httpx.HTTPError, ValueError): + return False + + async def models(self) -> list[RouterModel]: + async with self._client() as client: + reply = await client.get("/models") + reply.raise_for_status() + rows = reply.json().get("data", []) + return [ + RouterModel( + id=row["id"], + loaded=(row.get("status") or {}).get("value") == "loaded", + args=tuple((row.get("status") or {}).get("args") or ()), + ) + for row in rows + ] + + async def raw_models(self) -> dict: + """The `/models` payload as sent, for fields `RouterModel` does not carry.""" + async with self._client() as client: + reply = await client.get("/models") + reply.raise_for_status() + return reply.json() + + async def props(self, model_id: str) -> dict: + """One model's template capabilities and loaded settings.""" + try: + async with self._client() as client: + reply = await client.get("/props", params={"model": model_id}) + reply.raise_for_status() + return reply.json() + except (httpx.HTTPError, ValueError): + # An older build, or a model that is not resident. Capabilities read + # generously from an empty payload rather than failing the turn. + return {} + + async def load(self, model_id: str) -> None: + """Bring a model into memory. + + Deliberately sends no arguments. The router accepts an `args` field and + ignores it, measured; per-model flags come from the preset INI instead. + """ + async with self._client() as client: + reply = await client.post("/models/load", json={"model": model_id}) + reply.raise_for_status() + + async def unload(self, model_id: str) -> None: + async with self._client() as client: + reply = await client.post("/models/unload", json={"model": model_id}) + reply.raise_for_status() + + async def delete(self, model_id: str) -> None: + async with self._client() as client: + reply = await client.request( + "DELETE", "/models", params={"model": model_id} + ) + reply.raise_for_status() diff --git a/surfsense_local/backend/modules/llm/providers/llamacpp/thinking.py b/surfsense_local/backend/modules/llm/providers/llamacpp/thinking.py new file mode 100644 index 0000000000..63f63c627e --- /dev/null +++ b/surfsense_local/backend/modules/llm/providers/llamacpp/thinking.py @@ -0,0 +1,34 @@ +"""How this runtime is told not to think. + +A thinking model emits its whole trace before its first answer token, so a call +with a small `max_tokens` returns nothing. Measured against Qwen3 1.7B at +b11050: a 12 token title request came back `content: ''`, `finish_reason: +length`, and a full `reasoning_content`. + +Two mechanisms are sent, because each covers the other's blind spot and both +were measured to work: + +- `chat_template_kwargs` is the documented one, listed in the server README as + a chat completions parameter with this exact example. It is inert on a model + whose Jinja template never reads `enable_thinking`, which is most of them + outside the Qwen line. +- `thinking_budget_tokens` is llama.cpp's own end of thinking injection, so it + holds whatever the template does. It is confirmed upstream but absent from + the README, which is why it is not relied on alone. + +Rejected, both measured: + +- `reasoning_budget` as a request field is accepted and ignored. That is the + failure mode worth naming, because it looks like it worked. +- `--reasoning-budget 0` on the command line applies to the whole router, so it + would buy a title by removing reasoning from every answer. Worse, setting it + at all makes the server ignore `thinking_budget_tokens`, whose handler runs + only while the flag is at its `-1` default. The sidecar must never pass it, + and a test in the Electron tree holds that. +""" + +# Per call, so the answer path keeps the reasoning it actually benefits from. +THINKING_OFF: dict[str, object] = { + "thinking_budget_tokens": 0, + "chat_template_kwargs": {"enable_thinking": False}, +} diff --git a/surfsense_local/backend/tests/unit/llm/providers/__init__.py b/surfsense_local/backend/tests/unit/llm/providers/__init__.py new file mode 100644 index 0000000000..e69de29bb2 diff --git a/surfsense_local/backend/tests/unit/llm/providers/llamacpp/__init__.py b/surfsense_local/backend/tests/unit/llm/providers/llamacpp/__init__.py new file mode 100644 index 0000000000..e69de29bb2 diff --git a/surfsense_local/backend/tests/unit/llm/providers/llamacpp/fake_router.py b/surfsense_local/backend/tests/unit/llm/providers/llamacpp/fake_router.py new file mode 100644 index 0000000000..2ece1c347b --- /dev/null +++ b/surfsense_local/backend/tests/unit/llm/providers/llamacpp/fake_router.py @@ -0,0 +1,125 @@ +"""A stand-in for llama-server in router mode. + +Shapes come from the real thing at b11050: `/models` reports each discovered +model with a `status` carrying the worker's argv, which is the only reliable way +to see what the router actually did with a load request. +""" + +import json + +import httpx + + +class FakeRouter: + """Records what was asked of it, so tests can assert on the calls.""" + + def __init__(self, models: list[str] | None = None) -> None: + self.models = models or [] + self.loaded: set[str] = set() + self.deleted: list[str] = [] + self.chat_bodies: list[dict] = [] + # What `/props` reports about the chat template. Defaults to a template + # that carries everything, which is the common case. + self.template_caps: dict = {"supports_system_role": True} + # llama.cpp issue #29006: some templates 400 on a json_schema request. + self.reject_response_format = False + self.fail_chat_with: int | None = None + # A thinking model. Measured against Qwen3 1.7B at b11050 with + # `--reasoning-format deepseek`: every token of the trace arrives as + # `reasoning_content`, and a short `max_tokens` is spent before a + # single `content` token exists. + self.thinks = False + + def transport(self) -> httpx.MockTransport: + return httpx.MockTransport(self._handle) + + def _handle(self, request: httpx.Request) -> httpx.Response: + path = request.url.path + if path == "/health": + return httpx.Response(200, json={"status": "ok"}) + if path == "/props": + return httpx.Response( + 200, + json={ + "role": "router", + "chat_template_caps": self.template_caps, + "default_generation_settings": {"n_ctx": 16384}, + }, + ) + if path == "/models" and request.method == "GET": + return httpx.Response(200, json={"object": "list", "data": [ + { + "id": name, + "status": { + "value": "loaded" if name in self.loaded else "unloaded", + "args": ["llama-server", "--alias", name], + }, + } + for name in self.models + ]}) + if path == "/models" and request.method == "DELETE": + # The real router refuses anything it did not download into its own + # cache. Measured: `model name=... is not removable (not from + # cache)`, 500. Everything SurfSense installs lands in --models-dir, + # so this always refuses, and a fake that said otherwise is what let + # a broken delete path ship. + self.deleted.append(request.url.params.get("model", "")) + return httpx.Response( + 500, + json={ + "error": { + "code": 500, + "message": "model name=x is not removable (not from cache)", + "type": "server_error", + } + }, + ) + if path == "/models/load": + self.loaded.add(json.loads(request.content)["model"]) + return httpx.Response(200, json={"success": True}) + if path == "/models/unload": + self.loaded.discard(json.loads(request.content)["model"]) + return httpx.Response(200, json={"success": True}) + if path == "/v1/chat/completions": + body = json.loads(request.content) + self.chat_bodies.append(body) + if self.fail_chat_with is not None: + return httpx.Response(self.fail_chat_with, json={"error": "nope"}) + if self.reject_response_format and "response_format" in body: + return httpx.Response( + 400, json={"error": {"message": "unsupported response_format"}} + ) + chunks = _stream(self.thinks and not _thinking_off(body)) + return httpx.Response( + 200, + text="\n\n".join(chunks) + "\n\n", + headers={"content-type": "text/event-stream"}, + ) + return httpx.Response(404, json={"error": path}) + + +def _thinking_off(body: dict) -> bool: + """Either mechanism alone stops the trace, which is why both are sent. + + The real server reads them independently: one is a chat template variable, + the other is its own end of thinking injection. + """ + kwargs = body.get("chat_template_kwargs") or {} + return body.get("thinking_budget_tokens") == 0 or kwargs.get( + "enable_thinking" + ) is False + + +def _stream(thinking: bool) -> list[str]: + """What a thinking model emits, against what one answering plainly does.""" + if thinking: + return [ + 'data: {"choices":[{"delta":{"reasoning_content":"Okay, the user"}}]}', + 'data: {"choices":[{"finish_reason":"length","delta":{}}]}', + "data: [DONE]", + ] + return [ + 'data: {"choices":[{"delta":{"content":"Hel"}}]}', + 'data: {"choices":[{"delta":{"content":"lo"}}]}', + "data: [DONE]", + ] diff --git a/surfsense_local/backend/tests/unit/llm/providers/llamacpp/test_capabilities.py b/surfsense_local/backend/tests/unit/llm/providers/llamacpp/test_capabilities.py new file mode 100644 index 0000000000..4c1c8db75d --- /dev/null +++ b/surfsense_local/backend/tests/unit/llm/providers/llamacpp/test_capabilities.py @@ -0,0 +1,81 @@ +"""What a model can accept, and what its chat template can express. + +Two different questions from two different endpoints, and only one of them is a +capability a person should ever see. +""" + +import pytest + +from modules.llm.providers.llamacpp import Capabilities, Modality, read_capabilities + +pytestmark = pytest.mark.unit + + +def caps(**overrides) -> Capabilities: + """A text only model whose template supports everything.""" + models = { + "data": [ + { + "id": "m", + "architecture": {"input_modalities": overrides.pop("inputs", ["text"])}, + } + ] + } + props = { + "chat_template_caps": { + "supports_system_role": overrides.pop("system_role", True), + "supports_typed_content": overrides.pop("typed_content", True), + "supports_tools": overrides.pop("tools", False), + }, + "default_generation_settings": {"n_ctx": overrides.pop("n_ctx", 16384)}, + } + return read_capabilities("m", models, props) + + +def test_a_text_model_reports_no_vision() -> None: + """A model declaring only text input cannot be sent a picture.""" + assert caps().inputs == (Modality.TEXT,) + assert not caps().can_see + + +def test_vision_needs_the_model_and_the_template_to_agree() -> None: + """A model can accept images architecturally while its template takes only + string content, leaving no way to hand it one. Checking `input_modalities` + alone puts a vision badge on a model that cannot be sent a picture. + """ + both = caps(inputs=["text", "image"], typed_content=True) + architecture_only = caps(inputs=["text", "image"], typed_content=False) + + assert both.can_see + assert not architecture_only.can_see + + +def test_a_template_without_a_system_role_is_recorded() -> None: + """A live bug this fixes: `build_messages()` always emits a system message + and it is silently dropped on templates that cannot carry one, taking the + grounding and the citation instructions with it. + """ + assert not caps(system_role=False).system_role + + +def test_the_loaded_window_is_reported_so_the_history_budget_can_respect_it() -> None: + """Context is fixed at load, so this is the real ceiling for a turn.""" + assert caps(n_ctx=8192).context_tokens == 8192 + + +def test_a_runtime_that_says_nothing_is_assumed_capable_of_the_basics() -> None: + """An older build reporting no template caps must not lose the system + message, which is the one field whose absence silently degrades an answer. + """ + permissive = read_capabilities("m", {"data": [{"id": "m"}]}, {}) + + assert permissive.system_role + assert permissive.inputs == (Modality.TEXT,) + assert not permissive.can_see + + +def test_only_the_user_facing_capability_is_named_as_one() -> None: + """`system_role` and `typed_content` are constraints on how a request is + built. Neither is meaningful to a person, so neither is a badge.""" + assert caps(inputs=["text", "image"]).user_facing == ("vision",) + assert caps().user_facing == () diff --git a/surfsense_local/backend/tests/unit/llm/providers/llamacpp/test_constrained_output.py b/surfsense_local/backend/tests/unit/llm/providers/llamacpp/test_constrained_output.py new file mode 100644 index 0000000000..094a8d7c33 --- /dev/null +++ b/surfsense_local/backend/tests/unit/llm/providers/llamacpp/test_constrained_output.py @@ -0,0 +1,91 @@ +"""Asking for JSON and getting JSON. + +48 of the 69 prompt files demand a JSON object. `response_format` masks every +token that would produce invalid JSON, so a malformed answer stops being +something to repair and becomes something that cannot be emitted. +""" + +import httpx +import pytest + +from modules.llm.providers.llamacpp import LlamaCppProvider +from modules.llm.providers.types import Message +from tests.unit.llm.providers.llamacpp.fake_router import FakeRouter + +pytestmark = pytest.mark.unit + +SCHEMA = { + "type": "object", + "properties": {"title": {"type": "string"}}, + "required": ["title"], +} + + +def provider_for(fake: FakeRouter) -> LlamaCppProvider: + """A provider wired to a fake router rather than a live sidecar.""" + return LlamaCppProvider("http://127.0.0.1:1234", transport=fake.transport()) + + +async def drain(stream) -> str: + """Collect a streamed reply into one string.""" + return "".join([chunk async for chunk in stream]) + + +@pytest.mark.asyncio +async def test_a_plain_turn_asks_for_no_particular_shape() -> None: + """Chat is prose, and constraining it would be a mistake.""" + fake = FakeRouter(["m"]) + fake.loaded.add("m") + + await drain(provider_for(fake).chat("m", [Message("user", "hi")])) + + assert "response_format" not in fake.chat_bodies[-1] + + +@pytest.mark.asyncio +async def test_a_schema_is_sent_as_a_response_format() -> None: + """The schema reaches the runtime in the shape the OpenAI API defines.""" + fake = FakeRouter(["m"]) + fake.loaded.add("m") + + await drain( + provider_for(fake).chat("m", [Message("user", "hi")], json_schema=SCHEMA) + ) + + body = fake.chat_bodies[-1] + assert body["response_format"]["type"] == "json_schema" + assert body["response_format"]["json_schema"]["schema"] == SCHEMA + + +@pytest.mark.asyncio +async def test_a_template_that_refuses_a_schema_still_answers() -> None: + """llama.cpp issue #29006: json_schema on the chat endpoint returns 400 for + some templates. Studio would lose a whole format over a template quirk, so + the request is retried unconstrained and the parser's repair path catches + what the mask would have prevented. + """ + fake = FakeRouter(["m"]) + fake.loaded.add("m") + fake.reject_response_format = True + + text = await drain( + provider_for(fake).chat("m", [Message("user", "hi")], json_schema=SCHEMA) + ) + + assert text == "Hello" + assert "response_format" not in fake.chat_bodies[-1] + assert len(fake.chat_bodies) == 2 + + +@pytest.mark.asyncio +async def test_an_unrelated_failure_is_not_swallowed_by_the_fallback() -> None: + """Only a schema rejection is worth retrying. Retrying everything would turn + a broken runtime into a silently worse answer.""" + fake = FakeRouter(["m"]) + fake.loaded.add("m") + fake.fail_chat_with = 503 + + with pytest.raises(httpx.HTTPStatusError): + await drain( + provider_for(fake).chat("m", [Message("user", "hi")], json_schema=SCHEMA) + ) diff --git a/surfsense_local/backend/tests/unit/llm/providers/llamacpp/test_download.py b/surfsense_local/backend/tests/unit/llm/providers/llamacpp/test_download.py new file mode 100644 index 0000000000..2ef3aeb95c --- /dev/null +++ b/surfsense_local/backend/tests/unit/llm/providers/llamacpp/test_download.py @@ -0,0 +1,135 @@ +"""Fetching a GGUF into the models directory. + +SurfSense does this itself rather than asking the runtime to. llama-server is a +second process we do not proxy, so an in-process fetch is the only place +`egress.require()` can hold, and it also buys resume, checksums and the header +as the file lands. +""" + +import hashlib + +import httpx +import pytest + +from modules.llm.providers.llamacpp import download_gguf + +pytestmark = pytest.mark.unit + +BODY = b"GGUF" + b"x" * 4092 +DIGEST = hashlib.sha256(BODY).hexdigest() + + +def serving(body: bytes, *, accept_ranges: bool = True, record: list | None = None): + """A Hugging Face style file endpoint that honours Range when asked.""" + + def handler(request: httpx.Request) -> httpx.Response: + if record is not None: + record.append(request.headers.get("Range")) + span = request.headers.get("Range") + headers = {"content-type": "application/octet-stream"} + if accept_ranges: + headers["accept-ranges"] = "bytes" + if span and accept_ranges: + start = int(span.removeprefix("bytes=").split("-")[0]) + return httpx.Response(206, content=body[start:], headers=headers) + return httpx.Response(200, content=body, headers=headers) + + return httpx.MockTransport(handler) + + +@pytest.mark.asyncio +async def test_a_download_lands_under_its_real_filename(tmp_path) -> None: + """Real names, not llama.cpp's content addressed cache layout, because the + router discovers the directory and reports what it finds.""" + steps = [ + s + async for s in download_gguf( + "https://hf.invalid/repo/resolve/main/m.gguf", + tmp_path / "m.gguf", + transport=serving(BODY), + ) + ] + + assert (tmp_path / "m.gguf").read_bytes() == BODY + assert steps[-1].completed == len(BODY) + + +@pytest.mark.asyncio +async def test_progress_is_reported_as_it_arrives(tmp_path) -> None: + """The install route streams these straight through its NDJSON frames.""" + steps = [ + s + async for s in download_gguf( + "https://hf.invalid/repo/resolve/main/m.gguf", + tmp_path / "m.gguf", + transport=serving(BODY), + ) + ] + + assert steps + assert all(s.total == len(BODY) for s in steps) + assert steps[-1].completed >= steps[0].completed + + +@pytest.mark.asyncio +async def test_an_interrupted_download_resumes_instead_of_restarting(tmp_path) -> None: + """A model is gigabytes. Starting over because a laptop slept is not an option.""" + partial = tmp_path / "m.gguf.part" + partial.write_bytes(BODY[:2000]) + ranges: list = [] + + async for _ in download_gguf( + "https://hf.invalid/repo/resolve/main/m.gguf", + tmp_path / "m.gguf", + transport=serving(BODY, record=ranges), + ): + pass + + assert ranges == ["bytes=2000-"] + assert (tmp_path / "m.gguf").read_bytes() == BODY + + +@pytest.mark.asyncio +async def test_a_file_that_does_not_match_its_checksum_is_not_installed(tmp_path) -> None: + """A truncated or tampered file must never be left where the router will + discover it and try to load it.""" + with pytest.raises(ValueError, match="checksum"): + async for _ in download_gguf( + "https://hf.invalid/repo/resolve/main/m.gguf", + tmp_path / "m.gguf", + sha256="0" * 64, + transport=serving(BODY), + ): + pass + + assert not (tmp_path / "m.gguf").exists() + + +@pytest.mark.asyncio +async def test_a_matching_checksum_installs(tmp_path) -> None: + """The happy path, so the guard above cannot pass by rejecting everything.""" + async for _ in download_gguf( + "https://hf.invalid/repo/resolve/main/m.gguf", + tmp_path / "m.gguf", + sha256=DIGEST, + transport=serving(BODY), + ): + pass + + assert (tmp_path / "m.gguf").exists() + + +@pytest.mark.asyncio +async def test_a_cancelled_download_leaves_nothing_selectable(tmp_path) -> None: + """Cancelling mid stream keeps the partial file for a later resume, and never + puts a half model where the router would list it as installed.""" + stream = download_gguf( + "https://hf.invalid/repo/resolve/main/m.gguf", + tmp_path / "m.gguf", + transport=serving(BODY), + ) + async for _ in stream: + break + await stream.aclose() + + assert not (tmp_path / "m.gguf").exists() diff --git a/surfsense_local/backend/tests/unit/llm/providers/llamacpp/test_preset.py b/surfsense_local/backend/tests/unit/llm/providers/llamacpp/test_preset.py new file mode 100644 index 0000000000..f09ba537b4 --- /dev/null +++ b/surfsense_local/backend/tests/unit/llm/providers/llamacpp/test_preset.py @@ -0,0 +1,85 @@ +"""The INI the router reads per-model arguments from. + +`POST /models/load` accepts an `args` field and ignores it: measured at b11050, +every payload shape returns 200 while the worker's argv stays byte identical. +`--models-preset` is the mechanism that works, so this file is where the fit +calculation reaches the runtime. +""" + +import pytest + +from modules.llm.fit import KvPrecision +from modules.llm.providers.llamacpp import ModelPreset, render_presets + +pytestmark = pytest.mark.unit + + +def a_preset(**overrides) -> ModelPreset: + """A resident 8B at the context floor unless a test says otherwise.""" + fields = { + "model_id": "Qwen3-8B-Q4_K_M", + "path": "/models/Qwen3-8B-Q4_K_M.gguf", + "n_ctx": 16384, + "precision": KvPrecision.F16, + } + return ModelPreset(**{**fields, **overrides}) + + +def test_the_window_the_fit_calculation_chose_reaches_the_worker() -> None: + """The whole point: llama.cpp would otherwise use the model's own default, + and would shrink it to 4096 on its own if we left it unset.""" + ini = render_presets([a_preset(n_ctx=24576)]) + + assert "ctx-size = 24576" in ini + + +def test_a_section_is_named_for_the_model_it_configures() -> None: + """The router matches sections by the id it reports for a discovered file.""" + ini = render_presets([a_preset()]) + + assert "[Qwen3-8B-Q4_K_M]" in ini + assert "model = /models/Qwen3-8B-Q4_K_M.gguf" in ini + + +def test_quantized_cache_sets_both_halves_and_turns_on_flash_attention() -> None: + """A mismatched pair silently falls back to an unoptimised path, and quantized + cache without a working fused kernel collapses to CPU attention with no + warning at all. Both halves, or neither.""" + ini = render_presets([a_preset(precision=KvPrecision.Q8_0)]) + + assert "cache-type-k = q8_0" in ini + assert "cache-type-v = q8_0" in ini + assert "flash-attn = on" in ini + + +def test_an_unquantized_cache_names_no_cache_type_at_all() -> None: + """f16 is llama.cpp's default, and naming it would take on the flash-attention + dependency for nothing.""" + ini = render_presets([a_preset(precision=KvPrecision.F16)]) + + assert "cache-type" not in ini + + +def test_every_model_is_pinned_to_one_slot() -> None: + """llama-server defaults to four parallel slots and sizes the KV cache for + all of them. This app has one user asking one question.""" + assert "parallel = 1" in render_presets([a_preset()]) + + +def test_several_models_each_get_their_own_section() -> None: + """One installed model must not inherit another one's window.""" + ini = render_presets( + [a_preset(), a_preset(model_id="Qwen3-4B-Q4_K_M", path="/models/4b.gguf")] + ) + + assert ini.count("ctx-size") == 2 + assert "[Qwen3-4B-Q4_K_M]" in ini + + +def test_no_explicit_layer_count_is_ever_written() -> None: + """Setting n_gpu_layers by hand aborts --fit, after which the model loads + entirely on the CPU with no error and exit 0. It looks like it worked.""" + ini = render_presets([a_preset()]) + + assert "n-gpu-layers" not in ini + assert "ngl" not in ini diff --git a/surfsense_local/backend/tests/unit/llm/providers/llamacpp/test_provider.py b/surfsense_local/backend/tests/unit/llm/providers/llamacpp/test_provider.py new file mode 100644 index 0000000000..11690d509b --- /dev/null +++ b/surfsense_local/backend/tests/unit/llm/providers/llamacpp/test_provider.py @@ -0,0 +1,157 @@ +"""The adapter the rest of the app sees. + +It satisfies the same protocol the previous runtime did, so nothing above it learns that +the runtime changed. +""" + +import pytest + +from modules.llm.providers.llamacpp import LlamaCppProvider +from modules.llm.providers.protocols import ModelStore +from modules.llm.providers.types import Message +from tests.unit.llm.providers.llamacpp.fake_router import FakeRouter + +pytestmark = pytest.mark.unit + + +def provider_for(fake: FakeRouter) -> LlamaCppProvider: + """A provider wired to a fake router rather than a live sidecar.""" + return LlamaCppProvider("http://127.0.0.1:1234", transport=fake.transport()) + + +def test_it_is_the_registry_entry_the_selection_column_will_hold() -> None: + """Revision 0012 rewrites `selected_models.provider` to exactly this.""" + assert LlamaCppProvider("http://x").name == "llamacpp" + + +def test_it_answers_and_keeps_inventory_without_being_a_model_store() -> None: + """`ModelStore` requires `pull()`, and this runtime deliberately has none. + + The previous runtime pulled its own weights, so a name was enough. Here SurfSense fetches + the GGUF in process, which is the only place `egress.require()` actually + holds, and it also buys resume, checksums and the header as the file lands. + Downloads are a catalog concern, not a runtime one, so claiming ModelStore + would advertise a capability that belongs somewhere else. + """ + provider = LlamaCppProvider("http://x") + + assert not isinstance(provider, ModelStore) + assert callable(provider.health) + assert callable(provider.models) + assert callable(provider.chat) + assert callable(provider.delete) + + +@pytest.mark.asyncio +async def test_installed_models_come_from_the_router() -> None: + """The router auto-discovers the models directory, so disk is the source of truth.""" + models = await provider_for(FakeRouter(["Qwen3-8B-Q4_K_M"])).models() + + assert [m.name for m in models] == ["Qwen3-8B-Q4_K_M"] + assert models[0].installed + + +@pytest.mark.asyncio +async def test_a_model_is_brought_into_memory_before_it_is_asked_anything() -> None: + """The router discovers models as `unloaded`, so the first turn would + otherwise go to a model that is not resident.""" + fake = FakeRouter(["m"]) + + chunks = [c async for c in provider_for(fake).chat("m", [Message("user", "hi")])] + + assert "m" in fake.loaded + assert "".join(chunks) == "Hello" + + +@pytest.mark.asyncio +async def test_an_already_resident_model_is_not_reloaded() -> None: + """Reloading would evict and re-read gigabytes between two turns.""" + fake = FakeRouter(["m"]) + fake.loaded.add("m") + calls_before = len(fake.chat_bodies) + + async for _ in provider_for(fake).chat("m", [Message("user", "hi")]): + pass + + assert len(fake.chat_bodies) == calls_before + 1 + + +@pytest.mark.asyncio +async def test_the_request_names_the_model_and_asks_for_a_stream() -> None: + """Chat is composed from the OpenAI provider, so the body must be its shape.""" + fake = FakeRouter(["m"]) + + async for _ in provider_for(fake).chat("m", [Message("user", "hi")]): + pass + + body = fake.chat_bodies[-1] + assert body["model"] == "m" + assert body["stream"] is True + + +@pytest.mark.asyncio +async def test_a_template_without_a_system_role_still_gets_the_grounding() -> None: + """End to end through the adapter: the fold happens at this seam, so nothing + above it has to know which template the chosen model uses.""" + fake = FakeRouter(["m"]) + fake.loaded.add("m") + fake.template_caps = {"supports_system_role": False} + + async for _ in provider_for(fake).chat( + "m", + [Message("system", "Cite with [n]."), Message("user", "hi")], + ): + pass + + sent = fake.chat_bodies[-1]["messages"] + assert [m["role"] for m in sent] == ["user"] + assert "Cite with [n]." in sent[0]["content"] + + +@pytest.mark.asyncio +async def test_a_normal_template_is_left_alone() -> None: + """The common case: nothing is folded and nothing moves.""" + fake = FakeRouter(["m"]) + fake.loaded.add("m") + + async for _ in provider_for(fake).chat( + "m", + [Message("system", "Cite with [n]."), Message("user", "hi")], + ): + pass + + assert [m["role"] for m in fake.chat_bodies[-1]["messages"]] == ["system", "user"] + + +@pytest.mark.asyncio +async def test_deleting_removes_the_weights_from_disk(tmp_path) -> None: + """The router cannot do this for us. + + `DELETE /models` only removes models the router downloaded into its own + cache; anything staged into `--models-dir` comes back `not removable (not + from cache)` with a 500 and the file untouched. Everything SurfSense + installs lands there, so deletion is ours, symmetrically with install. + """ + models = tmp_path / "models" + models.mkdir() + (models / "m.gguf").write_bytes(b"GGUF weights") + fake = FakeRouter(["m"]) + provider = LlamaCppProvider( + "http://127.0.0.1:1234", models, transport=fake.transport() + ) + + await provider.delete("m") + + assert not (models / "m.gguf").exists() + assert fake.deleted == [] + + +@pytest.mark.asyncio +async def test_deleting_something_absent_says_so(tmp_path) -> None: + """A stale row, or a file removed outside the app.""" + models = tmp_path / "models" + models.mkdir() + provider = LlamaCppProvider("http://127.0.0.1:1234", models) + + with pytest.raises(FileNotFoundError): + await provider.delete("never-installed") diff --git a/surfsense_local/backend/tests/unit/llm/providers/llamacpp/test_router_client.py b/surfsense_local/backend/tests/unit/llm/providers/llamacpp/test_router_client.py new file mode 100644 index 0000000000..7606186674 --- /dev/null +++ b/surfsense_local/backend/tests/unit/llm/providers/llamacpp/test_router_client.py @@ -0,0 +1,88 @@ +"""Talking to llama-server in router mode.""" + +import httpx +import pytest + +from modules.llm.providers.llamacpp import RouterClient +from tests.unit.llm.providers.llamacpp.fake_router import FakeRouter + +pytestmark = pytest.mark.unit + + +def client_for(fake: FakeRouter) -> RouterClient: + """A client wired to a fake router rather than a live sidecar.""" + return RouterClient( + "http://127.0.0.1:1234", + transport=fake.transport(), + ) + + +@pytest.mark.asyncio +async def test_a_healthy_router_answers() -> None: + """The sidecar is up. Everything else is gated on this.""" + assert await client_for(FakeRouter()).health() is True + + +@pytest.mark.asyncio +async def test_router_mode_is_confirmed_rather_than_assumed() -> None: + """`/props` reporting role=router is the reliable check that the sidecar came + up in router mode rather than single-model mode, which answers /health too.""" + assert await client_for(FakeRouter()).is_router() is True + + +@pytest.mark.asyncio +async def test_an_empty_models_directory_is_a_working_router_not_a_failure() -> None: + """Verified against the real binary: the whole lifecycle is testable before + any model exists, which is what makes the sidecar shippable on its own.""" + assert await client_for(FakeRouter([])).models() == [] + + +@pytest.mark.asyncio +async def test_discovered_models_report_whether_they_are_resident() -> None: + """Residency decides whether the next turn pays a reload.""" + fake = FakeRouter(["Qwen3-8B-Q4_K_M"]) + fake.loaded.add("Qwen3-8B-Q4_K_M") + + models = await client_for(fake).models() + + assert models[0].id == "Qwen3-8B-Q4_K_M" + assert models[0].loaded is True + + +@pytest.mark.asyncio +async def test_loading_and_unloading_move_a_model_in_and_out_of_memory() -> None: + """The two calls the chat path and a cancelled download rely on.""" + fake = FakeRouter(["m"]) + client = client_for(fake) + + await client.load("m") + assert fake.loaded == {"m"} + + await client.unload("m") + assert fake.loaded == set() + + +@pytest.mark.asyncio +async def test_the_routers_own_delete_refuses_models_we_installed() -> None: + """Recorded because the contract table said this would work and it does not. + + `DELETE /models` only removes what the router downloaded into its own cache. + Anything staged into `--models-dir`, which is everything SurfSense installs, + comes back `not removable (not from cache)`. Deletion is ours to do. + """ + fake = FakeRouter(["m"]) + + with pytest.raises(httpx.HTTPStatusError): + await client_for(fake).delete("m") + + +@pytest.mark.asyncio +async def test_an_unreachable_router_is_unhealthy_rather_than_an_exception() -> None: + """The catalog stays visible when the runtime is down; installs disable.""" + + def refuse(request: httpx.Request) -> httpx.Response: + raise httpx.ConnectError("no route to host") + + client = RouterClient("http://127.0.0.1:1", transport=httpx.MockTransport(refuse)) + + assert await client.health() is False diff --git a/surfsense_local/backend/tests/unit/llm/providers/llamacpp/test_thinking.py b/surfsense_local/backend/tests/unit/llm/providers/llamacpp/test_thinking.py new file mode 100644 index 0000000000..e765034a43 --- /dev/null +++ b/surfsense_local/backend/tests/unit/llm/providers/llamacpp/test_thinking.py @@ -0,0 +1,72 @@ +"""Turning thinking off where a caller asked for a short answer. + +A thinking model spends its whole budget on `reasoning_content` before it emits +a single `content` token, so any call with a small `max_tokens` comes back +empty. Measured against Qwen3 1.7B at b11050: a 12 token title request returned +`content: ''`, `finish_reason: length`, and a populated trace. +""" + +import pytest + +from modules.chat.title import generate_title +from modules.llm.providers.llamacpp import LlamaCppProvider +from modules.llm.providers.openai_compatible.chat import OpenAICompatibleChatProvider +from modules.llm.providers.types import Message +from tests.unit.llm.providers.llamacpp.fake_router import FakeRouter + +pytestmark = pytest.mark.unit + + +def thinking_provider() -> tuple[LlamaCppProvider, FakeRouter]: + """A provider whose model reasons before it answers.""" + fake = FakeRouter(["qwen3"]) + fake.thinks = True + return LlamaCppProvider("http://127.0.0.1:1234", transport=fake.transport()), fake + + +@pytest.mark.asyncio +async def test_a_thinking_model_still_produces_a_title() -> None: + """The bug this exists for: 12 tokens of budget, all of it spent thinking. + + Asserted through `generate_title` rather than on the request body, because + the defect is that a caller's `reasoning=False` reached nothing. A test that + only checked the field would pass with the translation wired to the wrong + end. + """ + provider, _ = thinking_provider() + + assert await generate_title(provider, "qwen3", "explain the refund policy") + + +@pytest.mark.asyncio +async def test_the_answer_path_keeps_its_reasoning() -> None: + """Thinking is switched off per call, never for the runtime. + + The answer has no token cap and is the one place the trace earns its cost, + so a router level `--reasoning-budget 0` would pay for a title with every + reply. + """ + provider, fake = thinking_provider() + + async for _ in provider.chat("qwen3", [Message("user", "hi")]): + pass + + assert "thinking_budget_tokens" not in fake.chat_bodies[-1] + + +@pytest.mark.asyncio +async def test_a_remote_endpoint_is_never_sent_the_field() -> None: + """`thinking_budget_tokens` is llama.cpp's, and strict endpoints 400 on it. + + Only a caller that knows what it is talking to may add it, so the shared + OpenAI client keeps ignoring `reasoning` as it does today. + """ + fake = FakeRouter(["qwen3"]) + remote = OpenAICompatibleChatProvider( + "http://127.0.0.1:1234/v1", transport=fake.transport() + ) + + async for _ in remote.chat("qwen3", [Message("user", "hi")], reasoning=False): + pass + + assert fake.chat_bodies[-1].keys() == {"model", "messages", "stream"} diff --git a/surfsense_local/backend/tests/unit/llm/test_system_role.py b/surfsense_local/backend/tests/unit/llm/test_system_role.py new file mode 100644 index 0000000000..39e223c7cd --- /dev/null +++ b/surfsense_local/backend/tests/unit/llm/test_system_role.py @@ -0,0 +1,75 @@ +"""Templates that cannot carry a system message. + +`build_messages()` always emits one. On a template with no system role it is +dropped, and it is carrying the grounding and the citation instructions, so the +model answers from nothing and nothing says why. +""" + +import pytest + +from modules.llm.providers.llamacpp import Capabilities, Modality +from modules.llm.providers.llamacpp.messages import for_template +from modules.llm.providers.types import Message + +pytestmark = pytest.mark.unit + +GROUNDING = "Answer from the sources. Cite with [n]." +TURNS = [ + Message("system", GROUNDING), + Message("user", "what did the report say?"), +] + + +def caps(system_role: bool) -> Capabilities: + """A text only model whose template may or may not carry a system role.""" + return Capabilities( + inputs=(Modality.TEXT,), + system_role=system_role, + typed_content=False, + tools=False, + context_tokens=16384, + ) + + +def test_a_template_with_a_system_role_is_left_alone() -> None: + """The common case, so the fold cannot fire where it is not needed.""" + assert for_template(TURNS, caps(system_role=True)) == TURNS + + +def test_a_template_without_one_keeps_the_instructions_in_the_first_user_turn() -> None: + """Folded in, not dropped. The alternative is an ungrounded answer that + looks confident and cites nothing.""" + adjusted = for_template(TURNS, caps(system_role=False)) + + assert [m.role for m in adjusted] == ["user"] + assert GROUNDING in adjusted[0].content + assert "what did the report say?" in adjusted[0].content + + +def test_folding_preserves_the_order_the_model_reads_in() -> None: + """Instructions before the question, as the system message was.""" + folded = for_template(TURNS, caps(system_role=False))[0].content + + assert folded.index(GROUNDING) < folded.index("what did the report say?") + + +def test_a_conversation_with_no_system_message_is_untouched() -> None: + """Nothing to fold, so nothing is rewritten.""" + turns = [Message("user", "hello")] + + assert for_template(turns, caps(system_role=False)) == turns + + +def test_history_after_the_system_message_survives_the_fold() -> None: + """Only the system message moves; the conversation is not rewritten.""" + turns = [ + Message("system", GROUNDING), + Message("user", "first"), + Message("assistant", "reply"), + Message("user", "second"), + ] + + adjusted = for_template(turns, caps(system_role=False)) + + assert [m.role for m in adjusted] == ["user", "assistant", "user"] + assert adjusted[-1].content == "second" From cdab93b6fbc18b94903e84556cc413874e087b19 Mon Sep 17 00:00:00 2001 From: Anish Sarkar <104695310+AnishSarkar22@users.noreply.github.com> Date: Sun, 20 Sep 2026 20:07:43 +0530 Subject: [PATCH 18/60] feat(llm/catalog): replace recommendations with a curated manifest and Hugging Face search Co-Authored-By: Claude Sonnet 5 --- .../backend/modules/llm/catalog/__init__.py | 28 ++ .../modules/llm/catalog/curated-models.json | 191 +++++++++++ .../modules/llm/catalog/dependencies.py | 35 ++ .../backend/modules/llm/catalog/manifest.py | 136 ++++++++ .../modules/llm/catalog/recommendation.py | 59 ++++ .../backend/modules/llm/catalog/router.py | 247 +++++++++++++ .../backend/modules/llm/catalog/rows.py | 79 +++++ .../backend/modules/llm/catalog/schemas.py | 134 ++++++++ .../modules/llm/catalog/search/__init__.py | 21 ++ .../llm/catalog/search/architectures.py | 41 +++ .../modules/llm/catalog/search/client.py | 142 ++++++++ .../modules/llm/catalog/search/tickets.py | 61 ++++ .../backend/modules/llm/catalog/service.py | 324 ++++++++++++++++++ .../integration/llm/test_catalog_routes.py | 106 ++++++ .../tests/unit/llm/catalog/__init__.py | 0 .../tests/unit/llm/catalog/test_manifest.py | 154 +++++++++ .../unit/llm/catalog/test_recommendation.py | 120 +++++++ .../tests/unit/llm/catalog/test_reprice.py | 151 ++++++++ .../tests/unit/llm/catalog/test_rows.py | 100 ++++++ .../unit/llm/catalog/test_search_client.py | 91 +++++ .../unit/llm/catalog/test_search_gates.py | 49 +++ .../tests/unit/llm/catalog/test_service.py | 62 ++++ 22 files changed, 2331 insertions(+) create mode 100644 surfsense_local/backend/modules/llm/catalog/__init__.py create mode 100644 surfsense_local/backend/modules/llm/catalog/curated-models.json create mode 100644 surfsense_local/backend/modules/llm/catalog/dependencies.py create mode 100644 surfsense_local/backend/modules/llm/catalog/manifest.py create mode 100644 surfsense_local/backend/modules/llm/catalog/recommendation.py create mode 100644 surfsense_local/backend/modules/llm/catalog/router.py create mode 100644 surfsense_local/backend/modules/llm/catalog/rows.py create mode 100644 surfsense_local/backend/modules/llm/catalog/schemas.py create mode 100644 surfsense_local/backend/modules/llm/catalog/search/__init__.py create mode 100644 surfsense_local/backend/modules/llm/catalog/search/architectures.py create mode 100644 surfsense_local/backend/modules/llm/catalog/search/client.py create mode 100644 surfsense_local/backend/modules/llm/catalog/search/tickets.py create mode 100644 surfsense_local/backend/modules/llm/catalog/service.py create mode 100644 surfsense_local/backend/tests/integration/llm/test_catalog_routes.py create mode 100644 surfsense_local/backend/tests/unit/llm/catalog/__init__.py create mode 100644 surfsense_local/backend/tests/unit/llm/catalog/test_manifest.py create mode 100644 surfsense_local/backend/tests/unit/llm/catalog/test_recommendation.py create mode 100644 surfsense_local/backend/tests/unit/llm/catalog/test_reprice.py create mode 100644 surfsense_local/backend/tests/unit/llm/catalog/test_rows.py create mode 100644 surfsense_local/backend/tests/unit/llm/catalog/test_search_client.py create mode 100644 surfsense_local/backend/tests/unit/llm/catalog/test_search_gates.py create mode 100644 surfsense_local/backend/tests/unit/llm/catalog/test_service.py diff --git a/surfsense_local/backend/modules/llm/catalog/__init__.py b/surfsense_local/backend/modules/llm/catalog/__init__.py new file mode 100644 index 0000000000..90cd017c9d --- /dev/null +++ b/surfsense_local/backend/modules/llm/catalog/__init__.py @@ -0,0 +1,28 @@ +from modules.llm.catalog.manifest import ( + SCHEMA_VERSION, + CuratedModel, + CuratedModelsManifest, + ModelShapeSpec, + Variant, + load_curated_models, +) +from modules.llm.catalog.recommendation import MAX_OFFLOAD, Recommendation, recommend +from modules.llm.catalog.rows import CatalogRow, curated_rows +from modules.llm.catalog.service import Catalog, CatalogService, InstalledRow + +__all__ = [ + "MAX_OFFLOAD", + "SCHEMA_VERSION", + "Catalog", + "CatalogRow", + "CatalogService", + "CuratedModel", + "CuratedModelsManifest", + "InstalledRow", + "ModelShapeSpec", + "Recommendation", + "Variant", + "curated_rows", + "load_curated_models", + "recommend", +] diff --git a/surfsense_local/backend/modules/llm/catalog/curated-models.json b/surfsense_local/backend/modules/llm/catalog/curated-models.json new file mode 100644 index 0000000000..59ea76878e --- /dev/null +++ b/surfsense_local/backend/modules/llm/catalog/curated-models.json @@ -0,0 +1,191 @@ +{ + "schema_version": 3, + "models": [ + { + "model_id": "Qwen/Qwen3-0.6B", + "family": "Qwen3", + "label": "Qwen3 0.6B", + "parameter_count": "0.6B", + "shape": { + "architecture": "qwen3", + "block_count": 28, + "head_count_kv": 8, + "key_length": 128, + "value_length": 128, + "context_length": 40960, + "n_vocab": 151936, + "sliding_window": 0, + "expert_count": 0 + }, + "capabilities": [], + "decode_fraction": 1.0, + "variants": [ + { + "repo": "unsloth/Qwen3-0.6B-GGUF", + "file": "Qwen3-0.6B-Q4_K_M.gguf", + "quantization": "Q4_K_M", + "size_bytes": 396705472, + "mmproj": null, + "rank": 33, + "rank_basis": "llmfit-1.1.11", + "validated": false + } + ] + }, + { + "model_id": "Qwen/Qwen3-1.7B", + "family": "Qwen3", + "label": "Qwen3 1.7B", + "parameter_count": "1.7B", + "shape": { + "architecture": "qwen3", + "block_count": 28, + "head_count_kv": 8, + "key_length": 128, + "value_length": 128, + "context_length": 40960, + "n_vocab": 151936, + "sliding_window": 0, + "expert_count": 0 + }, + "capabilities": [], + "decode_fraction": 1.0, + "variants": [ + { + "repo": "unsloth/Qwen3-1.7B-GGUF", + "file": "Qwen3-1.7B-Q4_K_M.gguf", + "quantization": "Q4_K_M", + "size_bytes": 1107409472, + "mmproj": null, + "rank": 48, + "rank_basis": "llmfit-1.1.11", + "validated": false + } + ] + }, + { + "model_id": "Qwen/Qwen3-4B", + "family": "Qwen3", + "label": "Qwen3 4B", + "parameter_count": "4B", + "shape": { + "architecture": "qwen3", + "block_count": 36, + "head_count_kv": 8, + "key_length": 128, + "value_length": 128, + "context_length": 40960, + "n_vocab": 151936, + "sliding_window": 0, + "expert_count": 0 + }, + "capabilities": [], + "decode_fraction": 1.0, + "variants": [ + { + "repo": "unsloth/Qwen3-4B-GGUF", + "file": "Qwen3-4B-Q4_K_M.gguf", + "quantization": "Q4_K_M", + "size_bytes": 2497281312, + "mmproj": null, + "rank": 63, + "rank_basis": "llmfit-1.1.11", + "validated": false + } + ] + }, + { + "model_id": "Qwen/Qwen3-8B", + "family": "Qwen3", + "label": "Qwen3 8B", + "parameter_count": "8B", + "shape": { + "architecture": "qwen3", + "block_count": 36, + "head_count_kv": 8, + "key_length": 128, + "value_length": 128, + "context_length": 40960, + "n_vocab": 151936, + "sliding_window": 0, + "expert_count": 0 + }, + "capabilities": [], + "decode_fraction": 1.0, + "variants": [ + { + "repo": "unsloth/Qwen3-8B-GGUF", + "file": "Qwen3-8B-Q4_K_M.gguf", + "quantization": "Q4_K_M", + "size_bytes": 5027784512, + "mmproj": null, + "rank": 78, + "rank_basis": "llmfit-1.1.11", + "validated": false + } + ] + }, + { + "model_id": "Qwen/Qwen3-14B", + "family": "Qwen3", + "label": "Qwen3 14B", + "parameter_count": "14B", + "shape": { + "architecture": "qwen3", + "block_count": 40, + "head_count_kv": 8, + "key_length": 128, + "value_length": 128, + "context_length": 40960, + "n_vocab": 151936, + "sliding_window": 0, + "expert_count": 0 + }, + "capabilities": [], + "decode_fraction": 1.0, + "variants": [ + { + "repo": "unsloth/Qwen3-14B-GGUF", + "file": "Qwen3-14B-Q4_K_M.gguf", + "quantization": "Q4_K_M", + "size_bytes": 9001753984, + "mmproj": null, + "rank": 85, + "rank_basis": "llmfit-1.1.11", + "validated": false + } + ] + }, + { + "model_id": "Qwen/Qwen3-32B", + "family": "Qwen3", + "label": "Qwen3 32B", + "parameter_count": "32B", + "shape": { + "architecture": "qwen3", + "block_count": 64, + "head_count_kv": 8, + "key_length": 128, + "value_length": 128, + "context_length": 40960, + "n_vocab": 151936, + "sliding_window": 0, + "expert_count": 0 + }, + "capabilities": [], + "decode_fraction": 1.0, + "variants": [ + { + "repo": "unsloth/Qwen3-32B-GGUF", + "file": "Qwen3-32B-Q4_K_M.gguf", + "quantization": "Q4_K_M", + "size_bytes": 19762150048, + "mmproj": null, + "rank": 92, + "rank_basis": "llmfit-1.1.11", + "validated": false + } + ] + } + ] +} diff --git a/surfsense_local/backend/modules/llm/catalog/dependencies.py b/surfsense_local/backend/modules/llm/catalog/dependencies.py new file mode 100644 index 0000000000..8bce7e5d03 --- /dev/null +++ b/surfsense_local/backend/modules/llm/catalog/dependencies.py @@ -0,0 +1,35 @@ +"""Wiring the catalog service once per process.""" + +from functools import lru_cache +from pathlib import Path +from typing import Annotated + +from fastapi import Depends + +from modules.llm.catalog.manifest import ( + CuratedModelsManifest, + load_curated_models, +) +from modules.llm.catalog.service import CatalogService +from shared.config import get_llm_settings + + +@lru_cache +def get_catalog_service() -> CatalogService: + settings = get_llm_settings() + try: + manifest = load_curated_models() + except (OSError, ValueError): + # A broken manifest must not take the screen down: installed models and + # a local .gguf import still work without it. + manifest = CuratedModelsManifest(schema_version=3, models=[]) + models_dir = settings.llamacpp_models_dir or Path("models") + return CatalogService( + manifest, + models_dir, + settings.llamacpp_library_dir or Path("."), + settings.llamacpp_base_url, + ) + + +CatalogServiceDep = Annotated[CatalogService, Depends(get_catalog_service)] diff --git a/surfsense_local/backend/modules/llm/catalog/manifest.py b/surfsense_local/backend/modules/llm/catalog/manifest.py new file mode 100644 index 0000000000..3c4e9f3473 --- /dev/null +++ b/surfsense_local/backend/modules/llm/catalog/manifest.py @@ -0,0 +1,136 @@ +"""The curated manifest: models SurfSense has tested, priced offline. + +**Source, not build output.** A person runs the authoring script, reads what it +proposes, and commits the result. That is deliberate: a rank moving 78 to 94 +shows up in a pull request where someone notices, a tag rebuilds to the same +manifest forever, and the cadence is honest, since these change when someone +adds a model rather than when someone cuts a release. + +The schema enforces one thing no reviewer reliably catches: a `rank` sits inside +the variant it was measured on, so it cannot describe a build the manifest does +not ship. +""" + +from pathlib import Path + +from pydantic import BaseModel, ConfigDict, Field, model_validator + +from modules.llm.fit import ModelShape + +SCHEMA_VERSION = 3 + + +class ModelShapeSpec(BaseModel): + """Header fields, committed so an airgapped machine can price the row. + + Quantization independent: measured, the architecture fields are identical + across Q4_K_M, Q8_0 and f16 of the same model, so one header read at + authoring time prices every build of it forever. + """ + + model_config = ConfigDict(extra="forbid") + + architecture: str = Field(min_length=1) + block_count: int = Field(gt=0) + head_count_kv: int = Field(gt=0) + key_length: int = Field(gt=0) + value_length: int = Field(gt=0) + context_length: int = Field(gt=0) + n_vocab: int = Field(gt=0) + sliding_window: int = Field(default=0, ge=0) + expert_count: int = Field(default=0, ge=0) + + +class Variant(BaseModel): + """One downloadable build, plus the three fields a person decides. + + Everything above `rank` is derived from the Hugging Face listing and the + GGUF header. Everything from `rank` down is judgement, and it lives here + rather than on the entry because quality is a function of (model, + quantization), not of the model alone. + """ + + model_config = ConfigDict(extra="forbid") + + repo: str = Field(min_length=1) + file: str = Field(min_length=1) + quantization: str = Field(min_length=1) + size_bytes: int = Field(gt=0) + mmproj: str | None = None + + rank: int = Field(gt=0) + rank_basis: str = Field(min_length=1) + validated: bool = False + + +class CuratedModel(BaseModel): + model_config = ConfigDict(extra="forbid") + + model_id: str = Field(min_length=1) + family: str = Field(min_length=1) + label: str = Field(min_length=1) + parameter_count: str = Field(min_length=1) + + shape: ModelShapeSpec + # User facing only. Today that means "vision" or nothing; everything else + # the template reports is an internal constraint, not a badge. + capabilities: list[str] = Field(default_factory=list) + # Fraction of the build's bytes read per decoded token: 1.0 dense, the + # active slice for a mixture of experts. Per model rather than per variant, + # because it is an expert to total byte ratio and barely moves with quant. + decode_fraction: float = Field(default=1.0, gt=0.0, le=1.0) + + # A list from the start, with one thing in it. Costs nothing now and is the + # only part of this schema that is expensive to add later, because changing + # it means schema_version 4 and re-authoring every entry. + variants: list[Variant] = Field(min_length=1) + + @property + def model_shape(self) -> ModelShape: + """The committed header fields, as the estimator wants them. + + This is what lets a curated row be priced on first paint with no network + and nothing downloaded. + """ + return ModelShape(**self.shape.model_dump()) + + +class CuratedModelsManifest(BaseModel): + model_config = ConfigDict(extra="forbid") + + schema_version: int + models: list[CuratedModel] + + @model_validator(mode="after") + def validate_manifest(self) -> "CuratedModelsManifest": + if self.schema_version != SCHEMA_VERSION: + raise ValueError(f"unsupported curated-model schema: {self.schema_version}") + + ids = [model.model_id for model in self.models] + if len(ids) != len(set(ids)): + raise ValueError("duplicate model_id in curated-model manifest") + + files = [ + (variant.repo, variant.file) + for model in self.models + for variant in model.variants + ] + if len(files) != len(set(files)): + raise ValueError("duplicate pinned file in curated-model manifest") + + bases = { + variant.rank_basis + for model in self.models + for variant in model.variants + } + if len(bases) > 1: + raise ValueError( + "ranks from different rank_basis values cannot be compared: " + f"{sorted(bases)}" + ) + return self + + +def load_curated_models(path: Path | None = None) -> CuratedModelsManifest: + manifest_path = path or Path(__file__).with_name("curated-models.json") + return CuratedModelsManifest.model_validate_json(manifest_path.read_text()) diff --git a/surfsense_local/backend/modules/llm/catalog/recommendation.py b/surfsense_local/backend/modules/llm/catalog/recommendation.py new file mode 100644 index 0000000000..bf3ed2cbd1 --- /dev/null +++ b/surfsense_local/backend/modules/llm/catalog/recommendation.py @@ -0,0 +1,59 @@ +"""Which curated build to star. + +Gated on predicted speed rather than on full residency. Those coincide on a +large card and diverge on a small one: measured on an RTX 3050, Qwen3 8B spills +roughly a fifth and runs without noticeable lag, while a residency-only rule +stars a 1.7B on the same machine. That rule is not conservative, it is wrong, +because it bans a configuration that demonstrably works. + +The policy ranges over builds rather than models, because a build is what the +user installs and what `rank` describes. With one variant per entry the +cross-product is the entry list and the behaviour is identical, but writing it +this way is the difference between adding a second build later as a manifest +edit and rewriting this module, its tests and every fixture. +""" + +from collections.abc import Sequence +from dataclasses import dataclass + +from modules.llm.catalog.manifest import CuratedModel, Variant +from modules.llm.fit import FitState, HardwareBudget, estimate +from modules.llm.fit.estimate import FitVerdict + +# > ponytail: provisional, and possibly unnecessary. On the one machine measured +# > every offload fraction stayed usable: fully on the CPU the model still +# > decoded at 13 t/s, above reading pace, and prefill held at half device speed. +# > No threshold would have fired anywhere in the curated range. Check whether +# > this gate ever triggers before treating the number as load bearing; a +# > mechanism that never fires reads as protection that was never tested. +MAX_OFFLOAD = 0.75 + + +@dataclass(frozen=True) +class Recommendation: + entry: CuratedModel + variant: Variant + verdict: FitVerdict + + +def recommend( + curated: Sequence[CuratedModel], budget: HardwareBudget +) -> Recommendation | None: + """The highest-ranked build predicted fast enough, or None. + + None is an honest answer rather than a failure: every build that physics does + not refuse stays installable, it simply goes unstarred. + """ + candidates: list[Recommendation] = [] + for entry in curated: + for variant in entry.variants: + verdict = estimate(entry.model_shape, variant.size_bytes, budget) + if verdict.state is FitState.TOO_BIG: + continue + if verdict.offload_fraction > MAX_OFFLOAD: + continue + candidates.append(Recommendation(entry, variant, verdict)) + + if not candidates: + return None + return max(candidates, key=lambda c: (c.variant.rank, -c.variant.size_bytes)) diff --git a/surfsense_local/backend/modules/llm/catalog/router.py b/surfsense_local/backend/modules/llm/catalog/router.py new file mode 100644 index 0000000000..beff919273 --- /dev/null +++ b/surfsense_local/backend/modules/llm/catalog/router.py @@ -0,0 +1,247 @@ +"""The model screen's HTTP surface. + +Two catalog endpoints, deliberately. Curated plus installed renders offline and +instantly; search needs `huggingface.co` and is paged. One response covering +both would either block on the network or return partial results behind a flag, +and that flag is the `scanned` flag this phase deletes. + +`GET /llm/system`, `GET /llm/catalog` and `POST /llm/install` **replace** routes +that already exist at those paths. This is an in place reshape, not an addition. +""" + +import json +import logging +from collections.abc import AsyncIterator + +import httpx +from fastapi import APIRouter, HTTPException, Request, status +from fastapi.responses import StreamingResponse + +from api.dependencies import SessionDep, transact +from modules.egress import service as egress +from modules.llm.catalog.dependencies import CatalogServiceDep +from modules.llm.catalog.schemas import ( + CatalogRead, + InstallRequest, + RepoRead, + SearchRead, + SystemRead, +) +from modules.llm.models import ModelRole, SelectedModel +from modules.llm.schemas import SelectionRead +from modules.llm.selection import choose_model + +router = APIRouter() +logger = logging.getLogger(__name__) + +STALE_ID = "catalog id is stale or unknown; refresh the catalog" + + +@router.get("/system", response_model=SystemRead, summary="This machine's memory") +def read_system(service: CatalogServiceDep) -> dict: + """No scan, no button, no network. The allocator's own view.""" + return { + "budget": _budget(service.budget()), + "devices": [ + { + "name": device.name, + "description": device.description, + "kind": device.type.name.lower(), + "total_bytes": device.total_bytes, + "free_bytes": device.free_bytes, + } + for device in service.devices() + ], + } + + +@router.get("/catalog", response_model=CatalogRead, summary="Tested and installed models") +def read_catalog(service: CatalogServiceDep, session: SessionDep) -> dict: + """Renders on first paint with no network call of any kind.""" + selected = _selected_local(session) + catalog = service.catalog() + return { + "budget": _budget(catalog.budget), + "curated": [ + _row(row, catalog, selected, service) for row in catalog.curated + ], + "installed": [ + { + "model_id": row.model_id, + "file": row.file, + "size_bytes": row.size_bytes, + "selected": row.model_id == selected, + } + for row in catalog.installed + ], + "recommended_model_id": catalog.recommended_model_id, + } + + +@router.get("/search", response_model=SearchRead, summary="Search Hugging Face") +async def search( + service: CatalogServiceDep, session: SessionDep, q: str = "", limit: int = 30 +) -> dict: + """Absent rather than degraded when egress is off. That is the airgapped + product: curated, installed and a local .gguf import all still work.""" + await transact(session, egress.require, egress.MODEL_SEARCH) + try: + hits = await service.search(q, limit=limit) + except httpx.HTTPError as error: + raise HTTPException( + status.HTTP_503_SERVICE_UNAVAILABLE, "huggingface.co is unreachable" + ) from error + return {"results": [_hit(hit) for hit in hits]} + + +@router.get("/search/{repo:path}", response_model=RepoRead, summary="Builds in a repo") +async def read_repo(repo: str, service: CatalogServiceDep, session: SessionDep) -> dict: + """Where the header read happens, so the trigger is opening a result rather + than hovering or typing. The list level badge stays approximate until here.""" + await transact(session, egress.require, egress.MODEL_SEARCH) + try: + return await service.repo(repo) + except httpx.HTTPError as error: + raise HTTPException( + status.HTTP_503_SERVICE_UNAVAILABLE, "huggingface.co is unreachable" + ) from error + + +@router.post("/install", summary="Download a model and optionally select it") +async def install( + payload: InstallRequest, + request: Request, + service: CatalogServiceDep, + session: SessionDep, +) -> StreamingResponse: + plan = service.resolve_install(payload.catalog_id) + if plan is None: + raise HTTPException(status.HTTP_422_UNPROCESSABLE_CONTENT, STALE_ID) + + await transact(session, egress.require, egress.MODEL_DOWNLOAD) + + lock = service.install_lock() + if lock.locked(): + raise HTTPException( + status.HTTP_409_CONFLICT, "a model is already being installed" + ) + await lock.acquire() + + async def progress() -> AsyncIterator[bytes]: + try: + yield _event("starting", message="Preparing download") + async for step in service.install(plan): + yield _event( + "downloading", + message=step.status, + completed=step.completed, + total=step.total, + ) + yield _event("verifying", message="Checking the model") + # Rewrite the preset, then wait for the runtime to come back with + # the model. The router only learns about it by restarting, and + # reporting complete before then tells the user a model is ready + # while a chat returns `model not found`. + service.reprice() + yield _event("preparing", message="Preparing the model runtime") + servable = await service.wait_until_servable(plan.model_id) + + selection = None + if payload.select: + yield _event("selecting", message="Selecting model") + with request.app.state.session_factory() as fresh: + chosen = await choose_model( + fresh, ModelRole.GENERATION, service.provider_name, plan.model_id + ) + selection = SelectionRead.model_validate(chosen).model_dump( + mode="json" + ) + yield _event( + "complete", + message=( + "Model is ready" + if servable + else "Downloaded. It becomes available once the runtime restarts." + ), + selection=selection, + ) + except Exception: + logger.exception("model install failed") + yield _event( + "error", + message="The model could not be installed. Retry the download.", + ) + finally: + lock.release() + + return StreamingResponse(progress(), media_type="application/x-ndjson") + + +def _selected_local(session: SessionDep) -> str | None: + selected = session.get(SelectedModel, ModelRole.GENERATION) + if selected is None or selected.connection_id is not None: + return None + return selected.name + + +def _budget(budget) -> dict: + return { + "device_total_bytes": budget.device_total_bytes, + "device_free_bytes": budget.device_free_bytes, + "usable_vram_bytes": budget.usable_vram_bytes, + "fit_reserve_bytes": budget.fit_reserve_bytes, + "ram_available_bytes": budget.ram_available_bytes, + "uma": budget.uma, + "has_gpu": budget.has_gpu, + } + + +def _fit(fit) -> dict: + return { + "state": fit.state.value, + "need_bytes": fit.need_bytes, + "budget_bytes": fit.budget_bytes, + "offload_fraction": fit.offload_fraction, + "approximate": fit.approximate, + } + + +def _row(row, catalog, selected: str | None, service) -> dict: + """One curated row. Note the absent `rank`: it orders this list and selects + the star, and the renderer never needs to know the number.""" + installed = {entry.model_id for entry in catalog.installed} + model_id = row.variant.file.removesuffix(".gguf") + return { + "catalog_id": service.catalog_id(row), + "model_id": row.model_id, + "variant_model_id": model_id, + "label": row.label, + "family": row.family, + "parameter_count": row.parameter_count, + "quantization": row.variant.quantization, + "size_bytes": row.variant.size_bytes, + "context_length": 0, + "fit": _fit(row.fit), + "badge": {"verdict": row.badge.verdict, "reason": row.badge.reason}, + "capabilities": list(row.capabilities), + "installed": model_id in installed, + "selected": model_id == selected, + "can_install": row.can_install, + "recommended": row.model_id == catalog.recommended_model_id, + } + + +def _hit(hit) -> dict: + return { + "repo": hit.repo, + "downloads": hit.downloads, + "likes": hit.likes, + "license": hit.license, + "gated": hit.gated, + "quantized_from": hit.quantized_from, + "last_modified": hit.last_modified, + } + + +def _event(kind: str, **payload: object) -> bytes: + return (json.dumps({"type": kind, **payload}) + "\n").encode() diff --git a/surfsense_local/backend/modules/llm/catalog/rows.py b/surfsense_local/backend/modules/llm/catalog/rows.py new file mode 100644 index 0000000000..cd131b92ce --- /dev/null +++ b/surfsense_local/backend/modules/llm/catalog/rows.py @@ -0,0 +1,79 @@ +"""Curated entries as rows a screen can render. + +Priced entirely from the manifest: `shape` is committed, so this runs on first +paint with no network, no download and no scan. That is what makes the curated +tier the whole product on an airgapped machine. +""" + +from collections.abc import Sequence +from dataclasses import dataclass + +from modules.llm.catalog.manifest import CuratedModel, Variant +from modules.llm.fit import Badge, FitState, FitVerdict, HardwareBudget, badge, estimate + +_FIT_ORDER = {FitState.FITS: 0, FitState.PARTIAL: 1, FitState.TOO_BIG: 2} + + +@dataclass(frozen=True) +class CatalogRow: + """One model on the screen. + + Deliberately carries no `rank`. Rank orders this list and selects the star, + and it is never displayed; the surest way to keep that true is for the + renderer never to receive it. + """ + + model_id: str + label: str + family: str + parameter_count: str + variant: Variant + fit: FitVerdict + badge: Badge + capabilities: tuple[str, ...] + + @property + def can_install(self) -> bool: + """Only physics refuses. Reduced speed runs, so it never blocks.""" + return self.fit.state is not FitState.TOO_BIG + + +def curated_rows( + models: Sequence[CuratedModel], budget: HardwareBudget +) -> list[CatalogRow]: + """One row per model, best build first, ordered fit coarsely then rank finely. + + Sorting by rank alone would put a refused 32B at the top of a small machine's + screen, which is the one thing a model chooser must not do. + """ + rows = [_row(model, budget) for model in models] + return sorted( + rows, + key=lambda row: (_FIT_ORDER[row.fit.state], -row.variant.rank, row.model_id), + ) + + +def _row(model: CuratedModel, budget: HardwareBudget) -> CatalogRow: + """The best build of one model: a row is a model, not a file. + + A model with two builds is still one row, because the screen's job is + choosing a model. It takes the best state among its builds, and the install + action uses the build that produced it. + """ + priced = [ + (variant, estimate(model.model_shape, variant.size_bytes, budget)) + for variant in model.variants + ] + variant, fit = min( + priced, key=lambda pair: (_FIT_ORDER[pair[1].state], -pair[0].rank) + ) + return CatalogRow( + model_id=model.model_id, + label=model.label, + family=model.family, + parameter_count=model.parameter_count, + variant=variant, + fit=fit, + badge=badge(fit, budget), + capabilities=tuple(model.capabilities), + ) diff --git a/surfsense_local/backend/modules/llm/catalog/schemas.py b/surfsense_local/backend/modules/llm/catalog/schemas.py new file mode 100644 index 0000000000..95579de9fe --- /dev/null +++ b/surfsense_local/backend/modules/llm/catalog/schemas.py @@ -0,0 +1,134 @@ +"""What the model screen receives. + +Two things are deliberately absent from every row: a `rank`, and any hint of a +scan. Rank orders the curated list and selects the star and is never displayed, +so the renderer does not receive it. There is no `scanned` flag because there is +no scan: the budget comes from the runtime's own allocator in milliseconds. +""" + +from pydantic import BaseModel, ConfigDict, Field + + +class DeviceRead(BaseModel): + name: str + description: str + kind: str + total_bytes: int + free_bytes: int + + +class BudgetRead(BaseModel): + """One device, never a sum across devices.""" + + device_total_bytes: int + device_free_bytes: int + usable_vram_bytes: int + fit_reserve_bytes: int + ram_available_bytes: int + uma: bool + has_gpu: bool + + +class SystemRead(BaseModel): + budget: BudgetRead + devices: list[DeviceRead] + + +class BadgeRead(BaseModel): + """A verdict plus one plain line of why.""" + + verdict: str + reason: str + + +class FitRead(BaseModel): + state: str + need_bytes: int + budget_bytes: int + # Kept on the wire because the reason line is graded by it and the renderer + # must not recompute it. Three buckets lose the difference between a model + # that spills 5% and one that spills 70%. + offload_fraction: float + approximate: bool = False + + +class CatalogRowRead(BaseModel): + model_config = ConfigDict(from_attributes=True) + + catalog_id: str + model_id: str + # What the runtime calls the installed file, which is what Use and Delete + # act on. Distinct from `model_id`, which identifies the model rather than + # the build, and from `catalog_id`, which is an opaque install token. + variant_model_id: str + label: str + family: str + parameter_count: str + quantization: str + size_bytes: int + context_length: int + fit: FitRead + badge: BadgeRead + capabilities: list[str] + installed: bool + selected: bool + can_install: bool + recommended: bool + + +class InstalledRowRead(BaseModel): + model_id: str + file: str + size_bytes: int + selected: bool + + +class CatalogRead(BaseModel): + budget: BudgetRead + curated: list[CatalogRowRead] + installed: list[InstalledRowRead] + recommended_model_id: str | None + + +class SearchRowRead(BaseModel): + """A repo, described. No rank and no quality claim, ever.""" + + repo: str + downloads: int + likes: int + license: str | None + gated: bool + quantized_from: str | None + last_modified: str | None + + +class SearchRead(BaseModel): + results: list[SearchRowRead] + + +class BuildRead(BaseModel): + """One installable build inside a repo, priced exactly.""" + + catalog_id: str + file: str + quantization: str + size_bytes: int + fit: FitRead + badge: BadgeRead + can_install: bool + + +class RepoRead(BaseModel): + repo: str + architecture: str + context_length: int + supported: bool + builds: list[BuildRead] + # Eligibility is not fit: a model can be FITS and still refused here, and the + # two must never render as one thing. + ineligible_reason: str | None = None + + +class InstallRequest(BaseModel): + catalog_id: str = Field(min_length=1, max_length=128) + select: bool = True diff --git a/surfsense_local/backend/modules/llm/catalog/search/__init__.py b/surfsense_local/backend/modules/llm/catalog/search/__init__.py new file mode 100644 index 0000000000..a713982331 --- /dev/null +++ b/surfsense_local/backend/modules/llm/catalog/search/__init__.py @@ -0,0 +1,21 @@ +from modules.llm.catalog.search.architectures import SUPPORTED, is_supported +from modules.llm.catalog.search.client import ( + CACHE_SECONDS, + RepoFile, + SearchHit, + list_builds, + search_models, +) +from modules.llm.catalog.search.tickets import InstallTicket, TicketStore + +__all__ = [ + "CACHE_SECONDS", + "SUPPORTED", + "InstallTicket", + "RepoFile", + "SearchHit", + "TicketStore", + "is_supported", + "list_builds", + "search_models", +] diff --git a/surfsense_local/backend/modules/llm/catalog/search/architectures.py b/surfsense_local/backend/modules/llm/catalog/search/architectures.py new file mode 100644 index 0000000000..fc10d56ee6 --- /dev/null +++ b/surfsense_local/backend/modules/llm/catalog/search/architectures.py @@ -0,0 +1,41 @@ +"""Architectures llama.cpp can run. + +The gate is the header, not Hugging Face's tags. Counted on `library=gguf`, the +pipeline tags cover roughly 43,500 of 204,797 repos, so **about 80% carry no +useful tag at all** and filtering by tag would hide four fifths of the catalog, +including working chat models whose uploader left the field blank. + +Two header facts do the job structurally instead: a supported architecture +excludes Whisper-shaped models, and a present chat template excludes embedding +models. +""" + +# LLM_ARCH_* in llama.cpp. Kept as data rather than fetched, because the app is +# airgapped and this list changes only when the pinned build does. +SUPPORTED = frozenset( + { + "llama", "llama4", "deci", "falcon", "falcon-h1", "grok", "gpt2", "gptj", + "gptneox", "mpt", "baichuan", "starcoder", "refact", "bert", "nomic-bert", + "nomic-bert-moe", "neo-bert", "jina-bert-v2", "jina-bert-v3", "bloom", + "stablelm", "qwen", "qwen2", "qwen2moe", "qwen2vl", "qwen3", "qwen3moe", + "qwen3vl", "qwen3vlmoe", "qwen3next", "phi2", "phi3", "phimoe", "plamo", + "plamo2", "codeshell", "orion", "internlm2", "minicpm", "minicpm3", + "gemma", "gemma2", "gemma3", "gemma3n", "gemma-embedding", "starcoder2", + "mamba", "mamba2", "jamba", "xverse", "command-r", "cohere2", + "dbrx", "olmo", "olmo2", "olmoe", "openelm", "arctic", "deepseek", + "deepseek2", "deepseek-v4", "chatglm", "glm4", "glm4moe", "bitnet", + "t5", "t5encoder", "jais", "nemotron", "nemotron-h", "exaone", "exaone4", + "rwkv6", "rwkv6qwen2", "rwkv7", "arwkv7", "granite", "granite-moe", + "granite-hybrid", "chameleon", "wavtokenizer-dec", "plm", "bailingmoe", + "bailingmoe2", "dots1", "arcee", "ernie4_5", "ernie4_5-moe", "hunyuan-moe", + "hunyuan-dense", "smollm3", "openai-moe", "lfm2", "lfm2moe", "smallthinker", + "dream", "seed_oss", "grovemoe", "apertus", "minimax-m2", "cogvlm", + "pangu-embedded", "affine", "mistral", "mixtral", "solar", "stablelm2", + "internvl", "megrez", "gpt-oss", "seed-oss", "ling", "kimi-k2", + } +) + + +def is_supported(architecture: str) -> bool: + """Unknown means not installable: no amount of memory helps.""" + return architecture.lower() in SUPPORTED diff --git a/surfsense_local/backend/modules/llm/catalog/search/client.py b/surfsense_local/backend/modules/llm/catalog/search/client.py new file mode 100644 index 0000000000..292856bb64 --- /dev/null +++ b/surfsense_local/backend/modules/llm/catalog/search/client.py @@ -0,0 +1,142 @@ +"""Searching Hugging Face for anything llama.cpp can run. + +204,797 GGUF repos, against the 240 of the registry this replaced. A network +feature and is simply **absent** when egress is off, which is the airgapped +product rather than a degraded one. + +Ordered by downloads, which is the only sort usable as a default: every other +one surfaces abliterated derivatives or half finished uploads in the top few. +The count is presented as popularity, never as endorsement. +""" + +import re +from dataclasses import dataclass, field + +import httpx + +API = "https://huggingface.co/api/models" +TIMEOUT = httpx.Timeout(30.0, connect=10.0) + +# Hugging Face allows 500 requests per 5 minutes, and a search list is re-fetched +# on every keystroke a debounce lets through. +CACHE_SECONDS = 300 + + +@dataclass(frozen=True) +class SearchHit: + """A repo, described rather than judged. + + Carries **no rank and no quality claim**. Attaching one would put a base + model's score on a derivative that behaves differently: of the top 100 GGUF + repos by downloads, the matches include abliterated and uncensored builds + resolving to their parent's number. + """ + + repo: str + downloads: int + likes: int + last_modified: str | None = None + license: str | None = None + gated: bool = False + # 32 of the top 40 GGUF repos carry a base_model:quantized: tag, so a + # row can say where the file came from. That is a fact about provenance, not + # a grade, and it is most of what a quality score was doing for a reader. + quantized_from: str | None = None + pipeline_tag: str | None = None + tags: tuple[str, ...] = field(default_factory=tuple) + + +async def search_models( + client: httpx.AsyncClient, query: str, *, limit: int = 30 +) -> list[SearchHit]: + """GGUF repos matching `query`, most downloaded first.""" + reply = await client.get( + API, + params={ + "filter": "gguf", + "search": query, + "sort": "downloads", + "direction": -1, + "limit": min(limit, 50), + "full": "true", + }, + timeout=TIMEOUT, + ) + reply.raise_for_status() + return [_hit(row) for row in reply.json()] + + +def _hit(row: dict) -> SearchHit: + tags = tuple(row.get("tags") or ()) + return SearchHit( + repo=row.get("id", ""), + downloads=row.get("downloads", 0), + likes=row.get("likes", 0), + last_modified=row.get("lastModified"), + license=next( + (t.removeprefix("license:") for t in tags if t.startswith("license:")), None + ), + gated=bool(row.get("gated")), + quantized_from=next( + ( + t.removeprefix("base_model:quantized:") + for t in tags + if t.startswith("base_model:quantized:") + ), + None, + ), + pipeline_tag=row.get("pipeline_tag"), + tags=tags, + ) + + +@dataclass(frozen=True) +class RepoFile: + """One build inside a repo.""" + + file: str + size_bytes: int + quantization: str + + +async def list_builds(client: httpx.AsyncClient, repo: str) -> list[RepoFile]: + """Every single file GGUF in a repo, with its exact size. + + Split sets, projectors and draft models are excluded rather than offered: + none of them is a thing a user installs on its own. + """ + reply = await client.get( + f"{API}/{repo}/tree/main", params={"recursive": "true"}, timeout=TIMEOUT + ) + reply.raise_for_status() + + builds = [] + for row in reply.json(): + path = row.get("path", "") + lowered = path.lower() + if not lowered.endswith(".gguf"): + continue + if "mmproj" in lowered or "draft" in lowered or "-of-" in path: + continue + builds.append(RepoFile(path, row.get("size", 0), _quantization(path))) + return sorted(builds, key=lambda build: build.size_bytes) + + +# A quant name, not merely something that looks like one. Matching loosely reads +# "Qwen3" out of `Qwen3-Coder-30B-A3B-Instruct-UD-TQ1_0.gguf`, seen live: a model +# name can begin with Q and carry a digit exactly as a quant name does. +_QUANT = re.compile( + r"(?:^|[-_.])((?:I?Q\d(?:_[A-Z0-9]+)*|TQ\d_\d|BF16|F16|F32))(?=$|[-_.])", + re.IGNORECASE, +) + + +def _quantization(path: str) -> str: + """The quant name out of a filename, which is where uploaders put it. + + The last match wins, because the quant is conventionally the final token and + a repo name can contain something shaped like one. + """ + stem = path.rsplit("/", 1)[-1].removesuffix(".gguf") + matches = _QUANT.findall(stem) + return matches[-1].upper() if matches else "unknown" diff --git a/surfsense_local/backend/modules/llm/catalog/search/tickets.py b/surfsense_local/backend/modules/llm/catalog/search/tickets.py new file mode 100644 index 0000000000..f669964ad5 --- /dev/null +++ b/surfsense_local/backend/modules/llm/catalog/search/tickets.py @@ -0,0 +1,61 @@ +"""Install ids for models that no manifest names. + +`POST /llm/install` keys on a `catalog_id`, and the renderer is forbidden from +sending a repo, file, URL or path. A curated row's id comes from the manifest; a +search hit has no manifest entry, so there is nothing to key on. + +Closing that by letting the renderer post a repo and file would hand the +frontend the ability to name an arbitrary download, which is exactly the +capability the install contract exists to withhold. So the server mints an id +when it prices a search result, and hands back only that. +""" + +import secrets +import time +from dataclasses import dataclass + +# The same window the search cache uses. A ticket outliving its row would let a +# stale screen install something the user is no longer looking at. +TTL_SECONDS = 300.0 + + +@dataclass(frozen=True) +class InstallTicket: + """One downloadable build, resolved server side.""" + + repo: str + file: str + quantization: str + size_bytes: int + issued_at: float + + +class TicketStore: + """In memory and deliberately forgetful.""" + + def __init__(self, *, ttl_seconds: float = TTL_SECONDS) -> None: + self._ttl = ttl_seconds + self._tickets: dict[str, InstallTicket] = {} + + def mint( + self, repo: str, file: str, quantization: str, size_bytes: int, *, now: float | None = None + ) -> str: + self._expire(now) + token = secrets.token_urlsafe(18) + self._tickets[token] = InstallTicket( + repo, file, quantization, size_bytes, now if now is not None else time.monotonic() + ) + return token + + def resolve(self, token: str, *, now: float | None = None) -> InstallTicket | None: + """None for an expired or unknown id, which the route turns into the + existing "stale or unknown, refresh the catalog" 422 rather than a second + error class for the same failure.""" + self._expire(now) + return self._tickets.get(token) + + def _expire(self, now: float | None) -> None: + cutoff = (now if now is not None else time.monotonic()) - self._ttl + for token, ticket in list(self._tickets.items()): + if ticket.issued_at < cutoff: + del self._tickets[token] diff --git a/surfsense_local/backend/modules/llm/catalog/service.py b/surfsense_local/backend/modules/llm/catalog/service.py new file mode 100644 index 0000000000..b98f07bfa5 --- /dev/null +++ b/surfsense_local/backend/modules/llm/catalog/service.py @@ -0,0 +1,324 @@ +"""Assembling what the model screen shows. + +A new service rather than a parameter on the old one. `CatalogService` requires +an advisor and a scan; this tier has neither, and making one class serve both +would keep the scan-shaped vocabulary alive to describe something that no longer +scans. + +Curated and installed are assembled from the manifest and the models directory, +both local, so this answers with **no network** and nothing to wait for. +""" + +import asyncio +import logging +import secrets +import time +from collections.abc import AsyncIterator +from dataclasses import dataclass +from pathlib import Path + +import httpx + +from modules.llm.catalog.manifest import CuratedModelsManifest +from modules.llm.catalog.recommendation import recommend +from modules.llm.catalog.rows import CatalogRow, curated_rows +from modules.llm.catalog.search import ( + SearchHit, + TicketStore, + is_supported, + list_builds, + search_models, +) +from modules.llm.fit import HardwareBudget, badge, estimate, plan_load +from modules.llm.gguf import shape_from_file, shape_from_url +from modules.llm.hardware import ( + BudgetMode, + Device, + available_bytes, + build_budget, + probe_devices, +) +from modules.llm.providers.llamacpp import ( + PRESET_FILE, + PROVIDER, + ModelPreset, + RouterClient, + download_gguf, + write_presets, +) + +logger = logging.getLogger(__name__) + + +@dataclass(frozen=True) +class InstalledRow: + """A model already on disk, named as the router reports it.""" + + model_id: str + file: str + size_bytes: int + + +@dataclass(frozen=True) +class InstallPlan: + """A resolved build, named by the server rather than by the renderer.""" + + model_id: str + repo: str + file: str + size_bytes: int + + +@dataclass(frozen=True) +class Catalog: + budget: HardwareBudget + devices: tuple[Device, ...] + curated: tuple[CatalogRow, ...] + installed: tuple[InstalledRow, ...] + recommended_model_id: str | None + + +class CatalogService: + """Curated plus installed, priced against this machine, offline.""" + + def __init__( + self, + manifest: CuratedModelsManifest, + models_dir: Path, + library_dir: Path, + runtime_url: str = "http://127.0.0.1:8080", + ) -> None: + self._runtime_url = runtime_url + self._manifest = manifest + self._models_dir = models_dir + self._library_dir = library_dir + self._devices: tuple[Device, ...] | None = None + self._tickets = TicketStore() + self._catalog_ids: dict[str, tuple[str, str]] = {} + # One pull at a time: two concurrent downloads compete for the same disk + # and the screen has one progress bar. + self._install_lock = asyncio.Lock() + + provider_name = PROVIDER + + def install_lock(self) -> asyncio.Lock: + return self._install_lock + + def catalog_id(self, row: CatalogRow) -> str: + """A stable opaque id for a curated row, minted once per process. + + Opaque so the renderer cannot assemble one, which is the same reason + searched builds get tickets rather than repo and file. + """ + key = (row.variant.repo, row.variant.file) + for token, existing in self._catalog_ids.items(): + if existing == key: + return token + token = secrets.token_urlsafe(18) + self._catalog_ids[token] = key + return token + + def reprice(self) -> None: + """Write the preset for everything now on disk. + + The router reads its models directory, and this file, once at startup. + Measured: a model dropped into a running router's directory is still + invisible 26 seconds later and appears immediately after a restart. So + writing this is what makes a completed download reachable: Electron + watches the file and restarts the sidecar, which costs 0.15s because an + idle router holds no model memory. + + Two budgets, because they answer different questions. Capacity decides + the verdict, so the plan agrees with the badge the catalog showed. Live + caps how far the window widens past the floor, because that part is + opportunistic and on unified memory it comes out of the same pool the OS + is using. + """ + budget = self.budget(BudgetMode.CAPACITY) + live = self.budget(BudgetMode.LIVE) + presets = [] + for path in sorted(self._models_dir.glob("*.gguf")): + try: + shape = shape_from_file(path) + except (OSError, ValueError): + # A truncated or foreign file in the directory. Skipping it + # costs that one model; failing would leave the runtime dead + # over a file nobody asked it to load. + logger.warning("skipping unreadable model %s", path.name) + continue + plan = plan_load(shape, path.stat().st_size, budget, live=live) + presets.append( + ModelPreset(path.stem, str(path), plan.n_ctx, plan.precision) + ) + write_presets(self._models_dir / PRESET_FILE, presets) + + def resolve_install(self, catalog_id: str) -> InstallPlan | None: + """Turn an id from either tier into a build, or None if it has gone stale.""" + pinned = self._catalog_ids.get(catalog_id) + if pinned is not None: + repo, file = pinned + size = next( + ( + variant.size_bytes + for model in self._manifest.models + for variant in model.variants + if (variant.repo, variant.file) == pinned + ), + 0, + ) + return InstallPlan(file.removesuffix(".gguf"), repo, file, size) + + ticket = self._tickets.resolve(catalog_id) + if ticket is None: + return None + return InstallPlan( + ticket.file.removesuffix(".gguf"), ticket.repo, ticket.file, ticket.size_bytes + ) + + async def install(self, plan: InstallPlan) -> AsyncIterator: + """Fetch the build into the models directory, under its real filename.""" + url = f"https://huggingface.co/{plan.repo}/resolve/main/{plan.file}" + async for step in download_gguf(url, self._models_dir / plan.file): + yield step + + async def wait_until_servable( + self, model_id: str, *, timeout: float = 30.0, interval: float = 0.5 + ) -> bool: + """Block until the runtime can actually answer for this model. + + The router learns about a new model only by restarting, which Electron + triggers when the preset changes, on a poll. Reporting the install + complete before then tells the user a model is ready while a chat + returns `model '' not found`, measured as a 400. + + Returns False on timeout rather than raising: the download did succeed, + the file is on disk, and the model becomes usable once the sidecar comes + back. A failed install would be the wrong thing to report. + """ + deadline = time.monotonic() + timeout + client = RouterClient(self._runtime_url) + while time.monotonic() < deadline: + try: + if any(m.id == model_id for m in await client.models()): + return True + except httpx.HTTPError: + pass # restarting, which is exactly what we are waiting for + await asyncio.sleep(interval) + return False + + async def search(self, query: str, *, limit: int = 30) -> list[SearchHit]: + async with httpx.AsyncClient(follow_redirects=True) as client: + return await search_models(client, query, limit=limit) + + async def repo(self, repo: str) -> dict: + """Price every build in a repo, exactly, from one header read. + + Architecture fields are identical across quantizations, measured, so one + read prices the whole ladder and the weights bytes come free from the + listing. + """ + budget = self.budget() + async with httpx.AsyncClient(follow_redirects=True) as client: + builds = await list_builds(client, repo) + if not builds: + return { + "repo": repo, + "architecture": "", + "context_length": 0, + "supported": False, + "builds": [], + "ineligible_reason": "no single file build in this repo", + } + url = f"https://huggingface.co/{repo}/resolve/main/{builds[0].file}" + shape = await shape_from_url(client, url) + + supported = is_supported(shape.architecture) + rows = [] + for build in builds: + fit = estimate(shape, build.size_bytes, budget) + text = badge(fit, budget) + rows.append( + { + "catalog_id": self._tickets.mint( + repo, build.file, build.quantization, build.size_bytes + ), + "file": build.file, + "quantization": build.quantization, + "size_bytes": build.size_bytes, + "fit": { + "state": fit.state.value, + "need_bytes": fit.need_bytes, + "budget_bytes": fit.budget_bytes, + "offload_fraction": fit.offload_fraction, + "approximate": False, + }, + "badge": {"verdict": text.verdict, "reason": text.reason}, + "can_install": fit.can_install and supported, + } + ) + return { + "repo": repo, + "architecture": shape.architecture, + "context_length": shape.context_length, + "supported": supported, + "builds": rows, + "ineligible_reason": None + if supported + else f"llama.cpp cannot run the {shape.architecture} architecture", + } + + def devices(self) -> tuple[Device, ...]: + """Probe once and remember. + + The first call compiles Metal shaders and costs about 19 seconds on a + Mac; every call after is tens of milliseconds. Warm this in the + background at launch rather than on the path of the first render. + """ + if self._devices is None: + try: + self._devices = tuple(probe_devices(self._library_dir)) + except OSError: + # No staged runtime. The catalog still renders, priced against + # the host, and installs are disabled higher up. + self._devices = () + return self._devices + + def budget(self, mode: BudgetMode = BudgetMode.CAPACITY) -> HardwareBudget: + """Capacity by default: a catalog is a shelf, not a launch decision. + + Pricing rows against whatever is free this second makes every large row + read as too big whenever a browser is open. + """ + return build_budget(self.devices(), available_bytes(), mode=mode) + + def catalog(self) -> Catalog: + budget = self.budget() + rows = curated_rows(self._manifest.models, budget) + pick = recommend(self._manifest.models, budget) + return Catalog( + budget=budget, + devices=self.devices(), + curated=tuple(rows), + installed=tuple(self._installed()), + recommended_model_id=pick.entry.model_id if pick else None, + ) + + def installed(self) -> list[InstalledRow]: + """What is on disk, which is the only inventory deletion needs.""" + return self._installed() + + def _installed(self) -> list[InstalledRow]: + """Whatever GGUF files are in the models directory. + + Disk is the source of truth: the router auto-discovers this directory, + so there is no separate inventory to keep in step with it. + """ + if not self._models_dir.exists(): + return [] + return sorted( + ( + InstalledRow(path.stem, path.name, path.stat().st_size) + for path in self._models_dir.glob("*.gguf") + ), + key=lambda row: row.model_id, + ) diff --git a/surfsense_local/backend/tests/integration/llm/test_catalog_routes.py b/surfsense_local/backend/tests/integration/llm/test_catalog_routes.py new file mode 100644 index 0000000000..0857581316 --- /dev/null +++ b/surfsense_local/backend/tests/integration/llm/test_catalog_routes.py @@ -0,0 +1,106 @@ +"""The model screen's routes. + +These replace `GET /llm/system`, `GET /llm/catalog` and `POST /llm/install` at +the same paths, so this file is also the check that the reshape landed. +""" + +import pytest +from httpx import AsyncClient + +pytestmark = [pytest.mark.integration, pytest.mark.asyncio] + + +async def test_the_catalog_renders_with_no_network_and_no_scan(client: AsyncClient) -> None: + """A clean machine sees a hardware line and badged rows on first paint.""" + reply = await client.get("/llm/catalog") + + assert reply.status_code == 200 + body = reply.json() + assert body["curated"] + assert all(row["badge"]["verdict"] for row in body["curated"]) + + +async def test_no_row_carries_a_rank_on_the_wire(client: AsyncClient) -> None: + """Asserted on the serialized response, so it cannot be reintroduced by an + accidental `from_attributes` widening.""" + body = (await client.get("/llm/catalog")).json() + + for row in body["curated"]: + assert "rank" not in row + assert "score" not in row + + +async def test_the_response_carries_no_scanned_flag(client: AsyncClient) -> None: + """There is no scan. The flag is what the frontend hung a button on.""" + body = (await client.get("/llm/catalog")).json() + + assert "scanned" not in body + + +async def test_the_offload_fraction_survives_to_the_renderer(client: AsyncClient) -> None: + """The graded reason line is selected from it, and the renderer must not + recompute what the estimator already knows.""" + body = (await client.get("/llm/catalog")).json() + + assert all("offload_fraction" in row["fit"] for row in body["curated"]) + + +async def test_the_system_route_describes_one_device_never_a_sum(client: AsyncClient) -> None: + """Summing backends reports 12 GB on a 6 GB card, wrong in the dangerous + direction: it tells a user a model fits when it cannot.""" + body = (await client.get("/llm/system")).json() + + budget = body["budget"] + assert budget["usable_vram_bytes"] <= budget["device_free_bytes"] + for device in body["devices"]: + assert budget["device_total_bytes"] <= max( + device["total_bytes"] for device in body["devices"] + ) + + +async def test_search_is_refused_until_its_destination_is_allowed( + client: AsyncClient, +) -> None: + """Typing into a search box sends that text to huggingface.co, which is a + consent of its own and separate from allowing a download.""" + reply = await client.get("/llm/search", params={"q": "qwen"}) + + assert reply.status_code == 403 + + +async def test_installing_an_unknown_id_says_the_catalog_is_stale( + client: AsyncClient, +) -> None: + """One staleness failure, not two: a searched build's ticket expires on the + same clock as the row it was minted for.""" + reply = await client.post("/llm/install", json={"catalog_id": "never-minted"}) + + assert reply.status_code == 422 + assert "stale" in reply.json()["detail"] + + +async def test_installing_is_refused_until_its_destination_is_allowed( + client: AsyncClient, +) -> None: + """Downloading a model reaches huggingface.co, and that is a consent the + user gives explicitly. Separate from search: this is a repo they named. + """ + body = (await client.get("/llm/catalog")).json() + catalog_id = body["curated"][0]["catalog_id"] + + reply = await client.post("/llm/install", json={"catalog_id": catalog_id}) + + assert reply.status_code == 403 + + +async def test_a_curated_row_can_be_installed_by_its_opaque_id( + client: AsyncClient, +) -> None: + """The renderer sends only that id: no repo, file, URL or path. A valid id + gets past resolution and fails on egress, not on being unrecognised.""" + body = (await client.get("/llm/catalog")).json() + catalog_id = body["curated"][0]["catalog_id"] + + reply = await client.post("/llm/install", json={"catalog_id": catalog_id}) + + assert reply.status_code != 422 diff --git a/surfsense_local/backend/tests/unit/llm/catalog/__init__.py b/surfsense_local/backend/tests/unit/llm/catalog/__init__.py new file mode 100644 index 0000000000..e69de29bb2 diff --git a/surfsense_local/backend/tests/unit/llm/catalog/test_manifest.py b/surfsense_local/backend/tests/unit/llm/catalog/test_manifest.py new file mode 100644 index 0000000000..2739bbce77 --- /dev/null +++ b/surfsense_local/backend/tests/unit/llm/catalog/test_manifest.py @@ -0,0 +1,154 @@ +"""The curated manifest, schema 3. + +Source, not build output: a person commits these numbers, so the schema is what +catches a mistake before review does. +""" + +import pytest +from pydantic import ValidationError + +from modules.llm.catalog import ( + SCHEMA_VERSION, + CuratedModelsManifest, + load_curated_models, +) + +pytestmark = pytest.mark.unit + + +def entry(**overrides) -> dict: + """One well formed entry, before a test breaks a specific field.""" + base = { + "model_id": "Qwen/Qwen3-8B", + "family": "Qwen3", + "label": "Qwen3 8B", + "parameter_count": "8B", + "shape": { + "architecture": "qwen3", + "block_count": 36, + "head_count_kv": 8, + "key_length": 128, + "value_length": 128, + "context_length": 40960, + "n_vocab": 151936, + }, + "capabilities": [], + "decode_fraction": 1.0, + "variants": [ + { + "repo": "unsloth/Qwen3-8B-GGUF", + "file": "Qwen3-8B-Q4_K_M.gguf", + "quantization": "Q4_K_M", + "size_bytes": 5027784512, + "rank": 78, + "rank_basis": "llmfit-1.1.11", + "validated": True, + } + ], + } + return {**base, **overrides} + + +def manifest(models: list[dict] | None = None, **overrides) -> dict: + """A whole manifest around one or more entries.""" + return {"schema_version": 3, "models": models or [entry()], **overrides} + + +def test_a_well_formed_manifest_loads() -> None: + """The baseline, so the rejection tests cannot pass by rejecting everything.""" + parsed = CuratedModelsManifest.model_validate(manifest()) + + assert parsed.models[0].variants[0].rank == 78 + assert parsed.models[0].shape.block_count == 36 + + +def test_schema_two_is_refused_rather_than_read_as_three() -> None: + """Silently reading a v2 entry would drop every shape and price nothing.""" + with pytest.raises(ValidationError, match="unsupported curated-model schema"): + CuratedModelsManifest.model_validate(manifest() | {"schema_version": 2}) + + +def test_an_entry_without_a_shape_is_refused() -> None: + """Shape is what prices a curated row offline, on first paint, with no + network. Without it the airgapped product has no badge on its only tier.""" + broken = entry() + del broken["shape"] + + with pytest.raises(ValidationError): + CuratedModelsManifest.model_validate(manifest([broken])) + + +def test_an_entry_must_ship_at_least_one_build() -> None: + """A row with nothing to install is a row that can only disappoint.""" + with pytest.raises(ValidationError): + CuratedModelsManifest.model_validate(manifest([entry(variants=[])])) + + +def test_a_rank_must_name_the_quantization_it_was_taken_at() -> None: + """Quality is a function of (model, quantization): the same model reads + 75 / 78 / 81 / 82 / 83 across its ladder. A rank beside its own build cannot + drift from the file it describes, which a top level field could not prevent. + """ + variant = dict(entry()["variants"][0]) + del variant["quantization"] + + with pytest.raises(ValidationError): + CuratedModelsManifest.model_validate(manifest([entry(variants=[variant])])) + + +def test_ranks_are_never_compared_across_different_bases() -> None: + """An llmfit number and an eval derived number are not on one scale, and + sorting them together would silently reorder the list.""" + mixed = [ + entry(), + entry( + model_id="Qwen/Qwen3-4B", + label="Qwen3 4B", + variants=[ + { + **entry()["variants"][0], + "file": "Qwen3-4B-Q4_K_M.gguf", + "rank": 63, + "rank_basis": "surfsense-eval-1", + } + ], + ), + ] + + with pytest.raises(ValidationError, match="rank_basis"): + CuratedModelsManifest.model_validate(manifest(mixed)) + + +def test_duplicate_model_ids_are_refused() -> None: + """Two rows for one model is a copy paste slip, not a choice.""" + with pytest.raises(ValidationError, match="duplicate"): + CuratedModelsManifest.model_validate(manifest([entry(), entry()])) + + +def test_two_entries_cannot_pin_the_same_file() -> None: + """Two rows installing identical bytes is a copy paste slip, not a choice.""" + twin = entry(model_id="Qwen/Qwen3-8B-Other", label="Other") # same repo + file + + with pytest.raises(ValidationError, match="duplicate"): + CuratedModelsManifest.model_validate(manifest([entry(), twin])) + + +def test_the_superseded_v2_fields_are_refused_rather_than_ignored() -> None: + """`minimum_fit` was a gate and only physics gates now. Leaving it readable + would let a stale manifest look valid while meaning something else.""" + with pytest.raises(ValidationError): + CuratedModelsManifest.model_validate(manifest([entry(minimum_fit="good")])) + + +def test_the_shipped_manifest_is_valid_and_prices_offline() -> None: + """The one that actually ships, parsed as the app parses it.""" + parsed = load_curated_models() + + assert parsed.schema_version == SCHEMA_VERSION + assert parsed.models + for model in parsed.models: + assert model.shape.block_count > 0 + assert model.variants + for variant in model.variants: + assert variant.rank > 0 + assert variant.quantization diff --git a/surfsense_local/backend/tests/unit/llm/catalog/test_recommendation.py b/surfsense_local/backend/tests/unit/llm/catalog/test_recommendation.py new file mode 100644 index 0000000000..9487466c96 --- /dev/null +++ b/surfsense_local/backend/tests/unit/llm/catalog/test_recommendation.py @@ -0,0 +1,120 @@ +"""Which curated build gets the star. + +The regression this file exists for: on a 6 GB card a residency-only rule stars +a 1.7B while the owner is happily running an 8B. Residency is a mechanism and +speed is the goal, and on small cards they disagree. +""" + +import pytest + +from modules.llm.catalog import CuratedModel, recommend +from modules.llm.fit import HardwareBudget + +pytestmark = pytest.mark.unit + +MIB = 1024**2 +GB = 1000**3 + + +def qwen(block_count: int) -> dict: + """Every shipped Qwen3 has 8 KV heads and 128 wide keys; only depth differs.""" + return { + "architecture": "qwen3", + "block_count": block_count, + "head_count_kv": 8, + "key_length": 128, + "value_length": 128, + "context_length": 40960, + "n_vocab": 151936, + } + + +def build(repo: str, file: str, quant: str, size: int, rank: int) -> dict: + """One downloadable build plus the judgement fields beside it.""" + return { + "repo": repo, + "file": file, + "quantization": quant, + "size_bytes": size, + "rank": rank, + "rank_basis": "llmfit-1.1.11", + } + + +def model(model_id: str, label: str, blocks: int, variants: list[dict]) -> CuratedModel: + """A manifest entry, validated the way the app validates it.""" + return CuratedModel.model_validate( + { + "model_id": model_id, + "family": "Qwen3", + "label": label, + "parameter_count": label.split()[-1], + "shape": qwen(blocks), + "variants": variants, + } + ) + + +CURATED = [ + model("Qwen/Qwen3-0.6B", "Qwen3 0.6B", 28, [build("u/q", "0.6b.gguf", "Q4_K_M", int(0.40 * GB), 33)]), + model("Qwen/Qwen3-1.7B", "Qwen3 1.7B", 28, [build("u/q", "1.7b.gguf", "Q4_K_M", int(1.11 * GB), 48)]), + model("Qwen/Qwen3-4B", "Qwen3 4B", 36, [build("u/q", "4b.gguf", "Q4_K_M", int(2.50 * GB), 63)]), + model("Qwen/Qwen3-8B", "Qwen3 8B", 36, [build("u/q", "8b.gguf", "Q4_K_M", int(5.03 * GB), 78)]), + model("Qwen/Qwen3-32B", "Qwen3 32B", 64, [build("u/q", "32b.gguf", "Q4_K_M", int(19.76 * GB), 92)]), +] + +# The measured machine: RTX 3050, 5234 MiB free, llama.cpp's own 1024 MiB margin. +RTX_3050 = HardwareBudget(5234 * MIB, 6002 * MIB, 1024 * MIB, 22750 * MIB, False, True) + + +def test_a_spilling_model_can_be_recommended_over_a_resident_one() -> None: + """The measured contradiction. A FITS-only rule fails this, which is the point. + + On this card the 8B spills about a fifth and runs without noticeable lag, + while a residency-only rule stars the 1.7B. + """ + pick = recommend(CURATED, RTX_3050) + + assert pick is not None + assert pick.entry.label == "Qwen3 8B" + + +def test_physics_still_refuses_whatever_the_rank() -> None: + """The ceiling relaxes residency, never physics.""" + tiny = HardwareBudget(2000 * MIB, 2000 * MIB, 1024 * MIB, 3000 * MIB, False, True) + + pick = recommend(CURATED, tiny) + + assert pick is not None + assert pick.entry.label in {"Qwen3 0.6B", "Qwen3 1.7B"} + + +def test_a_roomy_machine_takes_the_best_ranked_build() -> None: + """Where residency and speed agree, rank decides.""" + workstation = HardwareBudget( + 24000 * MIB, 24576 * MIB, 1024 * MIB, 64000 * MIB, False, True + ) + + assert recommend(CURATED, workstation).entry.label == "Qwen3 32B" + + +def test_nothing_is_recommended_when_nothing_can_run() -> None: + """No star is honest. Every non-refused build stays installable regardless.""" + postage_stamp = HardwareBudget(64 * MIB, 64 * MIB, 1024 * MIB, 64 * MIB, False, True) + + assert recommend(CURATED, postage_stamp) is None + + +def test_the_policy_ranges_over_builds_so_a_smaller_one_can_rescue_an_entry() -> None: + """A model whose big build is refused is still recommendable on its small one.""" + two_builds = [ + model("Qwen/Qwen3-8B", "Qwen3 8B", 36, [ + build("u/q", "8b-q8.gguf", "Q8_0", int(60.0 * GB), 83), + build("u/q", "8b-q4.gguf", "Q4_K_M", int(5.03 * GB), 78), + ]) + ] + + pick = recommend(two_builds, RTX_3050) + + assert pick is not None + assert pick.variant.quantization == "Q4_K_M" diff --git a/surfsense_local/backend/tests/unit/llm/catalog/test_reprice.py b/surfsense_local/backend/tests/unit/llm/catalog/test_reprice.py new file mode 100644 index 0000000000..951547ddef --- /dev/null +++ b/surfsense_local/backend/tests/unit/llm/catalog/test_reprice.py @@ -0,0 +1,151 @@ +"""Making a downloaded model reachable. + +The router reads its models directory once, at startup, and reads per-model +arguments from a preset INI once, at startup. Measured: a file dropped into a +running router's directory is still invisible 26 seconds later, and appears +immediately after a restart. + +So a completed download has to write the preset. That is what the Electron +watcher notices, and the restart it triggers is what surfaces the model. Without +this the file lands on disk, shows under Installed, and cannot be chatted with. +""" + +from pathlib import Path + +import pytest + +from modules.llm.catalog import CatalogService, load_curated_models +from modules.llm.providers.llamacpp import PRESET_FILE +from tests.unit.llm.gguf.build import STRING, UINT32, array, gguf, kv + +pytestmark = pytest.mark.unit + + +def a_model(path: Path, *, blocks: int = 28, ctx: int = 40960) -> None: + """A parseable GGUF on disk, which is all reprice needs to price one.""" + path.write_bytes( + gguf( + [ + kv("general.architecture", STRING, "qwen3"), + kv("qwen3.block_count", UINT32, blocks), + kv("qwen3.attention.head_count_kv", UINT32, 8), + kv("qwen3.attention.key_length", UINT32, 128), + kv("qwen3.attention.value_length", UINT32, 128), + kv("qwen3.context_length", UINT32, ctx), + array("tokenizer.ggml.tokens", STRING, ["a", "b"]), + ] + ) + ) + + +@pytest.fixture +def service(tmp_path: Path) -> CatalogService: + """A service over an empty models directory.""" + models = tmp_path / "models" + models.mkdir() + return CatalogService(load_curated_models(), models, tmp_path / "lib") + + +def test_a_downloaded_model_gets_a_preset_section(service, tmp_path: Path) -> None: + """The section is what the router matches a discovered file against.""" + a_model(tmp_path / "models" / "Qwen3-1.7B-Q4_K_M.gguf") + + service.reprice() + + ini = (tmp_path / "models" / PRESET_FILE).read_text() + assert "[Qwen3-1.7B-Q4_K_M]" in ini + assert "ctx-size" in ini + + +def test_the_window_comes_from_the_fit_calculation_not_a_default( + service, tmp_path: Path +) -> None: + """A model trained to 8192 must not be asked for more, and llama.cpp would + otherwise pick its own window and shrink it as far as 4096.""" + a_model(tmp_path / "models" / "small.gguf", ctx=8192) + + service.reprice() + + assert "ctx-size = 8192" in (tmp_path / "models" / PRESET_FILE).read_text() + + +def test_every_installed_model_is_priced_not_only_the_newest( + service, tmp_path: Path +) -> None: + """The file is rewritten whole, so a second install must not orphan the first.""" + a_model(tmp_path / "models" / "one.gguf") + a_model(tmp_path / "models" / "two.gguf") + + service.reprice() + + ini = (tmp_path / "models" / PRESET_FILE).read_text() + assert "[one]" in ini and "[two]" in ini + + +def test_a_partial_download_is_not_offered_to_the_runtime( + service, tmp_path: Path +) -> None: + """A `.part` file is a download in flight, not a model.""" + a_model(tmp_path / "models" / "done.gguf") + (tmp_path / "models" / "busy.gguf.part").write_bytes(b"GGUF not finished") + + service.reprice() + + ini = (tmp_path / "models" / PRESET_FILE).read_text() + assert "[done]" in ini + assert "busy" not in ini + + +def test_one_unreadable_file_does_not_cost_the_others(service, tmp_path: Path) -> None: + """A truncated or foreign file in the directory is skipped, not fatal: the + runtime would otherwise stay dead over a file nobody asked it to load.""" + a_model(tmp_path / "models" / "good.gguf") + (tmp_path / "models" / "corrupt.gguf").write_bytes(b"not a gguf at all") + + service.reprice() + + ini = (tmp_path / "models" / PRESET_FILE).read_text() + assert "[good]" in ini + assert "[corrupt]" not in ini + + +def test_repricing_an_empty_directory_writes_an_empty_preset( + service, tmp_path: Path +) -> None: + """Deleting the last model must not leave a section pointing at nothing.""" + service.reprice() + + assert (tmp_path / "models" / PRESET_FILE).read_text() == "" + + +def test_a_truncated_model_is_skipped_like_any_other_unreadable_one( + service, tmp_path: Path +) -> None: + """A file that starts with GGUF and stops early is the likelier corruption: + an interrupted copy, or a download that bypassed the `.part` path. It failed + differently from a file that was never a GGUF, and killed the whole preset. + """ + a_model(tmp_path / "models" / "whole.gguf") + (tmp_path / "models" / "cut.gguf").write_bytes(b"GGUF") + + service.reprice() + + ini = (tmp_path / "models" / PRESET_FILE).read_text() + assert "[whole]" in ini + assert "[cut]" not in ini + + +@pytest.mark.asyncio +async def test_waiting_for_the_runtime_gives_up_rather_than_hanging( + service, tmp_path: Path +) -> None: + """The install stream must finish even when the sidecar never restarts. + + A model the runtime cannot serve yet is a worse answer than a slow one, but + a progress bar that never ends is worse than both. + """ + a_model(tmp_path / "models" / "m.gguf") + + reachable = await service.wait_until_servable("m", timeout=0.3, interval=0.05) + + assert reachable is False diff --git a/surfsense_local/backend/tests/unit/llm/catalog/test_rows.py b/surfsense_local/backend/tests/unit/llm/catalog/test_rows.py new file mode 100644 index 0000000000..2935a9edc6 --- /dev/null +++ b/surfsense_local/backend/tests/unit/llm/catalog/test_rows.py @@ -0,0 +1,100 @@ +"""Turning the manifest into the rows a screen renders.""" + +import pytest + +from modules.llm.catalog import CuratedModel, curated_rows +from modules.llm.fit import FitState, HardwareBudget + +pytestmark = pytest.mark.unit + +MIB = 1024**2 +GB = 1000**3 + + +def model(label: str, blocks: int, variants: list[dict]) -> CuratedModel: + """A manifest entry, validated the way the app validates it.""" + return CuratedModel.model_validate({ + "model_id": f"Qwen/{label.replace(' ', '-')}", + "family": "Qwen3", + "label": label, + "parameter_count": label.split()[-1], + "shape": { + "architecture": "qwen3", "block_count": blocks, "head_count_kv": 8, + "key_length": 128, "value_length": 128, "context_length": 40960, + "n_vocab": 151936, + }, + "variants": variants, + }) + + +def build(file: str, quant: str, gb: float, rank: int) -> dict: + """One downloadable build plus the judgement fields beside it.""" + return { + "repo": "unsloth/q", "file": file, "quantization": quant, + "size_bytes": int(gb * GB), "rank": rank, "rank_basis": "llmfit-1.1.11", + } + + +CURATED = [ + model("Qwen3 0.6B", 28, [build("0.6b.gguf", "Q4_K_M", 0.40, 33)]), + model("Qwen3 8B", 36, [build("8b.gguf", "Q4_K_M", 5.03, 78)]), + model("Qwen3 32B", 64, [build("32b.gguf", "Q4_K_M", 19.76, 92)]), +] +SMALL = HardwareBudget(5234 * MIB, 6002 * MIB, 1024 * MIB, 8000 * MIB, False, True) + + +def test_every_row_carries_a_badge() -> None: + """Fit is subtraction, not judgement: every row has a size, so every row can + be priced. There is no `unknown` state to fall back to.""" + rows = curated_rows(CURATED, SMALL) + + assert len(rows) == len(CURATED) + assert all(row.badge.verdict for row in rows) + + +def test_rows_sort_by_fit_coarsely_then_by_rank_finely() -> None: + """Sorting by rank alone puts a TOO_BIG 32B at the top of a small machine's + screen, which is the one thing that screen must not do.""" + labels = [row.label for row in curated_rows(CURATED, SMALL)] + + assert labels.index("Qwen3 8B") < labels.index("Qwen3 32B") + + +def test_a_model_with_two_builds_is_one_row() -> None: + """The screen's job is choosing a model. Listing builds separately shows the + same model twice.""" + two = [model("Qwen3 8B", 36, [ + build("8b-q8.gguf", "Q8_0", 60.0, 83), + build("8b-q4.gguf", "Q4_K_M", 5.03, 78), + ])] + + rows = curated_rows(two, SMALL) + + assert len(rows) == 1 + assert rows[0].variant.quantization == "Q4_K_M" + + +def test_a_row_takes_the_best_state_among_its_builds() -> None: + """A model whose large build is refused is still offered on its small one.""" + two = [model("Qwen3 8B", 36, [ + build("8b-q8.gguf", "Q8_0", 60.0, 83), + build("8b-q4.gguf", "Q4_K_M", 5.03, 78), + ])] + + assert curated_rows(two, SMALL)[0].fit.state is not FitState.TOO_BIG + + +def test_only_physics_blocks_installing() -> None: + """Reduced speed installs exactly like Full speed. It runs, slower.""" + rows = {row.label: row for row in curated_rows(CURATED, SMALL)} + + assert rows["Qwen3 8B"].can_install + assert not rows["Qwen3 32B"].can_install + + +def test_no_row_carries_a_rank_where_a_renderer_could_read_it() -> None: + """Rank orders the list and selects the star. It is never displayed, and the + surest way to keep that true is not to send it.""" + row = curated_rows(CURATED, SMALL)[0] + + assert not hasattr(row, "rank") diff --git a/surfsense_local/backend/tests/unit/llm/catalog/test_search_client.py b/surfsense_local/backend/tests/unit/llm/catalog/test_search_client.py new file mode 100644 index 0000000000..90dcee3009 --- /dev/null +++ b/surfsense_local/backend/tests/unit/llm/catalog/test_search_client.py @@ -0,0 +1,91 @@ +"""Reading Hugging Face's listings.""" + +import httpx +import pytest + +from modules.llm.catalog.search import list_builds, search_models +from modules.llm.catalog.search.client import _quantization + +pytestmark = pytest.mark.unit + +HIT = { + "id": "unsloth/Qwen3-8B-GGUF", + "downloads": 412_000, + "likes": 91, + "lastModified": "2026-09-01T00:00:00.000Z", + "gated": False, + "pipeline_tag": "text-generation", + "tags": ["gguf", "license:apache-2.0", "base_model:quantized:Qwen/Qwen3-8B"], +} +TREE = [ + {"path": "Qwen3-8B-Q4_K_M.gguf", "size": 5_027_784_512}, + {"path": "Qwen3-8B-Q8_0.gguf", "size": 8_710_000_000}, + {"path": "mmproj-F16.gguf", "size": 400_000_000}, + {"path": "Qwen3-8B-Q2_K-00001-of-00003.gguf", "size": 1_000_000}, + {"path": "README.md", "size": 900}, +] + + +def serving(payload): + """A Hugging Face API returning one fixed payload.""" + return httpx.MockTransport(lambda request: httpx.Response(200, json=payload)) + + +@pytest.mark.asyncio +async def test_a_search_describes_a_repo_without_grading_it() -> None: + """Search rows are described, not judged: no rank, ever.""" + async with httpx.AsyncClient(transport=serving([HIT])) as client: + hits = await search_models(client, "qwen3") + + assert hits[0].repo == "unsloth/Qwen3-8B-GGUF" + assert hits[0].downloads == 412_000 + assert not hasattr(hits[0], "rank") + + +@pytest.mark.asyncio +async def test_provenance_is_surfaced_because_it_is_free_and_useful() -> None: + """It lets someone recognise an unfamiliar repo as a repackaging of a model + they know, which is most of what a score was doing for them.""" + async with httpx.AsyncClient(transport=serving([HIT])) as client: + hits = await search_models(client, "qwen3") + + assert hits[0].quantized_from == "Qwen/Qwen3-8B" + assert hits[0].license == "apache-2.0" + + +@pytest.mark.asyncio +async def test_only_single_file_builds_are_offered() -> None: + """A projector, a draft model and one part of a split set are not things a + user installs on their own.""" + async with httpx.AsyncClient(transport=serving(TREE)) as client: + builds = await list_builds(client, "unsloth/Qwen3-8B-GGUF") + + assert [b.file for b in builds] == ["Qwen3-8B-Q4_K_M.gguf", "Qwen3-8B-Q8_0.gguf"] + + +@pytest.mark.asyncio +async def test_builds_carry_the_quantization_out_of_their_filename() -> None: + """Which is where uploaders put it, there being nowhere else to look.""" + async with httpx.AsyncClient(transport=serving(TREE)) as client: + builds = await list_builds(client, "unsloth/Qwen3-8B-GGUF") + + assert [b.quantization for b in builds] == ["Q4_K_M", "Q8_0"] + + +@pytest.mark.parametrize( + ("file", "expected"), + [ + ("Qwen3-8B-Q4_K_M.gguf", "Q4_K_M"), + ("Qwen3-8B-Q8_0.gguf", "Q8_0"), + ("Qwen3-Coder-30B-A3B-Instruct-UD-TQ1_0.gguf", "TQ1_0"), + ("Qwen3-Coder-30B-A3B-Instruct-UD-IQ2_XXS.gguf", "IQ2_XXS"), + ("gemma-3-4b-it-BF16.gguf", "BF16"), + ("SmolLM2-135M-Instruct-F16.gguf", "F16"), + ("Qwen3-8B-UD-Q4_K_XL.gguf", "Q4_K_XL"), + ], +) +def test_the_quantization_is_read_and_not_the_model_name(file, expected) -> None: + """Found live: `Qwen3-Coder-...-UD-TQ1_0.gguf` parsed as "QWEN3", because a + model name can begin with Q and contain a digit just as a quant name does. + """ + assert _quantization(file) == expected diff --git a/surfsense_local/backend/tests/unit/llm/catalog/test_search_gates.py b/surfsense_local/backend/tests/unit/llm/catalog/test_search_gates.py new file mode 100644 index 0000000000..71e79e6a28 --- /dev/null +++ b/surfsense_local/backend/tests/unit/llm/catalog/test_search_gates.py @@ -0,0 +1,49 @@ +"""What a searched model has to satisfy before it can be installed. + +Eligibility is not fit: a model can be FITS and still gated, and the two must +never render as the same thing. +""" + +import pytest + +from modules.llm.catalog.search import TicketStore, is_supported + +pytestmark = pytest.mark.unit + + +def test_a_supported_architecture_passes() -> None: + """The 152 name list, matched without caring about case.""" + assert is_supported("qwen3") + assert is_supported("GEMMA3") + + +def test_an_architecture_llama_cpp_cannot_run_is_refused() -> None: + """No amount of memory helps, so this blocks rather than badges.""" + assert not is_supported("whisper") + assert not is_supported("something-invented") + + +def test_a_ticket_resolves_to_the_build_the_server_priced() -> None: + """The renderer never names a repo or a file, so the server has to.""" + store = TicketStore() + + token = store.mint("unsloth/Qwen3-8B-GGUF", "Qwen3-8B-Q4_K_M.gguf", "Q4_K_M", 5_000, now=0.0) + ticket = store.resolve(token, now=1.0) + + assert ticket is not None + assert ticket.repo == "unsloth/Qwen3-8B-GGUF" + assert ticket.file == "Qwen3-8B-Q4_K_M.gguf" + + +def test_a_ticket_older_than_the_search_cache_is_gone() -> None: + """A ticket outliving its row would install something the user stopped + looking at. The route reports this as the existing stale id failure.""" + store = TicketStore(ttl_seconds=300) + token = store.mint("r", "f.gguf", "Q4_K_M", 1, now=0.0) + + assert store.resolve(token, now=301.0) is None + + +def test_an_unknown_id_is_simply_absent() -> None: + """Nothing was minted, so nothing resolves. Same answer as expiry.""" + assert TicketStore().resolve("never-minted") is None diff --git a/surfsense_local/backend/tests/unit/llm/catalog/test_service.py b/surfsense_local/backend/tests/unit/llm/catalog/test_service.py new file mode 100644 index 0000000000..c10888d807 --- /dev/null +++ b/surfsense_local/backend/tests/unit/llm/catalog/test_service.py @@ -0,0 +1,62 @@ +"""The screen's data, assembled without a network call.""" + +from pathlib import Path + +import pytest + +from modules.llm.catalog import CatalogService, load_curated_models + +pytestmark = pytest.mark.unit + + +@pytest.fixture +def service(tmp_path: Path) -> CatalogService: + """The shipped manifest against an empty models directory.""" + return CatalogService(load_curated_models(), tmp_path / "models", tmp_path / "lib") + + +def test_the_catalog_renders_before_anything_is_installed(service) -> None: + """A clean machine sees badged rows on first paint, with no button to press + and nothing downloaded.""" + catalog = service.catalog() + + assert catalog.curated + assert all(row.badge.verdict for row in catalog.curated) + assert catalog.installed == () + + +def test_a_missing_runtime_does_not_stop_the_catalog(service) -> None: + """Installs disable and the screen still explains itself. The library path + here does not exist, which is what a broken or unstaged install looks like.""" + catalog = service.catalog() + + assert catalog.devices == () + assert catalog.curated + + +def test_installed_models_are_read_from_disk(tmp_path: Path) -> None: + """The router auto-discovers the directory, so disk is the inventory.""" + models = tmp_path / "models" + models.mkdir() + (models / "Qwen3-8B-Q4_K_M.gguf").write_bytes(b"GGUF" + b"0" * 100) + + catalog = CatalogService(load_curated_models(), models, tmp_path / "lib").catalog() + + assert [row.model_id for row in catalog.installed] == ["Qwen3-8B-Q4_K_M"] + + +def test_the_recommendation_names_a_curated_model_or_nothing(service) -> None: + """Never a searched model: recommending requires a rank, and only the + manifest carries one.""" + catalog = service.catalog() + ids = {row.model_id for row in catalog.curated} + + assert catalog.recommended_model_id is None or catalog.recommended_model_id in ids + + +def test_the_probe_runs_once_however_often_the_catalog_is_asked(service) -> None: + """A cold probe compiles Metal shaders and costs about 19 seconds.""" + service.catalog() + first = service.devices() + + assert service.devices() is first From 9ffe32deefa41f947eb520cbe64ec5a8359b2253 Mon Sep 17 00:00:00 2001 From: Anish Sarkar <104695310+AnishSarkar22@users.noreply.github.com> Date: Sun, 20 Sep 2026 20:07:43 +0530 Subject: [PATCH 19/60] refactor(llm): remove the Ollama provider and the llmfit recommendation module Co-Authored-By: Claude Sonnet 5 --- .../modules/llm/providers/ollama/catalog.py | 23 - .../modules/llm/providers/ollama/provider.py | 259 -------- .../modules/llm/recommendations/__init__.py | 1 - .../modules/llm/recommendations/catalog.py | 607 ------------------ .../llm/recommendations/curated-models.json | 98 --- .../llm/recommendations/curated_models.py | 70 -- .../llm/recommendations/dependencies.py | 58 -- .../modules/llm/recommendations/llmfit.py | 475 -------------- .../modules/llm/recommendations/protocols.py | 32 - .../modules/llm/recommendations/router.py | 166 ----- .../modules/llm/recommendations/types.py | 141 ---- .../backend/tests/fixtures/llmfit/fit.json | 49 -- .../backend/tests/fixtures/llmfit/system.json | 14 - .../unit/llm/recommendations/test_catalog.py | 442 ------------- .../recommendations/test_curated_models.py | 82 --- .../unit/llm/recommendations/test_llmfit.py | 410 ------------ .../recommendations/test_ollama_runtime.py | 107 --- .../backend/tests/unit/llm/test_catalog.py | 13 - .../tests/unit/llm/test_ollama_num_ctx.py | 22 - 19 files changed, 3069 deletions(-) delete mode 100644 surfsense_local/backend/modules/llm/providers/ollama/catalog.py delete mode 100644 surfsense_local/backend/modules/llm/providers/ollama/provider.py delete mode 100644 surfsense_local/backend/modules/llm/recommendations/__init__.py delete mode 100644 surfsense_local/backend/modules/llm/recommendations/catalog.py delete mode 100644 surfsense_local/backend/modules/llm/recommendations/curated-models.json delete mode 100644 surfsense_local/backend/modules/llm/recommendations/curated_models.py delete mode 100644 surfsense_local/backend/modules/llm/recommendations/dependencies.py delete mode 100644 surfsense_local/backend/modules/llm/recommendations/llmfit.py delete mode 100644 surfsense_local/backend/modules/llm/recommendations/protocols.py delete mode 100644 surfsense_local/backend/modules/llm/recommendations/router.py delete mode 100644 surfsense_local/backend/modules/llm/recommendations/types.py delete mode 100644 surfsense_local/backend/tests/fixtures/llmfit/fit.json delete mode 100644 surfsense_local/backend/tests/fixtures/llmfit/system.json delete mode 100644 surfsense_local/backend/tests/unit/llm/recommendations/test_catalog.py delete mode 100644 surfsense_local/backend/tests/unit/llm/recommendations/test_curated_models.py delete mode 100644 surfsense_local/backend/tests/unit/llm/recommendations/test_llmfit.py delete mode 100644 surfsense_local/backend/tests/unit/llm/recommendations/test_ollama_runtime.py delete mode 100644 surfsense_local/backend/tests/unit/llm/test_catalog.py delete mode 100644 surfsense_local/backend/tests/unit/llm/test_ollama_num_ctx.py diff --git a/surfsense_local/backend/modules/llm/providers/ollama/catalog.py b/surfsense_local/backend/modules/llm/providers/ollama/catalog.py deleted file mode 100644 index 2f4ce3683c..0000000000 --- a/surfsense_local/backend/modules/llm/providers/ollama/catalog.py +++ /dev/null @@ -1,23 +0,0 @@ -from typing import NamedTuple - - -class Offering(NamedTuple): - """A model this provider knows how to pull, and its download size.""" - - name: str - label: str - size_gb: float - - -# Ollama has no API that lists its library, so the models offered are curated. -# Qwen family for v1; sizes are the Q4_K_M builds Ollama serves by default. -OFFERINGS: tuple[Offering, ...] = ( - Offering("qwen3:0.6b", "Qwen3 0.6B", 0.5), - Offering("qwen3:1.7b", "Qwen3 1.7B", 1.4), - Offering("qwen3:4b", "Qwen3 4B", 2.6), - Offering("qwen3:8b", "Qwen3 8B", 5.2), - Offering("qwen3:14b", "Qwen3 14B", 9.3), - Offering("qwen3:32b", "Qwen3 32B", 20.0), - Offering("qwen2.5-coder:7b", "Qwen2.5 Coder 7B", 4.7), - Offering("qwen2.5-coder:32b", "Qwen2.5 Coder 32B", 20.0), -) diff --git a/surfsense_local/backend/modules/llm/providers/ollama/provider.py b/surfsense_local/backend/modules/llm/providers/ollama/provider.py deleted file mode 100644 index b02cc2b295..0000000000 --- a/surfsense_local/backend/modules/llm/providers/ollama/provider.py +++ /dev/null @@ -1,259 +0,0 @@ -import asyncio -import contextlib -import json -from collections.abc import AsyncIterator - -import httpx - -from modules.llm.profile import Fingerprint, from_ollama -from modules.llm.providers.ollama.catalog import OFFERINGS -from modules.llm.providers.types import CatalogEntry, DownloadProgress, Message, Model -from modules.llm.recommendations.types import ( - InstalledModel, - InstallPlan, - ScoredModel, -) -from shared.config import get_llm_settings - -# A pull runs for minutes; a tag lookup is instant. Long read, short connect. -TIMEOUT = httpx.Timeout(600.0, connect=5.0) - -# Ollama's default context window is 4096 tokens and it silently drops whatever -# does not fit, so a 24k-char Studio grounding reached the model at a third of its -# length (measured: 2050 of ~6900 tokens evaluated). Size the window to the prompt. -MIN_CTX = 4096 - - -def num_ctx(messages: list[Message], max_tokens: int | None) -> int: - """The smallest power-of-two window that holds the prompt and the reply. - - ~3 chars per token is conservative for prose; Ollama clamps anything above the - model's trained maximum, so overshooting is safe and undershooting is not. - """ - needed = sum(len(m.content) for m in messages) // 3 + (max_tokens or 2048) - return max(MIN_CTX, 1 << (needed - 1).bit_length()) - - -class OllamaProvider: - """Ollama over its native API: it both answers and holds models on disk.""" - - name = "ollama" - requires_key = False - - def __init__(self, base_url: str) -> None: - self._base_url = base_url.rstrip("/") - - def _client(self) -> httpx.AsyncClient: - return httpx.AsyncClient(base_url=self._base_url, timeout=TIMEOUT) - - async def health(self) -> bool: - try: - async with self._client() as client: - return (await client.get("/")).status_code == 200 - except httpx.HTTPError: - return False - - async def models(self) -> list[Model]: - return [ - Model(model.model_name, installed=True, capabilities=model.capabilities) - for model in await self.installed_models() - ] - - async def installed_models(self) -> list[InstalledModel]: - async with self._client() as client: - reply = await client.get("/api/tags") - reply.raise_for_status() - names = [entry["name"] for entry in reply.json().get("models", [])] - details = await asyncio.gather( - *(self._details(client, name) for name in names) - ) - - return [ - InstalledModel( - runtime=self.name, - model_name=name, - capabilities=caps, - quantization=quantization, - ) - for name, (caps, quantization) in zip(names, details, strict=True) - ] - - async def resolve(self, model: ScoredModel) -> InstallPlan | None: - if not model.ollama_name: - return None - expected_bytes = ( - round(model.disk_size_gb * 1_000_000_000) - if model.disk_size_gb is not None - else None - ) - quantization = model.ollama_quantization or ( - model.best_quant - if model.best_quant - and model.best_quant.lower() in model.ollama_name.lower() - else None - ) - return InstallPlan( - canonical_id=model.canonical_id, - runtime=self.name, - model_name=model.ollama_name, - expected_bytes=expected_bytes, - quantization=quantization, - ) - - def install(self, plan: InstallPlan) -> AsyncIterator[DownloadProgress]: - if plan.runtime != self.name: - raise ValueError(f"install plan belongs to {plan.runtime}, not {self.name}") - return self.pull(plan.model_name) - - async def cleanup_cancelled_download(self) -> None: - models_dir = get_llm_settings().ollama_models_dir - if models_dir is None: - return - - blobs_dir = models_dir / "blobs" - patterns = ("*-partial", "*-partial-*", "*.tmp") - referenced: set[str] | None = set() - try: - for path in (models_dir / "manifests").rglob("*"): - if not path.is_file(): - continue - manifest = json.loads(path.read_text()) - entries = [manifest["config"], *manifest.get("layers", [])] - referenced.update( - entry["digest"].replace(":", "-", 1) for entry in entries - ) - except (OSError, KeyError, TypeError, json.JSONDecodeError): - # A bad manifest must not risk deleting a layer an installed model uses. - referenced = None - - def cancelled_files(): - files = {path for pattern in patterns for path in blobs_dir.glob(pattern)} - if referenced is not None: - files.update( - path - for path in blobs_dir.glob("sha256-*") - if path.name not in referenced - ) - return files - - # Ollama closes its writers asynchronously after the pull disconnects. - # Retry briefly for Windows, where an open file cannot be unlinked. - for attempt in range(5): - pending = cancelled_files() - if not pending: - return - for path in pending: - with contextlib.suppress(PermissionError): - path.unlink(missing_ok=True) - if attempt < 4: - await asyncio.sleep(0.1 * (attempt + 1)) - - remaining = cancelled_files() - if remaining: - raise OSError("Ollama still has cancelled download files open") - - def catalog(self) -> list[CatalogEntry]: - return [ - CatalogEntry( - name=offering.name, - label=offering.label, - size_gb=offering.size_gb, - ) - for offering in OFFERINGS - ] - - async def pull(self, name: str) -> AsyncIterator[DownloadProgress]: - body = {"model": name, "stream": True} - async with ( - self._client() as client, - client.stream("POST", "/api/pull", json=body) as reply, - ): - reply.raise_for_status() - async for line in reply.aiter_lines(): - if line: - yield _progress(json.loads(line)) - - async def delete(self, name: str) -> None: - async with self._client() as client: - reply = await client.request("DELETE", "/api/delete", json={"model": name}) - reply.raise_for_status() - - async def chat( - self, - model: str, - messages: list[Message], - *, - max_tokens: int | None = None, - temperature: float | None = None, - reasoning: bool | None = None, - ) -> AsyncIterator[str]: - body = { - "model": model, - "messages": [{"role": m.role, "content": m.content} for m in messages], - "stream": True, - } - if reasoning is not None: - body["think"] = reasoning - options = { - key: value - for key, value in ( - ("num_predict", max_tokens), - ("temperature", temperature), - ) - if value is not None - } - body["options"] = {**options, "num_ctx": num_ctx(messages, max_tokens)} - async with ( - self._client() as client, - client.stream("POST", "/api/chat", json=body) as reply, - ): - reply.raise_for_status() - async for line in reply.aiter_lines(): - if not line: - continue - delta = json.loads(line).get("message", {}).get("content") - if delta: - yield delta - - async def inspect(self, name: str) -> Fingerprint: - """What this runtime knows about an installed model, for prompt tiering.""" - async with self._client() as client: - tag, show = await asyncio.gather( - self._tag(client, name), self._show(client, name) - ) - return from_ollama(name, tag, show) - - async def _capabilities( - self, client: httpx.AsyncClient, name: str - ) -> tuple[str, ...]: - capabilities, _ = await self._details(client, name) - return capabilities - - async def _details( - self, client: httpx.AsyncClient, name: str - ) -> tuple[tuple[str, ...], str | None]: - body = await self._show(client, name) - details = body.get("details", {}) - quantization = ( - details.get("quantization_level") if isinstance(details, dict) else None - ) - return tuple(body.get("capabilities", [])), quantization - - async def _show(self, client: httpx.AsyncClient, name: str) -> dict: - reply = await client.post("/api/show", json={"model": name}) - return reply.json() if reply.status_code == 200 else {} - - async def _tag(self, client: httpx.AsyncClient, name: str) -> dict: - reply = await client.get("/api/tags") - if reply.status_code != 200: - return {} - rows = reply.json().get("models", []) - return next((row for row in rows if row.get("name") == name), {}) - - -def _progress(event: dict) -> DownloadProgress: - return DownloadProgress( - status=event.get("status", ""), - completed=event.get("completed", 0), - total=event.get("total", 0), - ) diff --git a/surfsense_local/backend/modules/llm/recommendations/__init__.py b/surfsense_local/backend/modules/llm/recommendations/__init__.py deleted file mode 100644 index 51a7e12f56..0000000000 --- a/surfsense_local/backend/modules/llm/recommendations/__init__.py +++ /dev/null @@ -1 +0,0 @@ -"""Hardware-aware generation model recommendations.""" diff --git a/surfsense_local/backend/modules/llm/recommendations/catalog.py b/surfsense_local/backend/modules/llm/recommendations/catalog.py deleted file mode 100644 index b6ab80ce56..0000000000 --- a/surfsense_local/backend/modules/llm/recommendations/catalog.py +++ /dev/null @@ -1,607 +0,0 @@ -import asyncio -import logging -import secrets -import shutil -from dataclasses import replace -from pathlib import Path - -from modules.llm.recommendations.curated_models import ( - CuratedModel, - CuratedModelsManifest, -) -from modules.llm.recommendations.protocols import LocalRuntime, ModelAdvisor -from modules.llm.recommendations.types import ( - AdvisorCatalog, - CatalogResult, - CatalogRow, - FitLevel, - InstalledModel, - InstallPlan, - RecommendationWarning, - ScoredModel, -) - -FIT_ORDER = { - FitLevel.PERFECT: 0, - FitLevel.GOOD: 1, - FitLevel.MARGINAL: 2, - FitLevel.TOO_TIGHT: 3, - FitLevel.UNKNOWN: 4, -} -LOGGER = logging.getLogger(__name__) - - -class UnknownCatalogIdError(ValueError): - pass - - -class RuntimeBusyError(RuntimeError): - pass - - -class InsufficientDiskError(RuntimeError): - def __init__(self, required: int, available: int) -> None: - super().__init__("insufficient disk space") - self.required = required - self.available = available - - -class CatalogService: - def __init__( - self, - advisor: ModelAdvisor, - runtimes: list[LocalRuntime], - curated_models: CuratedModelsManifest, - *, - max_context: int, - reserve_gb: float, - runtime_storage: dict[str, Path] | None = None, - initial_warnings: tuple[RecommendationWarning, ...] = (), - ) -> None: - self._advisor = advisor - self._runtimes = {runtime.name: runtime for runtime in runtimes} - self._curated_models = { - model.model_id: model for model in curated_models.models - } - self._max_context = max_context - self._reserve_gb = reserve_gb - self._runtime_storage = runtime_storage or {} - self._initial_warnings = initial_warnings - self._scan: AdvisorCatalog | None = None - self._scan_lock = asyncio.Lock() - # Catalog ids are stable for the life of this service (effectively the - # process lifetime, since it's a singleton), including across a - # refresh: they key on (canonical_id, runtime), not on scan state, so - # a scan-free placeholder row and its later scanned counterpart get - # the exact same id — the frontend updates the row in place instead - # of tearing it down and mounting a new one. - self._ids: dict[tuple[str, str], str] = {} - self._plans: dict[str, tuple[LocalRuntime, ScoredModel, InstallPlan]] = {} - self._logged_collision_keys: set[tuple[str, str]] = set() - self._install_locks = {runtime.name: asyncio.Lock() for runtime in runtimes} - - async def advisor_catalog(self, *, refresh: bool = False) -> AdvisorCatalog: - async with self._scan_lock: - if refresh: - self._scan = None - self._ids.clear() - self._plans.clear() - self._logged_collision_keys.clear() - if self._scan is None or refresh: - self._scan = await self._advisor.scan( - self._max_context, refresh=refresh - ) - return self._scan - - async def catalog( - self, - *, - selected: tuple[str, str] | None, - refresh: bool = False, - ) -> CatalogResult: - scan = await self.advisor_catalog(refresh=refresh) - health_results = await asyncio.gather( - *(runtime.health() for runtime in self._runtimes.values()), - return_exceptions=True, - ) - runtime_status = dict( - zip( - self._runtimes, - ( - result if isinstance(result, bool) else False - for result in health_results - ), - strict=True, - ) - ) - inventories = await asyncio.gather( - *(runtime.installed_models() for runtime in self._runtimes.values()), - return_exceptions=True, - ) - installed_by_key: dict[tuple[str, str], InstalledModel] = {} - warnings = [*self._initial_warnings, *scan.warnings] - warnings.extend( - RecommendationWarning( - "runtime_health_failed", - f"Could not check {name}.", - ) - for name, result in zip(self._runtimes, health_results, strict=True) - if isinstance(result, BaseException) - ) - for (runtime_name, _runtime), inventory in zip( - self._runtimes.items(), inventories, strict=True - ): - if isinstance(inventory, BaseException): - warnings.append( - RecommendationWarning( - "runtime_inventory_failed", - f"Could not read installed models from {runtime_name}.", - ) - ) - continue - installed_by_key.update( - {(model.runtime, model.model_name): model for model in inventory} - ) - - curated: list[CatalogRow] = [] - explore: list[CatalogRow] = [] - installed: list[CatalogRow] = [] - matched_installed: set[tuple[str, str]] = set() - matched_curated: set[str] = set() - resolved_by_key: dict[ - tuple[str, str], - tuple[LocalRuntime, ScoredModel, InstallPlan, CuratedModel | None], - ] = {} - ambiguous_keys: set[tuple[str, str]] = set() - collisions_by_key: dict[ - tuple[str, str], - list[tuple[LocalRuntime, ScoredModel, InstallPlan, CuratedModel | None]], - ] = {} - self._plans.clear() - - for raw_model in scan.models: - if _is_embedding(raw_model): - continue - model, curated_model = self._apply_curated_model(raw_model) - model = _apply_reserve(model, self._reserve_gb) - resolved = await self._resolve(model) - if resolved is None: - continue - runtime, plan = resolved - key = (runtime.name, plan.model_name) - candidate = (runtime, model, plan, curated_model) - existing = resolved_by_key.get(key) - if existing is not None and existing[1].canonical_id != model.canonical_id: - collisions_by_key.setdefault(key, [existing]).append(candidate) - elif key in collisions_by_key and all( - item[1].canonical_id != model.canonical_id - for item in collisions_by_key[key] - ): - collisions_by_key[key].append(candidate) - if curated_model is not None: - if existing is not None and existing[3] is not None: - if existing[1].canonical_id == model.canonical_id: - continue - resolved_by_key.pop(key) - ambiguous_keys.add(key) - continue - ambiguous_keys.discard(key) - resolved_by_key[key] = candidate - continue - if key in ambiguous_keys or ( - existing is not None and existing[3] is not None - ): - continue - if existing is None: - resolved_by_key[key] = candidate - continue - if existing[1].canonical_id != model.canonical_id: - resolved_by_key.pop(key) - ambiguous_keys.add(key) - - for key in tuple(ambiguous_keys): - candidates = collisions_by_key[key] - if any(candidate[3] is not None for candidate in candidates): - continue - runtime = candidates[0][0] - model = _runtime_target_model(key, candidates) - plan = await runtime.resolve(model) - if plan is None or (plan.runtime, plan.model_name) != key: - continue - resolved_by_key[key] = (runtime, model, plan, None) - ambiguous_keys.remove(key) - - for key, candidates in collisions_by_key.items(): - if key in self._logged_collision_keys: - continue - self._logged_collision_keys.add(key) - canonical_ids = {candidate[1].canonical_id for candidate in candidates} - resolved = resolved_by_key.get(key) - if key in ambiguous_keys or resolved is None: - LOGGER.warning( - "Hiding ambiguous models %s for target %s:%s", - ", ".join(sorted(canonical_ids)), - *key, - ) - elif resolved[3] is not None: - LOGGER.warning( - "Keeping curated model %s over %s for target %s:%s", - resolved[1].canonical_id, - ", ".join(sorted(canonical_ids - {resolved[1].canonical_id})), - *key, - ) - else: - LOGGER.warning( - "Representing ambiguous models %s as runtime target %s:%s", - ", ".join(sorted(canonical_ids)), - *key, - ) - - for key, (runtime, model, plan, curated_model) in resolved_by_key.items(): - is_installed = key in installed_by_key - row = self._row( - model, - plan, - installed=is_installed, - selected=selected == key, - can_install=runtime_status.get(runtime.name, False), - ) - self._plans[row.catalog_id] = (runtime, model, plan) - if curated_model is not None: - matched_curated.add(curated_model.model_id) - if is_installed: - installed.append(row) - matched_installed.add(key) - elif curated_model is not None: - # A curated model is always shown in its own section, at - # whatever fit the scan found — never merged into "More - # models". A poor fit isn't hidden, it's just an honest badge - # (and the existing marginal/too-tight install confirmation - # already gates acting on it). - curated.append(row) - elif model.fit in { - FitLevel.PERFECT, - FitLevel.GOOD, - FitLevel.MARGINAL, - }: - explore.append(row) - - for key, local_model in installed_by_key.items(): - if key in matched_installed or "completion" not in local_model.capabilities: - continue - display_name = clean_runtime_name(key[1]) - installed.append( - self._placeholder_row( - canonical_id=f"{key[0]}:{key[1]}", - family=display_name.split(":", 1)[0], - label=display_name, - parameter_count=None, - runtime=key[0], - runtime_model=key[1], - quantization=local_model.quantization, - disk_size_gb=None, - installed=True, - selected=selected == key, - can_delete=True, - warning="No current llmfit estimate is available.", - ) - ) - matched_installed.add(key) - - for model_id, curated_model in self._curated_models.items(): - if model_id in matched_curated: - continue - ollama = curated_model.artifacts.ollama - if ollama is None: - continue - target = ("ollama", ollama.name) - if target in installed_by_key or target in matched_installed: - # Already represented in `installed` above (matched either - # through a real scan or the inventory fallback) — showing it - # again here would duplicate the row. - continue - row = self._placeholder_row( - canonical_id=model_id, - family=curated_model.family, - label=curated_model.label, - parameter_count=curated_model.parameter_count, - runtime="ollama", - runtime_model=ollama.name, - quantization=ollama.quantization, - disk_size_gb=curated_model.size_bytes / 1_000_000_000, - installed=False, - selected=selected == target, - can_delete=False, - ) - # A placeholder is still installable — resolving it needs only - # `ollama_name`, which the manifest already pins, no scan - # required. Register a real plan under the row's id so `/install` - # works exactly as it would for a scanned row. - runtime = self._runtimes.get("ollama") - if runtime is not None: - synthetic = _synthetic_curated_model(model_id, curated_model) - plan = await runtime.resolve(synthetic) - if plan is not None: - self._plans[row.catalog_id] = (runtime, synthetic, plan) - curated.append(row) - - return CatalogResult( - hardware=scan.system, - llmfit_version=scan.llmfit_version, - curated=tuple(sorted(curated, key=_sort_key)), - explore=tuple(sorted(explore, key=_sort_key)), - installed=tuple(sorted(installed, key=_sort_key)), - scanned=scan.scanned, - warnings=tuple(warnings), - runtime_status=runtime_status, - ) - - async def preflight( - self, catalog_id: str - ) -> tuple[LocalRuntime, ScoredModel, InstallPlan]: - found = self._plans.get(catalog_id) - if found is None: - raise UnknownCatalogIdError(catalog_id) - runtime, model, old_plan = found - plan = await runtime.resolve(model) - if plan is None or plan != old_plan: - raise UnknownCatalogIdError(catalog_id) - if not await runtime.health(): - raise RuntimeError(f"{runtime.name} is unavailable") - storage = self._runtime_storage.get(runtime.name) - if storage is not None and plan.expected_bytes is not None: - available = shutil.disk_usage(storage).free - if available < plan.expected_bytes: - raise InsufficientDiskError(plan.expected_bytes, available) - return runtime, model, plan - - def install_lock(self, runtime: str) -> asyncio.Lock: - return self._install_locks[runtime] - - def _apply_curated_model( - self, model: ScoredModel - ) -> tuple[ScoredModel, CuratedModel | None]: - curated_model = self._curated_models.get(model.canonical_id) - if curated_model is None: - return model, None - ollama = curated_model.artifacts.ollama - return ( - replace( - model, - family=curated_model.family, - ollama_name=ollama.name if ollama is not None else model.ollama_name, - ollama_quantization=( - ollama.quantization if ollama is not None else None - ), - # The curated pin always installs a fixed quantization, - # regardless of what llmfit itself would recommend for this - # hardware — so its size must come from the manifest too, not - # from the scan. Otherwise the badge would describe a - # different file than the one that actually gets installed. - disk_size_gb=curated_model.size_bytes / 1_000_000_000, - ), - curated_model, - ) - - async def _resolve( - self, model: ScoredModel - ) -> tuple[LocalRuntime, InstallPlan] | None: - for runtime in self._runtimes.values(): - plan = await runtime.resolve(model) - if plan is not None: - return runtime, plan - return None - - def _row( - self, - model: ScoredModel, - plan: InstallPlan, - *, - installed: bool, - selected: bool, - can_install: bool, - ) -> CatalogRow: - warnings = list(model.notes) - if model.fit is FitLevel.MARGINAL: - warnings.append("This model may be slow or fail at long context.") - if model.fit is FitLevel.TOO_TIGHT: - warnings.append("This model is too large for the available memory.") - return CatalogRow( - catalog_id=self._id(model.canonical_id, plan.runtime), - canonical_id=model.canonical_id, - family=model.family, - label=model.display_name, - publisher=model.publisher, - parameter_count=model.parameter_count, - fit=model.fit, - score=model.score, - memory_required_gb=model.memory_required_gb, - disk_size_gb=model.disk_size_gb, - estimated_tps=model.estimated_tps, - prefill_tps=model.prefill_tps, - ttft_ms=model.ttft_ms, - effective_context_length=model.effective_context_length, - estimate_confidence=model.estimate_confidence, - license=model.license, - runtime=plan.runtime, - runtime_model=plan.model_name, - quantization=plan.quantization, - installed=installed, - selected=selected, - can_install=can_install and model.fit is not FitLevel.TOO_TIGHT, - can_delete=installed, - warnings=tuple(warnings), - ) - - def _placeholder_row( - self, - *, - canonical_id: str, - family: str, - label: str, - parameter_count: str | None, - runtime: str, - runtime_model: str, - quantization: str | None, - disk_size_gb: float | None, - installed: bool, - selected: bool, - can_delete: bool, - warning: str | None = None, - ) -> CatalogRow: - """A row built without a `ScoredModel` at all — no fit/score/memory - data exists yet, either because nothing has been scanned, or (for the - installed-but-unscanned case) llmfit just didn't return this exact - model. Shared by both fallback paths in `catalog()` so there is one - place, not two, that knows what an unscored row looks like.""" - return CatalogRow( - catalog_id=self._id(canonical_id, runtime), - canonical_id=canonical_id, - family=family, - label=label, - publisher=None, - parameter_count=parameter_count, - fit=FitLevel.UNKNOWN, - score=None, - memory_required_gb=None, - disk_size_gb=disk_size_gb, - estimated_tps=None, - prefill_tps=None, - ttft_ms=None, - effective_context_length=None, - estimate_confidence=None, - license=None, - runtime=runtime, - runtime_model=runtime_model, - quantization=quantization, - installed=installed, - selected=selected, - can_install=not installed, - can_delete=can_delete, - warnings=(warning,) if warning else (), - ) - - def _id(self, canonical_id: str, runtime: str) -> str: - key = (canonical_id, runtime) - if key not in self._ids: - self._ids[key] = secrets.token_urlsafe(18) - return self._ids[key] - - -def _runtime_target_model( - key: tuple[str, str], - candidates: list[ - tuple[LocalRuntime, ScoredModel, InstallPlan, CuratedModel | None] - ], -) -> ScoredModel: - source = max( - (candidate[1] for candidate in candidates), - key=lambda model: ( - FIT_ORDER[model.fit], - model.memory_required_gb or 0, - model.canonical_id, - ), - ) - runtime, runtime_model = key - return replace( - source, - canonical_id=f"{runtime}:{runtime_model}", - family=runtime_model.split(":", 1)[0], - display_name=runtime_model, - publisher=None, - parameter_count=None, - params_b=None, - use_case=None, - score=None, - capability_ids=(), - license=None, - notes=( - *source.notes, - "Multiple catalog entries map to this exact runtime model.", - ), - ollama_name=runtime_model if runtime == "ollama" else None, - ) - - -def _synthetic_curated_model( - model_id: str, curated_model: CuratedModel -) -> ScoredModel: - """Just enough of a `ScoredModel` to resolve an install plan pre-scan. - - `OllamaRuntime.resolve()` only reads `ollama_name` (and, optionally, - `disk_size_gb` for a progress-bar total) — every other field here is - unused by any runtime and left at a neutral default. - """ - ollama = curated_model.artifacts.ollama - return ScoredModel( - canonical_id=model_id, - publisher=None, - family=curated_model.family, - display_name=curated_model.label, - parameter_count=curated_model.parameter_count, - params_b=None, - use_case=None, - fit=FitLevel.UNKNOWN, - score=None, - runtime=None, - run_mode=None, - best_quant=None, - memory_required_gb=None, - memory_available_gb=None, - utilization_pct=None, - disk_size_gb=curated_model.size_bytes / 1_000_000_000, - estimated_tps=None, - prefill_tps=None, - ttft_ms=None, - estimate_confidence=None, - estimate_basis=None, - effective_context_length=None, - capability_ids=(), - license=None, - ollama_name=ollama.name if ollama is not None else None, - gguf_sources=(), - ollama_quantization=ollama.quantization if ollama is not None else None, - ) - - -def clean_runtime_name(model_name: str) -> str: - """A runtime's own model name, minus what's only meaningful to it. - - A native Ollama library name (`llama3.2:1b`) is already a name a person - picked — shown as-is, tag included, since the tag is the size/variant, - not noise. An `hf.co/[:]` fallback pull (built in - `llmfit.py` for a model Ollama's own library doesn't have) is different: - `hf.co/` only says where it came from, and the tag is either a - throwaway `:latest` or a quant code — neither means anything to a - person looking at a model list. Stripped down to `/`, - it still names who published it, just not through a URL. - """ - if not model_name.startswith("hf.co/"): - return model_name - return model_name.removeprefix("hf.co/").split(":", 1)[0] - - -def _is_embedding(model: ScoredModel) -> bool: - return (model.use_case is not None and "embedding" in model.use_case) or ( - bool(model.capability_ids) - and set(model.capability_ids).issubset({"embedding", "embeddings"}) - ) - - -def _apply_reserve(model: ScoredModel, reserve_gb: float) -> ScoredModel: - required = model.memory_required_gb - available = model.memory_available_gb - if required is None or available is None: - return model - adjusted = max(0.0, available - reserve_gb) - utilization = required / adjusted if adjusted else float("inf") - fit = model.fit - if utilization > 1: - fit = FitLevel.TOO_TIGHT - elif utilization > 0.9 and FIT_ORDER[fit] < FIT_ORDER[FitLevel.MARGINAL]: - fit = FitLevel.MARGINAL - return replace(model, fit=fit, utilization_pct=utilization * 100) - - -def _sort_key(row: CatalogRow) -> tuple[int, float, str]: - return (FIT_ORDER[row.fit], -(row.score or 0), row.canonical_id) diff --git a/surfsense_local/backend/modules/llm/recommendations/curated-models.json b/surfsense_local/backend/modules/llm/recommendations/curated-models.json deleted file mode 100644 index 0ec6a64cfd..0000000000 --- a/surfsense_local/backend/modules/llm/recommendations/curated-models.json +++ /dev/null @@ -1,98 +0,0 @@ -{ - "schema_version": 2, - "advisor_providers": ["Alibaba"], - "models": [ - { - "model_id": "Qwen/Qwen3-0.6B", - "family": "Qwen3", - "minimum_fit": "good", - "minimum_context": 8192, - "allowed_quantizations": ["Q4_K_M"], - "artifacts": {"ollama": {"name": "qwen3:0.6b", "quantization": "Q4_K_M"}}, - "label": "Qwen3 0.6B", - "parameter_count": "0.6B", - "size_bytes": 522640096 - }, - { - "model_id": "Qwen/Qwen3-1.7B", - "family": "Qwen3", - "minimum_fit": "good", - "minimum_context": 8192, - "allowed_quantizations": ["Q4_K_M"], - "artifacts": {"ollama": {"name": "qwen3:1.7b", "quantization": "Q4_K_M"}}, - "label": "Qwen3 1.7B", - "parameter_count": "1.7B", - "size_bytes": 1359279776 - }, - { - "model_id": "Qwen/Qwen3-4B", - "family": "Qwen3", - "minimum_fit": "good", - "minimum_context": 8192, - "allowed_quantizations": ["Q4_K_M"], - "artifacts": {"ollama": {"name": "qwen3:4b", "quantization": "Q4_K_M"}}, - "label": "Qwen3 4B", - "parameter_count": "4B", - "size_bytes": 2497280480 - }, - { - "model_id": "Qwen/Qwen3-8B", - "family": "Qwen3", - "minimum_fit": "good", - "minimum_context": 8192, - "allowed_quantizations": ["Q4_K_M"], - "artifacts": {"ollama": {"name": "qwen3:8b", "quantization": "Q4_K_M"}}, - "label": "Qwen3 8B", - "parameter_count": "8B", - "size_bytes": 5225374496 - }, - { - "model_id": "Qwen/Qwen3-14B", - "family": "Qwen3", - "minimum_fit": "good", - "minimum_context": 8192, - "allowed_quantizations": ["Q4_K_M"], - "artifacts": {"ollama": {"name": "qwen3:14b", "quantization": "Q4_K_M"}}, - "label": "Qwen3 14B", - "parameter_count": "14B", - "size_bytes": 9276184896 - }, - { - "model_id": "Qwen/Qwen3-32B", - "family": "Qwen3", - "minimum_fit": "good", - "minimum_context": 8192, - "allowed_quantizations": ["Q4_K_M"], - "artifacts": {"ollama": {"name": "qwen3:32b", "quantization": "Q4_K_M"}}, - "label": "Qwen3 32B", - "parameter_count": "32B", - "size_bytes": 20201240160 - }, - { - "model_id": "Qwen/Qwen2.5-Coder-7B-Instruct", - "family": "Qwen2.5 Coder", - "minimum_fit": "good", - "minimum_context": 8192, - "allowed_quantizations": ["Q4_K_M"], - "artifacts": { - "ollama": {"name": "qwen2.5-coder:7b", "quantization": "Q4_K_M"} - }, - "label": "Qwen2.5 Coder 7B", - "parameter_count": "7B", - "size_bytes": 4683074048 - }, - { - "model_id": "Qwen/Qwen2.5-Coder-32B-Instruct", - "family": "Qwen2.5 Coder", - "minimum_fit": "good", - "minimum_context": 8192, - "allowed_quantizations": ["Q4_K_M"], - "artifacts": { - "ollama": {"name": "qwen2.5-coder:32b", "quantization": "Q4_K_M"} - }, - "label": "Qwen2.5 Coder 32B", - "parameter_count": "32B", - "size_bytes": 19851336384 - } - ] -} diff --git a/surfsense_local/backend/modules/llm/recommendations/curated_models.py b/surfsense_local/backend/modules/llm/recommendations/curated_models.py deleted file mode 100644 index bbccdd7de4..0000000000 --- a/surfsense_local/backend/modules/llm/recommendations/curated_models.py +++ /dev/null @@ -1,70 +0,0 @@ -from pathlib import Path - -from pydantic import BaseModel, ConfigDict, Field, model_validator - -from modules.llm.recommendations.types import FitLevel - -SCHEMA_VERSION = 2 - - -class OllamaArtifact(BaseModel): - model_config = ConfigDict(extra="forbid") - - name: str = Field(min_length=1) - quantization: str | None = None - - -class CuratedModelArtifacts(BaseModel): - model_config = ConfigDict(extra="forbid") - - ollama: OllamaArtifact | None = None - - -class CuratedModel(BaseModel): - model_config = ConfigDict(extra="forbid") - - model_id: str = Field(min_length=1) - family: str = Field(min_length=1) - minimum_fit: FitLevel - minimum_context: int = Field(gt=0) - allowed_quantizations: list[str] - artifacts: CuratedModelArtifacts - - # Static display metadata, shown before any hardware scan has run — see - # CatalogService's scan-free curated rows. `size_bytes` is the pinned - # `artifacts.ollama` tag's exact size (from Ollama's own registry - # manifest, e.g. `GET registry.ollama.ai/v2/library//manifests/`), - # not an estimate: a curated pin always installs the same fixed tag - # regardless of the end user's hardware, so this number never varies. - label: str = Field(min_length=1) - parameter_count: str = Field(min_length=1) - size_bytes: int = Field(gt=0) - - -class CuratedModelsManifest(BaseModel): - model_config = ConfigDict(extra="forbid") - - schema_version: int - advisor_providers: list[str] = Field(default_factory=list) - models: list[CuratedModel] - - @model_validator(mode="after") - def validate_manifest(self) -> "CuratedModelsManifest": - if self.schema_version != SCHEMA_VERSION: - raise ValueError(f"unsupported curated-model schema: {self.schema_version}") - ids = [model.model_id for model in self.models] - if len(ids) != len(set(ids)): - raise ValueError("duplicate model_id in curated-model manifest") - targets = [ - ("ollama", model.artifacts.ollama.name) - for model in self.models - if model.artifacts.ollama is not None - ] - if len(targets) != len(set(targets)): - raise ValueError("duplicate runtime target in curated-model manifest") - return self - - -def load_curated_models(path: Path | None = None) -> CuratedModelsManifest: - manifest_path = path or Path(__file__).with_name("curated-models.json") - return CuratedModelsManifest.model_validate_json(manifest_path.read_text()) diff --git a/surfsense_local/backend/modules/llm/recommendations/dependencies.py b/surfsense_local/backend/modules/llm/recommendations/dependencies.py deleted file mode 100644 index 879072d1de..0000000000 --- a/surfsense_local/backend/modules/llm/recommendations/dependencies.py +++ /dev/null @@ -1,58 +0,0 @@ -from functools import lru_cache -from typing import Annotated - -from fastapi import Depends -from pydantic import ValidationError - -from modules.llm.providers import get_provider -from modules.llm.recommendations.catalog import CatalogService -from modules.llm.recommendations.curated_models import ( - CuratedModelsManifest, - load_curated_models, -) -from modules.llm.recommendations.llmfit import LlmfitAdvisor -from modules.llm.recommendations.protocols import LocalRuntime -from modules.llm.recommendations.types import RecommendationWarning -from shared.config import get_llm_settings, get_storage_settings - - -@lru_cache -def get_catalog_service() -> CatalogService: - settings = get_llm_settings() - provider = get_provider("ollama") - runtimes = [provider] if isinstance(provider, LocalRuntime) else [] - warnings: tuple[RecommendationWarning, ...] = () - try: - curated_models = load_curated_models() - except (OSError, ValidationError, ValueError): - curated_models = CuratedModelsManifest(schema_version=1, models=[]) - warnings = ( - RecommendationWarning( - "invalid_curated_models", - "SurfSense Recommended models are temporarily unavailable.", - ), - ) - advisor = LlmfitAdvisor( - settings.llmfit_path, - settings.llmfit_expected_version, - settings.llmfit_timeout_seconds, - tuple(curated_models.advisor_providers), - cache_path=get_storage_settings().data_dir / "llmfit-scan.json", - ) - storage = ( - {"ollama": settings.ollama_models_dir} - if settings.ollama_models_dir is not None - else {} - ) - return CatalogService( - advisor, - runtimes, - curated_models, - max_context=settings.llmfit_max_context, - reserve_gb=settings.recommendation_reserve_gb, - runtime_storage=storage, - initial_warnings=warnings, - ) - - -CatalogServiceDep = Annotated[CatalogService, Depends(get_catalog_service)] diff --git a/surfsense_local/backend/modules/llm/recommendations/llmfit.py b/surfsense_local/backend/modules/llm/recommendations/llmfit.py deleted file mode 100644 index 018482ea73..0000000000 --- a/surfsense_local/backend/modules/llm/recommendations/llmfit.py +++ /dev/null @@ -1,475 +0,0 @@ -"""The llmfit adapter: shells out to the pinned binary and caches its verdict. - -This is the only module that knows llmfit's CLI/JSON contract. Everything -above `LlmfitAdvisor` (CatalogService and up) only ever sees `ScoredModel`/ -`SystemProfile`/`AdvisorCatalog` — plain, versioned dataclasses, not llmfit's -own vocabulary. That seam is what lets the cache live here too: the cache's -job is "did we already ask llmfit this exact question," which only this -module has enough context to answer. -""" - -import asyncio -import json -import logging -import os -import re -import subprocess -from dataclasses import asdict -from datetime import UTC, datetime -from pathlib import Path -from typing import Any - -from modules.llm.recommendations.types import ( - AdvisorCatalog, - FitLevel, - RecommendationWarning, - ScoredModel, - SystemProfile, -) - -LOGGER = logging.getLogger(__name__) -MAX_OUTPUT_BYTES = 64 * 1024 * 1024 -VERSION_PATTERN = re.compile(r"(\d+\.\d+\.\d+)") - -# Bumped whenever ScoredModel's shape changes. A mismatch discards the cache -# file and rescans: a cached row is an optimisation, never something worth -# migrating. -CACHE_VERSION = 1 - -# Tuple fields survive a JSON round-trip as lists. -_TUPLE_FIELDS = ("capability_ids", "gguf_sources", "notes") - - -class LlmfitError(RuntimeError): - def __init__(self, code: str, public_message: str, detail: str = "") -> None: - super().__init__(detail or public_message) - self.code = code - self.public_message = public_message - - -class LlmfitAdvisor: - """Scores models with the pinned llmfit binary and caches the result. - - The cache lives here rather than in CatalogService because it stores the - hardware profile llmfit itself produced, and nothing above this seam - should re-derive it. `scan(refresh=False)` never spawns llmfit at all — - not the per-model scoring subprocess, not even the cheap version/system - probe — it only ever answers from whatever the last explicit scan wrote - to disk, or reports "not scanned" so the caller can show scan-free - content instead. Only `refresh=True` (an explicit "Scan hardware" click) - probes hardware and pays for scoring, then refreshes the cache. - """ - - def __init__( - self, - executable: Path, - expected_version: str, - timeout_seconds: float, - priority_providers: tuple[str, ...] = (), - *, - cache_path: Path | None = None, - ) -> None: - self._executable = executable - self._expected_version = expected_version - self._timeout_seconds = timeout_seconds - self._priority_providers = priority_providers - self._cache_path = cache_path - - async def scan(self, max_context: int, *, refresh: bool = False) -> AdvisorCatalog: - if not refresh: - # An unrefreshed call is never a user action (it fires on every - # page load), so it must not spawn llmfit at all — not even the - # cheap version/system probe. It only ever answers from what the - # last explicit "Scan hardware" wrote, or reports "not scanned - # yet" so the caller shows scan-free content (installed + - # curated) and a "Scan hardware" action instead. - cached = self._load_cache() - if cached is not None: - return cached - return AdvisorCatalog( - system=None, models=(), llmfit_version=None, scanned=False - ) - - try: - version = await self._version() - if version != self._expected_version: - return AdvisorCatalog( - system=None, - models=(), - llmfit_version=version, - scanned=True, - warnings=( - RecommendationWarning( - "version_mismatch", - "Model recommendations are unavailable until llmfit is updated.", - ), - ), - ) - system = _parse_system(await self._json("--json", "system")) - except LlmfitError as error: - LOGGER.warning("llmfit probe failed (%s): %s", error.code, error) - return AdvisorCatalog( - system=None, - models=(), - llmfit_version=None, - scanned=True, - warnings=(RecommendationWarning(error.code, error.public_message),), - ) - - try: - fit_payloads = await asyncio.gather( - self._fit(max_context), - *( - self._fit(max_context, provider) - for provider in self._priority_providers - ), - ) - except LlmfitError as error: - LOGGER.warning("llmfit scan failed (%s): %s", error.code, error) - return AdvisorCatalog( - system=system, - models=(), - llmfit_version=version, - scanned=True, - warnings=(RecommendationWarning(error.code, error.public_message),), - ) - - models_by_id: dict[str, ScoredModel] = {} - for payload in fit_payloads: - for row in _model_rows(payload): - model = _parse_model(row) - models_by_id.setdefault(model.canonical_id, model) - models = tuple(models_by_id.values()) - self._write_cache(system, version, models) - return AdvisorCatalog(system, models, version, scanned=True) - - async def _fit(self, max_context: int, provider: str | None = None) -> Any: - args = [ - "--max-context", - str(max_context), - "--json", - "fit", - ] - if provider is not None: - args.extend(("--providers", provider)) - return await self._json(*args) - - async def _version(self) -> str: - output = await self._run("--version") - match = VERSION_PATTERN.search(output) - if match is None: - raise LlmfitError( - "invalid_output", - "Model recommendations are temporarily unavailable.", - "llmfit --version returned no semantic version", - ) - return match.group(1) - - async def _json(self, *args: str) -> Any: - output = await self._run(*args) - try: - return json.loads(output) - except json.JSONDecodeError as error: - raise LlmfitError( - "invalid_output", - "Model recommendations are temporarily unavailable.", - f"invalid llmfit JSON: {error}", - ) from error - - async def _run(self, *args: str) -> str: - try: - process = await asyncio.create_subprocess_exec( - str(self._executable), - *args, - stdout=asyncio.subprocess.PIPE, - stderr=asyncio.subprocess.PIPE, - limit=MAX_OUTPUT_BYTES, - creationflags=getattr(subprocess, "CREATE_NO_WINDOW", 0), - ) - except FileNotFoundError as error: - raise LlmfitError( - "missing", - "Hardware recommendations are unavailable because llmfit is missing.", - ) from error - try: - stdout, stderr = await asyncio.wait_for( - process.communicate(), timeout=self._timeout_seconds - ) - except TimeoutError as error: - process.kill() - await process.wait() - raise LlmfitError( - "timeout", - "Hardware detection took too long. Try rescanning.", - ) from error - - if len(stdout) > MAX_OUTPUT_BYTES or len(stderr) > MAX_OUTPUT_BYTES: - raise LlmfitError( - "invalid_output", - "Model recommendations returned too much data.", - ) - if process.returncode != 0: - detail = stderr.decode(errors="replace").strip() - raise LlmfitError( - "scan_failed", - "Hardware recommendations could not be calculated.", - detail, - ) - return stdout.decode().strip() - - def _load_cache(self) -> AdvisorCatalog | None: - """The last explicit "Scan hardware" result, or None to show as unscanned. - - This is the only thing `scan(refresh=False)` ever reads — it never - re-probes hardware to validate freshness, since re-probing is exactly - the subprocess spawn an unrefreshed call must not pay for. A stale - cache (e.g. after a hardware change) simply stays stale until the - next explicit rescan. Every failure mode is "not scanned", never - "broken": a missing, truncated, unreadable, or version-stale file is - just a miss. - - The llmfit *version* check below is free (a string compare against - the pinned constant, no subprocess), unlike the hardware fingerprint - this used to also require — that's why it's still done here even - though nothing else on this path probes anything: an app update that - bumps the bundled binary must not keep serving scores computed by the - binary it replaced. - """ - if self._cache_path is None: - return None - try: - document = json.loads(self._cache_path.read_text(encoding="utf-8")) - if document.get("cache_version") != CACHE_VERSION: - return None - if document.get("llmfit_version") != self._expected_version: - return None - system = SystemProfile(**document["system"]) - models = tuple(_model_from_json(row) for row in document["models"]) - return AdvisorCatalog(system, models, document["llmfit_version"]) - except FileNotFoundError: - return None - except (OSError, ValueError, TypeError, KeyError) as error: - LOGGER.warning("discarding unreadable llmfit cache: %s", error) - return None - - def _write_cache( - self, system: SystemProfile, version: str, models: tuple[ScoredModel, ...] - ) -> None: - if self._cache_path is None or not models: - return - document = { - "cache_version": CACHE_VERSION, - "system": asdict(system), - "llmfit_version": version, - "scanned_at": datetime.now(UTC).isoformat(), - "models": [_model_to_json(model) for model in models], - } - # Rename onto the final path so an interrupted multi-megabyte write - # cannot leave JSON that parses as garbage. - temporary = self._cache_path.with_suffix(".json.tmp") - try: - self._cache_path.parent.mkdir(parents=True, exist_ok=True) - temporary.write_text(json.dumps(document), encoding="utf-8") - os.replace(temporary, self._cache_path) - except OSError as error: - LOGGER.warning("could not write llmfit cache: %s", error) - temporary.unlink(missing_ok=True) - - -def _model_to_json(model: ScoredModel) -> dict[str, Any]: - data = asdict(model) - data["fit"] = model.fit.value - return data - - -def _model_from_json(data: dict[str, Any]) -> ScoredModel: - data = dict(data) - data["fit"] = FitLevel(data["fit"]) - for name in _TUPLE_FIELDS: - data[name] = tuple(data.get(name) or ()) - return ScoredModel(**data) - - -def _code(value: Any) -> str | None: - if not isinstance(value, str): - return None - return re.sub(r"[^a-z0-9]+", "_", value.strip().lower()).strip("_") - - -def _number(value: Any) -> float | None: - return float(value) if isinstance(value, int | float) else None - - -def _integer(value: Any) -> int | None: - return int(value) if isinstance(value, int | float) else None - - -def _parse_system(payload: Any) -> SystemProfile: - if not isinstance(payload, dict) or not isinstance(payload.get("system"), dict): - raise LlmfitError( - "invalid_output", - "Hardware recommendations returned an unsupported system profile.", - ) - row = payload["system"] - return SystemProfile( - cpu_name=row.get("cpu_name") if isinstance(row.get("cpu_name"), str) else None, - cpu_cores=_integer(row.get("cpu_cores")), - total_ram_gb=_number(row.get("total_ram_gb")), - available_ram_gb=_number(row.get("available_ram_gb")), - has_gpu=bool(row.get("has_gpu", False)), - gpu_name=row.get("gpu_name") if isinstance(row.get("gpu_name"), str) else None, - gpu_vram_gb=_number(row.get("gpu_vram_gb")), - gpu_count=_integer(row.get("gpu_count")) or 0, - backend=row.get("backend") if isinstance(row.get("backend"), str) else None, - unified_memory=bool(row.get("unified_memory", False)), - ) - - -def _model_rows(payload: Any) -> list[dict[str, Any]]: - if not isinstance(payload, dict) or not isinstance(payload.get("models"), list): - raise LlmfitError( - "invalid_output", - "Model recommendations returned an unsupported catalog.", - ) - if not all(isinstance(row, dict) for row in payload["models"]): - raise LlmfitError( - "invalid_output", - "Model recommendations contained an invalid model entry.", - ) - return payload["models"] - - -def _gguf_sources(value: Any) -> tuple[str, ...]: - """Hugging Face GGUF repos llmfit found for this model. - - Real llmfit output is a list of `{"provider": ..., "repo": ...}` dicts, - not bare strings — this is the repo half of each one, which is all a - runtime needs to build an `hf.co/` pull target from. - """ - if not isinstance(value, list): - return () - repos: list[str] = [] - for item in value: - if isinstance(item, dict) and isinstance(item.get("repo"), str): - repos.append(item["repo"]) - return tuple(repos) - - -def _fallback_ollama_name( - gguf_sources: tuple[str, ...], runtime: str | None, best_quant: str | None -) -> str | None: - """`hf.co/[:]` for a model Ollama's own library doesn't have. - - Ollama can pull any GGUF straight from Hugging Face this way. The quant - suffix is only trustworthy when llmfit scored the model against - `llama_cpp` specifically: on other runtimes (e.g. `mlx`, only ever seen - on Apple Silicon, where MLX beats llama.cpp) `best_quant` is a label in - that runtime's own vocabulary (`mlx-4bit`), not a GGUF file suffix, and - would make the pull fail. Without a trusted quant, Ollama's own default - for an untagged `hf.co/...` pull is to prefer `Q4_K_M` when present in - the repo, else one reasonable quant — a sane fallback, not a guess. - - The tag is always explicit, never omitted: a bare `hf.co/` pull - request is accepted by Ollama, but it silently stores the result as - `hf.co/:latest` (confirmed against a real pull) — every later - exact-string match against this `ollama_name` (install verification, - "already installed" detection on a rescan) would then permanently fail - against an identifier that never matches what Ollama actually named it. - Requesting `:latest` up front keeps the string we hand out identical to - what comes back. - """ - if not gguf_sources: - return None - repo = gguf_sources[0] - if runtime == "llama_cpp" and best_quant: - return f"hf.co/{repo}:{best_quant}" - return f"hf.co/{repo}:latest" - - -def _parse_model(row: dict[str, Any]) -> ScoredModel: - name = row.get("name") - if not isinstance(name, str) or not name.strip(): - raise LlmfitError( - "invalid_output", - "Model recommendations contained a model without an id.", - ) - fit_code = _code(row.get("fit_level")) - try: - fit = FitLevel(fit_code) - except (TypeError, ValueError): - fit = FitLevel.UNKNOWN - - publisher = row.get("provider") - publisher = publisher if isinstance(publisher, str) else None - display = name.rsplit("/", 1)[-1] - family = display.split("-", 1)[0] or display - capabilities = row.get("capability_ids", row.get("capabilities", [])) - capability_ids = ( - tuple(code for item in capabilities if (code := _code(item))) - if isinstance(capabilities, list) - else () - ) - notes = row.get("notes", []) - estimate_basis = row.get("estimate_basis") - runtime = _code(row.get("runtime")) - best_quant = ( - row.get("best_quant") if isinstance(row.get("best_quant"), str) else None - ) - gguf_sources = _gguf_sources(row.get("gguf_sources")) - - native_ollama_name = row.get("ollama_name") - native_ollama_name = ( - native_ollama_name if isinstance(native_ollama_name, str) else None - ) - ollama_name = native_ollama_name or _fallback_ollama_name( - gguf_sources, runtime, best_quant - ) - # A fallback pull's fit badge must not overstate confidence: it's honest - # only when the quant used to build the pull target is the same one the - # fit/memory numbers above were computed for (the `runtime == "llama_cpp"` - # case in `_fallback_ollama_name`). Otherwise the badge describes a - # different runtime's memory profile than the file Ollama will actually - # fetch — relabel to Marginal (a hedge, not a promise) rather than either - # hiding the model entirely or carrying forward a number that doesn't - # match what gets installed. - trusted_quant = native_ollama_name is not None or runtime == "llama_cpp" - if native_ollama_name is None and gguf_sources and not trusted_quant: - fit = FitLevel.MARGINAL - - return ScoredModel( - canonical_id=name, - publisher=publisher, - family=family, - display_name=display, - parameter_count=( - row.get("parameter_count") - if isinstance(row.get("parameter_count"), str) - else None - ), - params_b=_number(row.get("params_b")), - use_case=_code(row.get("use_case") or row.get("category")), - fit=fit, - score=_number(row.get("score")), - runtime=runtime, - run_mode=_code(row.get("run_mode")), - best_quant=best_quant, - memory_required_gb=_number(row.get("memory_required_gb")), - memory_available_gb=_number(row.get("memory_available_gb")), - utilization_pct=_number(row.get("utilization_pct")), - disk_size_gb=_number(row.get("disk_size_gb")), - estimated_tps=_number(row.get("estimated_tps")), - prefill_tps=_number(row.get("prefill_tps")), - ttft_ms=_number(row.get("ttft_ms")), - estimate_confidence=_code(row.get("estimate_confidence")), - estimate_basis=estimate_basis if isinstance(estimate_basis, dict) else None, - effective_context_length=_integer(row.get("effective_context_length")), - capability_ids=capability_ids, - license=row.get("license") if isinstance(row.get("license"), str) else None, - ollama_name=ollama_name, - gguf_sources=gguf_sources, - notes=( - tuple(item for item in notes if isinstance(item, str)) - if isinstance(notes, list) - else () - ), - ) diff --git a/surfsense_local/backend/modules/llm/recommendations/protocols.py b/surfsense_local/backend/modules/llm/recommendations/protocols.py deleted file mode 100644 index 1327d887ee..0000000000 --- a/surfsense_local/backend/modules/llm/recommendations/protocols.py +++ /dev/null @@ -1,32 +0,0 @@ -from collections.abc import AsyncIterator -from typing import Protocol, runtime_checkable - -from modules.llm.providers.types import DownloadProgress -from modules.llm.recommendations.types import ( - AdvisorCatalog, - InstalledModel, - InstallPlan, - ScoredModel, -) - - -class ModelAdvisor(Protocol): - async def scan(self, max_context: int) -> AdvisorCatalog: ... - - -@runtime_checkable -class LocalRuntime(Protocol): - name: str - - async def health(self) -> bool: ... - - async def resolve(self, model: ScoredModel) -> InstallPlan | None: ... - - async def installed_models(self) -> list[InstalledModel]: ... - - def install(self, plan: InstallPlan) -> AsyncIterator[DownloadProgress]: ... - - -@runtime_checkable -class CancelledDownloadCleaner(Protocol): - async def cleanup_cancelled_download(self) -> None: ... diff --git a/surfsense_local/backend/modules/llm/recommendations/router.py b/surfsense_local/backend/modules/llm/recommendations/router.py deleted file mode 100644 index 5927e7d843..0000000000 --- a/surfsense_local/backend/modules/llm/recommendations/router.py +++ /dev/null @@ -1,166 +0,0 @@ -import asyncio -import json -import logging -from collections.abc import AsyncIterator - -from fastapi import APIRouter, HTTPException, Request, status -from fastapi.responses import StreamingResponse -from sqlalchemy.orm import Session - -from api.dependencies import SessionDep, transact -from modules.egress import service as egress -from modules.llm.models import ModelRole, SelectedModel -from modules.llm.recommendations.catalog import ( - InsufficientDiskError, - UnknownCatalogIdError, -) -from modules.llm.recommendations.dependencies import CatalogServiceDep -from modules.llm.recommendations.protocols import CancelledDownloadCleaner -from modules.llm.schemas import ( - InstallRequest, - RecommendationCatalogRead, - RecommendationSystemRead, - SelectionRead, -) -from modules.llm.selection import choose_model - -router = APIRouter() -logger = logging.getLogger(__name__) - - -@router.get("/system", response_model=RecommendationSystemRead) -async def recommendation_system(service: CatalogServiceDep) -> dict: - scan = await service.advisor_catalog() - return { - "hardware": scan.system, - "llmfit_version": scan.llmfit_version, - "estimates_available": bool(scan.models), - "warnings": scan.warnings, - } - - -@router.get("/catalog", response_model=RecommendationCatalogRead) -async def recommendation_catalog( - service: CatalogServiceDep, - session: SessionDep, - refresh: bool = False, -): - selected_key = await transact(session, _selected_local_model) - return await service.catalog(selected=selected_key, refresh=refresh) - - -def _selected_local_model(session: Session) -> tuple[str, str] | None: - """The chosen generation model, if it is a local one the catalog can mark.""" - selected = session.get(SelectedModel, ModelRole.GENERATION) - if selected is None or selected.connection_id is not None: - return None - return selected.provider, selected.name - - -@router.post("/install", summary="Install and optionally select a catalog model") -async def install_model( - payload: InstallRequest, - request: Request, - service: CatalogServiceDep, - session: SessionDep, -) -> StreamingResponse: - try: - runtime, _model, plan = await service.preflight(payload.catalog_id) - except UnknownCatalogIdError as error: - raise HTTPException( - status.HTTP_422_UNPROCESSABLE_CONTENT, - "catalog id is stale or unknown; refresh the catalog", - ) from error - except InsufficientDiskError as error: - raise HTTPException( - status.HTTP_507_INSUFFICIENT_STORAGE, - { - "message": "insufficient disk space", - "required": error.required, - "available": error.available, - }, - ) from error - except RuntimeError as error: - raise HTTPException(status.HTTP_409_CONFLICT, str(error)) from error - - await transact( - session, - egress.require, - egress.OLLAMA_PULL, - egress.ollama_pull_host(plan.model_name), - ) - - lock = service.install_lock(runtime.name) - if lock.locked(): - raise HTTPException( - status.HTTP_409_CONFLICT, - f"{runtime.name} is already installing a model", - ) - await lock.acquire() - - async def progress() -> AsyncIterator[bytes]: - try: - yield _event("starting", message="Preparing download") - async for step in runtime.install(plan): - yield _event( - "downloading", - message=step.status, - completed=step.completed, - total=step.total, - ) - yield _event("verifying", message="Verifying installed model") - installed = next( - ( - model - for model in await runtime.installed_models() - if model.model_name == plan.model_name - ), - None, - ) - if installed is None or "completion" not in installed.capabilities: - raise RuntimeError("runtime did not verify a generation model") - if ( - plan.quantization is not None - and installed.quantization is not None - and installed.quantization.casefold() != plan.quantization.casefold() - ): - raise RuntimeError("runtime installed an unexpected quantization") - - selection = None - if payload.select: - yield _event("selecting", message="Selecting model") - with request.app.state.session_factory() as session: - selection = await choose_model( - session, - ModelRole.GENERATION, - runtime.name, - plan.model_name, - ) - selection = SelectionRead.model_validate(selection).model_dump( - mode="json" - ) - yield _event( - "complete", - message="Model is ready", - selection=selection, - ) - except asyncio.CancelledError: - if isinstance(runtime, CancelledDownloadCleaner): - try: - await runtime.cleanup_cancelled_download() - except OSError: - logger.exception("Could not remove cancelled model download") - raise - except Exception: - yield _event( - "error", - message="The model could not be installed. Retry the download.", - ) - finally: - lock.release() - - return StreamingResponse(progress(), media_type="application/x-ndjson") - - -def _event(kind: str, **payload: object) -> bytes: - return (json.dumps({"type": kind, **payload}) + "\n").encode() diff --git a/surfsense_local/backend/modules/llm/recommendations/types.py b/surfsense_local/backend/modules/llm/recommendations/types.py deleted file mode 100644 index f1237ebcfe..0000000000 --- a/surfsense_local/backend/modules/llm/recommendations/types.py +++ /dev/null @@ -1,141 +0,0 @@ -from dataclasses import dataclass, field -from enum import StrEnum -from typing import Any - - -class FitLevel(StrEnum): - PERFECT = "perfect" - GOOD = "good" - MARGINAL = "marginal" - TOO_TIGHT = "too_tight" - UNKNOWN = "unknown" - - -@dataclass(frozen=True) -class RecommendationWarning: - code: str - message: str - - -@dataclass(frozen=True) -class SystemProfile: - cpu_name: str | None = None - cpu_cores: int | None = None - total_ram_gb: float | None = None - available_ram_gb: float | None = None - has_gpu: bool = False - gpu_name: str | None = None - gpu_vram_gb: float | None = None - gpu_count: int = 0 - backend: str | None = None - unified_memory: bool = False - - -@dataclass(frozen=True) -class ScoredModel: - canonical_id: str - publisher: str | None - family: str - display_name: str - parameter_count: str | None - params_b: float | None - use_case: str | None - fit: FitLevel - score: float | None - runtime: str | None - run_mode: str | None - best_quant: str | None - memory_required_gb: float | None - memory_available_gb: float | None - utilization_pct: float | None - disk_size_gb: float | None - estimated_tps: float | None - prefill_tps: float | None - ttft_ms: float | None - estimate_confidence: str | None - estimate_basis: dict[str, Any] | None - effective_context_length: int | None - capability_ids: tuple[str, ...] - license: str | None - ollama_name: str | None - gguf_sources: tuple[str, ...] - notes: tuple[str, ...] = () - ollama_quantization: str | None = None - - -@dataclass(frozen=True) -class AdvisorCatalog: - system: SystemProfile | None - models: tuple[ScoredModel, ...] - llmfit_version: str | None - # False only for "haven't scanned this hardware yet" (no cached result, - # scan not requested) — distinct from an empty `models` tuple caused by a - # real scan failure or version mismatch, which is still `scanned=True` - # since an attempt was made and `warnings` explains why it came back empty. - scanned: bool = True - warnings: tuple[RecommendationWarning, ...] = () - - -@dataclass(frozen=True) -class InstallPlan: - canonical_id: str - runtime: str - model_name: str - expected_bytes: int | None - quantization: str | None - - -@dataclass(frozen=True) -class InstalledModel: - runtime: str - model_name: str - capabilities: tuple[str, ...] - quantization: str | None = None - - -@dataclass(frozen=True) -class CatalogRow: - catalog_id: str - canonical_id: str - family: str - label: str - publisher: str | None - parameter_count: str | None - fit: FitLevel - score: float | None - memory_required_gb: float | None - disk_size_gb: float | None - estimated_tps: float | None - prefill_tps: float | None - ttft_ms: float | None - effective_context_length: int | None - estimate_confidence: str | None - license: str | None - runtime: str - runtime_model: str - quantization: str | None - installed: bool - selected: bool - can_install: bool - can_delete: bool - warnings: tuple[str, ...] = () - - -@dataclass(frozen=True) -class CatalogResult: - hardware: SystemProfile | None - llmfit_version: str | None - # A curated model is always in exactly this bucket, never `explore` — - # scanning only ever adds a fit badge to a row already here, it never - # moves the row elsewhere. Populated even when `scanned` is False, from - # the manifest alone. - curated: tuple[CatalogRow, ...] - explore: tuple[CatalogRow, ...] - installed: tuple[CatalogRow, ...] - # False when this result was served without running the (expensive) - # hardware scan — no cached scan existed yet and none was requested. - # `curated`/`installed` are still fully populated in that case; only - # `explore` and curated fit badges are scan-derived. - scanned: bool = True - warnings: tuple[RecommendationWarning, ...] = () - runtime_status: dict[str, bool] = field(default_factory=dict) diff --git a/surfsense_local/backend/tests/fixtures/llmfit/fit.json b/surfsense_local/backend/tests/fixtures/llmfit/fit.json deleted file mode 100644 index 8e31957828..0000000000 --- a/surfsense_local/backend/tests/fixtures/llmfit/fit.json +++ /dev/null @@ -1,49 +0,0 @@ -{ - "models": [ - { - "name": "Qwen/Qwen3-8B", - "provider": "Qwen", - "parameter_count": "8B", - "params_b": 8.0, - "use_case": "Chat", - "fit_level": "Good", - "score": 86.5, - "runtime": "llama.cpp", - "run_mode": "GPU", - "best_quant": "Q4_K_M", - "memory_required_gb": 6.2, - "memory_available_gb": 24.5, - "utilization_pct": 25.3, - "disk_size_gb": 5.2, - "estimated_tps": 42.5, - "prefill_tps": null, - "ttft_ms": null, - "estimate_confidence": "estimated", - "estimate_basis": {"method": "roofline"}, - "effective_context_length": 8192, - "capability_ids": ["tool_use"], - "license": "apache-2.0", - "ollama_name": "qwen3:8b", - "gguf_sources": [], - "notes": ["fixture"] - }, - { - "name": "BAAI/bge-small-en-v1.5", - "provider": "BAAI", - "parameter_count": "33M", - "use_case": "Embedding", - "fit_level": "perfect", - "score": 99.0, - "runtime": "llamacpp", - "run_mode": "cpu", - "memory_required_gb": 0.2, - "memory_available_gb": 24.5, - "disk_size_gb": 0.1, - "effective_context_length": 512, - "capability_ids": ["embedding"], - "ollama_name": null, - "gguf_sources": [], - "future_field": "ignored" - } - ] -} diff --git a/surfsense_local/backend/tests/fixtures/llmfit/system.json b/surfsense_local/backend/tests/fixtures/llmfit/system.json deleted file mode 100644 index 6297088667..0000000000 --- a/surfsense_local/backend/tests/fixtures/llmfit/system.json +++ /dev/null @@ -1,14 +0,0 @@ -{ - "system": { - "cpu_name": "Apple M2 Max", - "cpu_cores": 12, - "total_ram_gb": 32.0, - "available_ram_gb": 24.5, - "has_gpu": true, - "gpu_name": "Apple M2 Max", - "gpu_vram_gb": 32.0, - "gpu_count": 1, - "backend": "Metal", - "unified_memory": true - } -} diff --git a/surfsense_local/backend/tests/unit/llm/recommendations/test_catalog.py b/surfsense_local/backend/tests/unit/llm/recommendations/test_catalog.py deleted file mode 100644 index c4b9f42bb2..0000000000 --- a/surfsense_local/backend/tests/unit/llm/recommendations/test_catalog.py +++ /dev/null @@ -1,442 +0,0 @@ -from dataclasses import replace -from pathlib import Path -from types import SimpleNamespace - -import pytest - -from modules.llm.providers.types import DownloadProgress -from modules.llm.recommendations.catalog import ( - CatalogService, - InsufficientDiskError, - UnknownCatalogIdError, -) -from modules.llm.recommendations.curated_models import CuratedModelsManifest -from modules.llm.recommendations.types import ( - AdvisorCatalog, - FitLevel, - InstalledModel, - InstallPlan, - ScoredModel, - SystemProfile, -) - -pytestmark = pytest.mark.unit - - -def _scored(**changes: object) -> ScoredModel: - base = ScoredModel( - canonical_id="Qwen/Qwen3-8B", - publisher="Qwen", - family="Qwen", - display_name="Qwen3-8B", - parameter_count="8B", - params_b=8.0, - use_case="chat", - fit=FitLevel.GOOD, - score=80, - runtime="llamacpp", - run_mode="gpu", - best_quant="Q4_K_M", - memory_required_gb=6, - memory_available_gb=16, - utilization_pct=37.5, - disk_size_gb=5.2, - estimated_tps=30, - prefill_tps=None, - ttft_ms=None, - estimate_confidence="estimated", - estimate_basis=None, - effective_context_length=8192, - capability_ids=("tool_use",), - license="apache-2.0", - ollama_name="qwen3:8b", - gguf_sources=(), - ) - return replace(base, **changes) - - -class Advisor: - def __init__(self, models: tuple[ScoredModel, ...]) -> None: - self.models = models - self.calls = 0 - - async def scan(self, max_context: int, *, refresh: bool = False) -> AdvisorCatalog: - self.calls += 1 - return AdvisorCatalog( - SystemProfile(available_ram_gb=16), - self.models, - "1.1.11", - scanned=True, - ) - - -class Runtime: - name = "ollama" - - def __init__(self, installed: list[InstalledModel] | None = None) -> None: - self._installed = installed or [] - - async def health(self) -> bool: - return True - - async def resolve(self, model: ScoredModel) -> InstallPlan | None: - if not model.ollama_name: - return None - return InstallPlan( - model.canonical_id, - self.name, - model.ollama_name, - 100, - model.ollama_quantization or model.best_quant, - ) - - async def installed_models(self) -> list[InstalledModel]: - return self._installed - - async def install(self, plan: InstallPlan): - yield DownloadProgress("success") - - -class LlamaRuntime(Runtime): - name = "llamacpp" - - async def resolve(self, model: ScoredModel) -> InstallPlan | None: - if not model.gguf_sources: - return None - return InstallPlan( - model.canonical_id, - self.name, - model.gguf_sources[0], - 100, - model.best_quant, - ) - - -def _manifest() -> CuratedModelsManifest: - return CuratedModelsManifest.model_validate( - { - "schema_version": 2, - "models": [ - { - "model_id": "Qwen/Qwen3-8B", - "family": "Qwen3", - "minimum_fit": "good", - "minimum_context": 8192, - "allowed_quantizations": ["Q4_K_M"], - "artifacts": { - "ollama": { - "name": "qwen3:8b", - "quantization": "Q4_K_M", - } - }, - "label": "Qwen3 8B", - "parameter_count": "8B", - "size_bytes": 5_000_000_000, - } - ], - } - ) - - -def _qwen_1_7_manifest() -> CuratedModelsManifest: - return CuratedModelsManifest.model_validate( - { - "schema_version": 2, - "models": [ - { - "model_id": "Qwen/Qwen3-1.7B", - "family": "Qwen3", - "minimum_fit": "good", - "minimum_context": 8192, - "allowed_quantizations": ["Q4_K_M"], - "artifacts": { - "ollama": { - "name": "qwen3:1.7b", - "quantization": "Q4_K_M", - } - }, - "label": "Qwen3 1.7B", - "parameter_count": "1.7B", - "size_bytes": 1_400_000_000, - } - ], - } - ) - - -async def test_catalog_partitions_curated_explore_and_embeddings() -> None: - """Curated matches always land in `curated`; embeddings stay excluded.""" - other = _scored( - canonical_id="Other/Chat-3B", - display_name="Chat-3B", - ollama_name="other:3b", - ) - embedding = _scored( - canonical_id="BAAI/bge", - use_case="embedding", - capability_ids=("embedding",), - ollama_name="bge", - ) - service = CatalogService( - Advisor((other, embedding, _scored(best_quant="mlx-4bit"))), - [Runtime()], - _manifest(), - max_context=8192, - reserve_gb=2, - ) - - result = await service.catalog(selected=None) - - assert [row.canonical_id for row in result.curated] == ["Qwen/Qwen3-8B"] - assert [row.canonical_id for row in result.explore] == ["Other/Chat-3B"] - assert result.curated[0].family == "Qwen3" - assert result.curated[0].quantization == "Q4_K_M" - # The manifest's own size wins over whatever the scan estimated. - assert result.curated[0].disk_size_gb == pytest.approx(5.0) - - -async def test_memory_reserve_can_only_downgrade_fit() -> None: - """SurfSense peak memory turns a nominal good fit into a warning.""" - service = CatalogService( - Advisor((_scored(memory_required_gb=13.0),)), - [Runtime()], - _manifest(), - max_context=8192, - reserve_gb=2, - ) - - result = await service.catalog(selected=None) - - assert result.explore == () - assert result.curated[0].fit is FitLevel.MARGINAL - - -async def test_installed_models_are_authoritative_and_not_duplicated() -> None: - """Runtime inventory wins over llmfit's installed metadata.""" - runtime = Runtime([InstalledModel("ollama", "qwen3:8b", ("completion",), "Q4_K_M")]) - service = CatalogService( - Advisor((_scored(),)), - [runtime], - _manifest(), - max_context=8192, - reserve_gb=2, - ) - - result = await service.catalog(selected=("ollama", "qwen3:8b")) - - assert result.curated == () - assert len(result.installed) == 1 - assert result.installed[0].selected is True - assert result.installed[0].can_delete is True - - -async def test_unscanned_hf_co_install_shows_a_clean_name() -> None: - """The placeholder fallback strips `hf.co/` and the tag, keeps the repo.""" - runtime = Runtime( - [ - InstalledModel( - "ollama", - "hf.co/bartowski/Meta-Llama-3.1-8B-Instruct-GGUF:latest", - ("completion",), - None, - ) - ] - ) - service = CatalogService( - Advisor(()), # no scan match at all — this is the placeholder path - [runtime], - _manifest(), - max_context=8192, - reserve_gb=2, - ) - - result = await service.catalog(selected=None) - - assert len(result.installed) == 1 - row = result.installed[0] - assert row.label == "bartowski/Meta-Llama-3.1-8B-Instruct-GGUF" - assert row.runtime_model == ( - "hf.co/bartowski/Meta-Llama-3.1-8B-Instruct-GGUF:latest" - ) - - -async def test_unscanned_native_install_keeps_its_tag() -> None: - """A native Ollama name's tag is size/variant, not noise — kept as-is.""" - runtime = Runtime( - [InstalledModel("ollama", "llama3.2:1b", ("completion",), None)] - ) - service = CatalogService( - Advisor(()), - [runtime], - _manifest(), - max_context=8192, - reserve_gb=2, - ) - - result = await service.catalog(selected=None) - - assert result.installed[0].label == "llama3.2:1b" - - -async def test_embedding_only_installed_models_stay_out_of_generation_catalog() -> None: - """An external Ollama embedding model is not a chat model.""" - runtime = Runtime( - [InstalledModel("ollama", "nomic-embed-text", ("embedding",), "F16")] - ) - service = CatalogService( - Advisor(()), - [runtime], - _manifest(), - max_context=8192, - reserve_gb=2, - ) - - result = await service.catalog(selected=None) - - assert result.installed == () - - -@pytest.mark.parametrize("curated_first", [False, True]) -async def test_curated_model_owns_a_colliding_runtime_target( - curated_first: bool, -) -> None: - """The explicit Qwen policy wins regardless of llmfit result order.""" - qwen_base = _scored( - canonical_id="Qwen/Qwen3-1.7B-Base", - display_name="Qwen3-1.7B-Base", - score=72.8, - ollama_name="qwen3:1.7b", - ) - qwen_chat = _scored( - canonical_id="Qwen/Qwen3-1.7B", - display_name="Qwen3-1.7B", - score=69.2, - ollama_name=None, - ) - models = (qwen_chat, qwen_base) if curated_first else (qwen_base, qwen_chat) - runtime = Runtime( - [InstalledModel("ollama", "qwen3:1.7b", ("completion",), "Q4_K_M")] - ) - service = CatalogService( - Advisor(models), - [runtime], - _qwen_1_7_manifest(), - max_context=8192, - reserve_gb=2, - ) - - result = await service.catalog(selected=("ollama", "qwen3:1.7b")) - - assert result.curated == () - assert result.explore == () - assert [row.canonical_id for row in result.installed] == ["Qwen/Qwen3-1.7B"] - assert result.installed[0].selected is True - assert not any( - warning.code == "ambiguous_runtime_target" for warning in result.warnings - ) - - -async def test_ambiguous_non_curated_models_use_exact_runtime_target( - caplog: pytest.LogCaptureFixture, -) -> None: - """Conflicting names become one exact runtime-target row.""" - instruct = _scored( - canonical_id="meta-llama/Llama-3.2-1B-Instruct", - display_name="Llama-3.2-1B-Instruct", - ollama_name="llama3.2:1b", - ) - base = _scored( - canonical_id="meta-llama/Llama-3.2-1B", - display_name="Llama-3.2-1B", - ollama_name="llama3.2:1b", - ) - service = CatalogService( - Advisor((instruct, base)), - [Runtime()], - CuratedModelsManifest(schema_version=2, models=[]), - max_context=8192, - reserve_gb=2, - ) - - with caplog.at_level("WARNING"): - result = await service.catalog(selected=None) - repeated = await service.catalog(selected=None) - - assert result.curated == () - assert len(result.explore) == 1 - assert result.explore[0].canonical_id == "ollama:llama3.2:1b" - assert result.explore[0].label == "llama3.2:1b" - assert result.explore[0].runtime_model == "llama3.2:1b" - assert result.installed == () - assert result.warnings == () - assert repeated.warnings == () - _, resolved_model, plan = await service.preflight(result.explore[0].catalog_id) - assert resolved_model.canonical_id == "ollama:llama3.2:1b" - assert plan.model_name == "llama3.2:1b" - assert ( - caplog.messages.count( - "Representing ambiguous models meta-llama/Llama-3.2-1B, " - "meta-llama/Llama-3.2-1B-Instruct as runtime target " - "ollama:llama3.2:1b" - ) - == 1 - ) - - -async def test_refresh_invalidates_opaque_install_ids() -> None: - """A catalog id cannot select an artifact after its scan generation changes.""" - advisor = Advisor((_scored(),)) - service = CatalogService( - advisor, - [Runtime()], - _manifest(), - max_context=8192, - reserve_gb=2, - ) - first = await service.catalog(selected=None) - old_id = first.curated[0].catalog_id - - await service.catalog(selected=None, refresh=True) - - with pytest.raises(UnknownCatalogIdError): - await service.preflight(old_id) - assert advisor.calls == 2 - - -async def test_a_second_runtime_needs_no_advisor_or_schema_change() -> None: - """A future GGUF runtime plugs into resolution without changing catalog DTOs.""" - model = _scored(ollama_name=None, gguf_sources=("org/model-q4.gguf",)) - service = CatalogService( - Advisor((model,)), - [Runtime(), LlamaRuntime()], - CuratedModelsManifest(schema_version=2, models=[]), - max_context=8192, - reserve_gb=2, - ) - - result = await service.catalog(selected=None) - - assert result.explore[0].runtime == "llamacpp" - assert result.explore[0].runtime_model == "org/model-q4.gguf" - - -async def test_disk_space_is_rejected_before_install_streaming( - tmp_path: Path, monkeypatch: pytest.MonkeyPatch -) -> None: - """Managed runtime storage is checked before response headers are sent.""" - service = CatalogService( - Advisor((_scored(),)), - [Runtime()], - _manifest(), - max_context=8192, - reserve_gb=2, - runtime_storage={"ollama": tmp_path}, - ) - catalog = await service.catalog(selected=None) - monkeypatch.setattr( - "modules.llm.recommendations.catalog.shutil.disk_usage", - lambda _path: SimpleNamespace(free=50), - ) - - with pytest.raises(InsufficientDiskError): - await service.preflight(catalog.curated[0].catalog_id) diff --git a/surfsense_local/backend/tests/unit/llm/recommendations/test_curated_models.py b/surfsense_local/backend/tests/unit/llm/recommendations/test_curated_models.py deleted file mode 100644 index 6da08dca7d..0000000000 --- a/surfsense_local/backend/tests/unit/llm/recommendations/test_curated_models.py +++ /dev/null @@ -1,82 +0,0 @@ -import pytest -from pydantic import ValidationError - -from modules.llm.recommendations.curated_models import ( - CuratedModelsManifest, - load_curated_models, -) - -pytestmark = pytest.mark.unit - - -def _model(**overrides: object) -> dict: - base = { - "model_id": "Qwen/Qwen3-8B", - "family": "Qwen3", - "minimum_fit": "good", - "minimum_context": 8192, - "allowed_quantizations": ["Q4_K_M"], - "artifacts": {"ollama": {"name": "qwen3:8b", "quantization": "Q4_K_M"}}, - "label": "Qwen3 8B", - "parameter_count": "8B", - "size_bytes": 5_225_374_496, - } - return {**base, **overrides} - - -def test_packaged_manifest_has_eight_unique_curated_models() -> None: - """The policy carries the eight existing Qwen offerings exactly once.""" - manifest = load_curated_models() - - assert manifest.schema_version == 2 - assert manifest.advisor_providers == ["Alibaba"] - assert len(manifest.models) == 8 - assert len({model.model_id for model in manifest.models}) == 8 - # Every entry has real display metadata, not a placeholder. - for model in manifest.models: - assert model.label - assert model.parameter_count - assert model.size_bytes > 0 - - -def test_manifest_rejects_duplicate_model_ids() -> None: - """Ambiguous support policy fails validation instead of silently overriding.""" - model = _model() - - with pytest.raises(ValidationError, match="duplicate model_id"): - CuratedModelsManifest.model_validate( - {"schema_version": 2, "models": [model, model]} - ) - - -def test_manifest_rejects_duplicate_ollama_targets() -> None: - """Two supported models cannot claim the same runtime artifact.""" - qwen_chat = _model(model_id="Qwen/Qwen3-1.7B") - qwen_base = _model(model_id="Qwen/Qwen3-1.7B-Base") - - with pytest.raises(ValidationError, match="duplicate runtime target"): - CuratedModelsManifest.model_validate( - {"schema_version": 2, "models": [qwen_chat, qwen_base]} - ) - - -def test_manifest_rejects_missing_display_metadata() -> None: - """A curated entry without display fields can't render a scan-free row.""" - incomplete = { - "model_id": "Qwen/Qwen3-8B", - "family": "Qwen3", - "minimum_fit": "good", - "minimum_context": 8192, - "allowed_quantizations": ["Q4_K_M"], - "artifacts": {"ollama": {"name": "qwen3:8b", "quantization": "Q4_K_M"}}, - } - - with pytest.raises(ValidationError): - CuratedModelsManifest.model_validate( - {"schema_version": 2, "models": [incomplete]} - ) - - -def test_manifest_rejects_unsupported_schema_version() -> None: - with pytest.raises(ValidationError, match="unsupported curated-model schema"): - CuratedModelsManifest.model_validate({"schema_version": 1, "models": []}) diff --git a/surfsense_local/backend/tests/unit/llm/recommendations/test_llmfit.py b/surfsense_local/backend/tests/unit/llm/recommendations/test_llmfit.py deleted file mode 100644 index 0a10745b61..0000000000 --- a/surfsense_local/backend/tests/unit/llm/recommendations/test_llmfit.py +++ /dev/null @@ -1,410 +0,0 @@ -import json -from pathlib import Path - -import pytest - -from modules.llm.recommendations.llmfit import ( - CACHE_VERSION, - LlmfitAdvisor, - _fallback_ollama_name, - _gguf_sources, - _parse_model, -) -from modules.llm.recommendations.types import FitLevel - -pytestmark = pytest.mark.unit -FIXTURES = Path(__file__).parents[3] / "fixtures" / "llmfit" - - -def _executable(tmp_path: Path, body: str) -> Path: - tmp_path.mkdir(parents=True, exist_ok=True) - path = tmp_path / "llmfit" - path.write_text("#!/usr/bin/env python3\n" + body) - path.chmod(0o755) - return path - - -def _stub(tmp_path: Path, system: dict, fit: dict, *, log: Path | None = None) -> Path: - """A fake llmfit that serves fixed `system`/`fit` JSON and, if `log` is - given, appends every subcommand it's asked to run — so a test can assert - the expensive `fit` call did or didn't happen.""" - system_json = json.dumps(system) - fit_json = json.dumps(fit) - log_line = ( - f"open({str(log)!r}, 'a').write(' '.join(sys.argv[1:]) + chr(10))\n" - if log is not None - else "" - ) - return _executable( - tmp_path, - f""" -import sys -{log_line} -if "--version" in sys.argv: - print("llmfit 1.1.11") -elif "system" in sys.argv: - print({system_json!r}) -else: - print({fit_json!r}) -""", - ) - - -async def test_adapter_normalizes_the_pinned_cli_contract(tmp_path: Path) -> None: - """CLI labels, null estimates, and future fields normalize at one seam.""" - system = json.loads((FIXTURES / "system.json").read_text()) - fit = json.loads((FIXTURES / "fit.json").read_text()) - executable = _executable( - tmp_path, - f""" -import sys -if "--version" in sys.argv: - print("llmfit 1.1.11") -elif "system" in sys.argv: - print({json.dumps(system)!r}) -elif "-n" in sys.argv or "--limit" in sys.argv: - print("unexpected result limit", file=sys.stderr) - sys.exit(2) -else: - print({json.dumps(fit)!r}) -""", - ) - - result = await LlmfitAdvisor(executable, "1.1.11", 2).scan(8192, refresh=True) - - assert result.warnings == () - assert result.scanned is True - assert result.system is not None - assert result.system.unified_memory is True - assert [model.fit for model in result.models] == [ - FitLevel.GOOD, - FitLevel.PERFECT, - ] - assert result.models[0].publisher == "Qwen" - assert result.models[0].prefill_tps is None - assert result.models[0].capability_ids == ("tool_use",) - - -async def test_missing_binary_degrades_to_a_warning(tmp_path: Path) -> None: - """A missing optional advisor cannot prevent the API from starting.""" - result = await LlmfitAdvisor(tmp_path / "missing", "1.1.11", 1).scan( - 8192, refresh=True - ) - - assert result.models == () - assert result.scanned is True - assert result.warnings[0].code == "missing" - - -async def test_version_drift_refuses_to_parse_unknown_output(tmp_path: Path) -> None: - """A binary upgrade requires fixture review before recommendations resume.""" - executable = _executable(tmp_path, 'print("llmfit 2.0.0")\n') - - result = await LlmfitAdvisor(executable, "1.1.11", 1).scan(8192, refresh=True) - - assert result.models == () - assert result.warnings[0].code == "version_mismatch" - - -async def test_timeout_kills_the_process(tmp_path: Path) -> None: - """A hung hardware probe returns promptly instead of leaking a child.""" - executable = _executable( - tmp_path, - "import time\ntime.sleep(10)\n", - ) - - result = await LlmfitAdvisor(executable, "1.1.11", 0.01).scan(8192, refresh=True) - - assert result.warnings[0].code == "timeout" - - -async def test_malformed_json_is_not_partially_accepted(tmp_path: Path) -> None: - """Malformed advisor output returns no recommendations.""" - executable = _executable( - tmp_path, - """ -import sys -print("llmfit 1.1.11" if "--version" in sys.argv else "{not-json") -""", - ) - - result = await LlmfitAdvisor(executable, "1.1.11", 2).scan(8192, refresh=True) - - assert result.models == () - assert result.warnings[0].code == "invalid_output" - - -async def test_nonzero_exit_is_a_scan_warning(tmp_path: Path) -> None: - """Advisor diagnostics stay in logs while the public API gets a safe warning.""" - executable = _executable( - tmp_path, - 'import sys\nprint("probe failed", file=sys.stderr)\nsys.exit(2)\n', - ) - - result = await LlmfitAdvisor(executable, "1.1.11", 1).scan(8192, refresh=True) - - assert result.models == () - assert result.warnings[0].code == "scan_failed" - - -# --- gguf_sources / hf.co fallback (Phase 1) ------------------------------- - - -def _row(**overrides: object) -> dict: - """A raw llmfit fit-result row, defaulting to a trusted-quant native model.""" - base = { - "name": "org/Some-Model-7B", - "fit_level": "perfect", - "provider": "org", - "runtime": "llama.cpp", - "best_quant": "Q4_K_M", - "ollama_name": None, - "gguf_sources": [], - } - return {**base, **overrides} - - -def test_gguf_sources_dict_entries_parse_into_repo_strings() -> None: - """Real llmfit output is a list of {provider, repo} dicts, not strings.""" - assert _gguf_sources( - [{"provider": "bartowski", "repo": "bartowski/Model-GGUF"}] - ) == ("bartowski/Model-GGUF",) - - -def test_gguf_sources_tolerates_malformed_entries() -> None: - """Malformed gguf_sources shapes are ignored, never raised on.""" - assert _gguf_sources("not-a-list") == () - assert _gguf_sources([{"provider": "x"}]) == () # missing repo - assert _gguf_sources(["a-bare-string"]) == () # not a dict - assert _gguf_sources(None) == () - - -def test_fallback_name_used_only_when_native_name_is_absent() -> None: - """A native ollama_name always wins over the gguf_sources fallback.""" - model = _parse_model( - _row(ollama_name="org:tag", gguf_sources=[{"repo": "org/Model-GGUF"}]) - ) - assert model.ollama_name == "org:tag" - - -def test_fallback_name_trusts_quant_only_for_llamacpp_runtime() -> None: - """An untrusted quant gets an explicit `:latest`, not a bare tag. - - Ollama silently stores a tag-less `hf.co/` pull as `:latest` on - disk — requesting it explicitly keeps `ollama_name` identical to what a - later `/api/tags` lookup reports, so install verification and "already - installed" detection on a rescan don't permanently mismatch. - """ - trusted = _parse_model( - _row( - runtime="llama.cpp", - best_quant="Q8_0", - gguf_sources=[{"repo": "org/Model-GGUF"}], - ) - ) - assert trusted.ollama_name == "hf.co/org/Model-GGUF:Q8_0" - assert trusted.fit is FitLevel.PERFECT # unchanged: quant is trustworthy - - untrusted = _parse_model( - _row( - runtime="MLX", - best_quant="mlx-4bit", - gguf_sources=[{"repo": "org/Model-GGUF"}], - ) - ) - # Explicit ":latest", not a bare tag — Ollama silently stores an - # untagged pull as ":latest" anyway, so requesting it up front keeps - # this string byte-identical to what a later /api/tags lookup returns. - assert untrusted.ollama_name == "hf.co/org/Model-GGUF:latest" - assert untrusted.fit is FitLevel.MARGINAL # downgraded: quant not trustworthy - - -def test_no_gguf_sources_and_no_native_name_leaves_ollama_name_none() -> None: - """Nothing to fall back to leaves ollama_name unset, fit untouched.""" - model = _parse_model(_row(ollama_name=None, gguf_sources=[])) - assert model.ollama_name is None - assert model.fit is FitLevel.PERFECT # nothing to downgrade for - - -def test_fallback_helper_directly() -> None: - """No sources -> None; only a trusted (llama_cpp) quant gets a tag.""" - assert _fallback_ollama_name((), "llama_cpp", "Q4_K_M") is None - assert ( - _fallback_ollama_name(("org/repo",), "llama_cpp", "Q4_K_M") - == "hf.co/org/repo:Q4_K_M" - ) - assert ( - _fallback_ollama_name(("org/repo",), "mlx", "mlx-4bit") - == "hf.co/org/repo:latest" - ) - assert ( - _fallback_ollama_name(("org/repo",), "llama_cpp", None) - == "hf.co/org/repo:latest" - ) - - -# --- persistent, fingerprinted cache ---------------------------------------- - - -def _system_fixture() -> dict: - return json.loads((FIXTURES / "system.json").read_text()) - - -def _fit_fixture() -> dict: - return json.loads((FIXTURES / "fit.json").read_text()) - - -async def test_no_cache_file_scans_when_refresh_is_true(tmp_path: Path) -> None: - """An explicit refresh scans and writes a cache even with none present yet.""" - cache_path = tmp_path / "cache.json" - log = tmp_path / "calls.log" - executable = _stub(tmp_path / "bin", _system_fixture(), _fit_fixture(), log=log) - - result = await LlmfitAdvisor( - executable, "1.1.11", 2, cache_path=cache_path - ).scan(8192, refresh=True) - - assert result.scanned is True - assert len(result.models) == 2 - assert "fit" in log.read_text() - assert cache_path.exists() - - -async def test_refresh_false_never_scans_without_a_cache(tmp_path: Path) -> None: - """An unrefreshed call never invokes llmfit at all, not even to probe.""" - cache_path = tmp_path / "cache.json" - log = tmp_path / "calls.log" - executable = _stub(tmp_path / "bin", _system_fixture(), _fit_fixture(), log=log) - - result = await LlmfitAdvisor( - executable, "1.1.11", 2, cache_path=cache_path - ).scan(8192, refresh=False) - - assert result.scanned is False - assert result.models == () - assert not log.exists() - assert not cache_path.exists() - - -async def test_cache_hit_skips_the_expensive_call(tmp_path: Path) -> None: - """A cache from the last explicit scan answers without invoking the real binary.""" - cache_path = tmp_path / "cache.json" - executable = _stub(tmp_path / "bin", _system_fixture(), _fit_fixture()) - advisor = LlmfitAdvisor(executable, "1.1.11", 2, cache_path=cache_path) - await advisor.scan(8192, refresh=True) - - log = tmp_path / "calls.log" - warm_executable = _stub( - tmp_path / "bin2", _system_fixture(), _fit_fixture(), log=log - ) - warm = LlmfitAdvisor(warm_executable, "1.1.11", 2, cache_path=cache_path) - - result = await warm.scan(8192, refresh=False) - - assert result.scanned is True - assert len(result.models) == 2 - assert not log.exists() - - -async def test_refresh_false_never_reprobes_hardware_to_validate_the_cache( - tmp_path: Path, -) -> None: - """An unrefreshed call trusts the cache as-is, even for a stale fingerprint. - - Re-validating freshness would mean re-probing hardware, which is exactly - the subprocess spawn a page-load call must not pay for. Staleness is only - ever resolved by the next explicit "Scan hardware". - """ - cache_path = tmp_path / "cache.json" - executable = _stub(tmp_path / "bin", _system_fixture(), _fit_fixture()) - await LlmfitAdvisor(executable, "1.1.11", 2, cache_path=cache_path).scan( - 8192, refresh=True - ) - - # A different max_context would have changed a fingerprint, but nothing - # re-probes hardware on an unrefreshed call, so the cache still answers. - result = await LlmfitAdvisor( - executable, "1.1.11", 2, cache_path=cache_path - ).scan(4096, refresh=False) - - assert result.scanned is True - assert len(result.models) == 2 - - -async def test_corrupt_cache_file_is_discarded_silently(tmp_path: Path) -> None: - """A corrupt cache file is treated as a miss, never raises.""" - cache_path = tmp_path / "cache.json" - cache_path.write_text("{not-json") - executable = _stub(tmp_path / "bin", _system_fixture(), _fit_fixture()) - - result = await LlmfitAdvisor( - executable, "1.1.11", 2, cache_path=cache_path - ).scan(8192, refresh=False) - - assert result.scanned is False - assert result.models == () - - -async def test_wrong_cache_version_is_discarded(tmp_path: Path) -> None: - """A cache written by an older schema is discarded, not misread.""" - cache_path = tmp_path / "cache.json" - executable = _stub(tmp_path / "bin", _system_fixture(), _fit_fixture()) - await LlmfitAdvisor(executable, "1.1.11", 2, cache_path=cache_path).scan( - 8192, refresh=True - ) - document = json.loads(cache_path.read_text()) - document["cache_version"] = CACHE_VERSION + 1 - cache_path.write_text(json.dumps(document)) - - result = await LlmfitAdvisor( - executable, "1.1.11", 2, cache_path=cache_path - ).scan(8192, refresh=False) - - assert result.scanned is False - assert result.models == () - - -async def test_cache_from_an_older_llmfit_binary_is_discarded( - tmp_path: Path, -) -> None: - """An app update that bumps the pinned binary can't serve stale scores. - - This check is a free string compare against the pinned constant, unlike - the hardware fingerprint it replaced — so it stays even on the - never-probe path. - """ - cache_path = tmp_path / "cache.json" - executable = _stub(tmp_path / "bin", _system_fixture(), _fit_fixture()) - await LlmfitAdvisor(executable, "1.1.11", 2, cache_path=cache_path).scan( - 8192, refresh=True - ) - - result = await LlmfitAdvisor( - executable, "1.2.0", 2, cache_path=cache_path - ).scan(8192, refresh=False) - - assert result.scanned is False - assert result.models == () - - -async def test_refresh_true_always_rescans_and_overwrites_cache( - tmp_path: Path, -) -> None: - """refresh=True re-scans even when a valid cache already exists.""" - cache_path = tmp_path / "cache.json" - executable = _stub(tmp_path / "bin", _system_fixture(), _fit_fixture()) - advisor = LlmfitAdvisor(executable, "1.1.11", 2, cache_path=cache_path) - await advisor.scan(8192, refresh=True) - - log = tmp_path / "calls.log" - logged_executable = _stub( - tmp_path / "bin2", _system_fixture(), _fit_fixture(), log=log - ) - result = await LlmfitAdvisor( - logged_executable, "1.1.11", 2, cache_path=cache_path - ).scan(8192, refresh=True) - - assert result.scanned is True - # A valid cache already existed, but refresh=True re-ran the expensive - # call anyway rather than trusting it. - assert "fit" in log.read_text() diff --git a/surfsense_local/backend/tests/unit/llm/recommendations/test_ollama_runtime.py b/surfsense_local/backend/tests/unit/llm/recommendations/test_ollama_runtime.py deleted file mode 100644 index 6e60850ba7..0000000000 --- a/surfsense_local/backend/tests/unit/llm/recommendations/test_ollama_runtime.py +++ /dev/null @@ -1,107 +0,0 @@ -import json -from dataclasses import replace -from pathlib import Path - -import pytest - -from modules.llm.providers.ollama.provider import OllamaProvider -from modules.llm.recommendations.types import FitLevel, ScoredModel -from shared.config import get_llm_settings - -pytestmark = pytest.mark.unit - - -def _model(ollama_name: str | None = "qwen3:8b") -> ScoredModel: - return ScoredModel( - canonical_id="Qwen/Qwen3-8B", - publisher="Qwen", - family="Qwen3", - display_name="Qwen3-8B", - parameter_count="8B", - params_b=8, - use_case="chat", - fit=FitLevel.GOOD, - score=80, - runtime="llamacpp", - run_mode="gpu", - best_quant="Q4_K_M", - memory_required_gb=6, - memory_available_gb=16, - utilization_pct=40, - disk_size_gb=5.2, - estimated_tps=30, - prefill_tps=None, - ttft_ms=None, - estimate_confidence="estimated", - estimate_basis=None, - effective_context_length=8192, - capability_ids=("tool_use",), - license="apache-2.0", - ollama_name=ollama_name, - gguf_sources=(), - ) - - -async def test_ollama_resolves_only_advisor_mappings() -> None: - """The renderer cannot invent the tag placed in an install plan.""" - runtime = OllamaProvider("http://127.0.0.1:1") - - assert await runtime.resolve(_model(None)) is None - plan = await runtime.resolve( - replace( - _model(), - best_quant="mlx-4bit", - ollama_quantization="Q4_K_M", - ) - ) - - assert plan is not None - assert plan.model_name == "qwen3:8b" - assert plan.quantization == "Q4_K_M" - assert plan.expected_bytes == 5_200_000_000 - - -async def test_ollama_rejects_another_runtimes_plan() -> None: - """A trusted plan remains bound to the runtime that resolved it.""" - runtime = OllamaProvider("http://127.0.0.1:1") - plan = await runtime.resolve(_model()) - assert plan is not None - - with pytest.raises(ValueError): - runtime.install(replace(plan, runtime="llamacpp")) - - -async def test_cancel_cleanup_removes_only_partial_downloads( - tmp_path: Path, monkeypatch: pytest.MonkeyPatch -) -> None: - """Cancelling reclaims partial and orphan files without deleting shared blobs.""" - blobs = tmp_path / "blobs" - blobs.mkdir() - manifests = tmp_path / "manifests" / "registry.ollama.ai" / "library" / "qwen" - manifests.mkdir(parents=True) - referenced = blobs / "sha256-referenced" - orphan = blobs / "sha256-orphan" - partials = [ - blobs / "sha256-first-partial", - blobs / "sha256-first-partial-0", - blobs / "sha256-second.tmp", - ] - referenced.write_bytes(b"keep") - orphan.write_bytes(b"discard") - for partial in partials: - partial.write_bytes(b"discard") - (manifests / "latest").write_text( - json.dumps( - { - "config": {"digest": "sha256:referenced"}, - "layers": [], - } - ) - ) - monkeypatch.setattr(get_llm_settings(), "ollama_models_dir", tmp_path) - - await OllamaProvider("http://127.0.0.1:1").cleanup_cancelled_download() - - assert referenced.exists() - assert not orphan.exists() - assert not any(partial.exists() for partial in partials) diff --git a/surfsense_local/backend/tests/unit/llm/test_catalog.py b/surfsense_local/backend/tests/unit/llm/test_catalog.py deleted file mode 100644 index d38a62e863..0000000000 --- a/surfsense_local/backend/tests/unit/llm/test_catalog.py +++ /dev/null @@ -1,13 +0,0 @@ -import pytest - -from modules.llm.providers.ollama.provider import OllamaProvider - -pytestmark = pytest.mark.unit - - -def test_the_catalog_is_the_curated_qwen_family() -> None: - """Ollama lists no library, so what it offers is exactly what is curated.""" - names = [entry.name for entry in OllamaProvider("http://unused").catalog()] - - assert names - assert all(name.startswith("qwen") for name in names) diff --git a/surfsense_local/backend/tests/unit/llm/test_ollama_num_ctx.py b/surfsense_local/backend/tests/unit/llm/test_ollama_num_ctx.py deleted file mode 100644 index 26e56f7ab2..0000000000 --- a/surfsense_local/backend/tests/unit/llm/test_ollama_num_ctx.py +++ /dev/null @@ -1,22 +0,0 @@ -import pytest - -from modules.llm.providers.ollama.provider import MIN_CTX, num_ctx -from modules.llm.providers.types import Message - -pytestmark = pytest.mark.unit - - -def test_a_short_chat_keeps_the_default_window() -> None: - """Small prompts must not pay for a window they do not need.""" - assert num_ctx([Message(role="user", content="hi")], max_tokens=12) == MIN_CTX - - -def test_a_full_studio_grounding_gets_a_window_that_holds_it() -> None: - """gather()'s 24k-char budget is ~7k tokens; 4096 silently truncates it.""" - messages = [ - Message(role="system", content="x" * 2_000), - Message(role="user", content="y" * 24_000), - ] - window = num_ctx(messages, max_tokens=None) - assert window >= 26_000 // 3 + 2048 - assert window & (window - 1) == 0 # a power of two, as Ollama sizes its caches From dea61e491e3e1ec8e7ffce30286c33bd2a5e3ffc Mon Sep 17 00:00:00 2001 From: Anish Sarkar <104695310+AnishSarkar22@users.noreply.github.com> Date: Sun, 20 Sep 2026 20:07:43 +0530 Subject: [PATCH 20/60] feat(llm): resolve and select generation models through the llamacpp provider Co-Authored-By: Claude Sonnet 5 --- surfsense_local/backend/api/main.py | 34 ++ .../backend/modules/llm/connections/router.py | 12 +- .../backend/modules/llm/dependencies.py | 21 +- surfsense_local/backend/modules/llm/models.py | 5 +- .../backend/modules/llm/profile/__init__.py | 4 +- .../backend/modules/llm/profile/classify.py | 7 +- .../modules/llm/profile/fingerprint.py | 43 +-- .../backend/modules/llm/profile/types.py | 10 + .../backend/modules/llm/resolution.py | 8 +- surfsense_local/backend/modules/llm/router.py | 127 ++----- .../backend/modules/llm/schemas.py | 73 ---- .../backend/modules/llm/selection.py | 18 +- .../backend/tests/integration/llm/conftest.py | 100 +++-- .../tests/integration/llm/test_connections.py | 67 +++- .../tests/integration/llm/test_routes.py | 341 +++--------------- .../tests/unit/llm/profile/test_classify.py | 14 +- .../unit/llm/profile/test_fingerprint.py | 56 +-- 17 files changed, 334 insertions(+), 606 deletions(-) diff --git a/surfsense_local/backend/api/main.py b/surfsense_local/backend/api/main.py index ade21cb06a..603fda3a09 100644 --- a/surfsense_local/backend/api/main.py +++ b/surfsense_local/backend/api/main.py @@ -1,3 +1,4 @@ +import logging from collections.abc import AsyncIterator from contextlib import asynccontextmanager @@ -16,6 +17,7 @@ from modules.events.router import router as events_router from modules.health.router import router as health_router from modules.license.router import router as license_router +from modules.llm.catalog.dependencies import get_catalog_service from modules.llm.router import router as llm_router from modules.migration.router import router as migration_router from modules.workspaces.router import router as workspaces_router @@ -28,6 +30,7 @@ serving_request, ) from shared.migrations import upgrade_to_head +from shared.secrets import UnreadableSecretError class MarkRequest: @@ -46,6 +49,9 @@ async def __call__(self, scope: Scope, receive: Receive, send: Send) -> None: serving_request.reset(token) +logger = logging.getLogger(__name__) + + @asynccontextmanager async def lifespan(app: FastAPI) -> AsyncIterator[None]: """The API owns migrations; the worker only ever reads and writes rows.""" @@ -57,11 +63,25 @@ async def lifespan(app: FastAPI) -> AsyncIterator[None]: ensure_default_workspace(session) session.commit() app.state.session_factory = session_factory + # The preset is derived from what is on disk, so rebuild it at boot + # rather than only after an install. Without this a model imported from + # disk, or installed before this existed, loads at llama.cpp's own + # default window instead of the one the fit calculation chose, and a + # stale section keeps advertising a model whose file is gone. + _rewrite_generation_preset() yield finally: engine.dispose() +def _rewrite_generation_preset() -> None: + """Best effort: a preset that cannot be written must not stop the API.""" + try: + get_catalog_service().reprice() + except Exception: + logger.exception("could not rewrite the generation preset at startup") + + def create_app() -> FastAPI: """Application factory; each call returns an app isolated from the others.""" import_models() @@ -89,9 +109,23 @@ def create_app() -> FastAPI: app.include_router(license_router) app.include_router(egress_router) app.add_exception_handler(EgressDeniedError, egress_denied) + app.add_exception_handler(UnreadableSecretError, unreadable_secret) return app +def unreadable_secret(_request: Request, error: UnreadableSecretError) -> JSONResponse: + """Handled once, here, because any route touching a stored key can hit it. + + 409 rather than 500: the request is well formed and the server is healthy. + What is wrong is a stored value, and the client is the one that can replace + it. + """ + return JSONResponse( + {"detail": {"code": "unreadable_secret", "message": str(error)}}, + status.HTTP_409_CONFLICT, + ) + + def egress_denied(_request: Request, error: EgressDeniedError) -> JSONResponse: return JSONResponse( { diff --git a/surfsense_local/backend/modules/llm/connections/router.py b/surfsense_local/backend/modules/llm/connections/router.py index fec9831833..01abcde676 100644 --- a/surfsense_local/backend/modules/llm/connections/router.py +++ b/surfsense_local/backend/modules/llm/connections/router.py @@ -32,8 +32,12 @@ DEFAULT_IMAGE_TEST_PROMPT = "A simple blue circle centered on a plain white background." DEFAULT_CHAT_TEST_PROMPT = "Reply with one short sentence confirming you can answer." -# Enough to show the model answers, little enough that testing cannot run a bill up. -CHAT_TEST_MAX_TOKENS = 64 +# Enough that a thinking model reaches its answer, and still little enough that +# testing cannot run a bill up. A plain answer never approaches this, because the +# reply is cut at its character limit as soon as text arrives; only a model +# spending the budget on a reasoning trace gets near it. At 64 such a model +# returned nothing at all, and was reported as broken. +CHAT_TEST_MAX_TOKENS = 1024 CHAT_TEST_MAX_CHARS = 600 @@ -239,7 +243,9 @@ async def test_connection_chat( reply = reply.strip() if not reply: raise HTTPException( - status.HTTP_502_BAD_GATEWAY, "the model answered with no text" + status.HTTP_502_BAD_GATEWAY, + "the model returned no text, which a reasoning model does when it " + "spends the whole reply thinking", ) return ChatTestRead(reply=reply[:CHAT_TEST_MAX_CHARS]) diff --git a/surfsense_local/backend/modules/llm/dependencies.py b/surfsense_local/backend/modules/llm/dependencies.py index d02f265d1b..09d631d7df 100644 --- a/surfsense_local/backend/modules/llm/dependencies.py +++ b/surfsense_local/backend/modules/llm/dependencies.py @@ -2,7 +2,7 @@ from fastapi import Depends, HTTPException, status -from modules.llm.providers import get_provider +from modules.llm.providers import get_provider, llamacpp from modules.llm.providers.protocols import Generator, ModelStore @@ -27,3 +27,22 @@ def get_store_or_409(provider: ProviderDep) -> ModelStore: StoreDep = Annotated[ModelStore, Depends(get_store_or_409)] + + +def get_local_runtime() -> Generator: + """The one runtime that holds models on this machine. + + Routes that manage local files no longer name a provider in their path: + there is exactly one, and asking the caller to spell it invites a request + that names a remote endpoint and gets a confusing error instead of a route + that could never have applied. + """ + found = get_provider(llamacpp.PROVIDER) + if found is None: # pragma: no cover - fixed registry invariant + raise HTTPException( + status.HTTP_503_SERVICE_UNAVAILABLE, "the local runtime is unavailable" + ) + return found + + +LocalRuntimeDep = Annotated[Generator, Depends(get_local_runtime)] diff --git a/surfsense_local/backend/modules/llm/models.py b/surfsense_local/backend/modules/llm/models.py index 3731ea6824..09ab5b7ac9 100644 --- a/surfsense_local/backend/modules/llm/models.py +++ b/surfsense_local/backend/modules/llm/models.py @@ -28,8 +28,9 @@ class SelectedModel(Base): __table_args__ = ( CheckConstraint( # A connection is required exactly when the runtime is remote; - # Ollama and the bundled sd-server both answer on this machine. - "(provider IN ('ollama', 'sdcpp') AND connection_id IS NULL) OR " + # The local text runtime and the bundled sd-server both answer + # on this machine, so neither carries a connection. + "(provider IN ('llamacpp', 'sdcpp') AND connection_id IS NULL) OR " "(provider = 'openai_compatible' AND connection_id IS NOT NULL)", name="provider_connection", ), diff --git a/surfsense_local/backend/modules/llm/profile/__init__.py b/surfsense_local/backend/modules/llm/profile/__init__.py index e0f9712a6f..39534ae31c 100644 --- a/surfsense_local/backend/modules/llm/profile/__init__.py +++ b/surfsense_local/backend/modules/llm/profile/__init__.py @@ -1,7 +1,7 @@ """How a model should be prompted, derived from what is known about it.""" from modules.llm.profile.classify import classify -from modules.llm.profile.fingerprint import from_name, from_ollama, from_remote +from modules.llm.profile.fingerprint import from_llamacpp, from_name, from_remote from modules.llm.profile.types import Fingerprint, Line, Tier __all__ = [ @@ -9,7 +9,7 @@ "Line", "Tier", "classify", + "from_llamacpp", "from_name", - "from_ollama", "from_remote", ] diff --git a/surfsense_local/backend/modules/llm/profile/classify.py b/surfsense_local/backend/modules/llm/profile/classify.py index 56dc4e7d73..dacdd05ea5 100644 --- a/surfsense_local/backend/modules/llm/profile/classify.py +++ b/surfsense_local/backend/modules/llm/profile/classify.py @@ -15,8 +15,11 @@ def classify(fingerprint: Fingerprint) -> Tier: return Tier.FRONTIER if fingerprint.line is not None: return Tier.FRONTIER if fingerprint.line is Line.FLAGSHIP else Tier.CAPABLE - # A hosted endpoint runs models too large for a laptop; Ollama runs the laptop. - return Tier.COMPACT if fingerprint.provider == "ollama" else Tier.CAPABLE + # A hosted endpoint runs models too large for a laptop; a local one runs the + # laptop. Keyed on where the endpoint is, not on which provider name it + # carries: keying on the bundled runtime's name broke the moment someone ran + # a 4B through LM Studio and got capable prompts for a compact model. + return Tier.COMPACT if fingerprint.local else Tier.CAPABLE def _by_size(params_b: float) -> Tier: diff --git a/surfsense_local/backend/modules/llm/profile/fingerprint.py b/surfsense_local/backend/modules/llm/profile/fingerprint.py index 119ab4a2e5..863e7f3ae4 100644 --- a/surfsense_local/backend/modules/llm/profile/fingerprint.py +++ b/surfsense_local/backend/modules/llm/profile/fingerprint.py @@ -20,45 +20,24 @@ # Bytes one weight occupies on disk at each quantization, so a blob size divided # by it is a parameter count. -_BYTES_PER_WEIGHT = (("q4", 0.6), ("q5", 0.7), ("q6", 0.82), ("q8", 1.06), ("f16", 2.0)) -def from_ollama(name: str, tag: dict, show: dict) -> Fingerprint: - """What Ollama knows about an installed model, from /api/tags and /api/show.""" - exact = (show.get("model_info") or {}).get("general.parameter_count") - stated = (tag.get("details") or {}).get("parameter_size") or "" +def from_llamacpp(name: str, props: dict) -> Fingerprint: + """What the local runtime knows about a loaded model, from `GET /props`. + + The previous runtime divided a blob of quantized weights by a bytes-per-weight + guess, which read `gemma3:4b` as 5.5B because embedding tables inflate it. + llama.cpp states the count, so the guess is gone. Where it does not, the + filename usually carries the size and that still beats measuring. + """ + stated = (props.get("model_info") or {}).get("general.parameter_count") return Fingerprint( - provider="ollama", + provider="llamacpp", name=name, params_b=( - round(exact / 1e9, 1) - if exact - # The tag itself usually states the size, and beats measuring the blob. - else _largest_size_b(stated) or _largest_size_b(name) or _estimated_b(tag) - ), - ) - - -def _estimated_b(tag: dict) -> float | None: - """The parameter count a blob of quantized weights implies. - - ponytail: it measures weights only, so large embedding tables inflate it - (gemma3:4b reads 5.5B) and a model within ~20% of a threshold can land in the - wrong tier. Upgrade path: read the count from the GGUF header. - """ - size = tag.get("size") - quantization = ((tag.get("details") or {}).get("quantization_level") or "").lower() - per_weight = next( - ( - bytes_ - for prefix, bytes_ in _BYTES_PER_WEIGHT - if quantization.startswith(prefix) + round(stated / 1e9, 1) if stated else _largest_size_b(name) ), - None, ) - if not size or per_weight is None: - return None - return round(size / per_weight / 1e9, 1) def from_remote(name: str, rows: list[dict]) -> Fingerprint: diff --git a/surfsense_local/backend/modules/llm/profile/types.py b/surfsense_local/backend/modules/llm/profile/types.py index 2387cfd343..58dfab8ac8 100644 --- a/surfsense_local/backend/modules/llm/profile/types.py +++ b/surfsense_local/backend/modules/llm/profile/types.py @@ -26,3 +26,13 @@ class Fingerprint: params_b: float | None = None vendor: str | None = None line: Line | None = None + + @property + def local(self) -> bool: + """Whether this model runs on this machine. + + The fallback tier keys on this rather than on the provider's name: a + hosted endpoint runs models too big for a laptop, a local one runs the + laptop, and that is a fact about where the endpoint is. + """ + return self.provider == "llamacpp" diff --git a/surfsense_local/backend/modules/llm/resolution.py b/surfsense_local/backend/modules/llm/resolution.py index 58d958af17..84af2e51bb 100644 --- a/surfsense_local/backend/modules/llm/resolution.py +++ b/surfsense_local/backend/modules/llm/resolution.py @@ -5,7 +5,7 @@ from modules.egress import service as egress from modules.llm.models import ModelRole, ProviderConnection, SelectedModel from modules.llm.profile import Tier -from modules.llm.providers import get_provider +from modules.llm.providers import get_provider, llamacpp from modules.llm.providers.kokoro import provider as kokoro from modules.llm.providers.openai_compatible import ( OpenAICompatibleChatProvider, @@ -40,10 +40,10 @@ def resolve_generation(session: Session) -> ResolvedGeneration: selected = session.get(SelectedModel, ModelRole.GENERATION) if selected is None: raise ModelResolutionError("no chat model selected") - if selected.provider == "ollama": - provider = get_provider("ollama") + if selected.provider == llamacpp.PROVIDER: + provider = get_provider(llamacpp.PROVIDER) if provider is None: # pragma: no cover - fixed registry invariant - raise ModelResolutionError("Ollama provider is unavailable") + raise ModelResolutionError("the local runtime is unavailable") return ResolvedGeneration(selected, provider) connection = _connection(session, selected) return ResolvedGeneration( diff --git a/surfsense_local/backend/modules/llm/router.py b/surfsense_local/backend/modules/llm/router.py index f27d5b89ed..d8c6ac0c5f 100644 --- a/surfsense_local/backend/modules/llm/router.py +++ b/surfsense_local/backend/modules/llm/router.py @@ -11,17 +11,14 @@ from modules.documents.models import Document, DocumentStatus, DocumentType from modules.egress import service as egress from modules.llm.activity import ModelBusyError, model_activity, model_key +from modules.llm.catalog.dependencies import CatalogServiceDep +from modules.llm.catalog.router import router as catalog_router from modules.llm.connections.router import router as connections_router -from modules.llm.dependencies import ProviderDep, StoreDep +from modules.llm.dependencies import LocalRuntimeDep, ProviderDep from modules.llm.models import ModelRole, OnboardingCompletion, SelectedModel -from modules.llm.providers import get_provider, provider_names -from modules.llm.providers.protocols import ModelStore +from modules.llm.providers import get_provider, llamacpp, provider_names from modules.llm.providers.sdcpp import provider as sdcpp -from modules.llm.recommendations.catalog import clean_runtime_name -from modules.llm.recommendations.dependencies import CatalogServiceDep -from modules.llm.recommendations.router import router as recommendations_router from modules.llm.schemas import ( - CatalogEntryRead, LocalImageCatalogRead, LocalImageModelRead, LocalImageRuntimeRead, @@ -29,14 +26,13 @@ ModelRead, OnboardingStatusRead, ProviderRead, - PullRequest, SelectionRead, SelectionWrite, ) from modules.llm.selection import choose_model, complete_onboarding router = APIRouter(prefix="/llm", tags=["llm"]) -router.include_router(recommendations_router) +router.include_router(catalog_router) router.include_router(connections_router) @@ -68,7 +64,10 @@ async def list_providers() -> list[ProviderRead]: ProviderRead( name=provider.name, healthy=await provider.health(), - can_download=isinstance(provider, ModelStore), + # Downloading is a catalog capability now, not a runtime one: the + # local runtime's models install through POST /llm/install, and a + # remote endpoint's cannot be installed at all. + can_download=provider.name == llamacpp.PROVIDER, requires_key=getattr(provider, "requires_key", False), configured=True, ) @@ -82,62 +81,50 @@ async def list_providers() -> list[ProviderRead]: response_model=list[ModelRead], summary="List installed models", ) -async def list_models( - provider: ProviderDep, catalog_service: CatalogServiceDep -) -> list[ModelRead]: - scan = await catalog_service.advisor_catalog() - display_names = { - model.ollama_name: model.display_name - for model in scan.models - if model.ollama_name - } +async def list_models(provider: ProviderDep) -> list[ModelRead]: + """The runtime reports files by their real names, so there is nothing to + clean up: the previous runtime's registry tags needed tidying, these do not.""" return [ ModelRead( name=model.name, installed=model.installed, capabilities=list(model.capabilities), - display_name=( - model.display_name - or display_names.get(model.name) - or clean_runtime_name(model.name) - ), + display_name=model.display_name or model.name, ) for model in await provider.models() ] @router.delete( - "/providers/{provider}/models/{model_name:path}", + "/models/{model_name:path}", response_model=ModelDeleteRead, summary="Delete an installed local generation model", ) async def delete_model( model_name: str, - store: StoreDep, + store: LocalRuntimeDep, service: CatalogServiceDep, session: SessionDep, ) -> ModelDeleteRead: + # Inventory from disk, not from the runtime. Asking the router first meant a + # dead router made models undeletable, which is backwards: one reason to + # delete a model is that things are broken, and removing a file needs + # nothing running. installed = next( - (model for model in await store.models() if model.name == model_name), - None, + (row for row in service.installed() if row.model_id == model_name), None ) if installed is None: raise HTTPException(status.HTTP_404_NOT_FOUND, f"model not found: {model_name}") - if "completion" not in installed.capabilities: - raise HTTPException( - status.HTTP_409_CONFLICT, - f"model does not support generation: {model_name}", - ) if await transact(session, _studio_running): raise HTTPException( status.HTTP_409_CONFLICT, "a model cannot be deleted while Studio is generating", ) - install_lock = service.install_lock(store.name) + install_lock = service.install_lock() if install_lock.locked(): raise HTTPException( status.HTTP_409_CONFLICT, - f"{store.name} is currently installing a model", + "a model is already being installed", ) await install_lock.acquire() @@ -147,6 +134,14 @@ async def delete_model( await store.delete(model_name) except ModelBusyError as error: raise HTTPException(status.HTTP_409_CONFLICT, str(error)) from error + except FileNotFoundError as error: + raise HTTPException( + status.HTTP_404_NOT_FOUND, f"model not found: {model_name}" + ) from error + # Rewrite the preset so the router drops it on the next restart, and so + # it never advertises a model whose file is gone: a stale section is + # served as a real entry, `source: preset`, and fails when chosen. + service.reprice() finally: install_lock.release() @@ -185,29 +180,6 @@ def _clear_selection(session: Session, provider: str, model_name: str) -> bool: return cleared -@router.get( - "/providers/{provider}/catalog", - response_model=list[CatalogEntryRead], - summary="List models on offer to download", -) -async def list_catalog(provider: ProviderDep) -> list[CatalogEntryRead]: - if not isinstance(provider, ModelStore): - raise HTTPException( - status.HTTP_409_CONFLICT, f"{provider.name} has no catalog to download" - ) - - installed = {model.name for model in await provider.models()} - return [ - CatalogEntryRead( - name=entry.name, - label=entry.label, - size_gb=entry.size_gb, - installed=entry.name in installed, - ) - for entry in provider.catalog() - ] - - @router.get( "/image/local", response_model=LocalImageCatalogRead, @@ -318,45 +290,6 @@ async def _image_server_healthy() -> bool: return False -@router.post( - "/providers/{provider}/pull", - summary="Download a model, streaming progress", -) -async def pull_model( - store: StoreDep, - payload: PullRequest, - service: CatalogServiceDep, - session: SessionDep, -) -> StreamingResponse: - await transact( - session, - egress.require, - egress.OLLAMA_PULL, - egress.ollama_pull_host(payload.name), - ) - lock = service.install_lock(store.name) - if lock.locked(): - raise HTTPException( - status.HTTP_409_CONFLICT, - f"{store.name} is already installing a model", - ) - await lock.acquire() - - async def progress() -> AsyncIterator[bytes]: - try: - async for step in store.pull(payload.name): - line = { - "status": step.status, - "completed": step.completed, - "total": step.total, - } - yield (json.dumps(line) + "\n").encode() - finally: - lock.release() - - return StreamingResponse(progress(), media_type="application/x-ndjson") - - @router.get( "/selection/{role}", response_model=SelectionRead, diff --git a/surfsense_local/backend/modules/llm/schemas.py b/surfsense_local/backend/modules/llm/schemas.py index 1b0cb5e412..441180e92d 100644 --- a/surfsense_local/backend/modules/llm/schemas.py +++ b/surfsense_local/backend/modules/llm/schemas.py @@ -5,7 +5,6 @@ from modules.llm.connections.service import CapabilitySource from modules.llm.models import ModelRole from modules.llm.profile import Tier -from modules.llm.recommendations.types import FitLevel class ProviderRead(BaseModel): @@ -137,78 +136,6 @@ class OnboardingStatusRead(BaseModel): completed: bool -class RecommendationWarningRead(BaseModel): - model_config = ConfigDict(from_attributes=True) - - code: str - message: str - - -class SystemProfileRead(BaseModel): - model_config = ConfigDict(from_attributes=True) - - cpu_name: str | None - cpu_cores: int | None - total_ram_gb: float | None - available_ram_gb: float | None - has_gpu: bool - gpu_name: str | None - gpu_vram_gb: float | None - gpu_count: int - backend: str | None - unified_memory: bool - - -class RecommendationSystemRead(BaseModel): - hardware: SystemProfileRead | None - llmfit_version: str | None - estimates_available: bool - warnings: list[RecommendationWarningRead] - - -class RecommendationRowRead(BaseModel): - model_config = ConfigDict(from_attributes=True) - - catalog_id: str - canonical_id: str - family: str - label: str - publisher: str | None - parameter_count: str | None - fit: FitLevel - score: float | None - memory_required_gb: float | None - disk_size_gb: float | None - estimated_tps: float | None - prefill_tps: float | None - ttft_ms: float | None - effective_context_length: int | None - estimate_confidence: str | None - license: str | None - runtime: str - runtime_model: str - quantization: str | None - installed: bool - selected: bool - can_install: bool - can_delete: bool - warnings: list[str] - - -class RecommendationCatalogRead(BaseModel): - hardware: SystemProfileRead | None - llmfit_version: str | None - curated: list[RecommendationRowRead] - explore: list[RecommendationRowRead] - installed: list[RecommendationRowRead] - # False when served without running the hardware scan (no cache existed - # yet and none was requested) — `curated`/`installed` are still fully - # populated, only `explore` and curated fit badges are scan-derived. - scanned: bool - warnings: list[RecommendationWarningRead] - runtime_status: dict[str, bool] - - class InstallRequest(BaseModel): catalog_id: str = Field(min_length=1, max_length=128) select: bool = True diff --git a/surfsense_local/backend/modules/llm/selection.py b/surfsense_local/backend/modules/llm/selection.py index c67fbb9475..d4dcd84f2d 100644 --- a/surfsense_local/backend/modules/llm/selection.py +++ b/surfsense_local/backend/modules/llm/selection.py @@ -9,7 +9,7 @@ from modules.llm.connections.router import allowed_connection from modules.llm.models import ModelRole, OnboardingCompletion, SelectedModel from modules.llm.profile import Fingerprint, from_name -from modules.llm.providers import get_provider +from modules.llm.providers import get_provider, llamacpp from modules.llm.providers.openai_compatible import OpenAICompatibleChatProvider from modules.llm.providers.sdcpp import provider as sdcpp @@ -32,7 +32,7 @@ async def choose_model( "model name must not be empty", ) - if provider_name == "ollama": + if provider_name == llamacpp.PROVIDER: await _validate_local(role, model_name, connection_id) elif provider_name == sdcpp.PROVIDER: _validate_local_image(role, model_name, connection_id) @@ -69,8 +69,8 @@ async def _collect( ) -> Fingerprint: """Ask the provider what it knows, once, so generation never has to.""" try: - if provider_name == "ollama": - provider = get_provider("ollama") + if provider_name == llamacpp.PROVIDER: + provider = get_provider(llamacpp.PROVIDER) return await provider.inspect(model_name) if provider_name == "openai_compatible": connection = await transact(session, allowed_connection, connection_id) @@ -154,16 +154,18 @@ async def _validate_local( if role is not ModelRole.GENERATION: raise HTTPException( status.HTTP_422_UNPROCESSABLE_CONTENT, - "Ollama does not provide image generation", + "the local text runtime does not generate images", ) if connection_id is not None: raise HTTPException( status.HTTP_422_UNPROCESSABLE_CONTENT, - "Ollama selections must not include a connection", + "a local selection must not name a connection", ) - provider = get_provider("ollama") + provider = get_provider(llamacpp.PROVIDER) if provider is None: # pragma: no cover - fixed registry invariant - raise HTTPException(status.HTTP_503_SERVICE_UNAVAILABLE, "Ollama unavailable") + raise HTTPException( + status.HTTP_503_SERVICE_UNAVAILABLE, "the local runtime is unavailable" + ) model = next( (entry for entry in await provider.models() if entry.name == model_name), None, diff --git a/surfsense_local/backend/tests/integration/llm/conftest.py b/surfsense_local/backend/tests/integration/llm/conftest.py index effb9d4949..9cea46e039 100644 --- a/surfsense_local/backend/tests/integration/llm/conftest.py +++ b/surfsense_local/backend/tests/integration/llm/conftest.py @@ -6,55 +6,54 @@ import pytest +from modules.llm.catalog.dependencies import get_catalog_service from shared.config import get_llm_settings -INSTALLED = ["qwen3:1.7b", "qwen3:4b"] +INSTALLED = ["Qwen3-1.7B-Q4_K_M", "Qwen3-4B-Q4_K_M"] DELETED: list[str] = [] -PULL_STEPS = [ - {"status": "pulling manifest"}, - {"status": "downloading", "completed": 40, "total": 100}, - {"status": "downloading", "completed": 100, "total": 100}, - {"status": "success"}, -] -class StubOllama(BaseHTTPRequestHandler): - """Enough of Ollama's native API for the router to talk to.""" +class StubRouter(BaseHTTPRequestHandler): + """Enough of llama-server's router mode for the adapter to talk to.""" def do_GET(self) -> None: - if self.path == "/": - self._send(b"Ollama is running") - elif self.path == "/api/tags": - self._json({"models": [{"name": name} for name in INSTALLED]}) + parsed = urlsplit(self.path) + if parsed.path == "/health": + self._json({"status": "ok"}) + elif parsed.path == "/props": + self._json({"role": "router", "model_info": {}}) + elif parsed.path == "/models": + self._json( + { + "object": "list", + "data": [ + {"id": name, "status": {"value": "unloaded", "args": []}} + for name in INSTALLED + ], + } + ) else: self.send_error(404) def do_POST(self) -> None: - # Drain the request body so the client's connection can be reused. body = json.loads(self.rfile.read(int(self.headers["Content-Length"]))) - if self.path == "/api/show": - self._json( - { - "capabilities": ["completion", "tools"], - "details": {"quantization_level": "Q4_K_M"}, - } - ) - elif self.path == "/api/pull": - if body["model"] not in INSTALLED: - INSTALLED.append(body["model"]) - lines = "".join(json.dumps(step) + "\n" for step in PULL_STEPS) - self._send(lines.encode()) + if self.path in ("/models/load", "/models/unload"): + self._json({"success": True}) + elif self.path == "/v1/chat/completions": + self._send(b'data: {"choices":[{"delta":{"content":"Hi"}}]}\n\ndata: [DONE]\n\n') else: self.send_error(404) + del body def do_DELETE(self) -> None: - body = json.loads(self.rfile.read(int(self.headers["Content-Length"]))) - if self.path != "/api/delete" or body["model"] not in INSTALLED: + parsed = urlsplit(self.path) + model = parse_qs(parsed.query).get("model", [""])[0] + if parsed.path != "/models" or model not in INSTALLED: self.send_error(404) return - INSTALLED.remove(body["model"]) - DELETED.append(body["model"]) - self._send(b"") + INSTALLED.remove(model) + DELETED.append(model) + self._json({"success": True}) def _json(self, payload: dict) -> None: self._send(json.dumps(payload).encode()) @@ -70,19 +69,32 @@ def log_message(self, *args: object) -> None: @pytest.fixture -def ollama_server(monkeypatch: pytest.MonkeyPatch) -> Iterator[str]: - """A real Ollama stand-in on a real port, pointed to by settings.""" - INSTALLED[:] = ["qwen3:1.7b", "qwen3:4b"] +def llamacpp_server( + monkeypatch: pytest.MonkeyPatch, tmp_path_factory: pytest.TempPathFactory +) -> Iterator[str]: + """A real llama-server stand-in on a real port, pointed to by settings. + + Also stages the same models on disk, because that is where the app reads + its inventory from: the runtime answers questions about models, it is not + the record of which ones exist. + """ + INSTALLED[:] = ["Qwen3-1.7B-Q4_K_M", "Qwen3-4B-Q4_K_M"] DELETED.clear() - server = ThreadingHTTPServer(("127.0.0.1", 0), StubOllama) + models = tmp_path_factory.mktemp("models") + for name in INSTALLED: + (models / f"{name}.gguf").write_bytes(b"GGUF") + monkeypatch.setattr(get_llm_settings(), "llamacpp_models_dir", models) + get_catalog_service.cache_clear() + server = ThreadingHTTPServer(("127.0.0.1", 0), StubRouter) threading.Thread(target=server.serve_forever, daemon=True).start() url = f"http://127.0.0.1:{server.server_port}" - monkeypatch.setattr(get_llm_settings(), "ollama_base_url", url) + monkeypatch.setattr(get_llm_settings(), "llamacpp_base_url", url) yield url server.shutdown() server.server_close() + get_catalog_service.cache_clear() REMOTE_MODELS = [ @@ -93,6 +105,12 @@ def ollama_server(monkeypatch: pytest.MonkeyPatch) -> Iterator[str]: {"id": "black-forest-labs/flux", "architecture": {"output_modalities": ["image"]}}, ] CHAT_DELTAS = ["Hel", "lo"] +# A thinking model on the far end of a connection. It cannot be told to stop, +# because the field that does that is llama.cpp's and a strict endpoint would +# reject it, so the only thing that gets an answer out is a budget it can +# finish thinking inside. +REASONING_DELTAS = ["Okay, "] * 200 +REMOTE_THINKS = False REMOTE_REQUESTS: list[tuple[str, str]] = [] @@ -112,9 +130,15 @@ def do_POST(self) -> None: body = self.rfile.read(int(self.headers["Content-Length"])) REMOTE_REQUESTS.append((self.path, body.decode())) if self.path == "/chat/completions": + budget = json.loads(body).get("max_tokens") + deltas = [("reasoning_content", text) for text in REASONING_DELTAS] * ( + 1 if REMOTE_THINKS else 0 + ) + [("content", text) for text in CHAT_DELTAS] + if budget is not None: + deltas = deltas[:budget] frames = [ - f"data: {json.dumps({'choices': [{'delta': {'content': text}}]})}\n\n" - for text in CHAT_DELTAS + f"data: {json.dumps({'choices': [{'delta': {field: text}}]})}\n\n" + for field, text in deltas ] frames.append("data: [DONE]\n\n") self._send("".join(frames).encode()) diff --git a/surfsense_local/backend/tests/integration/llm/test_connections.py b/surfsense_local/backend/tests/integration/llm/test_connections.py index 7918b8acce..53b3aceec1 100644 --- a/surfsense_local/backend/tests/integration/llm/test_connections.py +++ b/surfsense_local/backend/tests/integration/llm/test_connections.py @@ -5,11 +5,13 @@ import pytest from httpx import AsyncClient +from modules.llm.connections.router import CHAT_TEST_MAX_TOKENS from modules.llm.providers.openai_compatible import OpenAICompatibleChatProvider from modules.llm.providers.sdcpp import provider as sdcpp from modules.llm.providers.types import Message from shared.config import get_llm_settings +from . import conftest from .conftest import REMOTE_REQUESTS pytestmark = pytest.mark.integration @@ -179,12 +181,36 @@ async def test_chat_test_answers_without_selecting_or_running_up_a_bill( path, body = REMOTE_REQUESTS[-1] assert path == "/chat/completions" - assert json.loads(body)["max_tokens"] == 64 + # Only a model that spends the budget thinking ever reaches this cap: the + # reply is cut at its character limit as soon as text arrives, so a plain + # answer costs the same handful of tokens it always did. + assert json.loads(body)["max_tokens"] == CHAT_TEST_MAX_TOKENS # Trying a model is not choosing it. assert (await client.get("/llm/selection/generation")).status_code == 404 +async def test_a_thinking_model_on_a_connection_is_not_called_broken( + client: AsyncClient, openai_server: str, monkeypatch: pytest.MonkeyPatch +) -> None: + """It answers, after a trace longer than the old 64 token cap allowed. + + A remote endpoint cannot be told to stop thinking, so the budget is what + decides whether the user is shown the answer or told their working model + replied with nothing. + """ + monkeypatch.setattr(conftest, "REMOTE_THINKS", True) + connection = await _connect(client, openai_server) + + tested = await client.post( + f"/llm/connections/{connection['id']}/chat-test", + json={"model": "anthropic/claude-3.5-sonnet"}, + ) + + assert tested.status_code == 200 + assert tested.json() == {"reply": "Hello"} + + def _stage( directory: Path, monkeypatch: pytest.MonkeyPatch ) -> sdcpp.ImageModel: @@ -279,7 +305,7 @@ async def test_local_image_model_is_silent_on_a_host_without_sd_server( assert refused.status_code == 409 -async def test_image_model_downloads_are_listed_and_refusable_like_ollama( +async def test_image_model_downloads_are_listed_and_refusable_too( client: AsyncClient, tmp_path: Path, monkeypatch: pytest.MonkeyPatch ) -> None: """Weights come from huggingface.co, so Network must name it and hold it off.""" @@ -377,3 +403,40 @@ async def test_unverified_and_unlisted_paths_require_explicit_confirmation( }, ) assert confirmed.status_code == 200 + + +async def test_a_key_the_app_can_no_longer_read_is_explained_not_a_crash( + client: AsyncClient, openai_server: str, engine +) -> None: + """Reachable without tampering, so it cannot answer with a stack trace. + + The per install secret lives in the OS keychain. A keychain reset, or a + backup restored onto another machine, leaves every stored key + undecryptable. The key is gone either way; the only thing left to decide is + whether the app says so or returns an Internal Server Error and leaves the + connection list broken with no route to recovery. + """ + connection = await _connect(client, openai_server) + row = await client.get("/llm/connections") + assert row.status_code == 200 + + # Damage the stored ciphertext exactly as a changed secret would. + from sqlalchemy import update + + from modules.llm.models import ProviderConnection + from shared.db import create_session_factory + + with create_session_factory(engine)() as session: + session.execute( + update(ProviderConnection) + .where(ProviderConnection.id == connection["id"]) + .values(api_key_ciphertext=b"not-a-token-this-key-changed") + ) + session.commit() + + listed = await client.get(f"/llm/connections/{connection['id']}/models") + + assert listed.status_code != 500 + detail = listed.json()["detail"] + assert detail["code"] == "unreadable_secret" + assert "key" in detail["message"].casefold() diff --git a/surfsense_local/backend/tests/integration/llm/test_routes.py b/surfsense_local/backend/tests/integration/llm/test_routes.py index 50a80d08a9..3bcffdc12a 100644 --- a/surfsense_local/backend/tests/integration/llm/test_routes.py +++ b/surfsense_local/backend/tests/integration/llm/test_routes.py @@ -8,8 +8,8 @@ from modules.documents.models import Document, DocumentStatus, DocumentType from modules.llm import providers as registry from modules.llm.activity import model_activity, model_key +from modules.llm.catalog.dependencies import get_catalog_service from modules.llm.providers.types import Message, Model -from modules.llm.recommendations.dependencies import get_catalog_service from modules.workspaces.models import Workspace from shared.config import get_llm_settings from shared.db import create_session_factory @@ -18,55 +18,24 @@ async def test_providers_report_health_and_capability( - client: AsyncClient, ollama_server: str + client: AsyncClient, llamacpp_server: str ) -> None: """The download UI is shown by capability, not by a provider's name.""" body = (await client.get("/llm/providers")).json() - ollama = next(entry for entry in body if entry["name"] == "ollama") - assert ollama["healthy"] is True - assert ollama["can_download"] is True + local = next(entry for entry in body if entry["name"] == "llamacpp") + assert local["healthy"] is True + assert local["can_download"] is True async def test_installed_models_carry_their_capabilities( - client: AsyncClient, ollama_server: str + client: AsyncClient, llamacpp_server: str ) -> None: """A chosen model has to be one the runtime actually holds.""" - body = (await client.get("/llm/providers/ollama/models")).json() + body = (await client.get("/llm/providers/llamacpp/models")).json() - assert {model["name"] for model in body} == {"qwen3:1.7b", "qwen3:4b"} - assert "tools" in body[0]["capabilities"] - - -async def test_unscanned_hf_co_model_gets_a_clean_display_name( - client: AsyncClient, ollama_server: str -) -> None: - """No scan has resolved this model yet, so the route's own fallback - must clean the raw `hf.co/...` pull name itself, not just pass it - through — this is what the "Chat: ..." summary elsewhere reads.""" - from tests.integration.llm.conftest import INSTALLED - - INSTALLED.append("hf.co/bartowski/Meta-Llama-3.1-8B-Instruct-GGUF:latest") - - body = (await client.get("/llm/providers/ollama/models")).json() - - entry = next( - model - for model in body - if model["name"] == "hf.co/bartowski/Meta-Llama-3.1-8B-Instruct-GGUF:latest" - ) - assert entry["display_name"] == "bartowski/Meta-Llama-3.1-8B-Instruct-GGUF" - - -async def test_the_catalog_marks_what_is_installed( - client: AsyncClient, ollama_server: str -) -> None: - """The offer list greys what is already here.""" - body = (await client.get("/llm/providers/ollama/catalog")).json() - - entries = {entry["name"]: entry for entry in body} - assert entries["qwen3:1.7b"]["installed"] is True - assert entries["qwen3:32b"]["installed"] is False + assert {model["name"] for model in body} == {"Qwen3-1.7B-Q4_K_M", "Qwen3-4B-Q4_K_M"} + assert "completion" in body[0]["capabilities"] async def test_an_unknown_provider_is_a_404(client: AsyncClient) -> None: @@ -74,46 +43,30 @@ async def test_an_unknown_provider_is_a_404(client: AsyncClient) -> None: assert (await client.get("/llm/providers/openai/models")).status_code == 404 -async def test_pull_streams_progress(client: AsyncClient, ollama_server: str) -> None: - """The client needs progress, not one reply after minutes of silence.""" - await client.put("/egress/ollama_pull", json={"enabled": True}) - steps = [] - async with client.stream( - "POST", "/llm/providers/ollama/pull", json={"name": "qwen3:1.7b"} - ) as reply: - assert reply.status_code == 200 - async for line in reply.aiter_lines(): - if line: - steps.append(json.loads(line)) - - assert steps[0]["status"] == "pulling manifest" - assert steps[-1]["status"] == "success" - - async def test_the_selection_is_read_after_it_is_set( - client: AsyncClient, ollama_server: str + client: AsyncClient, llamacpp_server: str ) -> None: """Set persists the choice; read is how the rest of the app learns it.""" assert (await client.get("/llm/selection/generation")).status_code == 404 written = await client.put( "/llm/selection/generation", - json={"provider": "ollama", "name": "qwen3:4b"}, + json={"provider": "llamacpp", "name": "Qwen3-4B-Q4_K_M"}, ) assert written.status_code == 200 - assert written.json()["name"] == "qwen3:4b" + assert written.json()["name"] == "Qwen3-4B-Q4_K_M" read = await client.get("/llm/selection/generation") assert read.json() == written.json() async def test_the_selection_says_which_prompt_tier_the_model_gets( - client: AsyncClient, ollama_server: str + client: AsyncClient, llamacpp_server: str ) -> None: """A 1.7B model asks for a different prompt than a hosted frontier one.""" await client.put( "/llm/selection/generation", - json={"provider": "ollama", "name": "qwen3:1.7b"}, + json={"provider": "llamacpp", "name": "Qwen3-1.7B-Q4_K_M"}, ) read = (await client.get("/llm/selection/generation")).json() @@ -122,14 +75,14 @@ async def test_the_selection_says_which_prompt_tier_the_model_gets( async def test_selecting_a_chat_model_does_not_complete_onboarding( - client: AsyncClient, ollama_server: str + client: AsyncClient, llamacpp_server: str ) -> None: """Use persists the chat model; Continue writes the completion marker.""" assert (await client.get("/llm/onboarding")).json() == {"completed": False} await client.put( "/llm/selection/generation", - json={"provider": "ollama", "name": "qwen3:1.7b"}, + json={"provider": "llamacpp", "name": "Qwen3-1.7B-Q4_K_M"}, ) assert (await client.get("/llm/onboarding")).json() == {"completed": False} @@ -148,26 +101,29 @@ async def test_onboarding_cannot_complete_without_a_chat_model( async def test_deleting_the_selected_local_model_clears_only_the_selection( - client: AsyncClient, ollama_server: str + client: AsyncClient, llamacpp_server: str ) -> None: """Deleting the active model preserves completed onboarding.""" await client.put( "/llm/selection/generation", - json={"provider": "ollama", "name": "qwen3:1.7b"}, + json={"provider": "llamacpp", "name": "Qwen3-1.7B-Q4_K_M"}, ) assert (await client.post("/llm/onboarding")).status_code == 200 - deleted = await client.delete("/llm/providers/ollama/models/qwen3%3A1.7b") + deleted = await client.delete("/llm/models/Qwen3-1.7B-Q4_K_M") assert deleted.status_code == 200 assert deleted.json() == { - "name": "qwen3:1.7b", + "name": "Qwen3-1.7B-Q4_K_M", "selection_cleared": True, } assert (await client.get("/llm/selection/generation")).status_code == 404 assert (await client.get("/llm/onboarding")).json() == {"completed": True} - models = (await client.get("/llm/providers/ollama/models")).json() - assert [model["name"] for model in models] == ["qwen3:4b"] + # Disk is the inventory, and it is what the catalog reads. The router still + # lists the model until its next restart, which the preset rewrite triggers: + # it scans its directory once at startup and has no way to be told otherwise. + installed = (await client.get("/llm/catalog")).json()["installed"] + assert [row["model_id"] for row in installed] == ["Qwen3-4B-Q4_K_M"] async def test_remote_models_cannot_be_deleted(client: AsyncClient) -> None: @@ -179,53 +135,14 @@ async def test_remote_models_cannot_be_deleted(client: AsyncClient) -> None: assert reply.status_code == 404 -async def test_embedding_models_cannot_be_deleted_from_chat_management( - client: AsyncClient, monkeypatch: pytest.MonkeyPatch -) -> None: - """Deletion is limited to local generation models.""" - - class EmbeddingStore: - name = "embedding-store" - - async def health(self) -> bool: - return True - - async def models(self) -> list[Model]: - return [Model("nomic-embed-text", True, ("embedding",))] - - def catalog(self) -> list[object]: - return [] - - async def pull(self, name: str): - if False: # pragma: no cover - makes this an async iterator - yield name - - async def delete(self, name: str) -> None: # pragma: no cover - raise AssertionError(f"must not delete {name}") - - def chat(self, model: str, messages: list[Message]): # pragma: no cover - raise NotImplementedError - - monkeypatch.setitem(registry.REGISTRY, "embedding-store", EmbeddingStore) - - reply = await client.delete( - "/llm/providers/embedding-store/models/nomic-embed-text" - ) - - assert reply.status_code == 409 - assert ( - reply.json()["detail"] == "model does not support generation: nomic-embed-text" - ) - - async def test_a_model_in_use_cannot_be_deleted( - client: AsyncClient, ollama_server: str + client: AsyncClient, llamacpp_server: str ) -> None: """Deletion cannot race an active generation.""" - key = model_key("ollama", "qwen3:1.7b") + key = model_key("llamacpp", "Qwen3-1.7B-Q4_K_M") await model_activity.acquire_use(key) try: - reply = await client.delete("/llm/providers/ollama/models/qwen3%3A1.7b") + reply = await client.delete("/llm/models/Qwen3-1.7B-Q4_K_M") finally: await model_activity.release_use(key) @@ -233,23 +150,23 @@ async def test_a_model_in_use_cannot_be_deleted( assert reply.json()["detail"] == "model is currently in use" -async def test_a_model_cannot_be_deleted_while_ollama_is_installing( - client: AsyncClient, ollama_server: str +async def test_a_model_cannot_be_deleted_while_one_is_installing( + client: AsyncClient, llamacpp_server: str ) -> None: """One runtime mutation cannot overlap another.""" - lock = get_catalog_service().install_lock("ollama") + lock = get_catalog_service().install_lock() await lock.acquire() try: - reply = await client.delete("/llm/providers/ollama/models/qwen3%3A1.7b") + reply = await client.delete("/llm/models/Qwen3-1.7B-Q4_K_M") finally: lock.release() assert reply.status_code == 409 - assert reply.json()["detail"] == "ollama is currently installing a model" + assert reply.json()["detail"] == "a model is already being installed" async def test_a_model_cannot_be_deleted_while_studio_is_generating( - client: AsyncClient, engine: Engine, ollama_server: str + client: AsyncClient, engine: Engine, llamacpp_server: str ) -> None: """The API sees model work running in the separate Studio worker.""" with create_session_factory(engine)() as session: @@ -271,7 +188,7 @@ async def test_a_model_cannot_be_deleted_while_studio_is_generating( ) session.commit() - reply = await client.delete("/llm/providers/ollama/models/qwen3%3A1.7b") + reply = await client.delete("/llm/models/Qwen3-1.7B-Q4_K_M") assert reply.status_code == 409 assert ( @@ -280,17 +197,17 @@ async def test_a_model_cannot_be_deleted_while_studio_is_generating( async def test_choosing_again_updates_in_place( - client: AsyncClient, ollama_server: str + client: AsyncClient, llamacpp_server: str ) -> None: """One row per role: the second choice replaces the first, not adds to it.""" await client.put( - "/llm/selection/generation", json={"provider": "ollama", "name": "qwen3:1.7b"} + "/llm/selection/generation", json={"provider": "llamacpp", "name": "Qwen3-1.7B-Q4_K_M"} ) await client.put( - "/llm/selection/generation", json={"provider": "ollama", "name": "qwen3:4b"} + "/llm/selection/generation", json={"provider": "llamacpp", "name": "Qwen3-4B-Q4_K_M"} ) - assert (await client.get("/llm/selection/generation")).json()["name"] == "qwen3:4b" + assert (await client.get("/llm/selection/generation")).json()["name"] == "Qwen3-4B-Q4_K_M" async def test_a_selection_names_a_known_provider(client: AsyncClient) -> None: @@ -302,12 +219,12 @@ async def test_a_selection_names_a_known_provider(client: AsyncClient) -> None: async def test_a_selection_names_an_installed_model( - client: AsyncClient, ollama_server: str + client: AsyncClient, llamacpp_server: str ) -> None: """A stale or invented model name is rejected before it reaches chat.""" reply = await client.put( "/llm/selection/generation", - json={"provider": "ollama", "name": "does-not-exist"}, + json={"provider": "llamacpp", "name": "does-not-exist"}, ) assert reply.status_code == 422 @@ -321,7 +238,7 @@ async def test_a_generation_selection_requires_completion_capability( """An installed embedding model cannot be selected to answer chat.""" class EmbeddingOnly: - name = "ollama" + name = "llamacpp" async def health(self) -> bool: return True @@ -332,11 +249,11 @@ async def models(self) -> list[Model]: def chat(self, model: str, messages: list[Message]): # pragma: no cover raise NotImplementedError - monkeypatch.setitem(registry.REGISTRY, "ollama", EmbeddingOnly) + monkeypatch.setitem(registry.REGISTRY, "llamacpp", EmbeddingOnly) reply = await client.put( "/llm/selection/generation", - json={"provider": "ollama", "name": "embedder"}, + json={"provider": "llamacpp", "name": "embedder"}, ) assert reply.status_code == 422 @@ -344,32 +261,6 @@ def chat(self, model: str, messages: list[Message]): # pragma: no cover assert (await client.get("/llm/selection/generation")).status_code == 404 -async def test_a_chat_only_provider_hides_the_catalog( - client: AsyncClient, monkeypatch: pytest.MonkeyPatch -) -> None: - """A remote API that cannot download offers no catalog to download from.""" - - class Echo: - name = "echo" - - async def health(self) -> bool: - return True - - async def models(self) -> list[Model]: - return [] - - def chat(self, model: str, messages: list[Message]): # pragma: no cover - raise NotImplementedError - - monkeypatch.setitem(registry.REGISTRY, "echo", lambda: Echo()) - - listed = (await client.get("/llm/providers")).json() - echo = next(entry for entry in listed if entry["name"] == "echo") - assert echo["can_download"] is False - - assert (await client.get("/llm/providers/echo/catalog")).status_code == 409 - - def _configure_llmfit( tmp_path: Path, monkeypatch: pytest.MonkeyPatch, @@ -394,7 +285,7 @@ def _configure_llmfit( "disk_size_gb": 5.2, "effective_context_length": 8192, "capability_ids": ["tool_use"], - "ollama_name": "qwen3:8b", + "file": "Qwen3-8B-Q4_K_M.gguf", "gguf_sources": [], } ] @@ -412,145 +303,3 @@ def _configure_llmfit( monkeypatch.setattr(get_llm_settings(), "llmfit_path", executable) get_catalog_service.cache_clear() - -async def test_catalog_exposes_one_row_per_runtime_target( - client: AsyncClient, - ollama_server: str, - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - """The API keeps the curated Qwen row when llmfit maps Base to the same tag.""" - common = { - "provider": "Qwen", - "parameter_count": "1.7B", - "use_case": "general", - "fit_level": "perfect", - "runtime": "llamacpp", - "run_mode": "gpu", - "best_quant": "Q4_K_M", - "memory_required_gb": 2, - "memory_available_gb": 16, - "disk_size_gb": 2, - "effective_context_length": 8192, - "capability_ids": ["tool_use"], - "gguf_sources": [], - } - _configure_llmfit( - tmp_path, - monkeypatch, - [ - { - **common, - "name": "Qwen/Qwen3-1.7B-Base", - "score": 72.8, - "ollama_name": "qwen3:1.7b", - }, - { - **common, - "name": "Qwen/Qwen3-1.7B", - "score": 69.2, - "ollama_name": None, - }, - ], - ) - - catalog = (await client.get("/llm/catalog?refresh=true")).json() - - matching = [ - row - for section in ("curated", "explore", "installed") - for row in catalog[section] - if row["runtime_model"] == "qwen3:1.7b" - ] - assert [row["canonical_id"] for row in matching] == ["Qwen/Qwen3-1.7B"] - get_catalog_service.cache_clear() - - -async def test_ranked_catalog_installs_and_selects_in_one_stream( - client: AsyncClient, - ollama_server: str, - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - """The normalized route resolves an opaque id, installs, and persists selection.""" - _configure_llmfit(tmp_path, monkeypatch) - catalog = (await client.get("/llm/catalog?refresh=true")).json() - - # The real curated manifest has 8 entries; only this one was actually - # scanned with real fit data (the rest fall back to unscored placeholders - # for models the fixture's fit response didn't include). - scanned_curated = [ - row for row in catalog["curated"] if row["fit"] != "unknown" - ] - assert [row["canonical_id"] for row in scanned_curated] == ["Qwen/Qwen3-8B"] - catalog_id = scanned_curated[0]["catalog_id"] - - await client.put("/egress/ollama_pull", json={"enabled": True}) - events = [] - async with client.stream( - "POST", - "/llm/install", - json={"catalog_id": catalog_id, "select": True}, - ) as response: - assert response.status_code == 200 - async for line in response.aiter_lines(): - if line: - events.append(json.loads(line)) - - assert [event["type"] for event in events] == [ - "starting", - "downloading", - "downloading", - "downloading", - "downloading", - "verifying", - "selecting", - "complete", - ] - assert events[-1]["selection"]["name"] == "qwen3:8b" - assert (await client.get("/llm/selection/generation")).json()["name"] == "qwen3:8b" - get_catalog_service.cache_clear() - - -async def test_refresh_makes_old_catalog_ids_unusable( - client: AsyncClient, - ollama_server: str, - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - """A refresh revokes prior artifact choices before download starts.""" - _configure_llmfit(tmp_path, monkeypatch) - first = (await client.get("/llm/catalog?refresh=true")).json() - old_id = next( - row["catalog_id"] for row in first["curated"] if row["fit"] != "unknown" - ) - await client.get("/llm/catalog?refresh=true") - - response = await client.post( - "/llm/install", json={"catalog_id": old_id, "select": True} - ) - - assert response.status_code == 422 - get_catalog_service.cache_clear() - - -async def test_missing_llmfit_keeps_installed_models_available( - client: AsyncClient, - ollama_server: str, - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - """Recommendation failure cannot hide models that can still answer chat.""" - monkeypatch.setattr(get_llm_settings(), "llmfit_path", tmp_path / "missing") - get_catalog_service.cache_clear() - - # An unrefreshed call never probes llmfit, so a missing binary stays - # silent (scan-free content instead) until an explicit rescan asks. - catalog = (await client.get("/llm/catalog?refresh=true")).json() - - assert {row["runtime_model"] for row in catalog["installed"]} == { - "qwen3:1.7b", - "qwen3:4b", - } - assert catalog["warnings"][0]["code"] == "missing" - get_catalog_service.cache_clear() diff --git a/surfsense_local/backend/tests/unit/llm/profile/test_classify.py b/surfsense_local/backend/tests/unit/llm/profile/test_classify.py index d2a8621d53..7de3071ef4 100644 --- a/surfsense_local/backend/tests/unit/llm/profile/test_classify.py +++ b/surfsense_local/backend/tests/unit/llm/profile/test_classify.py @@ -7,14 +7,14 @@ def test_a_stated_parameter_count_decides_the_tier() -> None: """A count needs no inference, so it outranks every other signal.""" - tier = classify(Fingerprint(provider="ollama", name="qwen3:8b", params_b=8.0)) + tier = classify(Fingerprint(provider="llamacpp", name="Qwen3-8B-Q4_K_M", params_b=8.0)) assert tier is Tier.CAPABLE def test_a_model_too_small_for_a_scaffold_gets_the_compact_prompt() -> None: """Steps and worked examples cost a 3B model accuracy instead of buying it.""" - tier = classify(Fingerprint(provider="ollama", name="llama3.2:3b", params_b=3.2)) + tier = classify(Fingerprint(provider="llamacpp", name="Llama-3.2-3B-Q4_K_M", params_b=3.2)) assert tier is Tier.COMPACT @@ -22,7 +22,7 @@ def test_a_model_too_small_for_a_scaffold_gets_the_compact_prompt() -> None: def test_a_model_large_enough_to_outgrow_a_scaffold_gets_the_frontier_prompt() -> None: """Past this size a model writes better from judgement than from steps.""" tier = classify( - Fingerprint(provider="ollama", name="deepseek-v3:671b", params_b=671.0) + Fingerprint(provider="llamacpp", name="DeepSeek-V3-Q4_K_M", params_b=671.0) ) assert tier is Tier.FRONTIER @@ -53,8 +53,12 @@ def test_a_hosted_model_with_no_count_is_placed_by_its_line() -> None: def test_a_model_revealing_nothing_falls_back_to_what_its_provider_serves() -> None: - """Ollama runs on the user's own machine; a hosted endpoint runs on a cluster.""" - local = classify(Fingerprint(provider="ollama", name="custom:latest")) + """The local runtime runs on the user's machine; a hosted one runs a cluster. + + Keyed on locality rather than on the provider's name, so a small model served + through some other local endpoint is still compact. + """ + local = classify(Fingerprint(provider="llamacpp", name="custom-model")) remote = classify(Fingerprint(provider="openai_compatible", name="internal-v2")) assert (local, remote) == (Tier.COMPACT, Tier.CAPABLE) diff --git a/surfsense_local/backend/tests/unit/llm/profile/test_fingerprint.py b/surfsense_local/backend/tests/unit/llm/profile/test_fingerprint.py index 17e288f7af..48890d282a 100644 --- a/surfsense_local/backend/tests/unit/llm/profile/test_fingerprint.py +++ b/surfsense_local/backend/tests/unit/llm/profile/test_fingerprint.py @@ -1,6 +1,6 @@ import pytest -from modules.llm.profile import Line, from_name, from_ollama, from_remote +from modules.llm.profile import Line, from_llamacpp, from_name, from_remote pytestmark = pytest.mark.unit @@ -105,55 +105,29 @@ def test_a_name_from_an_endpoint_that_describes_nothing_reveals_nothing() -> Non assert fingerprint.line is None -def test_ollama_states_the_count_and_it_is_taken_as_stated() -> None: - """Any registry model carries an exact count, so nothing needs estimating.""" - fingerprint = from_ollama( - "qwen3:8b", - tag={"details": {"parameter_size": "8.0B", "quantization_level": "Q4_K_M"}}, - show={"model_info": {"general.parameter_count": 8030261248}}, +def test_a_stated_parameter_count_is_used_exactly() -> None: + """`GET /props` carries the real count, so nothing has to be inferred.""" + fingerprint = from_llamacpp( + "Qwen3-8B-Q4_K_M", + {"model_info": {"general.parameter_count": 8030261248}}, ) assert fingerprint.params_b == 8.0 -def test_an_ollama_tag_states_the_size_in_the_tag_itself() -> None: - """The tag is where a pulled model states its size, and it cannot be stale.""" - fingerprint = from_ollama("qwen3:1.7b", tag={}, show={}) +def test_a_filename_states_the_size_when_the_runtime_does_not() -> None: + """Quantizers put the size in the name, and it beats guessing from bytes.""" + fingerprint = from_llamacpp("Qwen3-1.7B-Q4_K_M", {}) assert fingerprint.params_b == 1.7 -def test_a_local_build_that_states_nothing_is_estimated_from_its_weights() -> None: - """A Modelfile build can strip every field, but the blob on disk cannot lie.""" - fingerprint = from_ollama( - "custom:latest", - tag={"size": 4_661_211_808, "details": {"quantization_level": "Q4_0"}}, - show={}, - ) - - assert fingerprint.params_b == 7.8 - - -def test_the_tag_row_still_states_a_size_when_show_fails() -> None: - """Stating beats estimating, so the blob is only the last resort.""" - fingerprint = from_ollama( - "qwen3:1.7b", - tag={ - "size": 1_400_000_000, - "details": {"parameter_size": "1.7B", "quantization_level": "Q4_K_M"}, - }, - show={}, - ) - - assert fingerprint.params_b == 1.7 - - -def test_a_size_in_the_name_outranks_the_vendor_prefix() -> None: - """gpt-oss is open weights OpenAI hosts, so 120B decides, not the prefix.""" - fingerprint = from_name("openai_compatible", "openai/gpt-oss-120b") - - assert fingerprint.params_b == 120.0 - assert fingerprint.vendor is None +def test_a_model_that_states_nothing_anywhere_stays_unknown() -> None: + """Unknown is honest. The old blob-size division read `gemma3:4b` as 5.5B, + because embedding tables inflate a count taken from weights alone, and a + wrong number is worse than none: it picks the wrong prompt with confidence. + """ + assert from_llamacpp("custom-model", {}).params_b is None def test_total_parameters_win_over_active_ones() -> None: From c0f9cd2bab0bc7d38c69901609dcd8327220b997 Mon Sep 17 00:00:00 2001 From: Anish Sarkar <104695310+AnishSarkar22@users.noreply.github.com> Date: Sun, 20 Sep 2026 20:07:43 +0530 Subject: [PATCH 21/60] feat(egress): split the ollama_pull grant into model_download and model_search Co-Authored-By: Claude Sonnet 5 --- .../backend/modules/egress/models.py | 2 +- .../backend/modules/egress/service.py | 27 ++++--- .../tests/integration/llm/test_egress.py | 70 +++---------------- .../backend/tests/unit/egress/test_service.py | 11 --- .../frontend/src/features/egress/api.ts | 13 +++- 5 files changed, 37 insertions(+), 86 deletions(-) diff --git a/surfsense_local/backend/modules/egress/models.py b/surfsense_local/backend/modules/egress/models.py index 135258a57d..8d98528b8d 100644 --- a/surfsense_local/backend/modules/egress/models.py +++ b/surfsense_local/backend/modules/egress/models.py @@ -8,7 +8,7 @@ class EgressDestination(Base): __tablename__ = "egress_destinations" - # "ollama_pull", or "host:" for a BYO provider. + # "model_download", or "host:" for a BYO provider. destination: Mapped[str] = mapped_column(primary_key=True) enabled: Mapped[bool] = mapped_column(default=False) last_call_at: Mapped[datetime | None] diff --git a/surfsense_local/backend/modules/egress/service.py b/surfsense_local/backend/modules/egress/service.py index 9852291e19..dcf9f4d0ff 100644 --- a/surfsense_local/backend/modules/egress/service.py +++ b/surfsense_local/backend/modules/egress/service.py @@ -8,13 +8,22 @@ from modules.egress.models import EgressDestination from modules.llm.models import ProviderConnection -OLLAMA_PULL = "ollama_pull" IMAGE_MODEL_PULL = "image_model_pull" +# Two destinations on one host, because they are two different consents. +# `model_download` is a repo the user named and asked for. `model_search` is +# text they are typing, sent as they type it. Collapsing them into one would +# make allowing a download also allow everything typed into a search box. +MODEL_DOWNLOAD = "model_download" +MODEL_SEARCH = "model_search" HOST_PREFIX = "host:" # Named, not host:-prefixed, so downloading weights is listed and revocable in # Settings > Network like any other call, and reads as a download rather than as # a connection that would carry prompts and documents. -HOSTS = {OLLAMA_PULL: "registry.ollama.ai", IMAGE_MODEL_PULL: "huggingface.co"} +HOSTS = { + IMAGE_MODEL_PULL: "huggingface.co", + MODEL_DOWNLOAD: "huggingface.co", + MODEL_SEARCH: "huggingface.co", +} class EgressDeniedError(Exception): @@ -28,13 +37,6 @@ def host_of(destination: str) -> str: return HOSTS.get(destination) or destination.removeprefix(HOST_PREFIX) -def ollama_pull_host(model_name: str) -> str: - """Ollama fetches an `hf.co/` pull straight from Hugging Face, not its own registry.""" - if model_name.startswith("hf.co/"): - return "huggingface.co" - return HOSTS[OLLAMA_PULL] - - def is_destination(value: str) -> bool: return value in HOSTS or ( value.startswith(HOST_PREFIX) and len(value) > len(HOST_PREFIX) @@ -88,5 +90,10 @@ def list_destinations(session: Session) -> list[EgressDestination]: return [ rows.get(destination) or EgressDestination(destination=destination, enabled=False) - for destination in [OLLAMA_PULL, IMAGE_MODEL_PULL, *sorted(hosts)] + for destination in [ + IMAGE_MODEL_PULL, + MODEL_DOWNLOAD, + MODEL_SEARCH, + *sorted(hosts), + ] ] diff --git a/surfsense_local/backend/tests/integration/llm/test_egress.py b/surfsense_local/backend/tests/integration/llm/test_egress.py index be89e2aa1d..d6ba17a554 100644 --- a/surfsense_local/backend/tests/integration/llm/test_egress.py +++ b/surfsense_local/backend/tests/integration/llm/test_egress.py @@ -7,8 +7,8 @@ from httpx import AsyncClient from sqlalchemy import Engine +from modules.llm.catalog.dependencies import get_catalog_service from modules.llm.models import ProviderConnection -from modules.llm.recommendations.dependencies import get_catalog_service from shared.config import get_llm_settings from shared.db import create_session_factory @@ -27,36 +27,6 @@ async def _destinations(client: AsyncClient) -> dict[str, dict]: return {row["destination"]: row for row in reply.json()} -async def test_pull_is_refused_until_allowed_then_recorded( - client: AsyncClient, ollama_server: str -) -> None: - """Refused is a 403 the UI can act on; allowed runs and stamps the last call.""" - refused = await client.post( - "/llm/providers/ollama/pull", json={"name": "qwen3:1.7b"} - ) - assert refused.status_code == 403 - assert refused.json()["detail"] == { - "code": "egress_disabled", - "message": "sending data to registry.ollama.ai is off in Settings > Network", - "destination": "ollama_pull", - "host": "registry.ollama.ai", - } - assert (await _destinations(client))["ollama_pull"]["last_call_at"] is None - - assert ( - await client.put("/egress/ollama_pull", json={"enabled": True}) - ).status_code == 200 - async with client.stream( - "POST", "/llm/providers/ollama/pull", json={"name": "qwen3:1.7b"} - ) as reply: - assert reply.status_code == 200 - await reply.aread() - - row = (await _destinations(client))["ollama_pull"] - assert row["enabled"] is True - assert row["last_call_at"] is not None - - async def test_remote_connection_is_refused_before_it_is_probed( client: AsyncClient, ) -> None: @@ -93,7 +63,12 @@ async def test_loopback_connections_are_not_egress( ) assert created.status_code == 201, created.text # The two weight downloads are always listed; the connection adds no third. - assert set(await _destinations(client)) == {"ollama_pull", "image_model_pull"} + assert set(await _destinations(client)) == { + "image_model_pull", + # One host, two consents: a repo the user named, and text they typed. + "model_download", + "model_search", + } async def test_unknown_destination_is_rejected(client: AsyncClient) -> None: @@ -121,7 +96,7 @@ def _configure_llmfit(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: "disk_size_gb": 5.2, "effective_context_length": 8192, "capability_ids": ["tool_use"], - "ollama_name": "qwen3:8b", + "file": "Qwen3-8B-Q4_K_M.gguf", "gguf_sources": [], } ] @@ -139,32 +114,3 @@ def _configure_llmfit(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: monkeypatch.setattr(get_llm_settings(), "llmfit_path", executable) get_catalog_service.cache_clear() - -async def test_catalog_install_is_refused_until_allowed( - client: AsyncClient, - ollama_server: str, - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, -) -> None: - """The catalog's own install route is gated the same as a manual pull.""" - _configure_llmfit(tmp_path, monkeypatch) - catalog = (await client.get("/llm/catalog?refresh=true")).json() - catalog_id = next( - row["catalog_id"] for row in catalog["curated"] if row["fit"] != "unknown" - ) - - refused = await client.post( - "/llm/install", json={"catalog_id": catalog_id, "select": False} - ) - assert refused.status_code == 403 - assert refused.json()["detail"]["destination"] == "ollama_pull" - assert (await _destinations(client))["ollama_pull"]["last_call_at"] is None - - await client.put("/egress/ollama_pull", json={"enabled": True}) - async with client.stream( - "POST", "/llm/install", json={"catalog_id": catalog_id, "select": False} - ) as reply: - assert reply.status_code == 200 - await reply.aread() - assert (await _destinations(client))["ollama_pull"]["last_call_at"] is not None - get_catalog_service.cache_clear() diff --git a/surfsense_local/backend/tests/unit/egress/test_service.py b/surfsense_local/backend/tests/unit/egress/test_service.py index cc5c3292b1..bc36764e2c 100644 --- a/surfsense_local/backend/tests/unit/egress/test_service.py +++ b/surfsense_local/backend/tests/unit/egress/test_service.py @@ -1,15 +1,4 @@ import pytest -from modules.egress import service as egress - pytestmark = pytest.mark.unit - -def test_a_library_pull_reports_the_ollama_registry() -> None: - """Ollama's own model library is served from its registry.""" - assert egress.ollama_pull_host("qwen3:8b") == "registry.ollama.ai" - - -def test_an_hf_co_fallback_pull_reports_hugging_face() -> None: - """Ollama fetches `hf.co/` pulls straight from Hugging Face, not its own registry.""" - assert egress.ollama_pull_host("hf.co/Qwen/Qwen3-8B-GGUF:Q4_K_M") == "huggingface.co" diff --git a/surfsense_local/frontend/src/features/egress/api.ts b/surfsense_local/frontend/src/features/egress/api.ts index fa00bb37e6..e53cdd5bfb 100644 --- a/surfsense_local/frontend/src/features/egress/api.ts +++ b/surfsense_local/frontend/src/features/egress/api.ts @@ -32,11 +32,20 @@ export function describeDestination(destination: string, host: string) { body: `Checking asks ${host} for the latest release. SurfSense sends your IP address and the version you are running, nothing else. Allowing also turns on the check at launch, which you can switch off in Settings › Network.`, } } - if (destination === "ollama_pull") { + if (destination === "model_download") { return { label: "Model downloads", title: "Allow model downloads?", - body: `Downloading models contacts ${host}. SurfSense sends the model name and your IP address, nothing else.`, + body: `Downloading a model contacts ${host}. SurfSense sends the name of the model you chose and your IP address, nothing else.`, + } + } + // Same host as downloads, and a separate consent on purpose: this one sends + // what you type as you type it, rather than the name of a model you picked. + if (destination === "model_search") { + return { + label: "Model search", + title: "Allow searching for models?", + body: `Searching sends what you type to ${host}, along with your IP address. Without it you can still install the models SurfSense ships with, and any .gguf file already on this machine.`, } } if (destination === "image_model_pull") { From 2004f02f811151b8691944b936358a5f31dc34c9 Mon Sep 17 00:00:00 2001 From: Anish Sarkar <104695310+AnishSarkar22@users.noreply.github.com> Date: Sun, 20 Sep 2026 20:07:43 +0530 Subject: [PATCH 22/60] feat(db): migrate selected models and egress grants off Ollama in revision 0012 Co-Authored-By: Claude Sonnet 5 --- .../versions/0012_llamacpp_provider.py | 125 +++++++++++++++ .../tests/integration/test_migration_0012.py | 146 ++++++++++++++++++ .../tests/integration/test_migrations.py | 21 ++- 3 files changed, 284 insertions(+), 8 deletions(-) create mode 100644 surfsense_local/backend/alembic/versions/0012_llamacpp_provider.py create mode 100644 surfsense_local/backend/tests/integration/test_migration_0012.py diff --git a/surfsense_local/backend/alembic/versions/0012_llamacpp_provider.py b/surfsense_local/backend/alembic/versions/0012_llamacpp_provider.py new file mode 100644 index 0000000000..2ffd4e0b11 --- /dev/null +++ b/surfsense_local/backend/alembic/versions/0012_llamacpp_provider.py @@ -0,0 +1,125 @@ +"""Replace the Ollama runtime with llama.cpp. + +The only migration in this phase that touches user data, and the only +irreversible step in the swap. Two things happen, both deliberate: + +**A generation selection pointing at Ollama is cleared, not remapped.** Its +weights live in a blob format the app no longer manages, so there is no honest +file to point at. Left alone it would make `resolve_generation()` raise +`unknown provider` and chat would die with no explanation. + +**An `ollama_pull` egress grant becomes `model_download` only.** That one +destination split into two on the same host, and the two are different +consents: `model_download` fetches a repo the user named, `model_search` sends +text they are typing. Carrying one grant across both would manufacture a +consent nobody gave, so `model_search` starts unset. + +Revisions 0004 and 0009 mention Ollama and are applied history on every +installed machine. They are not touched here; this does the data work. + +Revision ID: 0012 +Revises: 0011 +""" + +from collections.abc import Sequence + +import sqlalchemy as sa + +from alembic import op + +revision: str = "0012" +down_revision: str | Sequence[str] | None = "0011" +branch_labels: str | Sequence[str] | None = None +depends_on: str | Sequence[str] | None = None + +NOW = sa.text("(CURRENT_TIMESTAMP)") +_TEMP = "_selected_models_0012" + +LLAMACPP_AND_REMOTE = ( + "(provider IN ('llamacpp', 'sdcpp') AND connection_id IS NULL) OR " + "(provider = 'openai_compatible' AND connection_id IS NOT NULL)" +) +OLLAMA_AND_REMOTE = ( + "(provider IN ('ollama', 'sdcpp') AND connection_id IS NULL) OR " + "(provider = 'openai_compatible' AND connection_id IS NOT NULL)" +) + + +def _rebuild(check: str, keep: str) -> None: + """SQLite cannot alter a check constraint, so move the rows to a new table.""" + op.create_table( + _TEMP, + sa.Column( + "role", + sa.Enum( + "generation", + "image_generation", + name="modelrole", + native_enum=False, + create_constraint=True, + ), + nullable=False, + ), + sa.Column("provider", sa.String(), nullable=False), + sa.Column("connection_id", sa.Integer(), nullable=True), + sa.Column("name", sa.String(), nullable=False), + sa.Column("params_b", sa.Float(), nullable=True), + sa.Column("vendor", sa.String(), nullable=True), + sa.Column( + "line", + sa.Enum( + "flagship", + "small", + name="line", + native_enum=False, + create_constraint=True, + ), + nullable=True, + ), + sa.Column("updated_at", sa.DateTime(), server_default=NOW, nullable=False), + sa.CheckConstraint( + check, name=op.f("ck_selected_models_provider_connection") + ), + sa.ForeignKeyConstraint( + ["connection_id"], + ["provider_connections.id"], + name=op.f("fk_selected_models_connection_id_provider_connections"), + ondelete="CASCADE", + ), + sa.PrimaryKeyConstraint("role", name=op.f("pk_selected_models")), + ) + op.execute( + sa.text( + f""" + INSERT INTO {_TEMP} + (role, provider, connection_id, name, params_b, vendor, line, updated_at) + SELECT role, provider, connection_id, name, params_b, vendor, line, updated_at + FROM selected_models + WHERE {keep} + """ + ) + ) + op.drop_table("selected_models") + op.rename_table(_TEMP, "selected_models") + + +def upgrade() -> None: + _rebuild(LLAMACPP_AND_REMOTE, keep="provider <> 'ollama'") + op.execute( + sa.text( + "UPDATE egress_destinations SET destination = 'model_download' " + "WHERE destination = 'ollama_pull'" + ) + ) + + +def downgrade() -> None: + # A llamacpp selection cannot exist under the old rule, so drop it rather + # than fail, the same way 0009 dropped what it could not carry across. + _rebuild(OLLAMA_AND_REMOTE, keep="provider <> 'llamacpp'") + op.execute( + sa.text( + "UPDATE egress_destinations SET destination = 'ollama_pull' " + "WHERE destination = 'model_download'" + ) + ) diff --git a/surfsense_local/backend/tests/integration/test_migration_0012.py b/surfsense_local/backend/tests/integration/test_migration_0012.py new file mode 100644 index 0000000000..efa84366ab --- /dev/null +++ b/surfsense_local/backend/tests/integration/test_migration_0012.py @@ -0,0 +1,146 @@ +"""Revision 0012: the one migration in this phase that touches user data. + +Two things happen here and both are irreversible, so both are pinned: +a generation selection pointing at Ollama is cleared, and an `ollama_pull` +egress grant becomes `model_download` and nothing else. +""" + +from pathlib import Path + +import pytest +from alembic.config import Config +from sqlalchemy import text + +from alembic import command +from shared.db import create_db_engine + +pytestmark = pytest.mark.integration + + +def upgraded(tmp_path: Path, seed: str) -> object: + """A database seeded at 0011 and then upgraded to head.""" + engine = create_db_engine(tmp_path / "surfsense.db") + config = Config() + config.set_main_option( + "script_location", str(Path(__file__).parents[2] / "alembic") + ) + config.attributes["engine"] = engine + command.upgrade(config, "0011") + with engine.begin() as connection: + for statement in seed.strip().split(";"): + if statement.strip(): + connection.execute(text(statement)) + command.upgrade(config, "head") + return engine + + +def test_an_ollama_selection_is_cleared_rather_than_remapped(tmp_path: Path) -> None: + """The weights are in a blob format the app no longer manages, so there is + no honest way to point the selection at an equivalent file. Leaving it would + make `resolve_generation()` raise and chat would simply die. + """ + engine = upgraded( + tmp_path, + "INSERT INTO selected_models(role, provider, name) " + "VALUES ('generation', 'ollama', 'qwen3:4b')", + ) + + with engine.connect() as connection: + rows = connection.execute(text("SELECT * FROM selected_models")).all() + + assert rows == [] + + +def test_an_image_selection_is_untouched(tmp_path: Path) -> None: + """sd-server is not part of this swap and its weights still exist.""" + engine = upgraded( + tmp_path, + "INSERT INTO selected_models(role, provider, name) " + "VALUES ('image_generation', 'sdcpp', 'sd15')", + ) + + with engine.connect() as connection: + selected = connection.execute( + text("SELECT provider, name FROM selected_models") + ).one() + + assert tuple(selected) == ("sdcpp", "sd15") + + +def test_the_provider_check_now_names_llamacpp(tmp_path: Path) -> None: + """A selection the app can actually resolve must be insertable, and the + provider this runtime registers under is the one the column allows.""" + engine = upgraded(tmp_path, "") + + with engine.begin() as connection: + connection.execute( + text( + "INSERT INTO selected_models(role, provider, name) " + "VALUES ('generation', 'llamacpp', 'Qwen3-8B-Q4_K_M')" + ) + ) + selected = connection.execute( + text("SELECT provider FROM selected_models") + ).scalar_one() + + assert selected == "llamacpp" + + +def test_ollama_is_no_longer_an_allowed_provider(tmp_path: Path) -> None: + """Nothing can resolve it, so the column should not accept it either.""" + from sqlalchemy.exc import IntegrityError + + engine = upgraded(tmp_path, "") + + with pytest.raises(IntegrityError), engine.begin() as connection: + connection.execute( + text( + "INSERT INTO selected_models(role, provider, name) " + "VALUES ('generation', 'ollama', 'qwen3:4b')" + ) + ) + + +def test_a_download_grant_carries_over_and_a_search_grant_is_not_invented( + tmp_path: Path, +) -> None: + """One host, two consents. Someone who allowed model downloads allowed + fetching a file they named; search sends text they are typing to the same + host for a different reason. Carrying the grant to both would manufacture + a consent nobody gave. + """ + engine = upgraded( + tmp_path, + "INSERT INTO egress_destinations(destination, enabled) " + "VALUES ('ollama_pull', 1)", + ) + + with engine.connect() as connection: + rows = dict( + connection.execute( + text("SELECT destination, enabled FROM egress_destinations") + ).all() + ) + + assert rows.get("model_download") == 1 + assert "model_search" not in rows + assert "ollama_pull" not in rows + + +def test_a_refused_download_stays_refused(tmp_path: Path) -> None: + """Renaming a destination must not quietly grant what was declined.""" + engine = upgraded( + tmp_path, + "INSERT INTO egress_destinations(destination, enabled) " + "VALUES ('ollama_pull', 0)", + ) + + with engine.connect() as connection: + allowed = connection.execute( + text( + "SELECT enabled FROM egress_destinations " + "WHERE destination = 'model_download'" + ) + ).scalar_one() + + assert allowed == 0 diff --git a/surfsense_local/backend/tests/integration/test_migrations.py b/surfsense_local/backend/tests/integration/test_migrations.py index 68df1cff11..08f6caa109 100644 --- a/surfsense_local/backend/tests/integration/test_migrations.py +++ b/surfsense_local/backend/tests/integration/test_migrations.py @@ -55,10 +55,17 @@ def test_a_failed_migration_leaves_nothing_behind(tmp_path: Path) -> None: assert inspect(engine).get_table_names() == [] -def test_connection_migration_preserves_only_ollama_selection( +def test_connection_migration_drops_old_remote_secrets( tmp_path: Path, ) -> None: - """The breaking migration drops old remote secrets without harming local setup.""" + """The breaking migration at 0004 drops old remote secrets. + + It used to assert the local Ollama selection survived to head as well. It no + longer does: 0012 clears it, because the weights are in a format the app + stopped managing and a selection nothing can resolve kills chat. That the + selection is cleared rather than remapped is pinned in + `test_migration_0012.py`; what this still guards is the credentials table. + """ engine = create_db_engine(tmp_path / "surfsense.db") config = Config() config.set_main_option( @@ -83,12 +90,10 @@ def test_connection_migration_preserves_only_ollama_selection( command.upgrade(config, "head") with engine.connect() as connection: - selected = connection.execute( - text( - "SELECT role, provider, connection_id, name FROM selected_models" - ) - ).one() + selections = connection.execute( + text("SELECT role, provider FROM selected_models") + ).all() tables = inspect(connection).get_table_names() - assert tuple(selected) == ("generation", "ollama", None, "qwen3:4b") + assert selections == [] assert "provider_credentials" not in tables assert "provider_connections" in tables From f439630d943e866f3de57403a6c7d6706ea793c9 Mon Sep 17 00:00:00 2001 From: Anish Sarkar <104695310+AnishSarkar22@users.noreply.github.com> Date: Sun, 20 Sep 2026 20:07:43 +0530 Subject: [PATCH 23/60] fix(secrets): raise UnreadableSecretError instead of leaking InvalidToken Co-Authored-By: Claude Sonnet 5 --- surfsense_local/backend/shared/secrets.py | 25 +++++++++++++++++-- .../backend/tests/unit/shared/test_secrets.py | 16 ++++++++---- 2 files changed, 34 insertions(+), 7 deletions(-) diff --git a/surfsense_local/backend/shared/secrets.py b/surfsense_local/backend/shared/secrets.py index bbbdbb2996..5c201ca07a 100644 --- a/surfsense_local/backend/shared/secrets.py +++ b/surfsense_local/backend/shared/secrets.py @@ -4,13 +4,31 @@ import os from functools import lru_cache -from cryptography.fernet import Fernet +from cryptography.fernet import Fernet, InvalidToken from shared.config import get_storage_settings log = logging.getLogger(__name__) +class UnreadableSecretError(Exception): + """Stored ciphertext this install's secret cannot open. + + Not a corruption bug. The secret lives in the OS keychain, so a keychain + reset or a backup restored onto another machine leaves every stored key + undecryptable while the rows themselves are intact. The key is gone either + way, and the only recovery is entering it again, so this is raised as its + own condition rather than leaking `InvalidToken` to a caller that can only + treat it as a crash. + """ + + def __init__(self) -> None: + super().__init__( + "the stored key could not be read on this machine, so it has to be " + "entered again" + ) + + @lru_cache def _fernet() -> Fernet: secret = os.environ.get("SURFSENSE_LOCAL_SECRET") or _file_secret() @@ -37,4 +55,7 @@ def encrypt(value: str) -> bytes: def decrypt(token: bytes) -> str: """Recover a provider API key stored by ``encrypt``.""" - return _fernet().decrypt(token).decode() + try: + return _fernet().decrypt(token).decode() + except InvalidToken as error: + raise UnreadableSecretError from error diff --git a/surfsense_local/backend/tests/unit/shared/test_secrets.py b/surfsense_local/backend/tests/unit/shared/test_secrets.py index 2b4a60fc56..641f258190 100644 --- a/surfsense_local/backend/tests/unit/shared/test_secrets.py +++ b/surfsense_local/backend/tests/unit/shared/test_secrets.py @@ -1,8 +1,7 @@ import pytest -from cryptography.fernet import InvalidToken from modules.llm.models import ProviderConnection -from shared.secrets import decrypt, encrypt +from shared.secrets import UnreadableSecretError, decrypt, encrypt def test_api_key_round_trips_and_is_not_stored_in_clear() -> None: @@ -19,9 +18,16 @@ def test_api_key_round_trips_and_is_not_stored_in_clear() -> None: assert connection.api_key is None -def test_tampered_ciphertext_is_rejected() -> None: - """A flipped byte fails closed instead of yielding a wrong key.""" +def test_unreadable_ciphertext_fails_closed_as_a_named_condition() -> None: + """A flipped byte yields no key, and says which condition it hit. + + Named rather than left as the library's `InvalidToken`, because this is + reachable without any tampering: the per install secret lives in the OS + keychain, and a keychain reset or a backup restored onto another machine + leaves every stored key undecryptable. Callers have to tell that apart from + a bug to say anything useful about it. + """ token = bytearray(encrypt("sk-1")) token[-1] ^= 0xFF - with pytest.raises(InvalidToken): + with pytest.raises(UnreadableSecretError): decrypt(bytes(token)) From a5da65c34fe6e318bac066b3381aeebdba4134bf Mon Sep 17 00:00:00 2001 From: Anish Sarkar <104695310+AnishSarkar22@users.noreply.github.com> Date: Sun, 20 Sep 2026 20:07:43 +0530 Subject: [PATCH 24/60] feat(chat): classify an unreadable stored key as a provider auth error Co-Authored-By: Claude Sonnet 5 --- .../backend/modules/chat/errors.py | 11 ++- .../integration/artifacts/test_routes.py | 4 +- .../tests/integration/chat/conftest.py | 70 ++++++++++++++----- .../tests/integration/chat/test_chat.py | 35 +++++----- .../backend/tests/unit/chat/test_errors.py | 39 +++++++++++ 5 files changed, 121 insertions(+), 38 deletions(-) create mode 100644 surfsense_local/backend/tests/unit/chat/test_errors.py diff --git a/surfsense_local/backend/modules/chat/errors.py b/surfsense_local/backend/modules/chat/errors.py index ffb3345e2e..7390ab6347 100644 --- a/surfsense_local/backend/modules/chat/errors.py +++ b/surfsense_local/backend/modules/chat/errors.py @@ -2,6 +2,8 @@ import httpx +from shared.secrets import UnreadableSecretError + class ChatErrorKind(enum.StrEnum): """Buckets a failed generation call so the UI can offer the right fix. @@ -36,9 +38,9 @@ class ChatErrorKind(enum.StrEnum): } # `network` is the one kind whose fix depends on the provider: a bad base URL -# is a Model setup problem, an unreachable local Ollama is not. +# is a Model setup problem, an unreachable local runtime is not. _NETWORK_MESSAGES: dict[str, str] = { - "ollama": "Couldn't reach Ollama — make sure it's running locally.", + "llamacpp": "Couldn't reach the local model runtime. Restart SurfSense to start it again.", } _DEFAULT_NETWORK_MESSAGE = ( "Couldn't reach the model provider — " @@ -55,6 +57,11 @@ def classify_chat_error(exc: Exception, provider: str) -> tuple[ChatErrorKind, s the exception's text, so this holds for any provider that raises through httpx (every provider in modules/llm/providers does). """ + if isinstance(exc, UnreadableSecretError): + # Not `unknown`: the fix is specific and the user can do it. The stored + # key is unrecoverable once this install's secret changes, so the only + # useful answer names the key rather than reporting a fault. + return ChatErrorKind.PROVIDER_AUTH, _MESSAGES[ChatErrorKind.PROVIDER_AUTH] if isinstance(exc, httpx.HTTPStatusError): status_code = exc.response.status_code if status_code in _AUTH_STATUS_CODES: diff --git a/surfsense_local/backend/tests/integration/artifacts/test_routes.py b/surfsense_local/backend/tests/integration/artifacts/test_routes.py index 8a5bd83e18..e6d71456d5 100644 --- a/surfsense_local/backend/tests/integration/artifacts/test_routes.py +++ b/surfsense_local/backend/tests/integration/artifacts/test_routes.py @@ -26,7 +26,7 @@ def choose_model(engine: Engine) -> None: """A generation model must be selected before a job can run.""" with create_session_factory(engine)() as session: session.add( - SelectedModel(role=ModelRole.GENERATION, provider="ollama", name="qwen3:4b") + SelectedModel(role=ModelRole.GENERATION, provider="llamacpp", name="Qwen3-4B-Q4_K_M") ) session.commit() @@ -76,7 +76,7 @@ async def test_infographic_needs_the_image_model_and_the_chat_model( with create_session_factory(engine)() as session: session.add( SelectedModel( - role=ModelRole.IMAGE_GENERATION, provider="ollama", name="x/flux2-klein" + role=ModelRole.IMAGE_GENERATION, provider="llamacpp", name="x/flux2-klein" ) ) session.commit() diff --git a/surfsense_local/backend/tests/integration/chat/conftest.py b/surfsense_local/backend/tests/integration/chat/conftest.py index 7709df3e7b..a979ce03df 100644 --- a/surfsense_local/backend/tests/integration/chat/conftest.py +++ b/surfsense_local/backend/tests/integration/chat/conftest.py @@ -14,25 +14,51 @@ _REQUESTS: list[dict] = [] -class StubOllamaChat(BaseHTTPRequestHandler): - """Just the chat endpoint of Ollama's native API, streaming its reply.""" +class StubRouterChat(BaseHTTPRequestHandler): + """The router's OpenAI chat endpoint, streaming its reply as SSE. + + The local runtime speaks OpenAI now, so the adapter composes the same + provider a remote endpoint uses and this stub is shaped accordingly. + """ + + def do_GET(self) -> None: + if self.path == "/models": + self._send( + json.dumps( + { + "object": "list", + "data": [ + {"id": "Qwen3-4B-Q4_K_M", "status": {"value": "loaded"}} + ], + } + ).encode() + ) + else: + self.send_error(404) def do_POST(self) -> None: raw = self.rfile.read(int(self.headers["Content-Length"])) - if self.path != "/api/chat": + if self.path not in ("/v1/chat/completions", "/models/load"): self.send_error(404) return + if self.path == "/models/load": + self._send(b'{"success": true}') + return request = json.loads(raw) _REQUESTS.append(request) deltas = ( ["Revenue ", "Growth"] - if request.get("options", {}).get("num_predict") == 12 + if request.get("max_tokens") == 12 else REPLY_DELTAS ) - body = "".join( - json.dumps({"message": {"content": delta}}) + "\n" for delta in deltas - ).encode() + chunks = [ + "data: " + json.dumps({"choices": [{"delta": {"content": delta}}]}) + for delta in deltas + ] + ["data: [DONE]"] + self._send(("\n\n".join(chunks) + "\n\n").encode()) + + def _send(self, body: bytes) -> None: self.send_response(200) self.send_header("Content-Length", str(len(body))) self.end_headers() @@ -43,13 +69,13 @@ def log_message(self, *args: object) -> None: @pytest.fixture -def ollama_server(monkeypatch: pytest.MonkeyPatch) -> Iterator[list[dict]]: - """A real Ollama chat stand-in on a real port; yields the requests it sees.""" +def llamacpp_server(monkeypatch: pytest.MonkeyPatch) -> Iterator[list[dict]]: + """A real llama-server stand-in on a real port; yields the requests it sees.""" _REQUESTS.clear() - server = ThreadingHTTPServer(("127.0.0.1", 0), StubOllamaChat) + server = ThreadingHTTPServer(("127.0.0.1", 0), StubRouterChat) threading.Thread(target=server.serve_forever, daemon=True).start() url = f"http://127.0.0.1:{server.server_port}" - monkeypatch.setattr(get_llm_settings(), "ollama_base_url", url) + monkeypatch.setattr(get_llm_settings(), "llamacpp_base_url", url) yield _REQUESTS @@ -57,8 +83,20 @@ def ollama_server(monkeypatch: pytest.MonkeyPatch) -> Iterator[list[dict]]: server.server_close() -class StubOllamaUnauthorized(BaseHTTPRequestHandler): - """A chat endpoint that always answers 401, as if the connection were bad.""" +class StubRouterUnauthorized(BaseHTTPRequestHandler): + """A router that always answers 401, as if the connection were bad. + + It still answers `GET /models`, because the adapter checks residency before + it asks a question and a 501 there would fail the request for the wrong + reason entirely. + """ + + def do_GET(self) -> None: + body = b'{"object": "list", "data": []}' + self.send_response(200) + self.send_header("Content-Length", str(len(body))) + self.end_headers() + self.wfile.write(body) def do_POST(self) -> None: self.rfile.read(int(self.headers["Content-Length"])) @@ -73,12 +111,12 @@ def log_message(self, *args: object) -> None: @pytest.fixture -def ollama_server_unauthorized(monkeypatch: pytest.MonkeyPatch) -> Iterator[None]: +def llamacpp_server_unauthorized(monkeypatch: pytest.MonkeyPatch) -> Iterator[None]: """A chat stand-in that fails every request, for exercising error handling.""" - server = ThreadingHTTPServer(("127.0.0.1", 0), StubOllamaUnauthorized) + server = ThreadingHTTPServer(("127.0.0.1", 0), StubRouterUnauthorized) threading.Thread(target=server.serve_forever, daemon=True).start() url = f"http://127.0.0.1:{server.server_port}" - monkeypatch.setattr(get_llm_settings(), "ollama_base_url", url) + monkeypatch.setattr(get_llm_settings(), "llamacpp_base_url", url) yield diff --git a/surfsense_local/backend/tests/integration/chat/test_chat.py b/surfsense_local/backend/tests/integration/chat/test_chat.py index 3ecb3beba2..2941675977 100644 --- a/surfsense_local/backend/tests/integration/chat/test_chat.py +++ b/surfsense_local/backend/tests/integration/chat/test_chat.py @@ -40,7 +40,7 @@ def _seed( ids[title] = doc.id session.add( SelectedModel( - role=ModelRole.GENERATION, provider="ollama", name="qwen3:1.7b" + role=ModelRole.GENERATION, provider="llamacpp", name="Qwen3-1.7B-Q4_K_M" ) ) session.commit() @@ -55,7 +55,7 @@ def _choose_generation_model(engine: Engine) -> None: with create_session_factory(engine)() as session: session.add( SelectedModel( - role=ModelRole.GENERATION, provider="ollama", name="qwen3:1.7b" + role=ModelRole.GENERATION, provider="llamacpp", name="Qwen3-1.7B-Q4_K_M" ) ) session.commit() @@ -113,7 +113,7 @@ async def test_a_thread_can_be_renamed(client: AsyncClient) -> None: async def test_a_message_streams_a_grounded_reply( - client: AsyncClient, engine: Engine, real_model: object, ollama_server: list[dict] + client: AsyncClient, engine: Engine, real_model: object, llamacpp_server: list[dict] ) -> None: """Deltas arrive, the citation tail names the source doc, both turns persist.""" workspace_id, ids = _seed(engine) @@ -166,17 +166,16 @@ async def test_a_message_streams_a_grounded_reply( assert stored[1]["completed_at"] == completed["assistant_completed_at"] threads = (await client.get(f"/workspaces/{workspace_id}/chat/threads")).json() assert threads[0]["title"] == "Revenue Growth" - assert ollama_server[0]["think"] is False - assert ollama_server[0]["options"] == { - "num_predict": 12, - "temperature": 0, - "num_ctx": 4096, - } - assert ollama_server[1]["options"] == {"num_ctx": 4096} + # The window is fixed at load now, not sized per request: llama.cpp takes + # `-c` once when the model is loaded, so there is no per-call equivalent of + # `num_ctx` to assert here. + assert llamacpp_server[0]["max_tokens"] == 12 + assert llamacpp_server[0]["temperature"] == 0 + assert llamacpp_server[0]["stream"] is True async def test_a_message_retrieves_only_from_selected_sources( - client: AsyncClient, engine: Engine, real_model: object, ollama_server: list[dict] + client: AsyncClient, engine: Engine, real_model: object, llamacpp_server: list[dict] ) -> None: """document_ids is the RAG scope: an unselected source cannot be cited.""" workspace_id, ids = _seed(engine, {"finance": FINANCE, "cat": CAT}) @@ -243,18 +242,18 @@ async def test_a_message_waits_for_a_source_to_index( async def test_a_followup_carries_the_earlier_turn( - client: AsyncClient, engine: Engine, real_model: object, ollama_server: list[dict] + client: AsyncClient, engine: Engine, real_model: object, llamacpp_server: list[dict] ) -> None: """The second message hands the model the first turn as history.""" workspace_id, _ids = _seed(engine) thread_id = await _open_thread(client, workspace_id) await _send(client, thread_id, "first question") - ollama_server.clear() + llamacpp_server.clear() await _send(client, thread_id, "second question") - assert len(ollama_server) == 1 - sent = ollama_server[-1]["messages"] + assert len(llamacpp_server) == 1 + sent = llamacpp_server[-1]["messages"] assert sent[0]["role"] == "system" assert sent[-1] == {"role": "user", "content": "second question"} assert "first question" in [message["content"] for message in sent] @@ -264,7 +263,7 @@ async def test_title_failure_does_not_block_the_answer( client: AsyncClient, engine: Engine, real_model: object, - ollama_server: list[dict], + llamacpp_server: list[dict], monkeypatch: pytest.MonkeyPatch, ) -> None: """Naming is best effort; its failure must not consume or fail the chat turn.""" @@ -291,7 +290,7 @@ async def test_a_failed_reply_is_classified_and_leaves_no_trace( client: AsyncClient, engine: Engine, real_model: object, - ollama_server_unauthorized: None, + llamacpp_server_unauthorized: None, ) -> None: """A generation failure is classified, not shown raw, and the turn is discarded.""" workspace_id, _ids = _seed(engine) @@ -331,7 +330,7 @@ async def test_missing_embedding_assets_are_an_actionable_503( with create_session_factory(engine)() as session: session.add( SelectedModel( - role=ModelRole.GENERATION, provider="ollama", name="qwen3:1.7b" + role=ModelRole.GENERATION, provider="llamacpp", name="Qwen3-1.7B-Q4_K_M" ) ) session.commit() diff --git a/surfsense_local/backend/tests/unit/chat/test_errors.py b/surfsense_local/backend/tests/unit/chat/test_errors.py new file mode 100644 index 0000000000..dead37be5e --- /dev/null +++ b/surfsense_local/backend/tests/unit/chat/test_errors.py @@ -0,0 +1,39 @@ +"""Sorting a failed generation into something the user can act on.""" + +import httpx +import pytest + +from modules.chat.errors import ChatErrorKind, classify_chat_error +from shared.secrets import UnreadableSecretError + +pytestmark = pytest.mark.unit + + +def test_a_key_the_app_cannot_read_asks_for_a_new_one() -> None: + """Not `unknown`, because the fix is specific and the user can do it. + + The stored key is unrecoverable once the per install secret changes, so the + only useful answer names the connection's key. "Something went wrong" sends + someone looking for a bug that is not there. + """ + # No message passed in: the condition owns its wording, so every caller + # says the same thing. + kind, message = classify_chat_error(UnreadableSecretError(), "openai_compatible") + + assert kind is ChatErrorKind.PROVIDER_AUTH + assert "key" in message.casefold() + + +def test_transport_and_status_failures_are_unchanged() -> None: + """The new branch sits beside the existing ones, not in front of them.""" + request = httpx.Request("POST", "http://x/v1/chat/completions") + unauthorized = httpx.HTTPStatusError( + "401", request=request, response=httpx.Response(401, request=request) + ) + + assert classify_chat_error(unauthorized, "x")[0] is ChatErrorKind.PROVIDER_AUTH + assert ( + classify_chat_error(httpx.ConnectError("down"), "llamacpp")[0] + is ChatErrorKind.NETWORK + ) + assert classify_chat_error(RuntimeError("?"), "x")[0] is ChatErrorKind.UNKNOWN From 12abfb0662080b13a3dbaa097f1832718efef6f8 Mon Sep 17 00:00:00 2001 From: Anish Sarkar <104695310+AnishSarkar22@users.noreply.github.com> Date: Sun, 20 Sep 2026 20:07:43 +0530 Subject: [PATCH 25/60] feat(frontend): show fit badges and model search in the model catalog Co-Authored-By: Claude Sonnet 5 --- .../src/features/model-catalog/api.ts | 190 ++++-- .../src/features/model-catalog/fit-badge.tsx | 52 ++ .../model-catalog/local-image-model.tsx | 4 +- .../src/features/model-catalog/model-card.tsx | 57 +- .../model-catalog/model-catalog-page.tsx | 444 ++++-------- .../model-catalog/model-catalog.test.tsx | 645 +++++++----------- .../features/model-catalog/model-search.tsx | 207 ++++++ .../model-catalog/use-model-catalog.ts | 29 +- 8 files changed, 790 insertions(+), 838 deletions(-) create mode 100644 surfsense_local/frontend/src/features/model-catalog/fit-badge.tsx create mode 100644 surfsense_local/frontend/src/features/model-catalog/model-search.tsx diff --git a/surfsense_local/frontend/src/features/model-catalog/api.ts b/surfsense_local/frontend/src/features/model-catalog/api.ts index 08ebd97845..36e61ea1ed 100644 --- a/surfsense_local/frontend/src/features/model-catalog/api.ts +++ b/surfsense_local/frontend/src/features/model-catalog/api.ts @@ -1,83 +1,123 @@ import { request, requestJson, requestVoid } from "@/lib/api" import type { ModelSelection } from "@/features/model-selection/api" -export type Fit = "perfect" | "good" | "marginal" | "too_tight" | "unknown" +/** + * Where a model's weights will live. Three states, and no `unknown`: every row + * has a file size, so every row can be priced. + */ +export type FitState = "fits" | "partial" | "too_big" + +export type Fit = { + state: FitState + need_bytes: number + budget_bytes: number + /** + * 0 fully resident, 1 entirely on the processor. The reason line is graded by + * this, and it is never recomputed here: the same number drives the badge and + * the recommendation, so they cannot disagree. + */ + offload_fraction: number + /** Priced from file size alone, before the header was read. */ + approximate: boolean +} + +/** A verdict plus one plain line of why. The API owns this copy. */ +export type Badge = { + verdict: string + reason: string +} export type CatalogRow = { + /** Opaque install token. The renderer never assembles one. */ catalog_id: string - canonical_id: string - family: string + /** Stable identity of the model, used as a React key. */ + model_id: string + /** What the runtime calls the installed file, for selection and deletion. */ + variant_model_id: string label: string - publisher: string | null - parameter_count: string | number | null + family: string + parameter_count: string + quantization: string + size_bytes: number + context_length: number fit: Fit - score: number | null - memory_required_gb: number | null - disk_size_gb: number | null - estimated_tps: number | null - prefill_tps: number | null - ttft_ms: number | null - effective_context_length: number | null - estimate_confidence: string | null - license: string | null - runtime: string - runtime_model: string - quantization: string | null + badge: Badge + capabilities: string[] installed: boolean selected: boolean can_install: boolean - can_delete: boolean - warnings: string[] + recommended: boolean } -export type RuntimeStatus = - | string - | boolean - | { - healthy?: boolean - available?: boolean - status?: string - message?: string - } - -export type RecommendationWarning = { - code: string - message: string +export type InstalledRow = { + model_id: string + file: string + size_bytes: number + selected: boolean } -export type HardwareProfile = { - cpu_name: string | null - cpu_cores: number | null - total_ram_gb: number | null - available_ram_gb: number | null +/** One device, never a sum across devices. */ +export type Budget = { + device_total_bytes: number + device_free_bytes: number + usable_vram_bytes: number + fit_reserve_bytes: number + ram_available_bytes: number + /** Unified memory: selects badge copy, since there is nothing to spill into. */ + uma: boolean has_gpu: boolean - gpu_name: string | null - gpu_vram_gb: number | null - gpu_count: number - backend: string | null - unified_memory: boolean } +/** + * Curated plus installed. No `scanned` flag, because there is no scan: the + * budget comes from the runtime's own allocator in about 180ms. + */ export type ModelCatalog = { - hardware: HardwareProfile | null - llmfit_version: string | null - // Always populated, scan-free — SurfSense's curated picks, shown with a - // fit badge once scanned and without one before. Never contains a row - // also present in `explore`. + budget: Budget curated: CatalogRow[] - explore: CatalogRow[] - installed: CatalogRow[] - // False when served without running the hardware scan (no cache existed - // yet and none was requested this session) — `curated`/`installed` are - // still fully populated either way, only `explore` and curated fit badges - // are scan-derived. - scanned: boolean - warnings: RecommendationWarning[] - runtime_status: Record + installed: InstalledRow[] + recommended_model_id: string | null +} + +/** A repo, described. Search rows carry no rank and no quality claim. */ +export type SearchRow = { + repo: string + downloads: number + likes: number + license: string | null + gated: boolean + /** "quantized from Qwen/Qwen3-8B", which is provenance and not a grade. */ + quantized_from: string | null + last_modified: string | null +} + +export type RepoBuild = { + catalog_id: string + file: string + quantization: string + size_bytes: number + fit: Fit + badge: Badge + can_install: boolean +} + +export type RepoDetail = { + repo: string + architecture: string + context_length: number + supported: boolean + builds: RepoBuild[] + /** Eligibility is not fit: a model can fit and still be refused here. */ + ineligible_reason: string | null } export type InstallEvent = - | { type: "starting" | "verifying" | "selecting"; message?: string } + | { + // `preparing` is the wait for the runtime to restart and pick the model + // up; the router learns about a new file only at startup. + type: "starting" | "verifying" | "preparing" | "selecting" + message?: string + } | { type: "downloading" message?: string @@ -178,16 +218,35 @@ export async function installLocalImageModel( } } -export function getModelCatalog( - refresh = false, +export function getModelCatalog(signal?: AbortSignal): Promise { + return requestJson("/llm/catalog", { signal }) +} + +export function getSystem(signal?: AbortSignal): Promise<{ budget: Budget }> { + return requestJson<{ budget: Budget }>("/llm/system", { signal }) +} + +export function searchModels( + query: string, signal?: AbortSignal -): Promise { - return requestJson( - refresh ? "/llm/catalog?refresh=true" : "/llm/catalog", +): Promise<{ results: SearchRow[] }> { + return requestJson<{ results: SearchRow[] }>( + `/llm/search?q=${encodeURIComponent(query)}`, { signal } ) } +/** + * Reads 2 to 4 MB of the model's header, so the trigger is opening a result + * rather than hovering or typing. The list badge stays approximate until then. + */ +export function getRepoDetail( + repo: string, + signal?: AbortSignal +): Promise { + return requestJson(`/llm/search/${repo}`, { signal }) +} + export async function installCatalogModel( catalogId: string, onEvent: (event: InstallEvent) => void, @@ -215,12 +274,9 @@ export async function installCatalogModel( throw new Error("The install stream ended before completion") } -export function deleteLocalModel( - provider: string, - modelName: string -): Promise { +export function deleteLocalModel(modelId: string): Promise { return requestJson( - `/llm/providers/${encodeURIComponent(provider)}/models/${encodeURIComponent(modelName)}`, + `/llm/models/${encodeURIComponent(modelId)}`, { method: "DELETE" } ) } diff --git a/surfsense_local/frontend/src/features/model-catalog/fit-badge.tsx b/surfsense_local/frontend/src/features/model-catalog/fit-badge.tsx new file mode 100644 index 0000000000..72e7ab0674 --- /dev/null +++ b/surfsense_local/frontend/src/features/model-catalog/fit-badge.tsx @@ -0,0 +1,52 @@ +import { Badge } from "@/components/ui/badge" +import { cn } from "@/lib/utils" + +import type { Fit, Badge as FitCopy } from "./api" + +/** + * Three states, and the API owns every word of them. Copy branches on whether + * the machine has a discrete card or unified memory, which is a fact about the + * budget rather than about the runtime, so it is decided server side and + * rendered verbatim here. + * + * "Full speed" is relative to the model, not a promise of speed in the + * abstract: a 32B running entirely on a 4090 is still slower than a 4B. + */ +const variantFor: Record< + Fit["state"], + "secondary" | "outline" | "destructive" +> = { + fits: "secondary", + // Not destructive. Reduced speed installs exactly like full speed, it just + // runs slower, and styling it as a failure would discourage a configuration + // that measurably works. + partial: "outline", + too_big: "destructive", +} + +export function FitBadge({ + fit, + copy, + className, +}: { + fit: Fit + copy: FitCopy + className?: string +}) { + return ( + + {fit.approximate ? `~ ${copy.verdict}` : copy.verdict} + + ) +} + +/** + * The explanation, dimmed and trailing. Kept separate from the verdict because + * the verdict is what someone choosing a model needs and the mechanism is not. + */ +export function FitReason({ copy }: { copy: FitCopy }) { + return

{copy.reason}

+} diff --git a/surfsense_local/frontend/src/features/model-catalog/local-image-model.tsx b/surfsense_local/frontend/src/features/model-catalog/local-image-model.tsx index 7c6cd2db14..0649e8e61b 100644 --- a/surfsense_local/frontend/src/features/model-catalog/local-image-model.tsx +++ b/surfsense_local/frontend/src/features/model-catalog/local-image-model.tsx @@ -138,7 +138,9 @@ export function LocalImageModel({ disabled = false }: { disabled?: boolean }) {

{model.detail}{" "} - {model.installed ? null : `${gigabytes(model.size_bytes)} download.`} + {model.installed + ? null + : `${gigabytes(model.size_bytes)} download.`}

{model.installed ? ( diff --git a/surfsense_local/frontend/src/features/model-catalog/model-card.tsx b/surfsense_local/frontend/src/features/model-catalog/model-card.tsx index 851959f4cb..9a7b18f8e9 100644 --- a/surfsense_local/frontend/src/features/model-catalog/model-card.tsx +++ b/surfsense_local/frontend/src/features/model-catalog/model-card.tsx @@ -1,34 +1,20 @@ -import { DownloadIcon, Trash2Icon } from "@/components/ui/icons" +import { DownloadIcon, SparklesIcon, Trash2Icon } from "@/components/ui/icons" import { Badge } from "@/components/ui/badge" import { Button } from "@/components/ui/button" import type { CatalogRow } from "./api" +import { FitBadge, FitReason } from "./fit-badge" import { InstallProgress } from "./install-progress" import type { InstallState } from "./use-model-catalog" -const formatSize = (sizeGb: number) => - `${new Intl.NumberFormat(undefined, { maximumFractionDigits: 1 }).format(sizeGb)} GB` - -const fitLabel: Record = { - perfect: "Great fit", - good: "Good fit", - marginal: "May be slow", - too_tight: "Doesn't fit", - unknown: "Fit unknown", -} +const formatSize = (bytes: number) => + `${new Intl.NumberFormat(undefined, { maximumFractionDigits: 1 }).format(bytes / 1e9)} GB` export function ModelCard({ row, installState, actionsDisabled, runtimeAvailable, - // Whether a hardware scan has ever run this session. An "unknown" fit - // before any scan is every row, all the time — repeating a badge that - // says nothing here 8+ times is noise, so it's suppressed. An "unknown" - // fit *after* a scan (a model llmfit just had no estimate for, despite a - // real scan having run) is the rare, actually-notable case and keeps its - // badge. - scanned, onAction, onCancel, onDelete, @@ -37,7 +23,6 @@ export function ModelCard({ installState: InstallState actionsDisabled: boolean runtimeAvailable: boolean - scanned: boolean onAction: (row: CatalogRow) => void onCancel: () => void onDelete?: (row: CatalogRow) => void @@ -45,25 +30,31 @@ export function ModelCard({ const isInstalling = installState.status === "installing" && installState.catalogId === row.catalog_id + // Only physics refuses. Reduced speed installs exactly like full speed. const cannotInstall = - !row.installed && - (!row.can_install || row.fit === "too_tight" || !runtimeAvailable) + !row.installed && (!row.can_install || !runtimeAvailable) return (
-
-

{row.label}

- {row.fit !== "unknown" || scanned ? ( - - {fitLabel[row.fit]} - - ) : null} - {row.disk_size_gb !== null ? ( +
+
+ {row.recommended ? ( + + ) : null} +

{row.label}

+ + {row.capabilities.includes("vision") ? ( + Reads images + ) : null} - {formatSize(row.disk_size_gb)} + {formatSize(row.size_bytes)} - ) : null} +
+
{row.selected ? ( @@ -84,7 +75,7 @@ export function ModelCard({ {row.installed ? "Use" : "Download"} )} - {row.can_delete && onDelete ? ( + {row.installed && onDelete ? (
diff --git a/surfsense_local/frontend/src/features/model-catalog/model-catalog-page.tsx b/surfsense_local/frontend/src/features/model-catalog/model-catalog-page.tsx index 206e5af1e7..6439bbfca0 100644 --- a/surfsense_local/frontend/src/features/model-catalog/model-catalog-page.tsx +++ b/surfsense_local/frontend/src/features/model-catalog/model-catalog-page.tsx @@ -1,10 +1,5 @@ import { Fragment, useId, useState, type ReactNode } from "react" -import { - CircleAlertIcon, - DotIcon, - RefreshCwIcon, - SearchIcon, -} from "@/components/ui/icons" +import { CircleAlertIcon, DotIcon, Trash2Icon } from "@/components/ui/icons" import { Alert, AlertDescription, AlertTitle } from "@/components/ui/alert" import { @@ -18,71 +13,43 @@ import { AlertDialogTitle, } from "@/components/ui/alert-dialog" import { Button } from "@/components/ui/button" -import { Input } from "@/components/ui/input" import { ScrollShadow } from "@/components/ui/scroll-shadow" import { Spinner } from "@/components/ui/spinner" import { Separator } from "@/components/ui/separator" import { cn } from "@/lib/utils" -import type { - CatalogRow, - HardwareProfile, - ModelCatalog, - RuntimeStatus, -} from "./api" +import type { Budget, CatalogRow, RepoBuild } from "./api" import { LocalImageModel } from "./local-image-model" import { ModelCard } from "./model-card" import { ModelFamilyGroup } from "./model-family-group" +import { ModelSearch } from "./model-search" import { useModelCatalog } from "./use-model-catalog" import type { ModelSelection } from "@/features/model-selection/api" -// Reserved while "More models" is being searched, so narrowing to a handful -// of matches (or none) leaves blank space below them instead of shrinking -// the page's scrollable area — that shrink is what clamps scrollTop and -// yanks the whole page upward. Fixed rather than measured: it doesn't chase -// this catalog's actual row count, so there's no DOM measurement, no ref, -// and no effect that has to race the catalog finishing its own load. -const EXPLORE_SEARCH_RESERVED_HEIGHT = 540 - function messageFrom(error: unknown) { return error instanceof Error ? error.message : "An unexpected error occurred" } -function runtimeAvailable(status: RuntimeStatus | undefined) { - if (status === undefined) { - return true - } - if (typeof status === "boolean") { - return status - } - if (typeof status === "string") { - return !["offline", "unavailable", "error", "stopped"].includes( - status.toLowerCase() - ) +/** + * What this machine has, in one line. No scan and no button: the figure comes + * from the runtime's own allocator in about 180ms, so there is nothing to wait + * for and nothing to trigger. + */ +function hardwareSummary(budget: Budget | undefined) { + const gb = (bytes: number) => + `${new Intl.NumberFormat(undefined, { maximumFractionDigits: 1 }).format(bytes / 1e9)} GB` + if (!budget) { + return ["Checking this computer"] } - if (status.available !== undefined) { - return status.available + if (!budget.has_gpu) { + return [ + "Runs on your processor", + `${gb(budget.ram_available_bytes)} memory`, + ] } - if (status.healthy !== undefined) { - return status.healthy - } - return !["offline", "unavailable", "error", "stopped"].includes( - status.status?.toLowerCase() ?? "" - ) -} - -function hardwareSummary(hardware: HardwareProfile | null, scanned: boolean) { - if (!hardware) { - // Never scanned (the common first-launch case, since nothing probes - // hardware until the user asks) reads differently from a scan that was - // attempted and failed — the latter also surfaces a warning banner. - return [scanned ? "Hardware profile unavailable" : "Not detected yet"] - } - const name = hardware.gpu_name ?? hardware.cpu_name - const memory = hardware.total_ram_gb - const parts = [name, memory !== null ? `${memory} GB memory` : null].filter( - (part): part is string => part !== null - ) - return parts.length > 0 ? parts : ["Hardware profile analyzed"] + return [ + budget.uma ? "Apple Silicon GPU" : "Graphics card", + `${gb(budget.device_total_bytes)} memory`, + ] } function grouped(rows: CatalogRow[]) { @@ -98,7 +65,6 @@ function CatalogSection({ title, description, rows, - catalog, headerAction, children, emptyMessage, @@ -107,9 +73,8 @@ function CatalogSection({ title: string description: string rows: CatalogRow[] - catalog: ModelCatalog headerAction?: ReactNode - children: (row: CatalogRow, available: boolean) => ReactNode + children: (row: CatalogRow) => ReactNode // Shown instead of the row list when `rows` is empty but the section // should still render (e.g. a search with no matches) — omit this prop to // keep the earlier behavior of hiding the section entirely when empty. @@ -145,18 +110,11 @@ function CatalogSection({ {[...grouped(rows)].map(([family, familyRows]) => ( {familyRows.map((row) => ( - // canonical_id, not catalog_id: the latter is an opaque, - // refresh-sensitive install-plan token (it deliberately goes - // stale on every rescan, so a stale install can't slip through) - // — using it as a React key would remount every row on every - // scan. canonical_id is stable across scan states for the same - // model, so the row updates in place instead. -
  • - {children( - row, - runtimeAvailable(catalog.runtime_status[row.runtime]) - )} -
    • + // model_id, not catalog_id: the latter is an opaque install + // token that the server may reissue, and using it as a key would + // remount the row each time. model_id is stable, so it updates + // in place instead. +
  • {children(row)}
    • ))}     ))} @@ -165,45 +123,6 @@ function CatalogSection({ ) } -function ScanHardwareCta({ - onScan, - pending, - search, -}: { - onScan: () => void - pending: boolean - search: ReactNode -}) { - return ( -
    -
    -
    -

    More models

    -

    - Other compatible models, ranked for your machine. -

    -
    - {search} -
    -
    -

    - Find every model that fits your machine, ranked best-first. -

    - -
    -
    - ) -} - export function ModelCatalogPage({ allowDelete = false, scrollable = true, @@ -223,22 +142,23 @@ export function ModelCatalogPage({ }) { const { catalog, - rescan, install, installState, cancelInstall, deleteModel, selectInstalled, } = useModelCatalog(onSelected, onModelUnavailable, onModelsChanged) - const [exploreQuery, setExploreQuery] = useState("") - const exploreQueryActive = exploreQuery.trim() !== "" - const [pendingConfirmation, setPendingConfirmation] = - useState(null) - const [pendingDelete, setPendingDelete] = useState(null) + // Either list can offer deletion, and both need the same confirmation, so + // this holds the minimum both rows share rather than a whole catalog row. + const [pendingDelete, setPendingDelete] = useState<{ + label: string + variant_model_id: string + selected: boolean + } | null>(null) - // No skeleton: the catalog GET no longer probes hardware on an unrefreshed - // load (see `LlmfitAdvisor.scan`), so it resolves as fast as any other - // page fetch and a loading state would only ever flash. + // No skeleton: the catalog is the manifest plus a directory listing, priced + // locally, so it resolves as fast as any other page fetch and a loading state + // would only ever flash. if (catalog.isPending) { return null } @@ -254,92 +174,31 @@ export function ModelCatalogPage({ } const data = catalog.data - const seen = new Set() - const unique = (rows: CatalogRow[]) => - rows.filter((row) => { - if (seen.has(row.catalog_id)) { - return false - } - seen.add(row.catalog_id) - return true - }) - // Curated is always populated — the manifest's own 8 models, scan-free — - // and never merges into "More models": scanning only ever adds a fit - // badge to a row already here, never moves it elsewhere. "More models" is - // fundamentally scan-derived, so it only exists once `data.scanned` is - // true; until then a "Scan hardware" prompt takes its place. - const curated = unique(data.curated) - const explore = unique(data.explore) - const installed = unique(data.installed) const curatedSection = { - title: "Curated models", - description: "Staff picks. Scan machine to see how well each one runs on your machine.", - rows: curated, - } - const exploreQueryNormalized = exploreQuery.trim().toLowerCase() - const filteredExplore = - exploreQueryNormalized === "" - ? explore - : explore.filter((row) => - row.label.toLowerCase().includes(exploreQueryNormalized) - ) - // Disabled (not hidden) until the first scan exists — there's nothing to - // search yet, and a persistent, disabled control previews the feature - // instead of the layout shifting once scanning finishes. - const exploreSearchInput = ( -
    - - setExploreQuery(event.target.value)} - /> -
    - ) - const exploreSection = { - title: "More models", - description: "Other compatible models, best fit first.", - rows: filteredExplore, - emptyMessage: `No models match "${exploreQuery.trim()}".`, - headerAction: exploreSearchInput, - } - const installedSection = { - title: "Installed", - description: "Local models already available on this machine.", - rows: installed, + title: "Tested by SurfSense", + description: "Models we have run, priced against this computer.", + rows: data.curated ?? [], } - // Estimates reserve resources for SurfSense and may vary by workload. const busy = disabled || install.isPending || selectInstalled.isPending || deleteModel.isPending - const act = (row: CatalogRow) => { - if (busy) { - return - } - if (row.fit === "marginal") { - setPendingConfirmation(row) - return - } - if (row.installed) { - selectInstalled.mutate(row) - } else { - install.mutate(row) + // A searched build and a curated row install through the same call, because + // the id is opaque either way and the server cannot tell them apart. + const installBuild = (build: RepoBuild) => { + if (!busy) { + install.mutate(build) } } - const confirm = () => { - const row = pendingConfirmation - setPendingConfirmation(null) - if (!row || busy) { + const act = (row: CatalogRow) => { + if (busy) { return } + // No confirmation for a partial fit. It runs, slower, and llama.cpp places + // the layers; only physics blocks, and that is already `can_install`. if (row.installed) { selectInstalled.mutate(row) } else { @@ -347,16 +206,22 @@ export function ModelCatalogPage({ } } - const card = (row: CatalogRow, available: boolean) => ( + const onDeleteRow = (row: CatalogRow) => + setPendingDelete({ + label: row.label, + variant_model_id: row.variant_model_id, + selected: row.selected, + }) + + const card = (row: CatalogRow) => ( ) @@ -374,112 +239,102 @@ export function ModelCatalogPage({ return (
    -
    -
    -

    - {hardwareSummary(data.hardware, data.scanned).map((part, index) => ( - - {index > 0 ? ( - - ))} -

    -

    - {data.scanned - ? "Only models compatible with this machine are shown." - : "Scan to filter by what fits your machine."} -

    -
    - +
    +

    + {hardwareSummary(data.budget).map((part, index) => ( + + {index > 0 ? ( + + ))} +

    +

    + Every model below is priced against this computer. +

    - {data.warnings.map((warning) => ( - - - Recommendations are degraded - {warning.message} - - ))} - {rescan.isError ? ( -

    - {messageFrom(rescan.error)} -

    - ) : null} - - {installedSection ? ( - - {card} - + {(data.installed ?? []).length > 0 ? ( +
    +
    +

    Installed

    +

    + Already on this machine. +

    +
    +
      + {(data.installed ?? []).map((row) => ( +
    • + + {row.model_id} + +
      + {row.selected ? ( + + ) : null} + {allowDelete ? ( + + ) : null} +
      +
      • + ))} +
    +
    ) : null} - {curated.length > 0 ? ( - - {card} - + {(data.curated ?? []).length > 0 ? ( + {card} ) : null} - {curated.length > 0 && (explore.length > 0 || !data.scanned) ? ( - - ) : null} + - {data.scanned ? ( - explore.length > 0 ? ( - - {card} - - ) : null - ) : ( - rescan.mutate()} - pending={rescan.isPending} - search={exploreSearchInput} - /> - )} + - {curated.length + explore.length + installed.length === 0 ? ( + {(data.curated ?? []).length + (data.installed ?? []).length === 0 ? ( No local models are available - This machine has no compatible local configuration right now. - You can still use an OpenAI-compatible connection. + Nothing is installed and no tested model could be read. You can + search for one above, add a .gguf file from disk, or use an + OpenAI compatible connection. ) : null} + {selectInstalled.isError ? (

    {messageFrom(selectInstalled.error)} @@ -493,31 +348,6 @@ export function ModelCatalogPage({

    - { - if (!open) { - setPendingConfirmation(null) - } - }} - > - - - Use a marginal-fit model? - - This model may respond slowly or fail with long conversations. You - can cancel and choose a better-fitting model. - - - - Cancel - - Continue anyway - - - - - { diff --git a/surfsense_local/frontend/src/features/model-catalog/model-catalog.test.tsx b/surfsense_local/frontend/src/features/model-catalog/model-catalog.test.tsx index 8681f0cabf..428e6101ae 100644 --- a/surfsense_local/frontend/src/features/model-catalog/model-catalog.test.tsx +++ b/surfsense_local/frontend/src/features/model-catalog/model-catalog.test.tsx @@ -5,63 +5,59 @@ import { toast } from "sonner" import { render } from "@/test-utils" import { ModelCatalogPage } from "./model-catalog-page" -import { parseNdjson, type CatalogRow, type ModelCatalog } from "./api" +import { + parseNdjson, + type CatalogRow, + type Fit, + type ModelCatalog, +} from "./api" vi.mock("sonner", () => ({ - toast: { - error: vi.fn(), - info: vi.fn(), - }, + toast: { error: vi.fn(), info: vi.fn() }, })) +const fit = (overrides: Partial = {}): Fit => ({ + state: "fits", + need_bytes: 7_000_000_000, + budget_bytes: 16_000_000_000, + offload_fraction: 0, + approximate: false, + ...overrides, +}) + const row = (overrides: Partial = {}): CatalogRow => ({ catalog_id: "opaque-qwen", - canonical_id: "Qwen/Qwen3-8B", - family: "Qwen", - label: "Qwen 3 8B", - publisher: "Qwen", - parameter_count: 8_000_000_000, - fit: "good", - score: 92, - memory_required_gb: 6.2, - disk_size_gb: 5.1, - estimated_tps: 24, - prefill_tps: 100, - ttft_ms: 240, - effective_context_length: 8192, - estimate_confidence: "high", - license: "Apache-2.0", - runtime: "ollama", - runtime_model: "qwen3:8b", + model_id: "Qwen/Qwen3-8B", + variant_model_id: "Qwen3-8B-Q4_K_M", + label: "Qwen3 8B", + family: "Qwen3", + parameter_count: "8B", quantization: "Q4_K_M", + size_bytes: 5_027_784_512, + context_length: 40960, + fit: fit(), + badge: { verdict: "Full speed", reason: "Runs entirely on the GPU" }, + capabilities: [], installed: false, selected: false, can_install: true, - can_delete: false, - warnings: [], + recommended: false, ...overrides, }) const catalog = (overrides: Partial = {}): ModelCatalog => ({ - hardware: { - cpu_name: "Apple M3", - cpu_cores: 8, - total_ram_gb: 16, - available_ram_gb: 12, + budget: { + device_total_bytes: 16_000_000_000, + device_free_bytes: 14_000_000_000, + usable_vram_bytes: 12_900_000_000, + fit_reserve_bytes: 1_073_741_824, + ram_available_bytes: 16_000_000_000, + uma: true, has_gpu: true, - gpu_name: "Apple M3", - gpu_vram_gb: 16, - gpu_count: 1, - backend: "Metal", - unified_memory: true, }, - llmfit_version: "1.1.11", curated: [row()], - explore: [], installed: [], - scanned: true, - warnings: [], - runtime_status: { ollama: { available: true } }, + recommended_model_id: null, ...overrides, }) @@ -69,21 +65,34 @@ function stream(chunks: string[]) { const encoder = new TextEncoder() return new ReadableStream({ start(controller) { - for (const chunk of chunks) { - controller.enqueue(encoder.encode(chunk)) - } + for (const chunk of chunks) controller.enqueue(encoder.encode(chunk)) controller.close() }, }) } +/** Routes the catalog GET and leaves everything else a 404. */ +function serving( + data: ModelCatalog, + extra: (path: string, init?: RequestInit) => Response | null = () => null +) { + return vi.fn(async (input: RequestInfo | URL, init?: RequestInit) => { + const path = String(input) + if (path === "/llm/catalog") return Response.json(data) + return ( + extra(path, init) ?? + Response.json({ detail: "not found" }, { status: 404 }) + ) + }) +} + afterEach(() => { cleanup() vi.clearAllMocks() vi.unstubAllGlobals() }) -describe("normalized model catalog", () => { +describe("model catalog", () => { it("parses NDJSON split across arbitrary chunks", async () => { const values = [] for await (const value of parseNdjson<{ type: string }>( @@ -94,461 +103,267 @@ describe("normalized model catalog", () => { expect(values).toEqual([{ type: "starting" }, { type: "complete" }]) }) - it("installs with only the opaque id and advances on complete", async () => { + it("renders a hardware line and badged rows with no scan button", async () => { + // There is no scan: the budget comes from the runtime's own allocator, so + // there is nothing to wait for and nothing for the user to press. + vi.stubGlobal("fetch", serving(catalog())) + + render() + + expect(await screen.findByText("Full speed")).toBeTruthy() + expect(screen.getByText("Runs entirely on the GPU")).toBeTruthy() + expect(screen.queryByRole("button", { name: /scan/i })).toBeNull() + }) + + it("installs with only the opaque id", async () => { + // The renderer never sends a repo, file, URL, path or quantization. const onSelected = vi.fn() - const fetchMock = vi.fn( - async (input: RequestInfo | URL, init?: RequestInit) => { - void init - const path = String(input) - if (path === "/llm/catalog") { - return Response.json(catalog()) - } - if (path === "/llm/install") { - return new Response( + const fetchMock = serving(catalog(), (path) => + path === "/llm/install" + ? new Response( stream([ - '{"type":"downloading","message":"Downloading","completed":5,', - '"total":10}\n', - '{"type":"complete","selection":{"role":"generation","provider":"ollama","name":"qwen3:8b","updated_at":"2026-09-07T00:00:00Z"}}\n', + '{"type":"downloading","completed":5,"total":10}\n', + '{"type":"complete","selection":{"role":"generation","provider":"llamacpp","name":"Qwen3-8B-Q4_K_M","updated_at":"2026-09-07T00:00:00Z"}}\n', ]) ) - } - return Response.json({ detail: "not found" }, { status: 404 }) - } + : null ) vi.stubGlobal("fetch", fetchMock) const user = userEvent.setup() render() - const action = await screen.findByRole("button", { - name: "Download", - }) - expect(screen.getByText("Curated models")).toBeTruthy() - expect( - screen.getByText("Only models compatible with this machine are shown.") - ).toBeTruthy() - expect( - screen.queryByText( - "Estimates reserve resources for SurfSense and may vary by workload." - ) - ).toBeNull() - expect( - screen.getByText("Apple M3").parentElement?.querySelector("svg") - ).toBeTruthy() - expect(screen.getByText("Good fit")).toBeTruthy() - expect(screen.getByRole("list", { name: "Qwen models" })).toBeTruthy() - expect(screen.getByRole("listitem")).toBeTruthy() - expect(screen.getByText("5.1 GB")).toBeTruthy() - await user.click(action) + await user.click(await screen.findByRole("button", { name: "Download" })) await waitFor(() => expect(onSelected).toHaveBeenCalledOnce()) - const installCall = fetchMock.mock.calls.find( - ([path]) => path === "/llm/install" - ) - expect(JSON.parse(String(installCall?.[1]?.body))).toEqual({ + const call = fetchMock.mock.calls.find(([path]) => path === "/llm/install") + expect(JSON.parse(String(call?.[1]?.body))).toEqual({ catalog_id: "opaque-qwen", select: true, }) }) - it("confirms marginal models and disables too-tight installs", async () => { - vi.stubGlobal( - "fetch", - vi.fn(async () => - Response.json( - catalog({ - curated: [], - explore: [ - row({ - catalog_id: "marginal", - label: "Marginal model", - fit: "marginal", - memory_required_gb: null, - }), - row({ - catalog_id: "tight", - label: "Too tight model", - fit: "too_tight", - }), - ], - }) - ) - ) - ) - const user = userEvent.setup() - render() - - const actions = await screen.findAllByRole("button", { - name: "Download", - }) - expect(screen.getByText("More models")).toBeTruthy() - expect(screen.getByText("May be slow")).toBeTruthy() - expect(screen.getByText("Doesn't fit")).toBeTruthy() - expect(screen.queryByText("Parameters")).toBeNull() - expect((actions[1] as HTMLButtonElement).disabled).toBe(true) - await user.click(actions[0]) - const dialog = screen.getByRole("alertdialog", { - name: "Use a marginal-fit model?", - }) - expect(dialog).toBeTruthy() - expect(dialog.className).toContain("select-none") - }) - - it("uses content-width separators between populated catalog sections", async () => { + it("installs a reduced speed model with no confirmation step", async () => { + // It runs, slower, and llama.cpp places the layers. Measured: an RTX 3050 + // runs an 8B at roughly 28% on the processor without noticeable lag, so + // asking the user to confirm would discourage a setup that works. vi.stubGlobal( "fetch", - vi.fn(async () => - Response.json( - catalog({ - explore: [row({ catalog_id: "explore", label: "Explore model" })], - installed: [ - row({ - catalog_id: "installed", - label: "Installed model", - installed: true, - }), - ], - }) - ) + serving( + catalog({ + curated: [ + row({ + fit: fit({ state: "partial", offload_fraction: 0.28 }), + badge: { + verdict: "Reduced speed", + reason: "A little too big for the GPU. Most of it still fits.", + }, + }), + ], + }) ) ) render() - await screen.findByText("Installed model") - // Installed leads on its own, ahead of the image-model block, so it no - // longer takes a separator against the curated sections that follow — - // only "Curated models" and "More models" get one between them. - const separators = document.querySelectorAll('[data-slot="separator"]') - expect(separators).toHaveLength(1) - for (const separator of separators) { - expect(separator.className).toContain("data-horizontal:w-full") - expect(separator.className).toContain("my-4") - expect(separator.className).not.toContain("mx-3") - } + const action = await screen.findByRole("button", { name: "Download" }) + expect(action.hasAttribute("disabled")).toBe(false) + expect(screen.getByText("Reduced speed")).toBeTruthy() + expect(screen.queryByRole("alertdialog")).toBeNull() }) - it("filters More models by search without affecting other sections", async () => { + it("grades the reason line by how much spills", async () => { + // One sentence is wrong at both ends of a range that runs from barely + // noticeable to unusable, and the API already knows the fraction. vi.stubGlobal( "fetch", - vi.fn(async () => - Response.json( - catalog({ - explore: [ - row({ - catalog_id: "explore-a", - canonical_id: "a", - label: "Llama Explorer", - }), - row({ - catalog_id: "explore-b", - canonical_id: "b", - label: "Mistral Ranger", - }), - ], - }) - ) + serving( + catalog({ + curated: [ + row({ + fit: fit({ state: "partial", offload_fraction: 0.7 }), + badge: { + verdict: "Reduced speed", + reason: "Well over the GPU's memory. Expect it to be slow.", + }, + }), + ], + }) ) ) - const user = userEvent.setup() render() - await screen.findByText("Llama Explorer") - expect(screen.getByText("Mistral Ranger")).toBeTruthy() - expect(screen.getByText("Qwen 3 8B")).toBeTruthy() - - const search = screen.getByRole("searchbox", { name: "Search more models" }) - await user.type(search, "llama") - - expect(screen.getByText("Llama Explorer")).toBeTruthy() - expect(screen.queryByText("Mistral Ranger")).toBeNull() - // Curated is unaffected by the "More models" search. - expect(screen.getByText("Qwen 3 8B")).toBeTruthy() - - await user.clear(search) - await user.type(search, "nothing matches this") - - expect(screen.getByText('No models match "nothing matches this".')).toBeTruthy() + expect( + await screen.findByText( + "Well over the GPU's memory. Expect it to be slow." + ) + ).toBeTruthy() }) - it("reserves a fixed height for More models while searching, so a search doesn't shrink the page", async () => { + it("blocks install only when physics refuses", async () => { vi.stubGlobal( "fetch", - vi.fn(async () => - Response.json( - catalog({ - explore: [ - row({ catalog_id: "explore-a", canonical_id: "a", label: "Llama Explorer" }), - row({ catalog_id: "explore-b", canonical_id: "b", label: "Mistral Ranger" }), - ], - }) - ) + serving( + catalog({ + curated: [ + row({ + fit: fit({ state: "too_big", offload_fraction: 1 }), + badge: { + verdict: "Won't fit", + reason: "Needs about 21 GB. This Mac has 13.6 GB", + }, + can_install: false, + }), + ], + }) ) ) - const user = userEvent.setup() render() - await screen.findByText("Llama Explorer") - - const listElement = () => - screen - .getByText("More models") - .closest("section") - ?.querySelector('[data-slot="catalog-section-list"]') as HTMLElement - - // No reservation while unfiltered — the natural row list is what shows. - expect(listElement().style.minHeight).toBe("") - - const search = screen.getByRole("searchbox", { name: "Search more models" }) - await user.type(search, "llama") - - expect(listElement().style.minHeight).toBe("320px") - await user.clear(search) - expect(listElement().style.minHeight).toBe("") + const action = await screen.findByRole("button", { name: "Download" }) + expect(action.hasAttribute("disabled")).toBe(true) + expect( + screen.getByText("Needs about 21 GB. This Mac has 13.6 GB") + ).toBeTruthy() }) - it("shows install failures as a toast instead of inside the model row", async () => { + it("marks the recommendation without displaying any rank", async () => { + // Rank orders the list and selects the star. It is never shown, and the + // surest way to keep that true is for the row never to carry it. vi.stubGlobal( "fetch", - vi.fn(async (input: RequestInfo | URL) => { - if (String(input) === "/llm/catalog") { - return Response.json(catalog()) - } - return Response.json( - { detail: "The model could not be installed. Retry the download." }, - { status: 500 } - ) - }) + serving( + catalog({ + curated: [row({ recommended: true })], + recommended_model_id: "Qwen/Qwen3-8B", + }) + ) ) - const user = userEvent.setup() render() - await user.click(await screen.findByRole("button", { name: "Download" })) - await waitFor(() => - expect(toast.error).toHaveBeenCalledWith( - "The model could not be installed. Retry the download.", - { id: "model-install-error" } - ) - ) expect( - screen.queryByText("The model could not be installed. Retry the download.") - ).toBeNull() + await screen.findByLabelText("Recommended for this computer") + ).toBeTruthy() + expect(screen.queryByText(/rank/i)).toBeNull() }) - it("shows a scan CTA instead of More models until scanned, and curated stays visible", async () => { - const fetchMock = vi.fn(async (input: RequestInfo | URL) => { - const path = String(input) - if (path === "/llm/catalog") { - return Response.json( - catalog({ - scanned: false, - curated: [row({ fit: "unknown", score: null })], - explore: [], - }) - ) - } - if (path === "/llm/catalog?refresh=true") { - return Response.json( - catalog({ scanned: true, explore: [row({ catalog_id: "explore" })] }) - ) - } - return Response.json({ detail: "not found" }, { status: 404 }) - }) - vi.stubGlobal("fetch", fetchMock) + it("explains that search is unavailable rather than erroring", async () => { + // With egress off, curated and installed still work. That is the airgapped + // product, not a degraded one. + vi.stubGlobal("fetch", serving(catalog())) const user = userEvent.setup() render() + await user.type( + await screen.findByRole("searchbox", { name: "Search all models" }), + "qwen" + ) - // Curated shows immediately, scan-free — with no fit badge at all rather - // than a claim we can't back up yet. - expect(await screen.findByText("Curated models")).toBeTruthy() - expect(screen.getByText("Qwen 3 8B")).toBeTruthy() - expect(screen.queryByText("Fit unknown")).toBeNull() - expect(screen.queryByText("Good fit")).toBeNull() - // "More models" keeps its heading, but its content is the scan prompt, - // not a row list, until scanned. - expect(screen.getByText("More models")).toBeTruthy() expect( - screen.getByText(/Find every model that fits your machine/) + await screen.findByText(/needs access to huggingface\.co/i) ).toBeTruthy() - // Two "Scan hardware" buttons exist pre-scan (the persistent header - // control and the "More models" CTA) — both trigger the same rescan. - const scanButtons = screen.getAllByRole("button", { name: "Scan hardware" }) - expect(scanButtons).toHaveLength(2) - - // The search box is present but disabled before the first scan — there's - // nothing to search yet, but the control doesn't pop in/out of the - // layout once scanning finishes. - const search = screen.getByRole("searchbox", { - name: "Search more models", - }) as HTMLInputElement - expect(search.disabled).toBe(true) - - await user.click(scanButtons[0]) - - await waitFor(() => - expect( - fetchMock.mock.calls.some( - ([path]) => path === "/llm/catalog?refresh=true" - ) - ).toBe(true) - ) - // The scan-free "More models" CTA is replaced by the real row list once - // scanned, so re-query rather than reuse the pre-scan input reference. - await waitFor(() => { - const rescanned = screen.getByRole("searchbox", { - name: "Search more models", - }) as HTMLInputElement - expect(rescanned.disabled).toBe(false) - }) - }) - - it("rescans through the explicit refresh endpoint", async () => { - const fetchMock = vi.fn(async (input: RequestInfo | URL) => { - void input - return Response.json(catalog()) - }) - vi.stubGlobal("fetch", fetchMock) - const user = userEvent.setup() - render() - - await user.click( - await screen.findByRole("button", { name: "Rescan hardware" }) - ) - await waitFor(() => - expect( - fetchMock.mock.calls.some( - ([path]) => path === "/llm/catalog?refresh=true" - ) - ).toBe(true) - ) }) - it("keeps installed controls when recommendations are degraded", async () => { + it("lists search results and prices a repo's builds when it is opened", async () => { + // The header read costs a few megabytes, so it happens on opening a row + // rather than for every result in a list. vi.stubGlobal( "fetch", - vi.fn(async () => - Response.json( - catalog({ - hardware: null, - curated: [], - installed: [ - row({ - catalog_id: "installed", - installed: true, - can_install: false, - }), - ], - warnings: [ + serving(catalog(), (path) => { + if (path.startsWith("/llm/search?")) { + return Response.json({ + results: [ { - code: "missing", - message: "Hardware recommendations are unavailable.", + repo: "unsloth/Qwen3-8B-GGUF", + downloads: 412000, + likes: 91, + license: "apache-2.0", + gated: false, + quantized_from: "Qwen/Qwen3-8B", + last_modified: null, }, ], }) - ) - ) - ) - - render() - - expect( - await screen.findByText("Hardware recommendations are unavailable.") - ).toBeTruthy() - expect(screen.getByRole("button", { name: "Use" })).toBeTruthy() - }) - - it("confirms deletion and reports when the selected model was removed", async () => { - const onModelUnavailable = vi.fn() - const fetchMock = vi.fn(async (input: RequestInfo | URL) => { - const path = String(input) - if (path === "/llm/catalog") { - return Response.json( - catalog({ - curated: [], - installed: [ - row({ - installed: true, - selected: true, - can_delete: true, - label: "Qwen 3 8B", - }), + } + if (path.startsWith("/llm/search/")) { + return Response.json({ + repo: "unsloth/Qwen3-8B-GGUF", + architecture: "qwen3", + context_length: 40960, + supported: true, + ineligible_reason: null, + builds: [ + { + catalog_id: "ticket-1", + file: "Qwen3-8B-Q4_K_M.gguf", + quantization: "Q4_K_M", + size_bytes: 5_027_784_512, + fit: fit(), + badge: { + verdict: "Full speed", + reason: "Runs entirely on the GPU", + }, + can_install: true, + }, ], }) - ) - } - if (path === "/llm/providers/ollama/models/qwen3%3A8b") { - return Response.json({ - name: "qwen3:8b", - selection_cleared: true, - }) - } - return Response.json({ detail: "not found" }, { status: 404 }) - }) - vi.stubGlobal("fetch", fetchMock) + } + return null + }) + ) const user = userEvent.setup() - render( - + render() + await user.type( + await screen.findByRole("searchbox", { name: "Search all models" }), + "qwen" ) - const deleteButton = await screen.findByRole("button", { - name: "Delete Qwen 3 8B", - }) - expect(deleteButton.getAttribute("data-variant")).toBe("destructive") - await user.click(deleteButton) - expect( - screen.getByText( - "This is your current model. Deleting it will require you to choose another model." - ) - ).toBeTruthy() - await user.click(screen.getByRole("button", { name: "Delete model" })) - await waitFor(() => expect(onModelUnavailable).toHaveBeenCalledOnce()) + const hit = await screen.findByText("unsloth/Qwen3-8B-GGUF") + expect(screen.getByText(/quantized from Qwen\/Qwen3-8B/)).toBeTruthy() + await user.click(hit) + + expect(await screen.findByText("Q4_K_M")).toBeTruthy() }) - it("cancels an in-flight install without selecting it", async () => { - const onSelected = vi.fn() + it("shows install failures as a toast instead of inside the model row", async () => { vi.stubGlobal( "fetch", - vi.fn(async (input: RequestInfo | URL, init?: RequestInit) => { - if (String(input) === "/llm/catalog") { - return Response.json(catalog()) - } - const body = new ReadableStream({ - start(controller) { - init?.signal?.addEventListener( - "abort", - () => - controller.error( - new DOMException("Installation cancelled", "AbortError") - ), - { once: true } + serving(catalog(), (path) => + path === "/llm/install" + ? new Response( + stream(['{"type":"error","message":"Download interrupted"}\n']) ) - }, - }) - return new Response(body) - }) + : null + ) ) const user = userEvent.setup() - render() + render() await user.click(await screen.findByRole("button", { name: "Download" })) - await user.click(await screen.findByRole("button", { name: "Cancel" })) await waitFor(() => - expect(toast.info).toHaveBeenCalledWith( - "Installation cancelled. You can retry.", - { id: "model-install-cancelled" } - ) + expect(toast.error).toHaveBeenCalledWith("Download interrupted", { + id: "model-install-error", + }) ) - expect( - screen.queryByText("Installation cancelled. You can retry.") - ).toBeNull() - expect(onSelected).not.toHaveBeenCalled() + }) + + it("confirms deletion of an installed model", async () => { + vi.stubGlobal( + "fetch", + serving(catalog({ curated: [row({ installed: true })] })) + ) + const user = userEvent.setup() + + render() + await user.click( + await screen.findByRole("button", { name: "Delete Qwen3 8B" }) + ) + + expect(await screen.findByRole("alertdialog")).toBeTruthy() + expect(screen.getByText("Delete Qwen3 8B?")).toBeTruthy() }) }) diff --git a/surfsense_local/frontend/src/features/model-catalog/model-search.tsx b/surfsense_local/frontend/src/features/model-catalog/model-search.tsx new file mode 100644 index 0000000000..1644b45f7d --- /dev/null +++ b/surfsense_local/frontend/src/features/model-catalog/model-search.tsx @@ -0,0 +1,207 @@ +import { useId, useState } from "react" +import { useQuery } from "@tanstack/react-query" + +import { Badge } from "@/components/ui/badge" +import { Button } from "@/components/ui/button" +import { Input } from "@/components/ui/input" +import { SearchIcon } from "@/components/ui/icons" +import { Spinner } from "@/components/ui/spinner" +import { getRepoDetail, searchModels, type RepoBuild } from "./api" +import { FitBadge, FitReason } from "./fit-badge" + +// Matches the API side cache. Hugging Face allows 500 requests per 5 minutes, +// and a list is refetched on every keystroke a debounce lets through. +const STALE_MS = 300_000 + +const formatSize = (bytes: number) => + `${new Intl.NumberFormat(undefined, { maximumFractionDigits: 1 }).format(bytes / 1e9)} GB` + +const formatDownloads = (count: number) => + new Intl.NumberFormat(undefined, { notation: "compact" }).format(count) + +/** + * One repo's builds, fetched when the row is opened rather than when it is + * listed: pricing a build exactly means reading two to four megabytes of its + * header, which is not something to do for every result in a list. + */ +function RepoBuilds({ + repo, + onInstall, + disabled, +}: { + repo: string + onInstall: (build: RepoBuild) => void + disabled: boolean +}) { + const detail = useQuery({ + queryKey: ["llm", "search", repo], + queryFn: ({ signal }) => getRepoDetail(repo, signal), + staleTime: STALE_MS, + }) + + if (detail.isPending) { + return ( +

    + + Reading this model's details +

    + ) + } + if (detail.isError || !detail.data) { + return ( +

    + Could not read this model's details. +

    + ) + } + if (!detail.data.supported) { + return ( +

    + {detail.data.ineligible_reason ?? "This model cannot run here."} +

    + ) + } + + return ( +
      + {detail.data.builds.map((build) => ( +
    • +
      +
      + {build.quantization} + + + {formatSize(build.size_bytes)} + +
      + +
      + +
      • + ))} +
    + ) +} + +export function ModelSearch({ + onInstall, + disabled, +}: { + onInstall: (build: RepoBuild) => void + disabled: boolean +}) { + const headingId = useId() + const [query, setQuery] = useState("") + const [openRepo, setOpenRepo] = useState(null) + const trimmed = query.trim() + + const results = useQuery({ + queryKey: ["llm", "search", "list", trimmed], + queryFn: ({ signal }) => searchModels(trimmed, signal), + enabled: trimmed.length > 1, + staleTime: STALE_MS, + }) + + return ( +
    +
    +
    +

    + All models +

    +

    + Every model llama.cpp can run. Ordered by how often each one is + downloaded, which is popularity and not a recommendation. +

    +
    +
    + + setQuery(event.target.value)} + /> +
    +
    + + {trimmed.length <= 1 ? ( +

    + Type to search every model llama.cpp can run. +

    + ) : results.isPending ? ( +

    + + Searching +

    + ) : results.isError ? ( +

    + Searching needs access to huggingface.co, which is turned off or + unreachable. Tested models and anything already installed still work. +

    + ) : results.data.results.length === 0 ? ( +

    + No models match “{trimmed}”. +

    + ) : ( +
      + {results.data.results.map((hit) => { + const open = openRepo === hit.repo + return ( +
    • + + {open ? ( +
      + +
      + ) : null} +
      • + ) + })} +
    + )} +
    + ) +} diff --git a/surfsense_local/frontend/src/features/model-catalog/use-model-catalog.ts b/surfsense_local/frontend/src/features/model-catalog/use-model-catalog.ts index 4b5caee0a9..d98e4706cc 100644 --- a/surfsense_local/frontend/src/features/model-catalog/use-model-catalog.ts +++ b/surfsense_local/frontend/src/features/model-catalog/use-model-catalog.ts @@ -8,6 +8,7 @@ import { installCatalogModel, type CatalogRow, type InstallEvent, + type RepoBuild, } from "./api" import { setGenerationSelection, @@ -44,31 +45,30 @@ export function useModelCatalog( status: "idle", }) + // No refresh parameter and no rescan mutation: the catalog is the manifest + // plus a directory listing, priced locally, so there is nothing to re-probe. const catalog = useQuery({ queryKey: catalogQueryKey, - queryFn: ({ signal }) => getModelCatalog(false, signal), - }) - - const rescan = useMutation({ - mutationFn: () => getModelCatalog(true), - onSuccess: (data) => queryClient.setQueryData(catalogQueryKey, data), + queryFn: ({ signal }) => getModelCatalog(signal), }) + // Curated rows and searched builds install through one path, because the id + // is opaque either way and the server cannot tell them apart. const install = useMutation({ - mutationFn: async (row: CatalogRow) => { + mutationFn: async (target: CatalogRow | RepoBuild) => { const nextController = new AbortController() controller.current = nextController setInstallState({ status: "installing", - catalogId: row.catalog_id, + catalogId: target.catalog_id, event: { type: "starting", message: "Preparing download" }, }) return installCatalogModel( - row.catalog_id, + target.catalog_id, (event) => setInstallState({ status: "installing", - catalogId: row.catalog_id, + catalogId: target.catalog_id, event, }), nextController.signal @@ -100,9 +100,9 @@ export function useModelCatalog( const selectInstalled = useMutation({ mutationFn: (row: CatalogRow) => setGenerationSelection({ - provider: row.runtime, + provider: "llamacpp", connection_id: null, - name: row.runtime_model, + name: row.variant_model_id, }), onSuccess: async (selection) => { await Promise.all([ @@ -114,8 +114,8 @@ export function useModelCatalog( }) const deleteModel = useMutation({ - mutationFn: (row: CatalogRow) => - deleteLocalModel(row.runtime, row.runtime_model), + mutationFn: (target: { variant_model_id: string }) => + deleteLocalModel(target.variant_model_id), onSuccess: async (result) => { if (result.selection_cleared) { onModelUnavailable?.() @@ -135,7 +135,6 @@ export function useModelCatalog( return { catalog, - rescan, install, installState, cancelInstall: () => controller.current?.abort(), From 768f4f113f4e6e2c929b0baf49c877edd19a608a Mon Sep 17 00:00:00 2001 From: Anish Sarkar <104695310+AnishSarkar22@users.noreply.github.com> Date: Sun, 20 Sep 2026 20:07:43 +0530 Subject: [PATCH 26/60] refactor(frontend): rename Ollama-era copy and tests for the llamacpp runtime Co-Authored-By: Claude Sonnet 5 --- .../src/features/chat/chat-error-notice.tsx | 4 +- .../src/features/chat/model-picker.test.tsx | 10 ++-- .../dashboard/dashboard-page.test.tsx | 54 +++++++++--------- .../model-selection/selected-roles.tsx | 4 +- .../onboarding/onboarding-page.test.tsx | 56 +++++++------------ .../settings/settings-dialog.test.tsx | 2 +- surfsense_local/frontend/src/lib/api.test.ts | 10 ++-- 7 files changed, 63 insertions(+), 77 deletions(-) diff --git a/surfsense_local/frontend/src/features/chat/chat-error-notice.tsx b/surfsense_local/frontend/src/features/chat/chat-error-notice.tsx index ea9631c640..ef1a78606c 100644 --- a/surfsense_local/frontend/src/features/chat/chat-error-notice.tsx +++ b/surfsense_local/frontend/src/features/chat/chat-error-notice.tsx @@ -12,9 +12,9 @@ function actionFor(error: ChatTurnError): Action { case "provider_not_found": return "model-setup" case "network": - // A bad base URL is a Model setup fix; a local Ollama that isn't + // A bad base URL is a Model setup fix; a local runtime that isn't // running isn't — there's no settings action that starts it. - return error.provider === "ollama" ? "none" : "model-setup" + return error.provider === "llamacpp" ? "none" : "model-setup" default: return "retry" } diff --git a/surfsense_local/frontend/src/features/chat/model-picker.test.tsx b/surfsense_local/frontend/src/features/chat/model-picker.test.tsx index 1cacd28baa..7d29b25868 100644 --- a/surfsense_local/frontend/src/features/chat/model-picker.test.tsx +++ b/surfsense_local/frontend/src/features/chat/model-picker.test.tsx @@ -21,7 +21,7 @@ describe("composer model picker", () => { if (path === "/llm/providers") { return Response.json([ { - name: "ollama", + name: "llamacpp", healthy: true, can_download: true, requires_key: false, @@ -29,7 +29,7 @@ describe("composer model picker", () => { }, ]) } - if (path === "/llm/providers/ollama/models") { + if (path === "/llm/providers/llamacpp/models") { return Response.json([ { name: "llama3.2:1b", @@ -46,7 +46,7 @@ describe("composer model picker", () => { if (path === "/llm/selection/generation" && init?.method === "PUT") { return Response.json({ role: "generation", - provider: "ollama", + provider: "llamacpp", connection_id: null, name: JSON.parse(String(init.body)).name, updated_at: "2026-09-09T00:00:00Z", @@ -62,7 +62,7 @@ describe("composer model picker", () => { { await waitFor(() => expect(onModelSelected).toHaveBeenCalledWith( - expect.objectContaining({ provider: "ollama", name: "qwen3:1.7b" }) + expect.objectContaining({ provider: "llamacpp", name: "qwen3:1.7b" }) ) ) diff --git a/surfsense_local/frontend/src/features/dashboard/dashboard-page.test.tsx b/surfsense_local/frontend/src/features/dashboard/dashboard-page.test.tsx index 7f8c45e733..cce43e95c5 100644 --- a/surfsense_local/frontend/src/features/dashboard/dashboard-page.test.tsx +++ b/surfsense_local/frontend/src/features/dashboard/dashboard-page.test.tsx @@ -84,7 +84,7 @@ describe("dashboard chat", () => { const path = String(input) if (path === "/llm/providers") { return Response.json([ - { name: "ollama", healthy: true, can_download: true }, + { name: "llamacpp", healthy: true, can_download: true }, ]) } if ( @@ -123,7 +123,7 @@ describe("dashboard chat", () => { initialProviderAvailable={true} selection={{ role: "generation", - provider: "ollama", + provider: "llamacpp", connection_id: null, name: "llama3.2:1b", updated_at: "2026-09-05T00:00:00Z", @@ -176,7 +176,7 @@ describe("dashboard chat", () => { const path = String(input) if (path === "/llm/providers") { return Response.json([ - { name: "ollama", healthy: true, can_download: true }, + { name: "llamacpp", healthy: true, can_download: true }, ]) } if ( @@ -204,7 +204,7 @@ describe("dashboard chat", () => { initialProviderAvailable={true} selection={{ role: "generation", - provider: "ollama", + provider: "llamacpp", connection_id: null, name: "llama3.2:1b", updated_at: "2026-09-05T00:00:00Z", @@ -259,7 +259,7 @@ describe("dashboard chat", () => { const path = String(input) if (path === "/llm/providers") { return Response.json([ - { name: "ollama", healthy: true, can_download: true }, + { name: "llamacpp", healthy: true, can_download: true }, ]) } if ( @@ -310,7 +310,7 @@ describe("dashboard chat", () => { initialProviderAvailable={true} selection={{ role: "generation", - provider: "ollama", + provider: "llamacpp", connection_id: null, name: "llama3.2:1b", updated_at: "2026-09-05T00:00:00Z", @@ -432,7 +432,7 @@ describe("dashboard chat", () => { const path = String(input) if (path === "/llm/providers") { return Response.json([ - { name: "ollama", healthy: true, can_download: true }, + { name: "llamacpp", healthy: true, can_download: true }, ]) } if ( @@ -453,7 +453,7 @@ describe("dashboard chat", () => { initialProviderAvailable={true} selection={{ role: "generation", - provider: "ollama", + provider: "llamacpp", connection_id: null, name: "llama3.2:1b", updated_at: "2026-09-05T00:00:00Z", @@ -515,7 +515,7 @@ describe("dashboard chat", () => { const path = String(input) if (path === "/llm/providers") { return Response.json([ - { name: "ollama", healthy: true, can_download: true }, + { name: "llamacpp", healthy: true, can_download: true }, ]) } if ( @@ -538,7 +538,7 @@ describe("dashboard chat", () => { initialProviderAvailable={true} selection={{ role: "generation", - provider: "ollama", + provider: "llamacpp", connection_id: null, name: "llama3.2:1b", updated_at: "2026-09-05T00:00:00Z", @@ -582,7 +582,7 @@ describe("dashboard chat", () => { const path = String(input) if (path === "/llm/providers") { return Response.json([ - { name: "ollama", healthy: true, can_download: true }, + { name: "llamacpp", healthy: true, can_download: true }, ]) } if (path === "/workspaces/1/chat/threads" && !init?.method) { @@ -690,7 +690,7 @@ describe("dashboard chat", () => { initialProviderAvailable={true} selection={{ role: "generation", - provider: "ollama", + provider: "llamacpp", connection_id: null, name: "llama3.2:1b", updated_at: "2026-09-05T00:00:00Z", @@ -771,7 +771,7 @@ describe("dashboard chat", () => { const path = String(input) if (path === "/llm/providers") { return Response.json([ - { name: "ollama", healthy: true, can_download: true }, + { name: "llamacpp", healthy: true, can_download: true }, ]) } if (path.includes("/chat/threads")) { @@ -792,7 +792,7 @@ describe("dashboard chat", () => { initialProviderAvailable={true} selection={{ role: "generation", - provider: "ollama", + provider: "llamacpp", connection_id: null, name: "llama3.2:1b", updated_at: "2026-09-05T00:00:00Z", @@ -825,13 +825,13 @@ describe("dashboard chat", () => { const path = String(input) if (path === "/llm/providers") { return Response.json([ - { name: "ollama", healthy: true, can_download: true }, + { name: "llamacpp", healthy: true, can_download: true }, ]) } if (path === "/llm/selection/generation") { return Response.json({ role: "generation", - provider: "ollama", + provider: "llamacpp", connection_id: null, name: "llama3.2:1b", updated_at: "2026-09-05T00:00:00Z", @@ -882,7 +882,7 @@ describe("dashboard chat", () => { initialProviderAvailable={true} selection={{ role: "generation", - provider: "ollama", + provider: "llamacpp", connection_id: null, name: "llama3.2:1b", updated_at: "2026-09-05T00:00:00Z", @@ -913,7 +913,7 @@ describe("dashboard chat", () => { const path = String(input) if (path === "/llm/providers") { return Response.json([ - { name: "ollama", healthy: true, can_download: true }, + { name: "llamacpp", healthy: true, can_download: true }, ]) } if (path.endsWith("/documents?document_type=FILE&document_type=NOTE")) { @@ -961,7 +961,7 @@ describe("dashboard chat", () => { initialProviderAvailable={true} selection={{ role: "generation", - provider: "ollama", + provider: "llamacpp", connection_id: null, name: "llama3.2:1b", updated_at: "2026-09-05T00:00:00Z", @@ -995,7 +995,7 @@ describe("dashboard chat", () => { const path = String(input) if (path === "/llm/providers") { return Response.json([ - { name: "ollama", healthy: true, can_download: true }, + { name: "llamacpp", healthy: true, can_download: true }, ]) } if ( @@ -1025,7 +1025,7 @@ describe("dashboard chat", () => { initialProviderAvailable={true} selection={{ role: "generation", - provider: "ollama", + provider: "llamacpp", connection_id: null, name: "llama3.2:1b", updated_at: "2026-09-05T00:00:00Z", @@ -1065,7 +1065,7 @@ describe("dashboard chat", () => { const path = String(input) if (path === "/llm/providers") { return Response.json([ - { name: "ollama", healthy: true, can_download: true }, + { name: "llamacpp", healthy: true, can_download: true }, ]) } if ( @@ -1116,7 +1116,7 @@ describe("dashboard chat", () => { initialProviderAvailable={true} selection={{ role: "generation", - provider: "ollama", + provider: "llamacpp", connection_id: null, name: "llama3.2:1b", updated_at: "2026-09-05T00:00:00Z", @@ -1152,7 +1152,7 @@ describe("dashboard chat", () => { const path = String(input) if (path === "/llm/providers") { return Response.json([ - { name: "ollama", healthy: true, can_download: true }, + { name: "llamacpp", healthy: true, can_download: true }, ]) } if ( @@ -1216,7 +1216,7 @@ describe("dashboard chat", () => { initialProviderAvailable={true} selection={{ role: "generation", - provider: "ollama", + provider: "llamacpp", connection_id: null, name: "llama3.2:1b", updated_at: "2026-09-05T00:00:00Z", @@ -1277,7 +1277,7 @@ describe("dashboard chat", () => { const path = String(input) if (path === "/llm/providers") { return Response.json([ - { name: "ollama", healthy: true, can_download: true }, + { name: "llamacpp", healthy: true, can_download: true }, ]) } if ( @@ -1298,7 +1298,7 @@ describe("dashboard chat", () => { initialProviderAvailable={true} selection={{ role: "generation", - provider: "ollama", + provider: "llamacpp", connection_id: null, name: "llama3.2:1b", updated_at: "2026-09-05T00:00:00Z", diff --git a/surfsense_local/frontend/src/features/model-selection/selected-roles.tsx b/surfsense_local/frontend/src/features/model-selection/selected-roles.tsx index 6d2f34d991..9c73610d22 100644 --- a/surfsense_local/frontend/src/features/model-selection/selected-roles.tsx +++ b/surfsense_local/frontend/src/features/model-selection/selected-roles.tsx @@ -16,7 +16,7 @@ import { // `null` connections mean the list is still loading or failed to load, which is // not the same as a connection that no longer exists. function sourceOf(selection: ModelSelection, connections: Connection[] | null) { - if (selection.provider === "ollama") return "Local" + if (selection.provider === "llamacpp") return "Local" if (connections === null) return null return ( connections.find((connection) => connection.id === selection.connection_id) @@ -122,7 +122,7 @@ export function SelectedRoles({ }, []) const connections = state.status === "ready" ? state.connections : null - // Same enrichment the composer's model picker uses: the raw ollama pull + // Same enrichment the composer's model picker uses: the raw model file // name can be an unreadable `hf.co/...` id, so prefer the scanned/curated // display name for the matching installed model when one is available. const generationDisplayName = diff --git a/surfsense_local/frontend/src/features/onboarding/onboarding-page.test.tsx b/surfsense_local/frontend/src/features/onboarding/onboarding-page.test.tsx index 9b09ef5337..7393b22507 100644 --- a/surfsense_local/frontend/src/features/onboarding/onboarding-page.test.tsx +++ b/surfsense_local/frontend/src/features/onboarding/onboarding-page.test.tsx @@ -11,7 +11,7 @@ function installApi() { if (path === "/llm/providers") { return Response.json([ { - name: "ollama", + name: "llamacpp", healthy: true, can_download: true, requires_key: false, @@ -19,7 +19,7 @@ function installApi() { }, ]) } - if (path === "/llm/providers/ollama/models") { + if (path === "/llm/providers/llamacpp/models") { return Response.json([ { name: "llama3.2:1b", @@ -31,7 +31,7 @@ function installApi() { if (path === "/llm/selection/generation") { return Response.json({ role: "generation", - provider: "ollama", + provider: "llamacpp", name: "llama3.2:1b", updated_at: "2026-09-05T00:00:00Z", }) @@ -47,41 +47,25 @@ function installApi() { } if (path === "/llm/catalog") { return Response.json({ - hardware: {}, - llmfit_version: "1.0", + budget: { + device_total_bytes: 16_000_000_000, + device_free_bytes: 14_000_000_000, + usable_vram_bytes: 12_900_000_000, + fit_reserve_bytes: 1_073_741_824, + ram_available_bytes: 16_000_000_000, + uma: true, + has_gpu: true, + }, curated: [], - explore: [], - scanned: true, installed: [ { - catalog_id: "opaque-llama", - canonical_id: "meta-llama/Llama-3.2-1B", - family: "Llama", - label: "Llama 3.2 1B", - publisher: "Meta", - parameter_count: 1_000_000_000, - fit: "perfect", - score: 90, - memory_required_gb: 2, - disk_size_gb: 1.2, - estimated_tps: 40, - prefill_tps: 100, - ttft_ms: 200, - effective_context_length: 8192, - estimate_confidence: "high", - license: "Llama", - runtime: "ollama", - runtime_model: "llama3.2:1b", - quantization: "Q4_K_M", - installed: true, + model_id: "Llama 3.2 1B", + file: "Llama-3.2-1B-Q4_K_M.gguf", + size_bytes: 1_200_000_000, selected: true, - can_install: true, - can_delete: true, - warnings: [], }, ], - warnings: [], - runtime_status: {}, + recommended_model_id: null, }) } if (init?.method === "PUT") { @@ -148,8 +132,10 @@ describe("model onboarding", () => { ) await user.click(screen.getByRole("button", { name: "Start setting up" })) + // The catalog no longer filters by fit, so this line stopped describing + // anything: every model is listed and every one carries an honest badge. await screen.findByText( - "Only models compatible with this machine are shown." + "Every model below is priced against this computer." ) const secondProgress = screen.getByLabelText("Onboarding step 2 of 2") expect(secondProgress.children[0]?.getAttribute("data-state")).toBe( @@ -221,7 +207,7 @@ describe("model onboarding", () => { if (path === "/llm/providers") { return Response.json([ { - name: "ollama", + name: "llamacpp", healthy: true, can_download: true, requires_key: false, @@ -229,7 +215,7 @@ describe("model onboarding", () => { }, ]) } - if (path === "/llm/providers/ollama/models") return Response.json([]) + if (path === "/llm/providers/llamacpp/models") return Response.json([]) if (path === "/llm/catalog") { return Response.json({ hardware: {}, diff --git a/surfsense_local/frontend/src/features/settings/settings-dialog.test.tsx b/surfsense_local/frontend/src/features/settings/settings-dialog.test.tsx index 1fcc4a7dbd..c9174f22f0 100644 --- a/surfsense_local/frontend/src/features/settings/settings-dialog.test.tsx +++ b/surfsense_local/frontend/src/features/settings/settings-dialog.test.tsx @@ -103,7 +103,7 @@ describe("SettingsDialog", () => { if (path === "/llm/selection/generation") { return Response.json({ role: "generation", - provider: "ollama", + provider: "llamacpp", name: "qwen3:1.7b", updated_at: "2026-09-09T00:00:00Z", }) diff --git a/surfsense_local/frontend/src/lib/api.test.ts b/surfsense_local/frontend/src/lib/api.test.ts index 554b158a59..4f70c9b087 100644 --- a/surfsense_local/frontend/src/lib/api.test.ts +++ b/surfsense_local/frontend/src/lib/api.test.ts @@ -66,8 +66,8 @@ describe("egress prompt", () => { detail: { code: "egress_disabled", message: "off", - destination: "ollama_pull", - host: "registry.ollama.ai", + destination: "model_download", + host: "huggingface.co", }, }, { status: 403 } @@ -91,7 +91,7 @@ describe("egress prompt", () => { ) setEgressPrompt(prompt) - const response = await request("/llm/providers/ollama/pull", { + const response = await request("/llm/install", { method: "POST", }) @@ -99,7 +99,7 @@ describe("egress prompt", () => { expect(prompt).toHaveBeenCalledTimes(1) expect(prompt.mock.calls[0]?.[0]).toMatchObject({ code: "egress_disabled", - detail: { destination: "ollama_pull", host: "registry.ollama.ai" }, + detail: { destination: "model_download", host: "huggingface.co" }, }) expect(fetchMock).toHaveBeenCalledTimes(2) }) @@ -110,7 +110,7 @@ describe("egress prompt", () => { setEgressPrompt(async () => false) await expect( - request("/llm/providers/ollama/pull", { method: "POST" }) + request("/llm/install", { method: "POST" }) ).rejects.toMatchObject({ code: "egress_disabled" }) expect(fetch).toHaveBeenCalledTimes(1) }) From 156437d15dddd7f6e94dae202296e9de8b14a324 Mon Sep 17 00:00:00 2001 From: Anish Sarkar <104695310+AnishSarkar22@users.noreply.github.com> Date: Sun, 20 Sep 2026 20:07:43 +0530 Subject: [PATCH 27/60] chore(scripts): add the curated model refresh script backed by Hugging Face Co-Authored-By: Claude Sonnet 5 --- .../backend/scripts/curated/__init__.py | 1 + .../backend/scripts/curated/huggingface.py | 44 ++++++ .../backend/scripts/refresh_curated_models.py | 125 ++++++++++++++++++ 3 files changed, 170 insertions(+) create mode 100644 surfsense_local/backend/scripts/curated/__init__.py create mode 100644 surfsense_local/backend/scripts/curated/huggingface.py create mode 100644 surfsense_local/backend/scripts/refresh_curated_models.py diff --git a/surfsense_local/backend/scripts/curated/__init__.py b/surfsense_local/backend/scripts/curated/__init__.py new file mode 100644 index 0000000000..8abe267530 --- /dev/null +++ b/surfsense_local/backend/scripts/curated/__init__.py @@ -0,0 +1 @@ +"""Authoring-time helpers for the curated manifest. Never imported by the app.""" diff --git a/surfsense_local/backend/scripts/curated/huggingface.py b/surfsense_local/backend/scripts/curated/huggingface.py new file mode 100644 index 0000000000..d037106961 --- /dev/null +++ b/surfsense_local/backend/scripts/curated/huggingface.py @@ -0,0 +1,44 @@ +"""Resolve a repo and quantization to an exact file, and read its header. + +Authoring time only. The app never calls this: it reads the committed JSON. +""" + +import httpx + +from modules.llm.fit import ModelShape +from modules.llm.gguf import shape_from_url + +API = "https://huggingface.co/api/models" +FILES = "https://huggingface.co/{repo}/resolve/main/{file}" + + +def find_variant(client: httpx.Client, repo: str, quantization: str) -> tuple[str, int]: + """The one file in `repo` built at `quantization`, and its exact size. + + A repo holds roughly twenty builds, and repo plus quantization does not name + one on its own: `Q4_K_M`, `UD-Q4_K_M` and a split set can all match. Split + parts, projectors and draft models are excluded rather than guessed between. + """ + reply = client.get(f"{API}/{repo}/tree/main", timeout=60) + reply.raise_for_status() + + candidates = [ + row + for row in reply.json() + if row["path"].endswith(".gguf") + and quantization.lower() in row["path"].lower() + and "mmproj" not in row["path"].lower() + and "draft" not in row["path"].lower() + and "-of-" not in row["path"] + ] + if not candidates: + raise SystemExit(f"{repo}: no single file build at {quantization}") + # Prefer the shortest name: "Qwen3-8B-Q4_K_M" over "Qwen3-8B-UD-Q4_K_M". + chosen = min(candidates, key=lambda row: len(row["path"])) + return chosen["path"], chosen["size"] + + +async def read_shape(repo: str, file: str) -> ModelShape: + """The architecture fields, over an HTTP range. No weights are downloaded.""" + async with httpx.AsyncClient(follow_redirects=True, timeout=120) as client: + return await shape_from_url(client, FILES.format(repo=repo, file=file)) diff --git a/surfsense_local/backend/scripts/refresh_curated_models.py b/surfsense_local/backend/scripts/refresh_curated_models.py new file mode 100644 index 0000000000..40534e5644 --- /dev/null +++ b/surfsense_local/backend/scripts/refresh_curated_models.py @@ -0,0 +1,125 @@ +"""Rewrite `curated-models.json` from real headers and listings. + +Run by hand, **never in CI**. The manifest is source: a person runs this, reads +what it proposes, downloads the file, chats with it, sets `validated`, and +commits. Four or five times a year. + + uv run scripts/refresh_curated_models.py + +Everything mechanical is derived here. The three judgement fields stay in +`ENTRIES` below, where a person edits them: + + rank preference order for *this app's job*, answering from the user's + documents with citations that resolve. Not general capability. + llmfit proposes it; a person decides. It is an int, and nothing + requires llmfit to have produced it. + rank_basis what produced the number, so a manifest part way through a + migration is detectable and two scales are never mixed. + validated somebody ran this exact file. + +> ponytail: llmfit has no flag to score at a named quantization, so proposing a +> rank means sweeping the device budget across the model's whole ladder and +> indexing the pinned rung out of it. `scripts/curated/ladder.py` is where that +> goes when llmfit is available; the ranks below were taken that way and are +> recorded with the quantization they were read at. +""" + +import asyncio +import json +from pathlib import Path + +import httpx +from curated.huggingface import find_variant, read_shape + +from modules.llm.catalog import SCHEMA_VERSION, CuratedModelsManifest + +OUT = Path(__file__).resolve().parents[1] / "modules/llm/catalog/curated-models.json" +RANK_BASIS = "llmfit-1.1.11" + +# The shipped ladder. Six rungs, one family, every machine gets a pick. +# +# The two Qwen2.5-Coder entries that were here are gone: SurfSense has no coding +# job. Chat is document Q&A with citations and the Studio formats are summary, +# flashcards, mindmap, quiz, podcast, office, web and visuals. A coder was +# taking two of eight slots in a list whose real gaps are a rung for 48 to 64 GB +# machines and a vision entry. Anyone who wants one searches for it. +ENTRIES = [ + # model_id, family, label, params, repo, quantization, rank + ("Qwen/Qwen3-0.6B", "Qwen3", "Qwen3 0.6B", "0.6B", "unsloth/Qwen3-0.6B-GGUF", "Q4_K_M", 33), + ("Qwen/Qwen3-1.7B", "Qwen3", "Qwen3 1.7B", "1.7B", "unsloth/Qwen3-1.7B-GGUF", "Q4_K_M", 48), + ("Qwen/Qwen3-4B", "Qwen3", "Qwen3 4B", "4B", "unsloth/Qwen3-4B-GGUF", "Q4_K_M", 63), + ("Qwen/Qwen3-8B", "Qwen3", "Qwen3 8B", "8B", "unsloth/Qwen3-8B-GGUF", "Q4_K_M", 78), + ("Qwen/Qwen3-14B", "Qwen3", "Qwen3 14B", "14B", "unsloth/Qwen3-14B-GGUF", "Q4_K_M", 85), + ("Qwen/Qwen3-32B", "Qwen3", "Qwen3 32B", "32B", "unsloth/Qwen3-32B-GGUF", "Q4_K_M", 92), +] + +# Files somebody has downloaded, run, and confirmed citations resolve on. +VALIDATED: set[str] = set() + + +async def build() -> dict: + models = [] + with httpx.Client(follow_redirects=True) as client: + for model_id, family, label, params, repo, quant, rank in ENTRIES: + file, size = find_variant(client, repo, quant) + shape = await read_shape(repo, file) + print( + f" {label:12s} {file:34s} {size / 1e9:5.2f} GB " + f"{shape.block_count} blocks, {shape.head_count_kv} kv-heads, " + f"ctx {shape.context_length}" + ) + models.append( + { + "model_id": model_id, + "family": family, + "label": label, + "parameter_count": params, + "shape": { + "architecture": shape.architecture, + "block_count": shape.block_count, + "head_count_kv": shape.head_count_kv, + "key_length": shape.key_length, + "value_length": shape.value_length, + "context_length": shape.context_length, + "n_vocab": shape.n_vocab, + "sliding_window": shape.sliding_window, + "expert_count": shape.expert_count, + }, + "capabilities": [], + "decode_fraction": 1.0, + "variants": [ + { + "repo": repo, + "file": file, + "quantization": quant, + "size_bytes": size, + "mmproj": None, + "rank": rank, + "rank_basis": RANK_BASIS, + "validated": file in VALIDATED, + } + ], + } + ) + return {"schema_version": SCHEMA_VERSION, "models": models} + + +def main() -> None: + print("reading headers over HTTP range, no weights downloaded:") + manifest = asyncio.run(build()) + # Fail here rather than shipping something the app will reject at startup. + CuratedModelsManifest.model_validate(manifest) + OUT.write_text(json.dumps(manifest, indent=2) + "\n") + unvalidated = [ + v["file"] + for m in manifest["models"] + for v in m["variants"] + if not v["validated"] + ] + print(f"\nwrote {OUT.relative_to(Path.cwd())} ({len(manifest['models'])} entries)") + if unvalidated: + print(f" {len(unvalidated)} builds are validated=false until someone runs them") + + +if __name__ == "__main__": + main() From 2d91219d517e242dbe53d7081a00101f8c95dff9 Mon Sep 17 00:00:00 2001 From: Anish Sarkar <104695310+AnishSarkar22@users.noreply.github.com> Date: Sun, 20 Sep 2026 20:07:43 +0530 Subject: [PATCH 28/60] docs(local): update the README and phase 7 spec for the llama.cpp runtime Co-Authored-By: Claude Sonnet 5 --- .../api/07-llamacpp-runtime.md | 308 ++++++++++++++---- surfsense_local/README.md | 4 +- 2 files changed, 244 insertions(+), 68 deletions(-) diff --git a/plans/community-local/api/07-llamacpp-runtime.md b/plans/community-local/api/07-llamacpp-runtime.md index 76734c056c..0dbd375220 100644 --- a/plans/community-local/api/07-llamacpp-runtime.md +++ b/plans/community-local/api/07-llamacpp-runtime.md @@ -428,23 +428,24 @@ is derived from the codebase rather than chosen. | Constant | Kind | Basis | |---|---|---| | **Context floor, 16K** | **runtime** | `HISTORY_BUDGET_TOKENS = 3000` plus ~8,000 tokens of grounding (~24k characters from `build_context`) plus a reply. Grounded, not chosen. | -| **Fit reserve** | prediction | **Measured: 1,028 MiB** on a Windows / Vulkan / discrete 6 GB card. A per-platform bootstrap, replaced by the machine's own figure after its first load. Two rows still blank. See **7.2**. | +| **Fit reserve** | **runtime** | **1024 MiB, and not ours.** It is llama.cpp's `fit_params_target` default, one GiB per device on every backend, read from source and observed on both Vulkan and Metal. Pass it as `-fitt` rather than predicting it. See **7.2**. | | **Compute buffers** | prediction | **Measured: ~170 MiB** for a 4B at 16K. Part of `need`, scales with context and batch. | -> **No longer a ponytail, with one caveat.** The reserve was an unmeasured -> placeholder at ~1,024 MiB. Measured on an RTX 3050, `--fit` holds back -> **1,028.34 MiB** of 5,234 free — the guess was within 4.3 MiB. Hermes' -> equivalent is `max(2 GiB, 9% of total)` (`_MARGIN_FLOOR = 2 << 30`, -> `_MARGIN_FRACTION = 0.09` in `local_runtime/hardware.py`), roughly twice as -> conservative. +> **No longer a ponytail, and no longer a prediction.** The reserve began as an +> unmeasured placeholder at ~1,024 MiB. On an RTX 3050 `--fit` held back +> **1,028.34 MiB** of 5,234 free, which read as a lucky guess. It was not a guess +> and it was not lucky: `fit_params_target` defaults to `1024 * 1024*1024` per +> device in `common/common.h`, so the true figure is **exactly 1024 MiB** and the +> Windows excess was allocator rounding. Confirmed independently on Metal, where +> the fitter prints `cannot meet free memory target of 1024 MiB`. > -> **The caveat is generality, not accuracy.** 1,028 MiB is 17% of a 6 GB card, so -> that single point cannot distinguish a constant reserve from a proportional -> one, and the two diverge badly at 24 GB. Metal and Linux are unmeasured and the -> numbers certainly do not carry — Metal has no separate VRAM at all. This is why -> 7.2 stores the reserve as a `(platform, backend)` table with blanks rather than -> a scalar, and why an unmeasured row rounds **up**: over-estimating costs a -> pessimistic badge on one screen, under-estimating ships the llmfit failure. +> **So the generality worry was unfounded and the table it motivated is gone.** +> One point could not distinguish a constant from a proportion, which was a fair +> objection; the answer is that the source says constant. Hermes' equivalent is +> `max(2 GiB, 9% of total)` (`_MARGIN_FLOOR = 2 << 30`, `_MARGIN_FRACTION = 0.09` +> in `local_runtime/hardware.py`), roughly twice as conservative, and it is +> *their* number rather than the runtime's, which is the difference that matters: +> ours is the one the allocator will actually apply, and `-fitt` lets us set it. #### Not in v1, and why @@ -550,10 +551,18 @@ header and a truncated-response retry. `modules/llm/hardware/` — `ctypes` into `libggml`, then `--list-devices`, then OS APIs. First hit wins, cached; warm the probe in the background at first -launch, not on the path of the first render (a cold `ggml_backend_load_all()` on -macOS compiles 20 Metal shader libraries, measured at ~20 s once, 180 ms after). -That cold penalty is **Metal-only**: measured off Apple Silicon the same call is -77 ms with one backend and 205 ms with two. +launch, not on the path of the first render: on macOS, 20 Metal shader libraries +are compiled on the first call, measured at **19.0 s once** and ~45 ms after. +That cold penalty is **Metal-only**: measured off Apple Silicon the equivalent +call is 77 ms with one backend and 205 ms with two. + +> **Warm the right call.** Earlier drafts attributed the cold cost to +> `ggml_backend_load_all()`. Measured on an M2, `load_all()` is **2.3 ms** and the +> compile happens inside the first **`ggml_backend_dev_count()`**, which is 19 s +> cold and 43 to 49 ms warm. A background warm that calls `load_all()` and stops +> therefore warms nothing, and the 19 s lands on whoever asks for the device list +> first, which is the first render. Enumerate the devices in the warm, not just +> load the backends. > **Folders, not two files.** Earlier drafts named `hardware.py` and `fit.py`. > Between them they carry library loading, the working-directory dance, device @@ -632,8 +641,33 @@ def select_device(devices): the 6 GB discrete card and places every layer on the slower device. - **Never aggregate.** No `sum()` across devices. One device becomes `HardwareBudget.usable_vram_bytes`. -- **`ACCEL` is not a GPU** for budgeting — an integrated part carving from system - RAM has no memory of its own to place layers in. +- **An integrated GPU is not a GPU** for budgeting — a part carving from system + RAM has no memory of its own to place layers in. It types as `IGPU`, not + `ACCEL`; see below. + +> **The device type enum has five members, not three, and getting that wrong is +> a crash rather than a wrong answer.** Read from `ggml-backend.h` at `b11050`: +> +> ```c +> GGML_BACKEND_DEVICE_TYPE_CPU = 0 // system memory +> GGML_BACKEND_DEVICE_TYPE_GPU = 1 // dedicated memory +> GGML_BACKEND_DEVICE_TYPE_IGPU = 2 // integrated, host memory +> GGML_BACKEND_DEVICE_TYPE_ACCEL = 3 // BLAS, AMX +> GGML_BACKEND_DEVICE_TYPE_META = 4 // wraps several devices for tensor parallelism +> ``` +> +> Earlier drafts of this document, and the listing in +> [`08-cuda-backend.md`](08-cuda-backend.md), label the integrated Radeon on the +> Windows machine `ACCEL`. Its raw type is `2`, so it is **`IGPU`**. The selection +> *behaviour* is unaffected, since both are skipped by `type == GPU`, and the +> reasoning was right about the hardware. Only the name was wrong. +> +> **A three-member Python enum crashes on macOS.** Every Mac lists an Accelerate +> `BLAS` device, measured at raw type `3`, so `DeviceType(3)` raises `ValueError` +> on a machine that is otherwise perfectly healthy. This is the probe running at +> first paint, so it is a startup failure, not a degraded badge. Model all five, +> and treat an unknown integer as "not a GPU" rather than as an error, because +> ggml has added a member twice and will again. This rule is required here, under Vulkan alone. It becomes load-bearing if [`08-cuda-backend.md`](08-cuda-backend.md) ever ships, because two loaded @@ -643,10 +677,18 @@ not deduplicate them. #### `ram_available_bytes` does not come from ggml reliably ggml's CPU device reports real available memory on native Windows (31884.6 MiB -total against 22750.2 MiB free, measured) but **`total == free` under WSL2**, -where the figure is virtualised. Keep the OS API in the chain for the RAM half -rather than trusting the CPU device, since `ram_available_bytes` is what -separates `PARTIAL` from `TOO_BIG`. +total against 22750.2 MiB free, measured) and **nowhere else that has been +tried**. Under WSL2 the figure is virtualised and `total == free`. On macOS it is +worse: measured on an M2 with roughly 2 GB genuinely free, the CPU device reports +**8192.0 MiB total against 8192.0 MiB free** — physical RAM restated twice, with +no live component at all. The Metal device is barely better, at 5461.3 total +against 5461.0 free. + +So **native Windows is the exception, not the rule**. Keep the OS API in the +chain for the RAM half rather than trusting the CPU device, since +`ram_available_bytes` is what separates `PARTIAL` from `TOO_BIG`, and treat a +`free == total` reading as "this device does not report live memory" rather than +as an idle machine. Two budget modes. **Capacity** (total − margin) for catalog pricing; **live** (free now) for launch decisions. Pricing against live-free while a model is @@ -686,38 +728,79 @@ not a rounding error on a 6 GB card. This replaces today's single `recommendation_reserve_gb: 2.0`, which was flagged `ponytail` as uncalibrated. -#### The fit reserve is a table, not a number +#### The fit reserve is not a measurement. It is a flag we can set + +**Corrected by reading the source and the log, on an M2 / 8 GB at `b11050`.** +Earlier drafts of this section carried a `(platform, backend, memory model)` +table with one measured row and two blanks, on the reasoning that Metal, Vulkan +and CUDA reserve different amounts and none of it transfers. **That reasoning was +wrong.** The reserve is a single hardcoded default in llama.cpp's own fitter, +identical on every backend: + +```cpp +// common/common.h — margin per device in bytes for fitting parameters to free memory +std::vector fit_params_target = + std::vector(llama_max_devices(), 1024 * 1024*1024); +``` + +**1 GiB per device, platform independent.** The Windows figure of 1,028.34 MiB +was that 1024 MiB plus 4.34 MiB of allocator rounding, so the spec's original +"~1,024 MiB" guess was not lucky to within 4.3 MiB, it was **exactly the +constant**. Metal names it out loud under `-v`: + +```text +common_params_fit_impl: projected to use 5752 MiB of device memory vs. 5460 MiB of free +common_params_fit_impl: cannot meet free memory target of 1024 MiB, + need to reduce device memory by 1315 MiB +``` + +`5752 − (5460 − 1024) = 1316`. The comparison `need > free − 1024` is exact, on a +second backend, with no fitting. -Keyed by `(platform, backend, memory model)`, because none of it transfers: Metal -has no separate VRAM at all, and a CUDA context alone costs ~541 MiB against -Vulkan's much lighter footprint. +**It is also a CLI flag: `-fitt` / `--fit-target`, in MiB, per device.** So the +reserve stops being a number to predict and becomes an input we control. Pass it +explicitly and `usable = device_free − fit_reserve` is true **by construction** +rather than by estimate, on every platform, including ones nobody has run. | platform / backend | reserve | basis | |---|---|---| -| windows / vulkan / discrete | **1028 MiB** | measured, `b11050`, RTX 3050 6 GB | -| linux / vulkan / discrete | — | unmeasured | -| macos / metal / uma | — | unmeasured | +| any backend, any platform | **1024 MiB** | llama.cpp default, read from source at `b11050`; observed on Vulkan and Metal | +| whatever we pass to `-fitt` | that value | the flag exists; prefer pinning it to inheriting it | | no GPU device | n/a | `PARTIAL` is unreachable; see **Fit states** | -> **6 GB cannot distinguish a constant from a proportion.** 1,028 MiB is 17% of -> that card; Hermes uses `max(2 GiB, 9%)`. One measurement cannot tell the two -> models apart, and they diverge badly on a 24 GB card. Record the figure with -> its device, and do not generalise from it. +**Three consequences.** The blank rows do not need measuring, and the plan to +measure them was work that could not have produced new information. The +self-calibration below is **no longer about the reserve**, only about the +compute-buffer half of the prediction. And pinning `-fitt` removes a class of +error rather than shrinking it, which is worth more than a better guess. + +> **Errors here are still asymmetric, and the direction still matters** for the +> compute-buffer term that remains predicted. Too large predicts `PARTIAL` where +> reality is `FITS`: the user sees *Reduced speed*, installs anyway (`PARTIAL` +> installs exactly like `FITS`), and is not harmed. Too small predicts `FITS` +> where reality is `PARTIAL`, which is precisely the llmfit failure this phase +> exists to delete. *Unknown shapes round up; never underestimate memory.* + +#### What `--fit` actually does when it cannot fit + +Measured, same session, Qwen3 1.7B at `-c 40960` against a 5,460 MiB working set. +It does **not** solve for a placement, it **searches** for one, reloading at each +step: -**Why shipping with two blank rows is safe.** An unmeasured row falls back to a -deliberately generous default, because the errors are asymmetric: +```text +29/29 layers → 0/29 → 29/29 → 22/29 → 23/29 (settles at 23/29, f ≈ 0.21) +``` -- **reserve too large** → predicts `PARTIAL` where reality is `FITS`. The user - sees *Reduced speed*, installs anyway (`PARTIAL` installs exactly like - `FITS`), and the self-calibration below replaces the guess after the first - load. Under-promise, self-correcting. -- **reserve too small** → predicts `FITS` where reality is `PARTIAL`. That is - precisely the llmfit failure this phase exists to delete: a confident badge - about a model that spills. +Two behaviours that matter to us fall out of that log: -So the bootstrap does not have to be right. It has to **not under-estimate** — -the same rule as **Boundaries**: *unknown shapes round up; never underestimate -memory*. +- **It never touched the context.** KV stayed at 4,480 MiB / 40,960 cells through + every step. Because `-c` was set explicitly, the fitter treats it as fixed and + spills layers instead, which is the same "explicitly set means hands off" rule + that `-ngl` follows. Good: our context floor survives contact with the fitter. +- **If we do not set `-c`, it will reduce context on its own, as far as + `fit_params_min_ctx = 4096`.** That is well below our 16K floor and it happens + silently. So `-c` is not optional for us, and not only because we want a + particular window: leaving it unset hands the floor to llama.cpp. Per-layer KV comes from the header; the formula is validated exactly in the **Appendix**. @@ -794,16 +877,43 @@ streaming. Install fetches the GGUF into `--models-dir` with real filenames, not llama.cpp's content-addressed HF cache layout (`LLAMA_CACHE` points at the app data dir so nothing writes to `~/.cache`). -> **ponytail: the mechanism below is unverified.** `POST /models/load` with -> `{"model": …, "args": ["-c", "16384"]}` was sent at `b11050` and the router -> **ignored the args** — the worker came up at the model's own default -> (`n_ctx_slot` 34304 / 28160 / 12288 for the 0.6B / 1.7B / 4B, not 16384). The -> same `-c 16384` passed directly to `llama-server` on the command line **is** -> honoured, so the flag works and the router's plumbing for it does not, at least -> not in this shape. Fixed-context-at-load is a Decisions-table entry and the -> reason the 16K floor exists, so **resolve this first in 7.3**: either the field -> name differs, the args need another shape, or context must be set per model at -> router startup. Do not build the context ladder on top of an unconfirmed call. +> **Resolved: `POST /models/load` cannot set per-model args. `--models-preset` +> can.** Measured on macOS at `b11050`. Every payload shape is accepted and none +> of them does anything: `{"args": ["-c","4096"]}`, `{"args": +> ["--ctx-size","4096","-fa","on"]}` and `{"preset": "..."}` all return +> `200 {"success":true}` while the worker's argv stays **byte identical**. The +> router reports that argv under `GET /models` as `status.args`, which is the +> only reliable way to see what it actually did. +> +> The mechanism is a separate flag, `--models-preset PATH`, pointing at an INI +> file with one section per model: +> +> ```ini +> [Qwen3-8B-Q4_K_M] +> model = /path/to/Qwen3-8B-Q4_K_M.gguf +> ctx-size = 16384 +> flash-attn = on +> cache-type-k = q8_0 +> cache-type-v = q8_0 +> parallel = 1 +> ``` +> +> Verified: the worker then launches with `--ctx-size 16384 --cache-type-k q8_0 +> --cache-type-v q8_0 --flash-attn on --parallel 1`, and reports the requested +> `n_ctx_slot` rather than its own default. **So every per-model decision in this +> phase is implementable**, including the KV precision row in **Decisions**, +> which rides the same mechanism as the context window. +> +> **The INI is read once at router startup.** Appending a section while the +> router runs does not surface the model, so installing a model or changing its +> load plan means rewriting the file and restarting the sidecar. That is the +> lifecycle `watchImageModel()` already implements for sd-server in +> `electron/src/main/index.ts`: poll a small endpoint, compare the arg set, then +> `stopNamed` plus `startOne` on a change. Reuse it rather than inventing a +> second supervisor pattern. +> +> Also measured and worth passing: llama-server defaults to `--parallel 4`, so +> the KV cache is sized for four concurrent slots this app never uses. **Context is fixed at load, not grown.** `POST /models/load {"args": ["-c", N]}` with `N` the largest window that keeps the verdict at **`FITS`** — not the @@ -1458,15 +1568,30 @@ pick = max(candidates, key=lambda ev: (ev[1].rank, -ev[1].size_bytes), default=None) ``` -> **`MAX_OFFLOAD` is provisional, and may not be needed at all.** On the one -> machine measured, *every* offload fraction stayed usable: even fully on the CPU -> this model decoded at 13 t/s, above reading pace, and prefill held at half -> device speed. No sensible threshold would have fired anywhere in the curated -> range. Before shipping a constant, check whether the gate ever triggers — a -> mechanism that never fires is worse than no mechanism, because it reads as -> protection that was never tested. The reason the `FITS`-only rule had to go is -> unchanged and does not depend on this number: it refused configurations that -> demonstrably work. +> **`MAX_OFFLOAD` fires, and it is load bearing.** An earlier draft suspected it +> might never trigger and should perhaps not ship. Run against the six profiles +> with the implemented estimator, the opposite holds: **without a ceiling, five +> of six machines star Qwen3 32B**, including a 6 GB RTX 3050 at `f = 0.82`. +> +> | machine | no ceiling | ceiling at 0.75 | `f` of the unceilinged pick | +> |---|---|---|---| +> | M2 8 GB | Qwen3 8B | Qwen3 8B | 0.38 | +> | RTX 3050 6 GB | **Qwen3 32B** | Qwen3 8B | **0.82** | +> | M4 16 GB | Qwen3 32B | Qwen3 32B | 0.57 | +> | RTX 4070 12 GB | Qwen3 32B | Qwen3 32B | 0.57 | +> | RTX 4090 24 GB | Qwen3 32B | Qwen3 32B | 0.05 | +> | M4 Max 64 GB | Qwen3 32B | Qwen3 32B | 0.00 | +> +> The earlier suspicion came from sweeping **one** model and finding every +> offload fraction usable. That is true, and it is the wrong population: the gate +> exists to stop the *largest* build being starred on a small card, and only the +> ladder exposes that. +> +> **0.75 is still too loose and should not ship as it stands.** It leaves the +> 4070 and the 16 GB Mac on a 32B at `f = 0.57`, which the roofline puts at about +> 37% of resident decode speed. Set the number from the roofline against a stated +> speed floor rather than picking a round one. The point here is only that the +> ceiling must exist. `MAX_OFFLOAD` is a v1 placeholder for the speed gate below, not a latency judgement. Set it generously: the failure it exists to prevent is starring a @@ -2171,6 +2296,57 @@ with 5,234 MiB free, from llama.cpp's own allocation log: 2,234 MiB, measured VRAM delta **2,234 MiB**. That is the evidence that a curated row can be priced offline from committed `shape` fields alone. +**Apple Silicon, measured on an M2 / 8 GB at `b11050`.** The platform the +reserve table listed as unmeasured. Qwen3 1.7B Q4_K_M, `llama-server -v`. + +| Measurement | Value | +|---|---| +| `ggml_backend_load_all()` | **2.3 ms** | +| first `ggml_backend_dev_count()` (compiles 20 Metal libraries) | **19.0 s** cold, 43 to 49 ms warm | +| `ggml_backend_dev_memory()` | 0.02 ms | +| devices listed | `MTL0` (GPU), `BLAS` (ACCEL, raw type 3), `CPU` | +| `recommendedMaxWorkingSetSize` | 5726.63 MB = **5461.3 MiB**, which is what ggml reports as MTL0 total | +| MTL0 total / free | 5461.3 / 5461.0 MiB — **not a live figure** | +| CPU device total / free | 8192.0 / 8192.0 MiB — physical RAM, twice, on a machine with ~2 GB free | +| fit margin the fitter names | **1024 MiB** | + +At `-c 16384`, `--parallel 1` would have been the right flag and was not passed; +llama-server defaults to **4 slots**, so pass `--parallel 1` or the KV cache is +sized for concurrency this app does not use. + +| buffer, `-c 16384` | MiB | +|---|---| +| `MTL0_Mapped` model | 1050.43 | +| `CPU_Mapped` model | 243.43 | +| `MTL0` KV | 1792.00 | +| `MTL0` compute | 102.24 | +| `CPU` compute | 24.01 | +| `CPU` output | 2.32 | +| **fitter's own projection** | **2944** | + +`1050 + 1792 + 102 = 2944`, against the fitter's `projected to use 2944 MiB`. +**The three-term `need` is exact on Metal**, as it was on Vulkan. + +**The KV formula is exact on Metal too.** `2 × 28 layers × 8 kv_heads × 128 × +2 bytes × 16384` = **1792 MiB**, reported as `1792.00`. At 40,960 cells it is +`4480.00`, again exact. Compute buffers scale with context as stated: 102.24 MiB +at 16K, 222.24 MiB at 40K. + +**243.43 MiB stayed on the CPU at "offloaded 29/29 layers to GPU".** The +non-repeating tensors are host-side even at full offload, so *29/29 offloaded* +does not mean *all weights on the device*, and an `offload_fraction` derived from +the layer count alone will read 0.00 when 19% of the file is elsewhere. + +**Metal reports `0.00 MiB` buffer sizes during the fitter's probe pass**, before +the real load. Anything scraping buffer sizes must ignore the first pass, and +`use shared buffers = true` means Metal allocations are not the separate arenas +Vulkan and CUDA report. + +**`--fit` searches rather than solves.** At `-c 40960`, needing 5752 MiB against +5460 free with a 1024 margin: `29/29 → 0/29 → 29/29 → 22/29 → 23/29`, reloading +at each step, settling at 23/29 (`f ≈ 0.21`). It never reduced the context, +because `-c` was explicit. + **Router lifecycle.** Router up with three models discovered: VRAM unchanged from idle. Worker spawn: `--port 0 --model `. Idle eviction ~30 s without `--sleep-idle-seconds`. Grandchild reaping via `taskkill /T /F`: four processes, diff --git a/surfsense_local/README.md b/surfsense_local/README.md index c4a4487adf..6c2dc7e13b 100644 --- a/surfsense_local/README.md +++ b/surfsense_local/README.md @@ -16,7 +16,7 @@ Local-first desktop app for research over your own documents. Runs fully offline |---|---| | Node.js | 22.12+ (Electron 44 engine floor) | | Python | 3.12+ | -| Local LLM | [Ollama](https://ollama.com) or `llama.cpp` on `127.0.0.1` | +| Local LLM | `llama.cpp` (`llama-server`, router mode) on `127.0.0.1` | ## Development @@ -68,7 +68,7 @@ Electron ─┬─> FastAPI (127.0.0.1) ──> surfsense.db ├─> Huey worker ingest (-w 1) ──> surfsense.db, huey.db │ └─> Docling, embeddings └─> Huey worker studio (-w 4) ──> surfsense.db, huey.db -Vite SPA ───> FastAPI Ollama / llama.cpp +Vite SPA ───> FastAPI llama.cpp ``` | Path | Contents | From accad5b1b552fec85db7d7e08ba204ee89831208 Mon Sep 17 00:00:00 2001 From: Anish Sarkar <104695310+AnishSarkar22@users.noreply.github.com> Date: Sun, 20 Sep 2026 20:11:22 +0530 Subject: [PATCH 29/60] refactor(llm): remove the Ollama provider and the llmfit recommendation module --- surfsense_local/backend/modules/llm/providers/ollama/__init__.py | 0 1 file changed, 0 insertions(+), 0 deletions(-) delete mode 100644 surfsense_local/backend/modules/llm/providers/ollama/__init__.py diff --git a/surfsense_local/backend/modules/llm/providers/ollama/__init__.py b/surfsense_local/backend/modules/llm/providers/ollama/__init__.py deleted file mode 100644 index e69de29bb2..0000000000 From e74d6162442dd661ff3c9558b05102cd13267225 Mon Sep 17 00:00:00 2001 From: Anish Sarkar <104695310+AnishSarkar22@users.noreply.github.com> Date: Mon, 21 Sep 2026 16:58:25 +0530 Subject: [PATCH 30/60] build(backend): pin llama.cpp's own gguf parser as a dependency Co-Authored-By: Claude Opus 5 (1M context) --- surfsense_local/backend/pyproject.toml | 4 ++++ surfsense_local/backend/uv.lock | 17 +++++++++++++++++ 2 files changed, 21 insertions(+) diff --git a/surfsense_local/backend/pyproject.toml b/surfsense_local/backend/pyproject.toml index 4e3b95da39..570555b94b 100644 --- a/surfsense_local/backend/pyproject.toml +++ b/surfsense_local/backend/pyproject.toml @@ -9,6 +9,10 @@ dependencies = [ "cryptography>=43", "docling>=2.125.0", "fastapi>=0.115.8", + # llama.cpp's own GGUF parser. Pinned exactly, not ranged: the prefix reader + # in modules/llm/gguf/header_prefix.py overrides two private methods, so a + # version bump is a deliberate act with a test that re-checks them. + "gguf==0.19.0", "httpx>=0.28.1", "huey>=3.3.4", # The offline podcast voice (Studio). Rides the bundled onnxruntime; its ~340MB diff --git a/surfsense_local/backend/uv.lock b/surfsense_local/backend/uv.lock index 24fb301f2c..71547d0932 100644 --- a/surfsense_local/backend/uv.lock +++ b/surfsense_local/backend/uv.lock @@ -746,6 +746,21 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/fd/3c/6a2bf344106328fd04963664a60b9bb6496fc25df8e962fcdc1367285fb9/fsspec-2026.7.0-py3-none-any.whl", hash = "sha256:b57ddbafedfaef7018c1ecab32aa200a9d7ca26b77965f64e48b70061249d279", size = 206583, upload-time = "2026-07-28T16:34:49.538Z" }, ] +[[package]] +name = "gguf" +version = "0.19.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "numpy" }, + { name = "pyyaml" }, + { name = "requests" }, + { name = "tqdm" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/48/ae/17f1308ae45cd7b08ebb521747d5b23f4efc4d172038a4e228dd5106c3ff/gguf-0.19.0.tar.gz", hash = "sha256:dbadcd6cc7ccd44256f2229fe7c2dff5e8aa5cf0612ab987fd2b1a57e428923f", size = 111220, upload-time = "2026-05-06T13:04:03.667Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/b3/bb/d71d6da82763528c2c2ed6b59a9d6142c6595545a4c448e2085d155e88c2/gguf-0.19.0-py3-none-any.whl", hash = "sha256:70bcd10edfe697fb2dad6e40af2234b9d8ece9a41a99761405121ebda1c3c1cd", size = 118475, upload-time = "2026-05-06T13:04:02.588Z" }, +] + [[package]] name = "greenlet" version = "3.5.5" @@ -2686,6 +2701,7 @@ dependencies = [ { name = "cryptography" }, { name = "docling" }, { name = "fastapi" }, + { name = "gguf" }, { name = "httpx" }, { name = "huey" }, { name = "kokoro-onnx" }, @@ -2720,6 +2736,7 @@ requires-dist = [ { name = "cryptography", specifier = ">=43" }, { name = "docling", specifier = ">=2.125.0" }, { name = "fastapi", specifier = ">=0.115.8" }, + { name = "gguf", specifier = "==0.19.0" }, { name = "httpx", specifier = ">=0.28.1" }, { name = "huey", specifier = ">=3.3.4" }, { name = "kokoro-onnx", specifier = ">=0.4.9" }, From 2e5670eff7c3817c320fa2fc4b833c274005633b Mon Sep 17 00:00:00 2001 From: Anish Sarkar <104695310+AnishSarkar22@users.noreply.github.com> Date: Mon, 21 Sep 2026 16:58:36 +0530 Subject: [PATCH 31/60] feat(fit): widen ModelShape with the fields an exact estimate needs --- .../backend/modules/llm/fit/cache_types.py | 27 +++++++++++ .../backend/modules/llm/fit/types.py | 47 +++++++++++++++++-- 2 files changed, 70 insertions(+), 4 deletions(-) create mode 100644 surfsense_local/backend/modules/llm/fit/cache_types.py diff --git a/surfsense_local/backend/modules/llm/fit/cache_types.py b/surfsense_local/backend/modules/llm/fit/cache_types.py new file mode 100644 index 0000000000..862a7d4997 --- /dev/null +++ b/surfsense_local/backend/modules/llm/fit/cache_types.py @@ -0,0 +1,27 @@ +"""Bytes one cache element costs, per storage type. + +A quantized cache stores a block of weights plus a scale, so the cost per +element is not the bit width divided by eight: `q8_0` is one byte per weight +plus one fp16 scale for every thirty two of them. + +The preset only ever writes `f16` or `q8_0`, which are the two the load plan +chooses between. The rest are here because a searched model's own header can +name any of them, and one table is better than a second one written later +somewhere else. +""" + +from modules.llm.fit.types import KvPrecision + +# Block layouts from ggml-common.h. Each entry is bytes per element: the block's +# total size divided by the number of weights it holds. +BYTES_PER_ELEMENT: dict[KvPrecision, float] = { + KvPrecision.F32: 4.0, + KvPrecision.F16: 2.0, + KvPrecision.BF16: 2.0, + KvPrecision.Q8_0: 34 / 32, # 32 int8 weights plus one fp16 scale + KvPrecision.Q5_1: 24 / 32, # 5 bits plus an fp16 scale and an fp16 min + KvPrecision.Q5_0: 22 / 32, # 5 bits plus an fp16 scale + KvPrecision.Q4_1: 20 / 32, # 4 bits plus an fp16 scale and an fp16 min + KvPrecision.Q4_0: 18 / 32, # 4 bits plus an fp16 scale + KvPrecision.IQ4_NL: 18 / 32, # same footprint, non linear codebook +} diff --git a/surfsense_local/backend/modules/llm/fit/types.py b/surfsense_local/backend/modules/llm/fit/types.py index 1096f6b233..23ebee0340 100644 --- a/surfsense_local/backend/modules/llm/fit/types.py +++ b/surfsense_local/backend/modules/llm/fit/types.py @@ -5,16 +5,30 @@ class KvPrecision(StrEnum): - """How the conversation cache is stored. Lossless either way; f16 costs twice - the memory of q8_0 and needs no flash-attention kernel to be fast.""" + """How the conversation cache is stored. + The load plan chooses between `F16` and `Q8_0`, and the preset writes only + those two: lossless either way, but a quantized cache needs a working flash + attention kernel and f16 does not. The others exist because a searched + model's header can name any of them and they have to be priceable. + """ + + F32 = "f32" F16 = "f16" + BF16 = "bf16" Q8_0 = "q8_0" + Q5_1 = "q5_1" + Q5_0 = "q5_0" + Q4_1 = "q4_1" + Q4_0 = "q4_0" + IQ4_NL = "iq4_nl" @property def bytes_per_element(self) -> float: - # q8_0 is one byte per weight plus a shared scale per 32-element block. - return 2.0 if self is KvPrecision.F16 else 1.0 + 2.0 / 32.0 + """Cost per cached element, from the block layout table.""" + from modules.llm.fit.cache_types import BYTES_PER_ELEMENT + + return BYTES_PER_ELEMENT[self] @dataclass(frozen=True) @@ -35,3 +49,28 @@ class ModelShape: n_vocab: int sliding_window: int = 0 expert_count: int = 0 + + # Everything below is optional so an older header, or a synthetic shape in a + # test, still constructs. Each one only ever makes an estimate sharper, and + # its absence is the conservative answer rather than a zero cost. + + # Graph scratch scales with the widest thing a layer computes, not with the + # model's parameter count. + embedding_length: int = 0 + feed_forward_length: int = 0 + expert_feed_forward_length: int = 0 + expert_shared_feed_forward_length: int = 0 + expert_used_count: int = 0 + + # Which layers hold the whole window. The header states this directly on + # newer files, as a period or as one flag per layer, and llama.cpp reads it + # before falling back to a per architecture default. + sliding_window_pattern: int = 0 + sliding_window_layers: tuple[bool, ...] = () + # Layers that reuse an earlier layer's cache and so allocate none of their own. + shared_kv_layers: int = 0 + + # Multi-head latent attention: one compressed entry per token per layer + # instead of a key and a value per head. + kv_lora_rank: int = 0 + key_length_mla: int = 0 From 0676f65b92452967ca4341509262dc8114b2c57c Mon Sep 17 00:00:00 2001 From: Anish Sarkar <104695310+AnishSarkar22@users.noreply.github.com> Date: Mon, 21 Sep 2026 16:58:44 +0530 Subject: [PATCH 32/60] refactor(gguf): read headers through llama.cpp's own key table --- .../backend/modules/llm/gguf/__init__.py | 19 ++- .../backend/modules/llm/gguf/header_prefix.py | 151 ++++++++++++++++++ .../backend/modules/llm/gguf/reader.py | 107 ------------- .../backend/modules/llm/gguf/shape.py | 66 ++++++-- .../backend/modules/llm/gguf/source.py | 48 ++++-- .../tests/unit/llm/gguf/test_header.py | 79 ++++++++- .../tests/unit/llm/gguf/test_header_prefix.py | 90 +++++++++++ 7 files changed, 419 insertions(+), 141 deletions(-) create mode 100644 surfsense_local/backend/modules/llm/gguf/header_prefix.py delete mode 100644 surfsense_local/backend/modules/llm/gguf/reader.py create mode 100644 surfsense_local/backend/tests/unit/llm/gguf/test_header_prefix.py diff --git a/surfsense_local/backend/modules/llm/gguf/__init__.py b/surfsense_local/backend/modules/llm/gguf/__init__.py index 35ef89d8ff..0b51bdbc42 100644 --- a/surfsense_local/backend/modules/llm/gguf/__init__.py +++ b/surfsense_local/backend/modules/llm/gguf/__init__.py @@ -1,12 +1,25 @@ -from modules.llm.gguf.reader import GgufHeader, TruncatedHeaderError, read_gguf_header +from modules.llm.gguf.header_prefix import ( + GgufHeader, + HeaderTensor, + TruncatedHeaderError, + read_header_prefix, +) from modules.llm.gguf.shape import read_header, to_shape -from modules.llm.gguf.source import shape_from_file, shape_from_url +from modules.llm.gguf.source import ( + header_from_file, + header_from_url, + shape_from_file, + shape_from_url, +) __all__ = [ "GgufHeader", + "HeaderTensor", "TruncatedHeaderError", - "read_gguf_header", + "header_from_file", + "header_from_url", "read_header", + "read_header_prefix", "shape_from_file", "shape_from_url", "to_shape", diff --git a/surfsense_local/backend/modules/llm/gguf/header_prefix.py b/surfsense_local/backend/modules/llm/gguf/header_prefix.py new file mode 100644 index 0000000000..1c23b5fb3f --- /dev/null +++ b/surfsense_local/backend/modules/llm/gguf/header_prefix.py @@ -0,0 +1,151 @@ +"""Read a GGUF header from a byte prefix, never the weights. + +llama.cpp's own parser is the one that gets hardened against malformed files, +and the search tier parses bytes from arbitrary repositories, so this wraps +`gguf.GGUFReader` rather than carrying a parser of our own. Two things stand in +the way, and this module is exactly the adapter around them: + +- `GGUFReader` memory maps a **path**, so a prefix held in memory is written to + a temporary file first. +- Its constructor always walks the tensor table and slices each tensor's data at + an offset inside the weights, which a prefix does not have. numpy shortens a + slice that runs past the end rather than raising, so the shortfall would + surface later as a confident wrong answer instead of a retry. + +Two private methods are therefore overridden: `_get` bounds checks every read, +and `_build_tensors` keeps names, dimensions and types without touching data. +Both are private upstream, which is why `gguf` is pinned to an exact version and +`test_header_prefix.py` asserts that version beside the two behaviours. +""" + +import os +import tempfile +from dataclasses import dataclass +from typing import Any, Literal, NamedTuple + +import numpy as np +import numpy.typing as npt +from gguf.gguf_reader import GGUFReader, ReaderField + +# The pseudo fields the reader synthesises for the file's own counts. They are +# not model metadata and no caller looks them up by these names. +_SYNTHETIC_PREFIX = "GGUF." + + +class TruncatedHeaderError(ValueError): + """The metadata block runs past the bytes we were given. + + A `ValueError`, because that is what it is: bytes that do not parse. Callers + that can widen the read catch it by name and retry; callers that merely need + to skip an unusable file catch `ValueError` and get both this and a file + that was never a GGUF, which is the behaviour they want. + + Distinct from a malformed file: the caller's remedy is a wider read, not a + rejection. Vocabulary lists make header size scale with vocabulary rather + than with model size, so a budget that suits a 135M model fails on a 30B one. + """ + + +class HeaderTensor(NamedTuple): + """One tensor-table row, without its weights. + + `ggml_type` is kept as the raw integer rather than the enum: an id this + build of the package does not know is a tensor we cannot size, not a header + we should refuse. + """ + + name: str + dims: tuple[int, ...] + ggml_type: int + + +@dataclass(frozen=True) +class GgufHeader: + metadata: dict[str, Any] + tensors: tuple[HeaderTensor, ...] + + +class HeaderPrefixReader(GGUFReader): + """`GGUFReader` over bytes that stop before the weights.""" + + header_tensors: tuple[HeaderTensor, ...] + + def _get( + self, + offset: int, + dtype: npt.DTypeLike, + count: int = 1, + override_order: Literal["I", "S", "<"] | None = None, + ): + """Every read, bounds checked. + + The base class slices a memory map, and numpy answers a slice past the + end with a shorter array rather than an error. Left alone, a prefix that + stopped mid header would parse into a confident wrong shape. + """ + itemsize = int(np.empty([], dtype=dtype).itemsize) + start = int(offset) + end = start + itemsize * int(count) + if start < 0 or end > len(self.data): + raise TruncatedHeaderError( + f"header needs {end} bytes, the prefix holds {len(self.data)}" + ) + return super()._get(offset, dtype, count, override_order) + + def _build_tensors(self, start_offs: int, fields: list[ReaderField]) -> None: + """Names, dimensions and types. Never the data. + + The base class slices each tensor at its offset in the weights, which a + prefix does not carry. Everything the estimator and the authoring script + want lives in the table itself. + """ + tensors: list[HeaderTensor] = [] + for field in fields: + _name_len, name_data, _n_dims, dims, raw_dtype, _offset = field.parts + tensors.append( + HeaderTensor( + name=str(bytes(name_data), encoding="utf-8"), + dims=tuple(int(d) for d in dims.tolist()), + ggml_type=int(raw_dtype[0]), + ) + ) + self.header_tensors = tuple(tensors) + self.tensors = [] + + def header(self) -> GgufHeader: + """The parsed header, with the reader's own bookkeeping fields dropped.""" + metadata = { + name: field.contents() + for name, field in self.fields.items() + if not name.startswith(_SYNTHETIC_PREFIX) + } + return GgufHeader(metadata=metadata, tensors=self.header_tensors) + + def close(self) -> None: + """Release the memory map so the temporary file can be removed. + + Windows refuses to unlink a mapped file, and this runs once per searched + model, so the map is closed explicitly rather than left to the collector. + """ + mapping = getattr(self.data, "_mmap", None) + if mapping is not None: + mapping.close() + + +def read_header_prefix(prefix: bytes) -> GgufHeader: + """Metadata and tensor table from the front of a GGUF file. + + Raises `TruncatedHeaderError` when the prefix stops inside the header, and + `ValueError` when it was never a GGUF at all. + """ + handle, name = tempfile.mkstemp(suffix=".gguf-header") + try: + with os.fdopen(handle, "wb") as file: + file.write(prefix) + reader = HeaderPrefixReader(name) + try: + return reader.header() + finally: + reader.close() + finally: + os.unlink(name) diff --git a/surfsense_local/backend/modules/llm/gguf/reader.py b/surfsense_local/backend/modules/llm/gguf/reader.py deleted file mode 100644 index 79990f991c..0000000000 --- a/surfsense_local/backend/modules/llm/gguf/reader.py +++ /dev/null @@ -1,107 +0,0 @@ -"""Parse a GGUF metadata block. Standard library only. - -The header is the only thing a fit estimate needs, and it sits at the front of -the file, so it can be read over an HTTP range without downloading weights. -""" - -import struct -from dataclasses import dataclass -from typing import Any - -MAGIC = b"GGUF" - -_UINT8, _INT8, _UINT16, _INT16, _UINT32, _INT32 = 0, 1, 2, 3, 4, 5 -_FLOAT32, _BOOL, _STRING, _ARRAY, _UINT64, _INT64, _FLOAT64 = 6, 7, 8, 9, 10, 11, 12 - -_SCALARS: dict[int, tuple[str, int]] = { - _UINT8: (" None: - self._data = data - self._at = 0 - - def take(self, n: int) -> bytes: - end = self._at + n - if end > len(self._data): - raise TruncatedHeaderError( - f"header needs {end} bytes, got {len(self._data)}" - ) - chunk = self._data[self._at : end] - self._at = end - return chunk - - def scalar(self, kind: int) -> Any: - fmt, size = _SCALARS[kind] - return struct.unpack(fmt, self.take(size))[0] - - def string(self) -> str: - return self.take(self.scalar(_UINT64)).decode("utf-8", errors="replace") - - def value(self, kind: int) -> Any: - if kind == _STRING: - return self.string() - if kind == _ARRAY: - elem = self.scalar(_UINT32) - count = self.scalar(_UINT64) - return [self.value(elem) for _ in range(count)] - if kind in _SCALARS: - return self.scalar(kind) - raise ValueError(f"unknown GGUF value type {kind}") - - -def read_gguf_header(data: bytes) -> GgufHeader: - """Metadata and tensor table, or TruncatedHeaderError if `data` is short.""" - cursor = _Cursor(data) - if cursor.take(4) != MAGIC: - raise ValueError("not a GGUF file") - cursor.scalar(_UINT32) # format version - tensor_count = cursor.scalar(_UINT64) - kv_count = cursor.scalar(_UINT64) - - metadata: dict[str, Any] = {} - for _ in range(kv_count): - key = cursor.string() - metadata[key] = cursor.value(cursor.scalar(_UINT32)) - - tensors: list[tuple[str, tuple[int, ...]]] = [] - for _ in range(tensor_count): - name = cursor.string() - dims = tuple(cursor.scalar(_UINT64) for _ in range(cursor.scalar(_UINT32))) - cursor.scalar(_UINT32) # element type - cursor.scalar(_UINT64) # offset - tensors.append((name, dims)) - - return GgufHeader(metadata=metadata, tensors=tuple(tensors)) diff --git a/surfsense_local/backend/modules/llm/gguf/shape.py b/surfsense_local/backend/modules/llm/gguf/shape.py index e2cc040dc3..9aec1b7a67 100644 --- a/surfsense_local/backend/modules/llm/gguf/shape.py +++ b/surfsense_local/backend/modules/llm/gguf/shape.py @@ -2,12 +2,20 @@ Architecture-scoped keys are named after the architecture itself, so every lookup goes through the `general.architecture` value rather than a fixed prefix. +The key names come from llama.cpp's own `Keys` table rather than from string +literals here, so a rename upstream is an import error rather than a field that +silently reads zero. """ from typing import Any +from gguf.constants import Keys + from modules.llm.fit import ModelShape -from modules.llm.gguf.reader import GgufHeader, read_gguf_header +from modules.llm.gguf.header_prefix import GgufHeader, read_header_prefix + +_LLM = Keys.LLM +_ATTENTION = Keys.Attention def _widest(value: Any, default: int = 0) -> int: @@ -25,33 +33,59 @@ def _widest(value: Any, default: int = 0) -> int: def to_shape(header: GgufHeader) -> ModelShape: meta = header.metadata - arch = str(meta.get("general.architecture", "unknown")) + arch = str(meta.get(Keys.General.ARCHITECTURE, "unknown")) - def field(name: str, default: int = 0) -> int: - return _widest(meta.get(f"{arch}.{name}"), default) + def field(key: str, default: int = 0) -> int: + return _widest(meta.get(key.format(arch=arch)), default) - key_length = field("attention.key_length") - value_length = field("attention.value_length") - embedding = field("embedding_length") - head_count = field("attention.head_count") + key_length = field(_ATTENTION.KEY_LENGTH) + value_length = field(_ATTENTION.VALUE_LENGTH) + embedding = field(_LLM.EMBEDDING_LENGTH) + head_count = field(_ATTENTION.HEAD_COUNT) # Older headers omit the per-head lengths and imply them from the embedding. if not key_length and embedding and head_count: key_length = value_length = embedding // head_count - tokens = meta.get("tokenizer.ggml.tokens") + period, layers = _sliding_pattern(meta.get(_ATTENTION.SLIDING_WINDOW_PATTERN.format(arch=arch))) + + tokens = meta.get(Keys.Tokenizer.LIST) return ModelShape( architecture=arch, - block_count=field("block_count"), - head_count_kv=field("attention.head_count_kv", 1), + block_count=field(_LLM.BLOCK_COUNT), + head_count_kv=field(_ATTENTION.HEAD_COUNT_KV, 1), key_length=key_length, value_length=value_length, - context_length=field("context_length"), - n_vocab=len(tokens) if isinstance(tokens, list) else field("vocab_size"), - sliding_window=field("attention.sliding_window"), - expert_count=field("expert_count"), + context_length=field(_LLM.CONTEXT_LENGTH), + n_vocab=len(tokens) if isinstance(tokens, list) else field(_LLM.VOCAB_SIZE), + sliding_window=field(_ATTENTION.SLIDING_WINDOW), + expert_count=field(_LLM.EXPERT_COUNT), + embedding_length=embedding, + feed_forward_length=field(_LLM.FEED_FORWARD_LENGTH), + expert_feed_forward_length=field(_LLM.EXPERT_FEED_FORWARD_LENGTH), + expert_shared_feed_forward_length=field(_LLM.EXPERT_SHARED_FEED_FORWARD_LENGTH), + expert_used_count=field(_LLM.EXPERT_USED_COUNT), + sliding_window_pattern=period, + sliding_window_layers=layers, + shared_kv_layers=field(_ATTENTION.SHARED_KV_LAYERS), + kv_lora_rank=field(_ATTENTION.KV_LORA_RANK), + key_length_mla=field(_ATTENTION.KEY_LENGTH_MLA), ) +def _sliding_pattern(value: Any) -> tuple[int, tuple[bool, ...]]: + """The window pattern in whichever of its two forms the header used. + + llama.cpp accepts both: one integer meaning "every nth layer sees the whole + window", or one flag per layer. A file that states either is the authority, + so the caller prefers this over any table. + """ + if isinstance(value, list): + return 0, tuple(bool(v) for v in value) + if value is None: + return 0, () + return int(value), () + + def read_header(data: bytes) -> ModelShape: """Bytes from the front of a GGUF file to the shape a fit estimate needs.""" - return to_shape(read_gguf_header(data)) + return to_shape(read_header_prefix(data)) diff --git a/surfsense_local/backend/modules/llm/gguf/source.py b/surfsense_local/backend/modules/llm/gguf/source.py index b4be310fb4..906198c9f6 100644 --- a/surfsense_local/backend/modules/llm/gguf/source.py +++ b/surfsense_local/backend/modules/llm/gguf/source.py @@ -3,6 +3,10 @@ Hugging Face serves `Range` requests on model files, so a model can be priced before a single weight is downloaded. That is what puts a real fit badge on a search result. + +The widening is the whole point of this module. Header size scales with +vocabulary rather than with model size, so no single prefix suits every model, +and a short read is a retry rather than a refusal. """ from pathlib import Path @@ -10,8 +14,12 @@ import httpx from modules.llm.fit import ModelShape -from modules.llm.gguf.reader import TruncatedHeaderError -from modules.llm.gguf.shape import read_header +from modules.llm.gguf.header_prefix import ( + GgufHeader, + TruncatedHeaderError, + read_header_prefix, +) +from modules.llm.gguf.shape import to_shape # A 135M model's metadata ends at 1.77 MB, but Qwen3-Coder-30B-A3B's runs to # 5.94 MB with 579 tensor entries after it. Header size scales with vocabulary, @@ -20,12 +28,26 @@ WIDENED_BYTES = 24 * 1024 * 1024 -def shape_from_file(path: Path) -> ModelShape: +def header_from_file(path: Path) -> GgufHeader: + """The whole header, for callers that need the tensor table. + + The authoring script's expert accounting reads tensors, which the shape + deliberately does not carry. + """ with path.open("rb") as handle: - return _read_widening(lambda n: _read_prefix(handle, n)) + def fetch(size: int) -> bytes: + handle.seek(0) + return handle.read(size) -async def shape_from_url(client: httpx.AsyncClient, url: str) -> ModelShape: + return _widening(fetch) + + +def shape_from_file(path: Path) -> ModelShape: + return to_shape(header_from_file(path)) + + +async def header_from_url(client: httpx.AsyncClient, url: str) -> GgufHeader: """Read only the header of a remote GGUF, over HTTP range requests.""" async def fetch(size: int) -> bytes: @@ -33,20 +55,18 @@ async def fetch(size: int) -> bytes: reply.raise_for_status() return reply.content - data = await fetch(INITIAL_BYTES) try: - return read_header(data) + return read_header_prefix(await fetch(INITIAL_BYTES)) except TruncatedHeaderError: - return read_header(await fetch(WIDENED_BYTES)) + return read_header_prefix(await fetch(WIDENED_BYTES)) -def _read_prefix(handle, size: int) -> bytes: - handle.seek(0) - return handle.read(size) +async def shape_from_url(client: httpx.AsyncClient, url: str) -> ModelShape: + return to_shape(await header_from_url(client, url)) -def _read_widening(fetch) -> ModelShape: +def _widening(fetch) -> GgufHeader: try: - return read_header(fetch(INITIAL_BYTES)) + return read_header_prefix(fetch(INITIAL_BYTES)) except TruncatedHeaderError: - return read_header(fetch(WIDENED_BYTES)) + return read_header_prefix(fetch(WIDENED_BYTES)) diff --git a/surfsense_local/backend/tests/unit/llm/gguf/test_header.py b/surfsense_local/backend/tests/unit/llm/gguf/test_header.py index 8f254e0d3c..c38eb3b542 100644 --- a/surfsense_local/backend/tests/unit/llm/gguf/test_header.py +++ b/surfsense_local/backend/tests/unit/llm/gguf/test_header.py @@ -3,7 +3,7 @@ import pytest from modules.llm.gguf import TruncatedHeaderError, read_header -from tests.unit.llm.gguf.build import FLOAT32, STRING, UINT32, array, gguf, kv +from tests.unit.llm.gguf.build import BOOL, FLOAT32, STRING, UINT32, array, gguf, kv pytestmark = pytest.mark.unit @@ -73,6 +73,83 @@ def test_a_mixture_of_experts_reports_its_expert_counts() -> None: assert shape.expert_count == 128 +def test_the_widths_a_compute_buffer_is_sized_from_are_read() -> None: + """Graph scratch scales with the widest thing a layer computes. Without + these the estimate is one constant for every model, which under-states a + large one by enough to flip a badge on a tight card.""" + entries = [ + *qwen3_entries(), + kv("qwen3.embedding_length", UINT32, 2048), + kv("qwen3.feed_forward_length", UINT32, 6144), + ] + shape = read_header(gguf(entries)) + + assert shape.embedding_length == 2048 + assert shape.feed_forward_length == 6144 + + +def test_a_window_pattern_stated_as_a_period_is_read() -> None: + """llama.cpp reads this key before falling back to its own table, so a file + that states its pattern is the authority on that model.""" + entries = [ + *qwen3_entries(), + kv("qwen3.attention.sliding_window", UINT32, 1024), + kv("qwen3.attention.sliding_window_pattern", UINT32, 6), + ] + shape = read_header(gguf(entries)) + + assert shape.sliding_window_pattern == 6 + assert shape.sliding_window_layers == () + + +def test_a_window_pattern_stated_per_layer_is_read_as_flags() -> None: + """The same key carries either form. An array is the exact answer and needs + no period arithmetic at all.""" + entries = [ + *qwen3_entries(), + kv("qwen3.attention.sliding_window", UINT32, 1024), + array("qwen3.attention.sliding_window_pattern", BOOL, [True, True, False]), + ] + shape = read_header(gguf(entries)) + + assert shape.sliding_window_layers == (True, True, False) + assert shape.sliding_window_pattern == 0 + + +def test_layers_that_share_a_cache_are_counted() -> None: + """Gemma 3n reuses an earlier layer's cache on its last blocks, so charging + those layers a cache of their own over-states the window's cost.""" + entries = [*qwen3_entries(), kv("qwen3.attention.shared_kv_layers", UINT32, 10)] + + assert read_header(gguf(entries)).shared_kv_layers == 10 + + +def test_a_latent_attention_model_reports_its_compressed_widths() -> None: + """DeepSeek-class models cache one compressed entry per token per layer, not + a key and a value per head. Priced by the usual formula they read far too + large.""" + entries = [ + *qwen3_entries(), + kv("qwen3.attention.kv_lora_rank", UINT32, 512), + kv("qwen3.attention.key_length_mla", UINT32, 64), + ] + shape = read_header(gguf(entries)) + + assert shape.kv_lora_rank == 512 + assert shape.key_length_mla == 64 + + +def test_an_older_header_without_the_new_fields_still_reads() -> None: + """Every field added for a sharper estimate is optional, and its absence + prices the conservative way rather than zero.""" + shape = read_header(gguf(qwen3_entries())) + + assert shape.embedding_length == 0 + assert shape.feed_forward_length == 0 + assert shape.kv_lora_rank == 0 + assert shape.block_count == 28 + + def test_unknown_value_types_do_not_stop_the_parse() -> None: """Uploaders put arbitrary keys in headers, and one we cannot read is not a reason to refuse a model we can otherwise price.""" diff --git a/surfsense_local/backend/tests/unit/llm/gguf/test_header_prefix.py b/surfsense_local/backend/tests/unit/llm/gguf/test_header_prefix.py new file mode 100644 index 0000000000..2cb88b139d --- /dev/null +++ b/surfsense_local/backend/tests/unit/llm/gguf/test_header_prefix.py @@ -0,0 +1,90 @@ +"""The adapter that lets llama.cpp's own parser read a header with no weights. + +Two private methods of `GGUFReader` are overridden, so this file is the contract +that says what those overrides must do. It also pins the package version: the +methods are private upstream and a bump is only safe once these still pass. +""" + +from importlib.metadata import version + +import pytest + +from modules.llm.gguf import HeaderTensor, TruncatedHeaderError, read_header_prefix +from tests.unit.llm.gguf.build import STRING, UINT32, array, gguf, kv, tensor + +pytestmark = pytest.mark.unit + +# F32 and Q4_K as ggml numbers them, so the table can be checked without the enum. +F32, Q4_K = 0, 12 + + +def entries() -> list[bytes]: + """The smallest header that is still a valid one.""" + return [ + kv("general.architecture", STRING, "qwen3"), + kv("qwen3.block_count", UINT32, 28), + array("tokenizer.ggml.tokens", STRING, ["a", "b", "c"]), + ] + + +def test_the_pinned_version_is_the_one_these_overrides_were_written_against() -> None: + """`_get` and `_build_tensors` are private upstream. + + Failing here is the signal to re-read them before taking the new version, + not a reason to widen the pin. + """ + assert version("gguf") == "0.19.0" + + +def test_a_header_without_its_weights_parses() -> None: + """The whole point: a prefix stops where the tensor data would start, and + the base class would slice into bytes that were never downloaded.""" + header = read_header_prefix(gguf(entries(), [tensor("blk.0.attn_q.weight", [64, 64])])) + + assert header.metadata["general.architecture"] == "qwen3" + assert header.metadata["qwen3.block_count"] == 28 + + +def test_the_tensor_table_keeps_names_dimensions_and_types() -> None: + """Sizing an expert or finding a projector reads the table, never the data.""" + tensors = [ + tensor("blk.0.attn_q.weight", [64, 128], dtype=F32), + tensor("blk.0.ffn_up_exps.weight", [128, 256, 8], dtype=Q4_K), + ] + + header = read_header_prefix(gguf(entries(), tensors)) + + assert header.tensors == ( + HeaderTensor("blk.0.attn_q.weight", (64, 128), F32), + HeaderTensor("blk.0.ffn_up_exps.weight", (128, 256, 8), Q4_K), + ) + + +def test_a_prefix_that_stops_inside_the_header_says_so() -> None: + """numpy shortens a slice past the end instead of raising, so without the + bounds check this would parse into a confident wrong shape.""" + whole = gguf(entries()) + + with pytest.raises(TruncatedHeaderError): + read_header_prefix(whole[: len(whole) // 2]) + + +def test_a_prefix_that_stops_inside_the_tensor_table_says_so() -> None: + """The table sits after the metadata, so a read can be long enough for one + and short for the other. Both are a wider read, not a rejection.""" + whole = gguf(entries(), [tensor("blk.0.attn_q.weight", [64, 64])]) + + with pytest.raises(TruncatedHeaderError): + read_header_prefix(whole[:-8]) + + +def test_the_reader_leaves_no_temporary_file_behind(tmp_path, monkeypatch) -> None: + """It writes the prefix to disk to hand the parser a path, once per searched + model, so a leak here would be one file per row the user scrolls past.""" + monkeypatch.setattr("tempfile.tempdir", str(tmp_path)) + + read_header_prefix(gguf(entries())) + with pytest.raises(TruncatedHeaderError): + read_header_prefix(gguf(entries())[:20]) + + assert list(tmp_path.iterdir()) == [] From 5214a26ed797336f95fd84f26faef106baac2bf7 Mon Sep 17 00:00:00 2001 From: Anish Sarkar <104695310+AnishSarkar22@users.noreply.github.com> Date: Mon, 21 Sep 2026 16:58:49 +0530 Subject: [PATCH 33/60] feat(fit): price sliding-window and latent KV caches per layer --- .../backend/modules/llm/fit/kv_cache.py | 67 +++++++++---- .../backend/modules/llm/fit/kv_cells.py | 40 ++++++++ .../backend/modules/llm/fit/mla_cache.py | 21 +++++ .../backend/modules/llm/fit/sliding_window.py | 85 +++++++++++++---- .../tests/unit/llm/fit/test_kv_cache.py | 36 +++++++ .../tests/unit/llm/fit/test_sliding_window.py | 93 ++++++++++++++++++- 6 files changed, 301 insertions(+), 41 deletions(-) create mode 100644 surfsense_local/backend/modules/llm/fit/kv_cells.py create mode 100644 surfsense_local/backend/modules/llm/fit/mla_cache.py diff --git a/surfsense_local/backend/modules/llm/fit/kv_cache.py b/surfsense_local/backend/modules/llm/fit/kv_cache.py index c77cb7d4cb..77c4d23f42 100644 --- a/surfsense_local/backend/modules/llm/fit/kv_cache.py +++ b/surfsense_local/backend/modules/llm/fit/kv_cache.py @@ -1,33 +1,64 @@ -"""The conversation cache's memory cost, which is architecture-derived and -independent of how the weights were quantized.""" +"""The conversation cache's memory cost. -from modules.llm.fit.sliding_window import global_layer_count +Architecture-derived and independent of how the weights were quantized, which is +what lets one header read price every build of a model. + +Three things decide it, and each has its own module: how wide one layer's entry +is (here, or `mla_cache` for a latent model), how many cells a layer allocates +(`kv_cells`), and which layers hold the whole window rather than a slice +(`sliding_window`). +""" + +from modules.llm.fit.kv_cells import global_cells, local_cells +from modules.llm.fit.mla_cache import is_latent, latent_bytes_per_layer_token +from modules.llm.fit.sliding_window import sliding_layers from modules.llm.fit.types import KvPrecision, ModelShape def kv_cache_bytes(shape: ModelShape, n_ctx: int, precision: KvPrecision) -> int: """Bytes the KV cache occupies for a window of `n_ctx` tokens. - One key and one value per attention head per layer per token. Verified exact - against llama.cpp's own allocation on two backends. - - A model with sliding-window attention holds the full context on only some of - its layers, but the header does not say which, so an architecture we cannot - name is priced as though every layer were global. That over-states memory, - which is the only direction it is safe to be wrong in. + Verified exact against llama.cpp's own allocation on two backends. """ - bytes_per_layer_token = int( + per_layer_token = _bytes_per_layer_token(shape, precision) + caching = _caching_layers(shape) + if caching <= 0 or per_layer_token <= 0: + return 0 + + window = shape.sliding_window + if window <= 0: + return per_layer_token * caching * global_cells(n_ctx) + + sliding = sliding_layers(shape) + if sliding is None: + # No pattern to be had, so every layer is priced at full width. + return per_layer_token * caching * global_cells(n_ctx) + + # The header's pattern describes the model's layers; the ones that cache are + # a prefix of those, since shared layers reuse an earlier layer's cache. + local = sum(1 for slides in sliding[:caching] if slides) + full = caching - local + return per_layer_token * ( + full * global_cells(n_ctx) + local * local_cells(window, n_ctx) + ) + + +def _bytes_per_layer_token(shape: ModelShape, precision: KvPrecision) -> int: + """One layer's cache entry for one token.""" + if is_latent(shape): + return latent_bytes_per_layer_token(shape, precision) + return int( shape.head_count_kv * (shape.key_length + shape.value_length) * precision.bytes_per_element ) - if shape.sliding_window <= 0 or shape.sliding_window >= n_ctx: - return bytes_per_layer_token * shape.block_count * n_ctx - globals_ = global_layer_count(shape.architecture, shape.block_count) - if globals_ is None: - return bytes_per_layer_token * shape.block_count * n_ctx +def _caching_layers(shape: ModelShape) -> int: + """Layers that allocate a cache of their own. - locals_ = shape.block_count - globals_ - return bytes_per_layer_token * (globals_ * n_ctx + locals_ * shape.sliding_window) + Gemma 3n and Gemma 4 reuse an earlier layer's cache on their last blocks, so + charging those layers a cache each over-states the window's cost on exactly + the models chosen to be cheap on a small machine. + """ + return max(0, shape.block_count - shape.shared_kv_layers) diff --git a/surfsense_local/backend/modules/llm/fit/kv_cells.py b/surfsense_local/backend/modules/llm/fit/kv_cells.py new file mode 100644 index 0000000000..f429f3476d --- /dev/null +++ b/surfsense_local/backend/modules/llm/fit/kv_cells.py @@ -0,0 +1,40 @@ +"""How many cache cells llama.cpp allocates for a window. + +Not the same as the number of tokens. The cache is padded, and a sliding window +layer is sized for its window plus a micro batch rather than for the window +alone, because a batch being processed has to sit in the cache beside the window +it attends to. Both come from `llama_kv_cache_iswa`, which sizes a sliding layer +as `pad(min(size_base, n_swa * n_seq_max + n_ubatch))`. + +`n_seq_max` is 1 here, pinned by the `parallel = 1` this app writes into every +preset: one user asking one question. +""" + +# CUDA wants the cache aligned, and llama.cpp pads every cache to this on every +# backend rather than branching per device. +CELL_PADDING = 256 + +# llama-server's default micro batch. A sliding layer holds the batch being +# processed alongside the window it attends to. +UBATCH_TOKENS = 512 + + +def pad(cells: int) -> int: + """Round up to the alignment llama.cpp allocates on.""" + if cells <= 0: + return 0 + return -(-cells // CELL_PADDING) * CELL_PADDING + + +def global_cells(n_ctx: int) -> int: + """Cells a layer that attends to the whole conversation allocates.""" + return pad(n_ctx) + + +def local_cells(sliding_window: int, n_ctx: int) -> int: + """Cells a sliding window layer allocates. + + Never more than a global layer: a window wider than the context is not a + window at all, and llama.cpp takes the smaller of the two for that reason. + """ + return min(global_cells(n_ctx), pad(sliding_window + UBATCH_TOKENS)) diff --git a/surfsense_local/backend/modules/llm/fit/mla_cache.py b/surfsense_local/backend/modules/llm/fit/mla_cache.py new file mode 100644 index 0000000000..0162b490e0 --- /dev/null +++ b/surfsense_local/backend/modules/llm/fit/mla_cache.py @@ -0,0 +1,21 @@ +"""The cache cost of multi-head latent attention. + +DeepSeek-class models do not cache a key and a value per head. They cache one +compressed latent per token per layer, plus the rotary part of the key, and +reconstruct the rest on the way past. Priced by the ordinary formula they read +enormously too large, because such a header reports a single KV head while the +model has a hundred and more. +""" + +from modules.llm.fit.types import KvPrecision, ModelShape + + +def is_latent(shape: ModelShape) -> bool: + """Whether this model caches a compressed latent rather than keys and values.""" + return shape.kv_lora_rank > 0 + + +def latent_bytes_per_layer_token(shape: ModelShape, precision: KvPrecision) -> int: + """Bytes one layer caches per token: the latent plus the rotary key.""" + width = shape.kv_lora_rank + shape.key_length_mla + return int(width * precision.bytes_per_element) diff --git a/surfsense_local/backend/modules/llm/fit/sliding_window.py b/surfsense_local/backend/modules/llm/fit/sliding_window.py index 5029ee6ffc..200f579ba8 100644 --- a/surfsense_local/backend/modules/llm/fit/sliding_window.py +++ b/surfsense_local/backend/modules/llm/fit/sliding_window.py @@ -1,29 +1,78 @@ """Which layers of a sliding-window model hold the full context. -The GGUF header states the window size but never the pattern, so the pattern is -a per-architecture fact. llama.cpp carries the same knowledge in `src/models`; -this is the subset that matters to the catalog, and anything absent is priced as -full attention rather than guessed at. +Three sources, in the order llama.cpp itself consults them. A header that states +its own pattern is the authority, as one flag per layer or as a period; only a +file that states neither falls back to the table below, which mirrors the +defaults each architecture passes to `load_swa_pattern`. + +Anything absent from all three is priced as full attention on every layer. That +over-states memory, which is the only direction it is safe to be wrong in. """ -# Global layers per cycle: a value of 6 means one layer in six attends to the -# whole context and the other five attend to `sliding_window` tokens. -_GLOBAL_LAYER_EVERY: dict[str, int] = { - "gemma2": 2, - "gemma3": 6, - "gemma3n": 6, - "cohere2": 4, - "phi3": 2, +from modules.llm.fit.types import ModelShape + +# Period and whether the cycle leads with the full attention layer, matching the +# `n_pattern` and `dense_first` arguments in `src/models/*.cpp`. Read from +# llama.cpp at b11050; an architecture whose default nobody has verified is +# deliberately absent rather than guessed. +_DEFAULT_PATTERN: dict[str, tuple[int, bool]] = { + "gemma2": (2, False), + "gemma3": (6, False), + "gemma3n": (5, False), + "cohere2": (4, False), + "gpt-oss": (2, False), + "llama4": (4, False), + "olmo2": (4, False), + "exaone4": (4, False), + "plamo3": (8, False), + "afmoe": (4, False), + "mellum": (4, False), + "cohere2moe": (4, True), + "modern-bert": (3, True), + "smallthinker": (4, True), } -def global_layer_count(architecture: str, block_count: int) -> int | None: - """How many layers attend to the whole window, or None when unknown. +def sliding_layers(shape: ModelShape) -> tuple[bool, ...] | None: + """One flag per layer, True where the layer slides, or None when unknown. None is the honest answer for an architecture nobody has characterised, and - the caller must then round up rather than assume a ratio. + the caller must then price every layer at full width rather than assume a + ratio. """ - cycle = _GLOBAL_LAYER_EVERY.get(architecture) - if cycle is None: + if shape.sliding_window_layers: + return _fitted(shape.sliding_window_layers, shape.block_count) + + if shape.sliding_window_pattern > 0: + return _from_period(shape.sliding_window_pattern, shape.block_count, False) + + default = _DEFAULT_PATTERN.get(shape.architecture) + if default is None: return None - return block_count // cycle + period, dense_first = default + return _from_period(period, shape.block_count, dense_first) + + +def _from_period(period: int, block_count: int, dense_first: bool) -> tuple[bool, ...]: + """Expand a cycle length into one flag per layer. + + A period of six means one layer in six attends to the whole context. Which + one depends on `dense_first`: normally the full attention layer closes the + cycle, and for a handful of architectures it opens it. + """ + if period <= 1: + return (False,) * block_count + if dense_first: + return tuple(index % period != 0 for index in range(block_count)) + return tuple((index + 1) % period != 0 for index in range(block_count)) + + +def _fitted(layers: tuple[bool, ...], block_count: int) -> tuple[bool, ...]: + """The header's flags, cut or repeated to the model's depth. + + A header that disagrees with its own block count is a file we cannot read + exactly, and repeating the cycle is what llama.cpp does with a short list. + """ + if len(layers) == block_count: + return layers + return tuple(layers[index % len(layers)] for index in range(block_count)) diff --git a/surfsense_local/backend/tests/unit/llm/fit/test_kv_cache.py b/surfsense_local/backend/tests/unit/llm/fit/test_kv_cache.py index 4983ee156a..c68083f9a2 100644 --- a/surfsense_local/backend/tests/unit/llm/fit/test_kv_cache.py +++ b/surfsense_local/backend/tests/unit/llm/fit/test_kv_cache.py @@ -42,3 +42,39 @@ def test_quantizing_the_cache_roughly_halves_it() -> None: is why precision is an output of the fit calculation rather than a default. """ assert kv_cache_bytes(QWEN3_1_7B, 16384, KvPrecision.Q8_0) == 952 * MIB + + +def test_cells_are_padded_the_way_the_allocator_pads_them() -> None: + """llama.cpp aligns the cache to 256 cells, so a window one token past a + boundary costs a whole block more. Pricing the raw token count under-states + every window that is not already aligned.""" + aligned = kv_cache_bytes(QWEN3_1_7B, 16384, KvPrecision.F16) + one_past = kv_cache_bytes(QWEN3_1_7B, 16385, KvPrecision.F16) + + assert one_past == kv_cache_bytes(QWEN3_1_7B, 16640, KvPrecision.F16) + assert one_past > aligned + + +def test_a_latent_attention_model_caches_one_compressed_entry_per_token() -> None: + """DeepSeek-class models cache a latent, not a key and a value per head, and + such a header reports one KV head while the model runs many. + + Priced by the ordinary formula this reads far too small, because it would + take that single head at face value. The latent's own width is what governs. + """ + deepseek_like = ModelShape( + architecture="deepseek2", + block_count=28, + head_count_kv=1, + key_length=192, + value_length=128, + context_length=163840, + n_vocab=129280, + kv_lora_rank=512, + key_length_mla=64, + ) + + per_layer_token = (512 + 64) * 2 + expected = per_layer_token * 28 * 16384 + + assert kv_cache_bytes(deepseek_like, 16384, KvPrecision.F16) == expected diff --git a/surfsense_local/backend/tests/unit/llm/fit/test_sliding_window.py b/surfsense_local/backend/tests/unit/llm/fit/test_sliding_window.py index 1fe042003d..1262720ca4 100644 --- a/surfsense_local/backend/tests/unit/llm/fit/test_sliding_window.py +++ b/surfsense_local/backend/tests/unit/llm/fit/test_sliding_window.py @@ -31,14 +31,19 @@ def test_a_known_pattern_prices_only_the_global_layers_at_full_width() -> None: """Gemma 3 is 5 local to 1 global, so 8 of 48 layers hold the whole window. Worked by hand: per layer per token is 8 heads * 256 = 2048 elements at - 2 bytes = 4096 bytes. Eight global layers * 16384 tokens = 512 MiB. Forty - local layers * 1024 tokens = 160 MiB. Total 672 MiB, against 3072 MiB if - every layer were priced at full width. That ratio is 4.57x, which is the - Gemma over-estimate the spec quotes at 4.6x from a separate measurement. + 2 bytes = 4096 bytes. Eight global layers * 16384 cells = 512 MiB. Forty + local layers * 1536 cells = 240 MiB. Total 752 MiB, against 3072 MiB if + every layer were priced at full width. + + 1536 rather than the window's own 1024: `llama_kv_cache_iswa` sizes a + sliding layer as `pad256(n_swa + n_ubatch)`, because the batch being + processed sits in the cache beside the window it attends to. Pricing the + window alone under-states this model by 80 MiB, and under-stating is the + direction that ships a confident badge about a model that spills. """ cost = kv_cache_bytes(gemma_like("gemma3"), 16384, KvPrecision.F16) - assert cost == 672 * MIB + assert cost == 752 * MIB def test_an_unknown_architecture_declining_to_guess_rounds_up() -> None: @@ -51,3 +56,81 @@ def test_an_unknown_architecture_declining_to_guess_rounds_up() -> None: cost = kv_cache_bytes(gemma_like("something-nobody-has-seen"), 16384, KvPrecision.F16) assert cost == 3072 * MIB + + +def test_a_pattern_stated_in_the_header_beats_the_table() -> None: + """llama.cpp reads this key before its own defaults, so a file that states + its pattern is the authority on that model, whatever we happen to know.""" + stated = ModelShape( + architecture="something-nobody-has-seen", + block_count=48, + head_count_kv=8, + key_length=128, + value_length=128, + context_length=131072, + n_vocab=262144, + sliding_window=1024, + sliding_window_pattern=6, + ) + + assert kv_cache_bytes(stated, 16384, KvPrecision.F16) == 752 * MIB + + +def test_a_pattern_stated_per_layer_is_counted_exactly() -> None: + """The other form the same key takes. No period arithmetic at all: the file + says which layers slide, and a model with an irregular pattern is priced + correctly rather than rounded up to the nearest cycle.""" + shape = ModelShape( + architecture="something-nobody-has-seen", + block_count=4, + head_count_kv=8, + key_length=128, + value_length=128, + context_length=131072, + n_vocab=262144, + sliding_window=1024, + sliding_window_layers=(True, True, True, False), + ) + + per_layer_token = 8 * 256 * 2 + expected = per_layer_token * (1 * 16384 + 3 * 1536) + + assert kv_cache_bytes(shape, 16384, KvPrecision.F16) == expected + + +def test_gemma_3n_is_priced_at_the_period_llama_cpp_uses() -> None: + """Our table said 6 and llama.cpp passes 5, so every gemma3n row was priced + with one global layer too few and read cheaper than it is.""" + at_five = kv_cache_bytes(gemma_like("gemma3n"), 16384, KvPrecision.F16) + at_six = kv_cache_bytes(gemma_like("gemma3"), 16384, KvPrecision.F16) + + assert at_five > at_six + + +def test_layers_that_share_a_cache_allocate_none_of_their_own() -> None: + """Gemma 3n reuses an earlier layer's cache on its last blocks. Charging + those layers a cache each over-states exactly the models chosen to be cheap + on a small machine.""" + shared = ModelShape( + architecture="qwen3", + block_count=48, + head_count_kv=8, + key_length=128, + value_length=128, + context_length=131072, + n_vocab=262144, + shared_kv_layers=16, + ) + alone = ModelShape( + architecture="qwen3", + block_count=32, + head_count_kv=8, + key_length=128, + value_length=128, + context_length=131072, + n_vocab=262144, + ) + + assert kv_cache_bytes(shared, 16384, KvPrecision.F16) == kv_cache_bytes( + alone, 16384, KvPrecision.F16 + ) From 0243635dd7bbbdd35bebb24b782e47877634cfb2 Mon Sep 17 00:00:00 2001 From: Anish Sarkar <104695310+AnishSarkar22@users.noreply.github.com> Date: Mon, 21 Sep 2026 16:58:55 +0530 Subject: [PATCH 34/60] feat(fit): size compute buffers from the widest tensor a layer builds --- .../modules/llm/fit/compute_buffers.py | 91 ++++++++++++--- .../unit/llm/fit/test_compute_buffers.py | 108 ++++++++++++++++++ 2 files changed, 186 insertions(+), 13 deletions(-) create mode 100644 surfsense_local/backend/tests/unit/llm/fit/test_compute_buffers.py diff --git a/surfsense_local/backend/modules/llm/fit/compute_buffers.py b/surfsense_local/backend/modules/llm/fit/compute_buffers.py index 0d906f987e..65e43acf96 100644 --- a/surfsense_local/backend/modules/llm/fit/compute_buffers.py +++ b/surfsense_local/backend/modules/llm/fit/compute_buffers.py @@ -1,25 +1,90 @@ """The scratch memory a graph needs beyond weights and cache. -Absent from earlier drafts of the estimator entirely, which under-stated `need` -by enough to flip a verdict on a 6 GB card. - -> ponytail: two measured points, one model, one backend. Qwen3 1.7B on Metal at -> b11050 reported 102.24 MiB at 16384 and 222.24 MiB at 40960, and a 4B on -> Vulkan reported 143.62 device plus 26.01 host at 16384. The line below passes -> through the Metal pair and is the best available answer, not a law. Replace it -> with the machine's own residual after the first load, which measures this term -> along with everything else that is neither weights nor cache. +Absent from earlier drafts of the estimator entirely, and then present as a +constant, which is nearly as wrong: scratch is sized by the widest thing one +layer computes, so it scales with the model rather than being a fee every model +pays alike. The constant was fitted to a 1.7B and under-stated a 4B by about +70 MiB, which is more than the margin that decided a 12 GB card's top row. + +Two parts. A flat term for the activations, the output rows and the safety the +allocator keeps, which is fixed once the model is known. And a term that grows +with the window, for the attention mask and the scratch that goes with it. + +> ponytail: the flat term's shape is llama.cpp's own graph, but the safety +> factor and the per token slope are fitted, not derived. Three measured points: +> Qwen3 1.7B on Metal reported 102.24 MiB at 16384 and 222.24 MiB at 40960; a 4B +> on Vulkan reported 143.62 device plus 26.01 host at 16384. The slope is exact +> for the Metal pair, 120 MiB over 24576 tokens. +> +> The two backends disagree about the flat half, and by more than the widths +> explain: the 4B's measured flat is 4.0x the 1.7B's where the modelled widths +> differ by 1.6x. Two models on two backends cannot separate a model effect from +> a backend effect, so this is fitted to cover both rather than to match either, +> and it over-states the 1.7B by a third to do it. Replace the whole term with +> the machine's own residual after a first load, which measures it along with +> everything else that is neither weights nor cache, on the machine that will +> run the model. """ -_BASE_BYTES = 23 * 1024**2 -_PER_TOKEN_BYTES = 5_120 +from modules.llm.fit.types import ModelShape + +# llama-server's defaults, and the `parallel = 1` this app pins in every preset. +UBATCH_TOKENS = 512 +SLOTS = 1 + +# What the allocator keeps beyond the modelled width. Fitted to cover every +# measured point, not derived: the smallest value at which all three land on the +# safe side. At 1.10 the 4B came out at 0.97 of what Vulkan really allocated, +# and under-stating is the direction that ships a confident badge about a model +# that spills. +SAFETY = 1.20 + +# The context-shaped half: attention mask and its scratch, per token. +PER_TOKEN_BYTES = 5_120 + +# The line the constant used to be, kept for a header that states no widths at +# all. Wrong in detail, but it is the measured floor for the smallest model and +# is certainly better than pricing scratch at nothing. +_FALLBACK_BASE_BYTES = 23 * 1024**2 + +def activation_width(shape: ModelShape) -> int: + """Elements the graph holds live per micro batch token. -def compute_buffer_bytes(n_ctx: int) -> int: + The widest of the three things a layer can be: the attention and residual + buffers, the dense feed forward, or the experts a router actually reads. + """ + embedding = shape.embedding_length + if embedding <= 0: + return 0 + + widths = [12 * embedding] + if shape.feed_forward_length > 0: + widths.append(4 * shape.feed_forward_length) + + used = shape.expert_used_count + expert_width = shape.expert_feed_forward_length or ( + shape.feed_forward_length if shape.expert_count else 0 + ) + if used > 0 and expert_width > 0: + shared = shape.expert_shared_feed_forward_length + widths.append(used * (2 * embedding + 3 * expert_width) + 3 * shared) + + return max(widths) + + +def compute_buffer_bytes(shape: ModelShape, n_ctx: int) -> int: """Scratch bytes for a graph over an `n_ctx` window, rounded up. Rounding up is the rule the whole estimator runs on: over-stating costs a pessimistic badge, under-stating ships a confident badge about a model that spills. """ - return _BASE_BYTES + _PER_TOKEN_BYTES * n_ctx + width = activation_width(shape) + if width <= 0 or shape.n_vocab <= 0: + return _FALLBACK_BASE_BYTES + PER_TOKEN_BYTES * n_ctx + + activations = width * UBATCH_TOKENS * 4 + outputs = shape.n_vocab * min(UBATCH_TOKENS, SLOTS) * 4 + flat = int((activations + outputs) * SAFETY) + return flat + PER_TOKEN_BYTES * n_ctx diff --git a/surfsense_local/backend/tests/unit/llm/fit/test_compute_buffers.py b/surfsense_local/backend/tests/unit/llm/fit/test_compute_buffers.py new file mode 100644 index 0000000000..98443d1038 --- /dev/null +++ b/surfsense_local/backend/tests/unit/llm/fit/test_compute_buffers.py @@ -0,0 +1,108 @@ +"""Graph scratch: the third term, and the one that was a constant. + +The measured points are what llama.cpp reported as its compute buffer at +b11050. A constant line passing through the smallest of them under-states a 4B +by roughly 70 MiB, which is more than the margin that decided a 12 GB card's +top row. +""" + +import pytest + +from modules.llm.fit import ModelShape, compute_buffer_bytes + +pytestmark = pytest.mark.unit + +MIB = 1024**2 + + +def qwen3(blocks: int, embedding: int, feed_forward: int) -> ModelShape: + """A dense Qwen3, whose widths are what the scratch term is sized from.""" + return ModelShape( + architecture="qwen3", + block_count=blocks, + head_count_kv=8, + key_length=128, + value_length=128, + context_length=40960, + n_vocab=151936, + embedding_length=embedding, + feed_forward_length=feed_forward, + ) + + +QWEN3_1_7B = qwen3(28, 2048, 6144) +QWEN3_4B = qwen3(36, 2560, 9728) + + +def test_a_wider_model_needs_more_scratch_at_the_same_window() -> None: + """The whole point. A constant charged these two the same and was wrong + about at least one of them.""" + narrow = compute_buffer_bytes(QWEN3_1_7B, 16384) + wide = compute_buffer_bytes(QWEN3_4B, 16384) + + assert wide > narrow + + +@pytest.mark.parametrize( + ("shape", "n_ctx", "measured_mib"), + [ + (QWEN3_1_7B, 16384, 102.24), # Metal, b11050 + (QWEN3_1_7B, 40960, 222.24), # Metal, same run + (QWEN3_4B, 16384, 169.63), # Vulkan: 143.62 device plus 26.01 host + ], +) +def test_no_measured_point_is_under_predicted(shape, n_ctx, measured_mib) -> None: + """The rule the whole estimator runs on, pinned to every point we have. + + Over-stating costs a pessimistic badge, which a user can install through. + Under-stating ships a confident badge about a model that spills, so this + asserts a floor rather than a match. The safety factor is fitted to keep + every one of these on the safe side. + """ + assert compute_buffer_bytes(shape, n_ctx) >= measured_mib * MIB + + +def test_scratch_is_never_priced_at_zero_when_the_widths_are_missing() -> None: + """A searched header can omit them. Zero would be the one answer that is + certainly wrong, so the old constant line is the floor in that case.""" + unknown = ModelShape( + architecture="something-nobody-has-seen", + block_count=28, + head_count_kv=8, + key_length=128, + value_length=128, + context_length=40960, + n_vocab=151936, + ) + + assert compute_buffer_bytes(unknown, 16384) > 0 + + +def test_scratch_grows_with_the_window() -> None: + """Part of it is the attention mask, which is context shaped.""" + at_floor = compute_buffer_bytes(QWEN3_4B, 16384) + at_full = compute_buffer_bytes(QWEN3_4B, 40960) + + assert at_full > at_floor + + +def test_a_mixture_of_experts_is_sized_by_what_one_token_activates() -> None: + """A router reads a few experts per token, and the graph holds those. Priced + by the dense width alone, a model built to be cheap reads cheaper still.""" + dense = qwen3(48, 2048, 6144) + moe = ModelShape( + architecture="qwen3moe", + block_count=48, + head_count_kv=8, + key_length=128, + value_length=128, + context_length=40960, + n_vocab=151936, + embedding_length=2048, + feed_forward_length=6144, + expert_count=128, + expert_used_count=8, + expert_feed_forward_length=768, + ) + + assert compute_buffer_bytes(moe, 16384) > compute_buffer_bytes(dense, 16384) From 8bc69f31090975a8620a5ec10e99d1d932a91826 Mon Sep 17 00:00:00 2001 From: Anish Sarkar <104695310+AnishSarkar22@users.noreply.github.com> Date: Mon, 21 Sep 2026 16:59:08 +0530 Subject: [PATCH 35/60] feat(fit): itemise what a load allocates and count offload in layers --- .../backend/modules/llm/fit/itemisation.py | 55 ++++++++++++ .../backend/modules/llm/fit/offload.py | 40 +++++++++ .../tests/unit/llm/fit/test_itemisation.py | 67 ++++++++++++++ .../tests/unit/llm/fit/test_offload.py | 88 +++++++++++++++++++ 4 files changed, 250 insertions(+) create mode 100644 surfsense_local/backend/modules/llm/fit/itemisation.py create mode 100644 surfsense_local/backend/modules/llm/fit/offload.py create mode 100644 surfsense_local/backend/tests/unit/llm/fit/test_itemisation.py create mode 100644 surfsense_local/backend/tests/unit/llm/fit/test_offload.py diff --git a/surfsense_local/backend/modules/llm/fit/itemisation.py b/surfsense_local/backend/modules/llm/fit/itemisation.py new file mode 100644 index 0000000000..1607268b0c --- /dev/null +++ b/surfsense_local/backend/modules/llm/fit/itemisation.py @@ -0,0 +1,55 @@ +"""What a load allocates, broken into the four things it is made of. + +The estimator compares one number against a budget, but that number is a sum, +and a sum is untestable: it can be right for compensating wrong reasons. Naming +the parts is what lets a test assert that the weights term does not move when +the window does, and that the parts still add up to what the verdict reports. + +It is also the shape the offload calculation needs, because only some of these +are things llama.cpp can move to the host. +""" + +from dataclasses import dataclass + +from modules.llm.fit.compute_buffers import compute_buffer_bytes +from modules.llm.fit.kv_cache import kv_cache_bytes +from modules.llm.fit.types import KvPrecision, ModelShape + + +@dataclass(frozen=True) +class NeedItems: + """The four terms, each priced by its own module.""" + + weights_bytes: int + mmproj_bytes: int + kv_bytes: int + compute_bytes: int + + @property + def total(self) -> int: + """What the model allocates, which is what the verdict compares.""" + return ( + self.weights_bytes + self.mmproj_bytes + self.kv_bytes + self.compute_bytes + ) + + +def itemise( + shape: ModelShape, + weights_bytes: int, + n_ctx: int, + precision: KvPrecision = KvPrecision.F16, + mmproj_bytes: int = 0, +) -> NeedItems: + """Price one build at one window. + + The projector is a term of its own because `--fit` does not count it + (llama.cpp issue #19980), so a vision model our sum calls resident can still + fail to allocate. It is also not a thing the fitter can move, which is why + the offload calculation needs it separated from the weights. + """ + return NeedItems( + weights_bytes=weights_bytes, + mmproj_bytes=mmproj_bytes, + kv_bytes=kv_cache_bytes(shape, n_ctx, precision), + compute_bytes=compute_buffer_bytes(shape, n_ctx), + ) diff --git a/surfsense_local/backend/modules/llm/fit/offload.py b/surfsense_local/backend/modules/llm/fit/offload.py new file mode 100644 index 0000000000..dc6f5d948c --- /dev/null +++ b/surfsense_local/backend/modules/llm/fit/offload.py @@ -0,0 +1,40 @@ +"""How much of a model ends up on the processor when it does not all fit. + +llama.cpp does not split a model by bytes. Its fitter fills devices with whole +layers, back to front, and stops at the last one that fits, so the answer is a +count of layers over the layer count rather than a ratio of bytes. + +Measured on an RTX 3050, Qwen3 4B spilled 602 MiB of weights plus 320 MiB of +cache: 0.19 of the model. The byte ratio said 0.12. Both agree on the verdict +and on the ordering, but the fraction is what grades the reason line and gates +the recommendation, so it is the fraction that has to be right. + +Only layers move. The projector is pinned by a flag rather than placed by the +fitter, and the compute buffer belongs to whichever device runs the graph, so +neither is part of what a layer costs. +""" + +from math import ceil + +from modules.llm.fit.itemisation import NeedItems +from modules.llm.fit.types import ModelShape + + +def offload_fraction(shape: ModelShape, items: NeedItems, resident_bytes: int) -> float: + """Fraction of the model's layers that will not fit on the device. + + Returns 0.0 when everything fits, and never more than 1.0. + """ + spilled = items.total - resident_bytes + if spilled <= 0: + return 0.0 + if shape.block_count <= 0: + return min(1.0, spilled / items.total) if items.total else 1.0 + + movable = items.weights_bytes + items.kv_bytes + per_layer = movable / shape.block_count + if per_layer <= 0: + return 1.0 + + layers_out = ceil(spilled / per_layer) + return min(1.0, layers_out / shape.block_count) diff --git a/surfsense_local/backend/tests/unit/llm/fit/test_itemisation.py b/surfsense_local/backend/tests/unit/llm/fit/test_itemisation.py new file mode 100644 index 0000000000..c4d780f9a8 --- /dev/null +++ b/surfsense_local/backend/tests/unit/llm/fit/test_itemisation.py @@ -0,0 +1,67 @@ +"""The four terms a load allocates, and that they still add up. + +A single `need` number can be right for compensating wrong reasons. Naming the +parts is what makes each one assertable on its own. +""" + +import pytest + +from modules.llm.fit import KvPrecision, ModelShape, estimate, itemise +from modules.llm.fit.budget import HardwareBudget + +pytestmark = pytest.mark.unit + +MIB = 1024**2 + +QWEN3_4B = ModelShape( + architecture="qwen3", + block_count=36, + head_count_kv=8, + key_length=128, + value_length=128, + context_length=40960, + n_vocab=151936, + embedding_length=2560, + feed_forward_length=9728, +) +ROOMY = HardwareBudget(24000 * MIB, 24576 * MIB, 1024 * MIB, 32000 * MIB, False, True) + + +def test_the_parts_are_what_the_verdict_compares() -> None: + """The property that keeps the itemisation honest: if these ever stop + summing to `need`, one of the two is describing a different model.""" + for n_ctx in (8192, 16384, 40960): + items = itemise(QWEN3_4B, 2500 * MIB, n_ctx) + verdict = estimate(QWEN3_4B, 2500 * MIB, ROOMY, n_ctx=n_ctx) + + assert items.total == verdict.need_bytes + + +def test_the_weights_term_does_not_move_with_the_window() -> None: + """The load bearing one. Weights and cache are priced by different modules, + and a window that moved the weights would mean the two had started naming + the same bytes.""" + narrow = itemise(QWEN3_4B, 2500 * MIB, 8192) + wide = itemise(QWEN3_4B, 2500 * MIB, 40960) + + assert narrow.weights_bytes == wide.weights_bytes + assert wide.kv_bytes > narrow.kv_bytes + + +def test_the_projector_is_its_own_term() -> None: + """`--fit` does not count it, and the fitter cannot move it either, so it is + neither part of the weights nor part of what a layer costs.""" + items = itemise(QWEN3_4B, 2500 * MIB, 16384, mmproj_bytes=600 * MIB) + + assert items.mmproj_bytes == 600 * MIB + assert items.weights_bytes == 2500 * MIB + + +def test_a_quantized_cache_moves_only_the_cache_term() -> None: + """Which is why precision is an output of the fit calculation.""" + f16 = itemise(QWEN3_4B, 2500 * MIB, 16384, KvPrecision.F16) + q8 = itemise(QWEN3_4B, 2500 * MIB, 16384, KvPrecision.Q8_0) + + assert q8.kv_bytes < f16.kv_bytes + assert q8.weights_bytes == f16.weights_bytes + assert q8.compute_bytes == f16.compute_bytes diff --git a/surfsense_local/backend/tests/unit/llm/fit/test_offload.py b/surfsense_local/backend/tests/unit/llm/fit/test_offload.py new file mode 100644 index 0000000000..37b0b7edcf --- /dev/null +++ b/surfsense_local/backend/tests/unit/llm/fit/test_offload.py @@ -0,0 +1,88 @@ +"""How much spills, counted the way llama.cpp actually places it. + +The fitter fills a device with whole layers and stops at the last one that fits, +so the fraction is a count of layers rather than a ratio of bytes. It matters +because this number grades the reason line and gates the recommendation. +""" + +import pytest + +from modules.llm.fit import FitState, HardwareBudget, ModelShape, estimate +from modules.llm.fit.itemisation import itemise +from modules.llm.fit.offload import offload_fraction + +pytestmark = pytest.mark.unit + +MIB = 1024**2 +GB = 1000**3 + +QWEN3_4B = ModelShape( + architecture="qwen3", + block_count=36, + head_count_kv=8, + key_length=128, + value_length=128, + context_length=40960, + n_vocab=151936, + embedding_length=2560, + feed_forward_length=9728, +) + +# The measured machine: RTX 3050, 5234 MiB free, llama.cpp's own 1024 MiB margin. +RTX_3050 = HardwareBudget(5234 * MIB, 6002 * MIB, 1024 * MIB, 22750 * MIB, False, True) + + +def test_the_measured_case_is_closer_in_layers_than_it_was_in_bytes() -> None: + """Qwen3 4B spilled 602 MiB of weights plus 320 of cache on that card: 0.19. + + The byte ratio said 0.12, which is not a rounding error at thresholds of + 0.25 and 0.5. Counting layers gives 5 of 36. + """ + verdict = estimate(QWEN3_4B, int(2.50 * GB), RTX_3050, n_ctx=16384) + + assert verdict.state is FitState.PARTIAL + assert verdict.offload_fraction == pytest.approx(5 / 36) + + +def test_the_fraction_is_always_a_whole_number_of_layers() -> None: + """A model cannot have four and a half layers on the processor.""" + for weights_gb in (2.4, 2.6, 3.0, 3.4, 4.0, 4.4): + verdict = estimate(QWEN3_4B, int(weights_gb * GB), RTX_3050, n_ctx=16384) + if verdict.state is not FitState.PARTIAL: + continue + + layers = verdict.offload_fraction * QWEN3_4B.block_count + + assert layers == pytest.approx(round(layers)) + + +def test_a_bigger_shortfall_spills_more_layers() -> None: + """Monotonic, which is what lets the reason line be graded on it.""" + smaller = estimate(QWEN3_4B, int(2.6 * GB), RTX_3050, n_ctx=16384) + larger = estimate(QWEN3_4B, int(4.0 * GB), RTX_3050, n_ctx=16384) + + assert larger.offload_fraction > smaller.offload_fraction + + +def test_a_model_that_fits_spills_nothing() -> None: + """The boundary, stated rather than assumed.""" + items = itemise(QWEN3_4B, int(2.50 * GB), 16384) + + assert offload_fraction(QWEN3_4B, items, items.total) == 0.0 + + +def test_only_layers_count_as_movable() -> None: + """The projector is pinned by a flag and the compute buffer belongs to + whichever device runs the graph, so neither is a thing the fitter can move. + + Counting them as movable would make each layer look larger than it is, and + so under-count the layers a given shortfall displaces. + """ + weights = int(2.50 * GB) + without = itemise(QWEN3_4B, weights, 16384) + with_projector = itemise(QWEN3_4B, weights, 16384, mmproj_bytes=600 * MIB) + resident = without.total - 300 * MIB + + assert offload_fraction(QWEN3_4B, with_projector, resident) > offload_fraction( + QWEN3_4B, without, resident + ) From 35f716ada1807549c5c9fd51aeabab836fd2807c Mon Sep 17 00:00:00 2001 From: Anish Sarkar <104695310+AnishSarkar22@users.noreply.github.com> Date: Mon, 21 Sep 2026 16:59:19 +0530 Subject: [PATCH 36/60] fix(fit): split residency from refusal so a CPU-only machine is judged right --- .../backend/modules/llm/fit/budget.py | 33 +++ .../backend/modules/llm/fit/copy.py | 13 +- .../backend/modules/llm/fit/estimate.py | 78 ++++--- .../tests/unit/llm/fit/test_badge_copy.py | 32 ++- .../tests/unit/llm/fit/test_estimate.py | 62 +++++- .../tests/unit/llm/fit/test_properties.py | 205 ++++++++++++++++++ 6 files changed, 376 insertions(+), 47 deletions(-) create mode 100644 surfsense_local/backend/tests/unit/llm/fit/test_properties.py diff --git a/surfsense_local/backend/modules/llm/fit/budget.py b/surfsense_local/backend/modules/llm/fit/budget.py index 4a6e666be8..865feb1bcd 100644 --- a/surfsense_local/backend/modules/llm/fit/budget.py +++ b/surfsense_local/backend/modules/llm/fit/budget.py @@ -23,3 +23,36 @@ class HardwareBudget: def usable_vram_bytes(self) -> int: """What --fit will actually place layers in.""" return max(0, self.device_free_bytes - self.fit_reserve_bytes) + + @property + def resident_bytes(self) -> int: + """What a model must fit inside to run at full speed. + + Without a GPU there is no device to be resident in, and the processor's + own memory is both where the model runs and all there is, so the two + questions below collapse into one answer. + """ + return self.usable_vram_bytes if self.has_gpu else self.ram_available_bytes + + @property + def refusal_bytes(self) -> int: + """What physics allows at all. Only this refuses a model. + + On unified memory the device and the host are the same chips, so this is + host memory alone and never a sum with the device: adding the two counts + one memory twice, which is what gave an 8 GB Mac a 10.3 GB threshold. + + Host memory rather than the device's own figure, because Metal's working + set bounds what Metal will allocate, not what the machine can hold. + Measured: a 1.7B at 40960 projected 6032 MiB against a 5460 MiB working + set, and llama.cpp ran it with 23 of 29 layers on the CPU backend, which + reads the same chips without that ceiling. + + On a discrete card the two really are separate pools, and spilling from + one into the other is what the middle state means. + """ + if not self.has_gpu: + return self.ram_available_bytes + if self.uma: + return self.ram_available_bytes + return self.usable_vram_bytes + self.ram_available_bytes diff --git a/surfsense_local/backend/modules/llm/fit/copy.py b/surfsense_local/backend/modules/llm/fit/copy.py index 646f1ffd1c..6814db4a0c 100644 --- a/surfsense_local/backend/modules/llm/fit/copy.py +++ b/surfsense_local/backend/modules/llm/fit/copy.py @@ -18,9 +18,11 @@ from modules.llm.fit.estimate import FitVerdict from modules.llm.fit.states import FitState -# Above this the spill is no longer something a user can ignore. Below it, most -# of the model is still resident and the difference is barely noticeable. -_NOTICEABLE_SPILL = 0.35 +# Three bands, because the middle state spans barely noticeable to unusable and +# one sentence is wrong at both ends. Below the first, most of the model is still +# resident; above the second, most of it is not. +_SLIGHT_SPILL = 0.25 +_HEAVY_SPILL = 0.5 @dataclass(frozen=True) @@ -44,8 +46,11 @@ def badge(fit: FitVerdict, budget: HardwareBudget) -> Badge: def _spill(fit: FitVerdict, budget: HardwareBudget) -> str: where = "the GPU" if budget.uma else "the graphics card" - if fit.offload_fraction <= _NOTICEABLE_SPILL: + elsewhere = "the CPU" if budget.uma else "the processor" + if fit.offload_fraction <= _SLIGHT_SPILL: return f"A little too big for {where}. Most of it still fits." + if fit.offload_fraction < _HEAVY_SPILL: + return f"Too big for {where}, so part runs on {elsewhere}." return f"Well over {where}'s memory. Expect it to be slow." diff --git a/surfsense_local/backend/modules/llm/fit/estimate.py b/surfsense_local/backend/modules/llm/fit/estimate.py index c404419c10..e128c2b7f1 100644 --- a/surfsense_local/backend/modules/llm/fit/estimate.py +++ b/surfsense_local/backend/modules/llm/fit/estimate.py @@ -1,18 +1,25 @@ """Does this model fit in this machine, and if not, how much spills. - need = weights + KV(window) + compute_buffers what the model allocates - usable = device_free - fit_reserve what --fit will use + need = weights + KV(window) + compute_buffers what the model allocates + resident = what it must fit inside to run at full speed + refusal = what physics allows at all The two subtractions sit on opposite sides of the comparison. Collapsing them into one `overhead` term predicts residency for a configuration that measurably spills, which is the specific failure this estimator exists to avoid. + +`resident` and `refusal` come from the budget rather than being assembled here, +because what they mean depends on the machine: they are two pools on a discrete +card, one pool on unified memory, and the same number on a machine with no GPU +at all. Assembling them here is what badged every model on a CPU only laptop as +spilling from a graphics card it does not have. """ from dataclasses import dataclass from modules.llm.fit.budget import HardwareBudget -from modules.llm.fit.compute_buffers import compute_buffer_bytes -from modules.llm.fit.kv_cache import kv_cache_bytes +from modules.llm.fit.itemisation import itemise +from modules.llm.fit.offload import offload_fraction from modules.llm.fit.states import FitState from modules.llm.fit.types import KvPrecision, ModelShape @@ -48,50 +55,53 @@ def estimate( a model that will not fit at 16K cannot be rescued by a smaller context, and the remedy to offer is a smaller build. """ - # The projector is counted here because `--fit` does not count it: - # llama.cpp issue #19980. Left out, a vision model our sum calls resident - # can still fail to allocate on a tight machine. - need = ( - weights_bytes - + mmproj_bytes - + kv_cache_bytes(shape, n_ctx, precision) - + compute_buffer_bytes(n_ctx) - ) - usable = budget.usable_vram_bytes + items = itemise(shape, weights_bytes, n_ctx, precision, mmproj_bytes) + need = items.total + # Two questions, two numbers, and the budget owns the difference between + # them. On a discrete card residency is the card and physics is the card plus + # the host; on unified memory both are the one pool; without a GPU both are + # the processor's own memory, which is what makes the middle state + # unreachable there rather than something the states have to special case. + resident = budget.resident_bytes + refusal = budget.refusal_bytes - floor_need = ( - weights_bytes - + mmproj_bytes - + kv_cache_bytes(shape, CONTEXT_FLOOR_TOKENS, precision) - + compute_buffer_bytes(CONTEXT_FLOOR_TOKENS) - ) - if floor_need > usable + budget.ram_available_bytes: + floor = itemise(shape, weights_bytes, CONTEXT_FLOOR_TOKENS, precision, mmproj_bytes) + if floor.total > refusal: return FitVerdict( state=FitState.TOO_BIG, need_bytes=need, - budget_bytes=usable + budget.ram_available_bytes, + budget_bytes=refusal, offload_fraction=1.0, ) - if need <= usable: + if need <= resident: return FitVerdict( state=FitState.FITS, need_bytes=need, - budget_bytes=usable, + budget_bytes=resident, offload_fraction=0.0, ) - # > ponytail: this is the theoretical minimum spill, and llama.cpp spills in - # > whole layers, so reality rounds up. Measured on an RTX 3050, Qwen3 4B - # > spilled 602 MiB of weights plus 320 MiB of cache, f = 0.19, where this - # > arithmetic gives 0.12. The states agree and the ordering agrees; the - # > fraction does not. It matters because the badge copy is graded on it at - # > 0.25 and 0.5, so calibrate against layer-granular placement before those - # > thresholds are trusted. - spilled = need - usable + if need > refusal: + # Past the floor check, which asked whether a smaller window would + # rescue this model. It would not rescue this *window*, and there is no + # third place to put the excess, so the requested window is refused + # rather than described as a partial offload. + # + # Reachable only where residency and physics are the same number, which + # means a machine with no GPU: the model is already entirely on the + # processor, so calling it partly offloaded names a device that is not + # there. Found by the property sweep, not by a fixture. + return FitVerdict( + state=FitState.TOO_BIG, + need_bytes=need, + budget_bytes=refusal, + offload_fraction=1.0, + ) + return FitVerdict( state=FitState.PARTIAL, need_bytes=need, - budget_bytes=usable, - offload_fraction=min(1.0, spilled / need), + budget_bytes=resident, + offload_fraction=offload_fraction(shape, items, resident), ) diff --git a/surfsense_local/backend/tests/unit/llm/fit/test_badge_copy.py b/surfsense_local/backend/tests/unit/llm/fit/test_badge_copy.py index 27355e3938..040b6c29c6 100644 --- a/surfsense_local/backend/tests/unit/llm/fit/test_badge_copy.py +++ b/surfsense_local/backend/tests/unit/llm/fit/test_badge_copy.py @@ -28,6 +28,10 @@ def test_no_user_facing_string_uses_an_em_dash_or_a_hyphen() -> None: for state in FitState: if state is FitState.PARTIAL and not budget.has_gpu: continue + for fraction in (0.1, 0.35, 0.7): + text = badge(verdict(state, fraction), budget) + assert "—" not in text.verdict + text.reason + assert "-" not in text.verdict + text.reason text = badge(verdict(state), budget) assert "—" not in text.verdict + text.reason assert "-" not in text.verdict + text.reason @@ -50,20 +54,34 @@ def test_apple_silicon_is_never_told_about_system_ram() -> None: """There is no separate pool to spill into. Some layers run on the CPU backend against the same physical memory, so "uses system RAM" would describe a transfer that does not happen.""" - text = badge(verdict(FitState.PARTIAL, 0.3), APPLE) + for fraction in (0.1, 0.35, 0.7): + text = badge(verdict(FitState.PARTIAL, fraction), APPLE) - assert "system RAM" not in text.reason - assert "graphics card" not in text.reason + assert "system RAM" not in text.reason + assert "graphics card" not in text.reason + assert "processor" not in text.reason -def test_a_small_spill_and_a_large_one_do_not_get_the_same_sentence() -> None: +def test_the_three_bands_each_get_their_own_sentence() -> None: """PARTIAL spans barely noticeable to unusable, and the fraction is already - on the verdict. One sentence is wrong at both ends.""" + on the verdict. One sentence is wrong at both ends, and two leave the middle + of the range described as though it were one extreme or the other.""" slight = badge(verdict(FitState.PARTIAL, 0.1), DISCRETE) + middling = badge(verdict(FitState.PARTIAL, 0.35), DISCRETE) severe = badge(verdict(FitState.PARTIAL, 0.7), DISCRETE) - assert slight.verdict == severe.verdict == "Reduced speed" - assert slight.reason != severe.reason + assert slight.verdict == middling.verdict == severe.verdict == "Reduced speed" + assert len({slight.reason, middling.reason, severe.reason}) == 3 + + +def test_the_bands_sit_where_the_spec_put_them() -> None: + """A quarter and a half, not a single threshold between them.""" + assert badge(verdict(FitState.PARTIAL, 0.25), DISCRETE).reason == badge( + verdict(FitState.PARTIAL, 0.1), DISCRETE + ).reason + assert badge(verdict(FitState.PARTIAL, 0.5), DISCRETE).reason == badge( + verdict(FitState.PARTIAL, 0.9), DISCRETE + ).reason def test_a_refusal_states_both_numbers() -> None: diff --git a/surfsense_local/backend/tests/unit/llm/fit/test_estimate.py b/surfsense_local/backend/tests/unit/llm/fit/test_estimate.py index 58cd7be975..40b3e4d82c 100644 --- a/surfsense_local/backend/tests/unit/llm/fit/test_estimate.py +++ b/surfsense_local/backend/tests/unit/llm/fit/test_estimate.py @@ -29,16 +29,30 @@ ) WEIGHTS = 1050 * MIB # MTL0_Mapped model buffer, measured -# MTL0: 5461.3 MiB total, 5460 MiB free as the fitter saw it. +# MTL0: 5461.3 MiB total, 5460 MiB free as the fitter saw it, on an 8 GiB +# machine. Two ceilings on one memory, not two pools: Metal will not allocate +# past its working set, and the CPU backend reads the same chips without that +# limit, which is why a projection above 5460 still loaded with layers spilled. M2 = HardwareBudget( device_free_bytes=5460 * MIB, device_total_bytes=5461 * MIB, fit_reserve_bytes=1024 * MIB, - ram_available_bytes=2000 * MIB, + ram_available_bytes=6144 * MIB, uma=True, has_gpu=True, ) +# An older laptop with no card ggml can use. Nothing to spill from, so the +# processor's memory is both where the model runs and all there is. +NO_GPU = HardwareBudget( + device_free_bytes=0, + device_total_bytes=0, + fit_reserve_bytes=1024 * MIB, + ram_available_bytes=14 * 1024 * MIB, + uma=False, + has_gpu=False, +) + def test_the_measured_resident_case_fits() -> None: """`projected to use 2944 MiB vs 5460 MiB free`, and it loaded 29/29.""" @@ -137,3 +151,47 @@ def test_a_projector_can_be_what_tips_a_model_over() -> None: assert resident.state is FitState.FITS assert with_projector.state is FitState.PARTIAL + + +def test_without_a_gpu_a_model_inside_memory_simply_fits() -> None: + """The measured bug: every row on a CPU only machine read `Reduced speed`. + + There is no graphics card to be too big for, so a model that fits in memory + runs, at the only speed this machine has. Calling that a partial offload + described a spill from a device the machine does not have, and the badge + named hardware the user could see they did not own. + """ + verdict = estimate(QWEN3_1_7B, WEIGHTS, NO_GPU, n_ctx=16384) + + assert verdict.state is FitState.FITS + assert verdict.offload_fraction == 0.0 + + +def test_without_a_gpu_the_middle_state_is_unreachable() -> None: + """Swept rather than sampled: partial means some layers moved to the CPU, + and on this machine every layer was already there.""" + for gib in range(1, 40): + state = estimate(QWEN3_1_7B, gib * 1024 * MIB, NO_GPU).state + + assert state in {FitState.FITS, FitState.TOO_BIG} + + +def test_without_a_gpu_physics_still_refuses_what_memory_cannot_hold() -> None: + """The one refusal that survives: a model larger than the machine.""" + verdict = estimate(QWEN3_1_7B, 20 * 1024 * MIB, NO_GPU) + + assert verdict.state is FitState.TOO_BIG + assert verdict.budget_bytes == NO_GPU.ram_available_bytes + + +def test_unified_memory_refuses_past_the_pool_not_past_the_pool_twice_over() -> None: + """The measured bug: one memory counted as two. + + An 8 GB Mac was priced with a 10.3 GB refusal threshold, because the working + set and the host reading, which describe the same chips, were added. A model + between the pool and that sum was offered to a machine that cannot hold it. + """ + need = estimate(QWEN3_1_7B, 5000 * MIB, M2).need_bytes + + assert need > M2.device_free_bytes + assert estimate(QWEN3_1_7B, 5000 * MIB, M2).state is FitState.TOO_BIG diff --git a/surfsense_local/backend/tests/unit/llm/fit/test_properties.py b/surfsense_local/backend/tests/unit/llm/fit/test_properties.py new file mode 100644 index 0000000000..8faab152ee --- /dev/null +++ b/surfsense_local/backend/tests/unit/llm/fit/test_properties.py @@ -0,0 +1,205 @@ +"""Properties the estimator has to hold whatever it is asked. + +The tests beside this one assert numbers, which a self-consistent wrong +estimator can also satisfy. These assert relationships instead, over a grid of +shapes, windows, precisions and machines, and they are the kind of check that +would have caught both of the verdict bugs this work started from: one of them +priced a machine with no GPU as spilling, and the other added a unified memory +pool to itself. + +Borrowed in design from Unsloth Studio's platform matrix, which asks the same +question: not "is the number N", but "is this estimator internally honest". +""" + +from itertools import pairwise + +import pytest + +from modules.llm.fit import ( + CONTEXT_FLOOR_TOKENS, + FitState, + HardwareBudget, + KvPrecision, + ModelShape, + estimate, + itemise, +) +from modules.llm.hardware import BudgetMode, build_budget + +pytestmark = pytest.mark.unit + +MIB = 1024**2 +GB = 1000**3 + +SHAPES = { + "qwen3-1.7b": ModelShape( + "qwen3", 28, 8, 128, 128, 40960, 151936, + embedding_length=2048, feed_forward_length=6144, + ), + "qwen3-4b": ModelShape( + "qwen3", 36, 8, 128, 128, 40960, 151936, + embedding_length=2560, feed_forward_length=9728, + ), + "gemma3-swa": ModelShape( + "gemma3", 48, 8, 128, 128, 131072, 262144, + sliding_window=1024, embedding_length=3840, feed_forward_length=15360, + ), + "deepseek-mla": ModelShape( + "deepseek2", 28, 1, 192, 128, 163840, 129280, + embedding_length=2048, feed_forward_length=10944, + kv_lora_rank=512, key_length_mla=64, + ), +} + +BUDGETS = { + # A 6 GB card with the host behind it: the only shape where residency and + # physics are genuinely two different pools. + "discrete-6gb": HardwareBudget(5234 * MIB, 6002 * MIB, 1024 * MIB, 22750 * MIB, False, True), + "discrete-24gb": HardwareBudget(23000 * MIB, 24576 * MIB, 1024 * MIB, 32000 * MIB, False, True), + # Unified memory states one pool twice, which is what `build_budget` now + # guarantees and what the second verdict bug got wrong. + "unified-8gb": HardwareBudget(5222 * MIB, 5461 * MIB, 1024 * MIB, 5222 * MIB, True, True), + "no-gpu-16gb": HardwareBudget(0, 0, 1024 * MIB, 14 * 1024 * MIB, False, False), + # What a machine with no staged runtime produces. + "probed-empty": build_budget([], 8 * 1024 * MIB, mode=BudgetMode.CAPACITY), +} + +WINDOWS = (8192, 16384, 16385, 32768, 131072) +PRECISIONS = (KvPrecision.F16, KvPrecision.Q8_0) +WEIGHTS = (int(0.4 * GB), int(2.5 * GB), int(9.0 * GB), int(19.76 * GB)) + +EVERY_CASE = [ + (shape_name, budget_name, n_ctx, precision, weights) + for shape_name in SHAPES + for budget_name in BUDGETS + for n_ctx in WINDOWS + for precision in PRECISIONS + for weights in WEIGHTS +] + + +@pytest.mark.parametrize(("shape_name", "budget_name", "n_ctx", "precision", "weights"), EVERY_CASE) +def test_the_itemisation_sums_to_what_the_verdict_reports( + shape_name, budget_name, n_ctx, precision, weights +) -> None: + """If these diverge, one of the two is describing a different model.""" + shape = SHAPES[shape_name] + items = itemise(shape, weights, n_ctx, precision) + verdict = estimate(SHAPES[shape_name], weights, BUDGETS[budget_name], n_ctx=n_ctx, precision=precision) + + assert items.total == verdict.need_bytes + + +@pytest.mark.parametrize(("shape_name", "budget_name", "n_ctx", "precision", "weights"), EVERY_CASE) +def test_nothing_is_ever_negative( + shape_name, budget_name, n_ctx, precision, weights +) -> None: + """A negative byte count is a subtraction that went the wrong way, and it + would reach the screen as a badge claiming a model needs less than nothing. + """ + verdict = estimate(SHAPES[shape_name], weights, BUDGETS[budget_name], n_ctx=n_ctx, precision=precision) + + assert verdict.need_bytes >= 0 + assert verdict.budget_bytes >= 0 + assert 0.0 <= verdict.offload_fraction <= 1.0 + + +@pytest.mark.parametrize("shape_name", SHAPES) +@pytest.mark.parametrize("precision", PRECISIONS) +def test_the_weights_term_never_moves_with_the_window(shape_name, precision) -> None: + """The load bearing one. Weights and cache are priced by separate modules, + and a window that moved the weights would mean they had started naming the + same bytes.""" + shape = SHAPES[shape_name] + at_each = [itemise(shape, 2500 * MIB, n_ctx, precision) for n_ctx in WINDOWS] + + assert len({items.weights_bytes for items in at_each}) == 1 + + +@pytest.mark.parametrize("shape_name", SHAPES) +@pytest.mark.parametrize("precision", PRECISIONS) +def test_cache_and_scratch_never_shrink_as_the_window_grows(shape_name, precision) -> None: + """Both are sized from the window, so a wider one cannot cost less.""" + shape = SHAPES[shape_name] + priced = [itemise(shape, 2500 * MIB, n_ctx, precision) for n_ctx in sorted(WINDOWS)] + + for narrower, wider in pairwise(priced): + assert wider.kv_bytes >= narrower.kv_bytes + assert wider.compute_bytes >= narrower.compute_bytes + + +@pytest.mark.parametrize("budget_name", BUDGETS) +def test_residency_is_never_more_generous_than_physics(budget_name) -> None: + """Fitting entirely on the device is a stronger claim than merely being + possible, so the budget it is measured against cannot be the larger one.""" + budget = BUDGETS[budget_name] + + assert budget.resident_bytes <= budget.refusal_bytes + + +@pytest.mark.parametrize(("shape_name", "n_ctx", "precision", "weights"), + [(s, c, p, w) for s in SHAPES for c in WINDOWS for p in PRECISIONS for w in WEIGHTS]) +def test_a_machine_with_no_gpu_never_reports_a_partial_offload( + shape_name, n_ctx, precision, weights +) -> None: + """The first verdict bug, as a property. There is no device to spill from, + so every layer was already on the processor.""" + verdict = estimate(SHAPES[shape_name], weights, BUDGETS["no-gpu-16gb"], n_ctx=n_ctx, precision=precision) + + assert verdict.state in {FitState.FITS, FitState.TOO_BIG} + assert verdict.offload_fraction in {0.0, 1.0} + + +@pytest.mark.parametrize(("shape_name", "n_ctx", "precision", "weights"), + [(s, c, p, w) for s in SHAPES for c in WINDOWS for p in PRECISIONS for w in WEIGHTS]) +def test_unified_memory_is_never_refused_later_than_its_own_pool( + shape_name, n_ctx, precision, weights +) -> None: + """The second verdict bug, as a property. The device reading and the host + reading name one memory, so a refusal threshold above either of them is that + memory counted twice.""" + budget = BUDGETS["unified-8gb"] + verdict = estimate(SHAPES[shape_name], weights, budget, n_ctx=n_ctx, precision=precision) + + if verdict.state is FitState.TOO_BIG: + assert verdict.budget_bytes <= budget.device_free_bytes + + +@pytest.mark.parametrize(("shape_name", "budget_name", "precision", "weights"), + [(s, b, p, w) for s in SHAPES for b in BUDGETS for p in PRECISIONS for w in WEIGHTS]) +def test_moving_work_off_the_device_never_shrinks_what_a_model_needs( + shape_name, budget_name, precision, weights +) -> None: + """An offloaded byte is not a freed byte. On unified memory in particular, + a layer on the processor sits in the same chips it sat in before.""" + verdict = estimate(SHAPES[shape_name], weights, BUDGETS[budget_name], precision=precision) + items = itemise(SHAPES[shape_name], weights, CONTEXT_FLOOR_TOKENS, precision) + + assert verdict.need_bytes == items.total + assert verdict.need_bytes >= weights + + +def test_a_machine_with_no_staged_runtime_prices_against_the_host() -> None: + """A probe that found nothing must not read as a device with no memory.""" + budget = BUDGETS["probed-empty"] + + assert budget.has_gpu is False + assert budget.usable_vram_bytes == 0 + assert budget.resident_bytes == budget.ram_available_bytes + + +@pytest.mark.parametrize(("shape_name", "budget_name", "precision", "weights"), + [(s, b, p, w) for s in SHAPES for b in BUDGETS for p in PRECISIONS for w in WEIGHTS]) +def test_a_partial_offload_is_always_a_whole_number_of_layers( + shape_name, budget_name, precision, weights +) -> None: + """llama.cpp fills devices with whole layers, so a fraction that is not a + multiple of one layer describes a placement the runtime cannot make.""" + shape = SHAPES[shape_name] + verdict = estimate(shape, weights, BUDGETS[budget_name], precision=precision) + if verdict.state is not FitState.PARTIAL: + return + + layers = verdict.offload_fraction * shape.block_count + + assert layers == pytest.approx(round(layers)) From 39cd1bb1c339a7ea3fe168d218c41924fd31b725 Mon Sep 17 00:00:00 2001 From: Anish Sarkar <104695310+AnishSarkar22@users.noreply.github.com> Date: Mon, 21 Sep 2026 16:59:41 +0530 Subject: [PATCH 37/60] feat(fit): choose one cache precision the badge and the loader share --- .../backend/modules/llm/fit/__init__.py | 8 ++ .../backend/modules/llm/fit/plan_load.py | 81 ++++++++++++++---- .../backend/modules/llm/fit/precision.py | 85 +++++++++++++++++++ .../tests/unit/llm/fit/test_plan_load.py | 15 +++- 4 files changed, 169 insertions(+), 20 deletions(-) create mode 100644 surfsense_local/backend/modules/llm/fit/precision.py diff --git a/surfsense_local/backend/modules/llm/fit/__init__.py b/surfsense_local/backend/modules/llm/fit/__init__.py index fee0b839be..29db150a7a 100644 --- a/surfsense_local/backend/modules/llm/fit/__init__.py +++ b/surfsense_local/backend/modules/llm/fit/__init__.py @@ -2,8 +2,11 @@ from modules.llm.fit.compute_buffers import compute_buffer_bytes from modules.llm.fit.copy import Badge, badge from modules.llm.fit.estimate import CONTEXT_FLOOR_TOKENS, FitVerdict, estimate +from modules.llm.fit.itemisation import NeedItems, itemise from modules.llm.fit.kv_cache import kv_cache_bytes +from modules.llm.fit.offload import offload_fraction from modules.llm.fit.plan_load import LoadPlan, plan_load +from modules.llm.fit.precision import planned_precision, resident_precision from modules.llm.fit.states import FitState from modules.llm.fit.types import KvPrecision, ModelShape @@ -16,9 +19,14 @@ "KvPrecision", "LoadPlan", "ModelShape", + "NeedItems", "badge", "compute_buffer_bytes", "estimate", + "itemise", "kv_cache_bytes", + "offload_fraction", "plan_load", + "planned_precision", + "resident_precision", ] diff --git a/surfsense_local/backend/modules/llm/fit/plan_load.py b/surfsense_local/backend/modules/llm/fit/plan_load.py index c6f8ad4546..58865c3b37 100644 --- a/surfsense_local/backend/modules/llm/fit/plan_load.py +++ b/surfsense_local/backend/modules/llm/fit/plan_load.py @@ -15,6 +15,7 @@ from modules.llm.fit.budget import HardwareBudget from modules.llm.fit.estimate import CONTEXT_FLOOR_TOKENS, FitVerdict, estimate +from modules.llm.fit.precision import FALLBACK, resident_precision from modules.llm.fit.states import FitState from modules.llm.fit.types import KvPrecision, ModelShape @@ -31,6 +32,8 @@ def plan_load( weights_bytes: int, budget: HardwareBudget, live: HardwareBudget | None = None, + *, + mmproj_bytes: int = 0, ) -> LoadPlan: """Choose the widest window that stays resident, at the cheapest precision. @@ -47,35 +50,71 @@ def plan_load( Without that split, a 1.7B took a 28,672 token window on an 8 GB Mac with 2.3 GB reclaimable, the load paged, and it took 43 seconds. + + A cap in both directions. `live` can be the more generous of the two on an + idle machine, and widening to fit a moment of free memory commits a window + that only fits while nothing else is running, because the cache is allocated + once at load and never shrinks. So widening stays inside whichever budget is + tighter, which is also what keeps the verdict equal to the badge. """ ceiling = shape.context_length or CONTEXT_FLOOR_TOKENS floor = min(CONTEXT_FLOOR_TOKENS, ceiling) - for precision in (KvPrecision.F16, KvPrecision.Q8_0): - at_floor = estimate( - shape, weights_bytes, budget, n_ctx=floor, precision=precision - ) - if at_floor.state is not FitState.FITS: - continue - n_ctx = _widest_resident( - shape, weights_bytes, live or budget, precision, ceiling, floor - ) + # The same rule the badge is drawn from, so the screen and the load cannot + # describe different configurations of the same model. + precision = resident_precision( + shape, weights_bytes, budget, n_ctx=floor, mmproj_bytes=mmproj_bytes + ) + + if precision is None: + # Nothing keeps it resident, so hold the floor and let --fit place the + # layers. A cheaper cache buys nothing once layers are spilling anyway. return LoadPlan( - n_ctx=n_ctx, - precision=precision, + n_ctx=floor, + precision=FALLBACK, verdict=estimate( - shape, weights_bytes, budget, n_ctx=n_ctx, precision=precision + shape, + weights_bytes, + budget, + n_ctx=floor, + precision=FALLBACK, + mmproj_bytes=mmproj_bytes, ), ) - # Nothing keeps it resident, so hold the floor and let --fit place the layers. + # Widened against whichever budget is tighter, because `live` is a cap and a + # cap must not raise a ceiling. On a busy machine that is `live`, which is + # what stops a window being sized against memory the OS is already using. On + # an idle one it is `budget`: widening to fit this moment's free memory + # would commit a window that only fits while nothing else is running, and + # the cache is allocated once at load and never shrinks. + # + # It also keeps the verdict below honest. Widening against a more generous + # `live` and then reporting against `budget` produced a plan that said + # PARTIAL for a row the catalog had badged FITS. + widening = budget if live is None else min(budget, live, key=_headroom) + n_ctx = _widest_resident( + shape, weights_bytes, widening, precision, ceiling, floor, mmproj_bytes + ) return LoadPlan( - n_ctx=floor, - precision=KvPrecision.F16, - verdict=estimate(shape, weights_bytes, budget, n_ctx=floor), + n_ctx=n_ctx, + precision=precision, + verdict=estimate( + shape, + weights_bytes, + budget, + n_ctx=n_ctx, + precision=precision, + mmproj_bytes=mmproj_bytes, + ), ) +def _headroom(budget: HardwareBudget) -> int: + """What a widening search has to stay inside: the resident ceiling.""" + return budget.resident_bytes + + def _widest_resident( shape: ModelShape, weights_bytes: int, @@ -83,6 +122,7 @@ def _widest_resident( precision: KvPrecision, ceiling: int, floor: int, + mmproj_bytes: int = 0, ) -> int: """Binary search the window, in whole thousands of tokens.""" low, high = floor, ceiling @@ -92,7 +132,14 @@ def _widest_resident( if mid <= low: break fits = ( - estimate(shape, weights_bytes, budget, n_ctx=mid, precision=precision).state + estimate( + shape, + weights_bytes, + budget, + n_ctx=mid, + precision=precision, + mmproj_bytes=mmproj_bytes, + ).state is FitState.FITS ) if fits: diff --git a/surfsense_local/backend/modules/llm/fit/precision.py b/surfsense_local/backend/modules/llm/fit/precision.py new file mode 100644 index 0000000000..c1bc7e758a --- /dev/null +++ b/surfsense_local/backend/modules/llm/fit/precision.py @@ -0,0 +1,85 @@ +"""Which cache precision a build will actually be loaded with. + +One rule, in one place, because four callers need the same answer and disagreeing +about it is a screen that describes a load the app will not perform. The badge +said `Reduced speed` for a model the loader then placed entirely on the GPU, +purely because the badge priced f16 and the loader chose q8_0. + +The rule: prefer f16, and take q8_0 only where it converts a spill into +residency. Quality is identical either way, but a quantized cache needs a +working fused flash attention kernel, and where one is missing llama.cpp falls +back to CPU attention silently. That is a dependency worth taking to keep a +model on the device, and not worth taking for nothing. + +Evaluated at the context floor, because that is the window the app guarantees +and the one the verdict is about. Widening past it is opportunistic and belongs +to `plan_load`. + +> ponytail: whether q8_0 is the right trade at all is the open measurement in +> `07-llamacpp-runtime.md`. Unsloth measured a quantized cache at 35 % slower +> generation, which if it reproduces here means preferring a small spill instead. +> Until then the badge's job is to agree with the loader, whatever the loader +> does, which is exactly what this shared rule buys. +""" + +from modules.llm.fit.budget import HardwareBudget +from modules.llm.fit.estimate import CONTEXT_FLOOR_TOKENS, estimate +from modules.llm.fit.states import FitState +from modules.llm.fit.types import KvPrecision, ModelShape + +# In preference order. The first that keeps the model resident wins. +_LADDER = (KvPrecision.F16, KvPrecision.Q8_0) + +# What a load falls back to when no precision keeps the model resident. The +# cheaper cache buys nothing once layers are spilling anyway, so it would take +# the flash attention dependency for no gain. +FALLBACK = KvPrecision.F16 + + +def resident_precision( + shape: ModelShape, + weights_bytes: int, + budget: HardwareBudget, + *, + n_ctx: int = CONTEXT_FLOOR_TOKENS, + mmproj_bytes: int = 0, +) -> KvPrecision | None: + """The cheapest cache that keeps this build resident, or None if none does. + + None is meaningful: it says no cache choice rescues this model, so the + caller reports whatever f16 gives rather than pretending a quantized cache + changed the answer. + """ + for precision in _LADDER: + verdict = estimate( + shape, + weights_bytes, + budget, + n_ctx=n_ctx, + precision=precision, + mmproj_bytes=mmproj_bytes, + ) + if verdict.state is FitState.FITS: + return precision + return None + + +def planned_precision( + shape: ModelShape, + weights_bytes: int, + budget: HardwareBudget, + *, + n_ctx: int = CONTEXT_FLOOR_TOKENS, + mmproj_bytes: int = 0, +) -> KvPrecision: + """The precision a load will use, resident or not. + + What a badge wants: it has to name one, and it has to name the one the + loader would pick. + """ + return ( + resident_precision( + shape, weights_bytes, budget, n_ctx=n_ctx, mmproj_bytes=mmproj_bytes + ) + or FALLBACK + ) diff --git a/surfsense_local/backend/tests/unit/llm/fit/test_plan_load.py b/surfsense_local/backend/tests/unit/llm/fit/test_plan_load.py index 51d6bcc3c2..358d2be142 100644 --- a/surfsense_local/backend/tests/unit/llm/fit/test_plan_load.py +++ b/surfsense_local/backend/tests/unit/llm/fit/test_plan_load.py @@ -22,8 +22,15 @@ def budget(free_mib: int) -> HardwareBudget: - """A unified-memory device with llama.cpp's own one GiB margin.""" - return HardwareBudget(free_mib * MIB, free_mib * MIB, 1024 * MIB, 2000 * MIB, True, True) + """A unified-memory device with llama.cpp's own one GiB margin. + + The host reading equals the device reading because on unified memory they + name the same chips, which is what `build_budget` guarantees. Pairing them + with different numbers here would test a budget the app cannot produce. + """ + return HardwareBudget( + free_mib * MIB, free_mib * MIB, 1024 * MIB, free_mib * MIB, True, True + ) def test_a_roomy_machine_keeps_the_lossless_cache() -> None: @@ -52,7 +59,9 @@ def test_the_window_never_drops_below_the_floor() -> None: llama.cpp would reduce context to 4096 on its own if we left it unset, well under what a grounded turn needs, so the floor is ours to hold. """ - plan = plan_load(QWEN3_1_7B, 4200 * MIB, budget(5460)) + # Too large to stay resident at either precision, small enough that physics + # does not refuse it, so the floor is what holds the window up. + plan = plan_load(QWEN3_1_7B, 3500 * MIB, budget(5460)) assert plan.n_ctx == CONTEXT_FLOOR_TOKENS assert plan.verdict.state is FitState.PARTIAL From 86cc6f874182db8213beb5a355eb38d914926e48 Mon Sep 17 00:00:00 2001 From: Anish Sarkar <104695310+AnishSarkar22@users.noreply.github.com> Date: Mon, 21 Sep 2026 16:59:50 +0530 Subject: [PATCH 38/60] feat(hardware): probe ggml devices out of process --- surfsense_local/backend/main.py | 25 +++- .../modules/llm/hardware/device_lines.py | 61 +++++++++ .../backend/modules/llm/hardware/probe.py | 9 +- .../modules/llm/hardware/probe_script.py | 43 ++++++ .../modules/llm/hardware/probe_subprocess.py | 122 ++++++++++++++++++ .../unit/llm/hardware/test_device_lines.py | 46 +++++++ .../tests/unit/llm/hardware/test_probe.py | 60 ++++++++- 7 files changed, 357 insertions(+), 9 deletions(-) create mode 100644 surfsense_local/backend/modules/llm/hardware/device_lines.py create mode 100644 surfsense_local/backend/modules/llm/hardware/probe_script.py create mode 100644 surfsense_local/backend/modules/llm/hardware/probe_subprocess.py create mode 100644 surfsense_local/backend/tests/unit/llm/hardware/test_device_lines.py diff --git a/surfsense_local/backend/main.py b/surfsense_local/backend/main.py index f4d7da5638..edce7f4b95 100644 --- a/surfsense_local/backend/main.py +++ b/surfsense_local/backend/main.py @@ -23,8 +23,31 @@ def check_retrieval_runtime() -> None: ) +def probe_devices(library_dir: str) -> int: + """List this machine's ggml devices, one per line, and exit. + + The frozen build has no interpreter to run `-m` with, so the device probe + re-executes this binary behind this flag. Losing it does not break anything + visibly: the parent falls back to probing in process and the answer stays + correct, while a GPU driver moves back into the API process and, on a Mac, + a 19 second shader compile with it. + """ + from modules.llm.hardware.probe_script import main + + return main([library_dir]) + + if __name__ == "__main__": - if sys.argv[1:] == ["--check-retrieval-runtime"]: + arguments = sys.argv[1:] + if arguments == ["--check-retrieval-runtime"]: check_retrieval_runtime() + elif arguments[:1] == ["--probe-devices"]: + # Matched on the flag rather than on the whole shape, so a malformed + # probe is an error instead of falling through and starting a server + # that nothing is waiting on and nothing will stop. + if len(arguments) != 2: + sys.stderr.write("usage: --probe-devices \n") + sys.exit(2) + sys.exit(probe_devices(arguments[1])) else: serve() diff --git a/surfsense_local/backend/modules/llm/hardware/device_lines.py b/surfsense_local/backend/modules/llm/hardware/device_lines.py new file mode 100644 index 0000000000..397b20a72f --- /dev/null +++ b/surfsense_local/backend/modules/llm/hardware/device_lines.py @@ -0,0 +1,61 @@ +"""How the probe child reports a device to its parent. + +One line per device, tab separated, because the child is a separate process and +its stdout is shared with whatever ggml decides to log on the way past. A format +this plain survives a stray log line: an unparseable one is skipped rather than +taking the listing with it. +""" + +from modules.llm.hardware.devices import Device, DeviceType + +_FIELDS = 5 +_SEPARATOR = "\t" + + +def encode_line(device: Device) -> str: + """One device as one line. + + A description can carry anything a driver put in it, so the separator is + replaced rather than escaped: the fields are read positionally and a space + changes nothing a caller reads. + """ + return _SEPARATOR.join( + ( + _clean(device.name), + _clean(device.description), + str(int(device.type)), + str(device.total_bytes), + str(device.free_bytes), + ) + ) + + +def decode_lines(text: str) -> list[Device]: + """Every device the child reported, skipping anything that is not one. + + ggml logs to stdout on some backends, so a line that does not parse is far + more likely to be a log line than a device. Dropping it keeps the machine's + real devices rather than refusing the whole listing over a stray one. + """ + devices = [] + for line in text.splitlines(): + parts = line.split(_SEPARATOR) + if len(parts) != _FIELDS: + continue + name, description, raw_type, total, free = parts + try: + device = Device( + name=name, + description=description, + type=DeviceType.parse(int(raw_type)), + total_bytes=int(total), + free_bytes=int(free), + ) + except ValueError: + continue + devices.append(device) + return devices + + +def _clean(text: str) -> str: + return text.replace(_SEPARATOR, " ").replace("\n", " ") diff --git a/surfsense_local/backend/modules/llm/hardware/probe.py b/surfsense_local/backend/modules/llm/hardware/probe.py index bd2e2634cc..58961c1328 100644 --- a/surfsense_local/backend/modules/llm/hardware/probe.py +++ b/surfsense_local/backend/modules/llm/hardware/probe.py @@ -1,4 +1,9 @@ -"""Ask ggml what this machine has. +"""Ask ggml what this machine has, inside this process. + +The fallback path. `probe_subprocess` is what the app calls: loading a GPU +driver into the API process means a driver crash takes the API with it, and on +Metal it means a 19 second shader compile inside the server. This module stays +because a child that cannot be spawned is still better answered than not. **The scan runs from the directory of the running executable, not the directory the libraries were loaded from.** Verified on a machine with a working RTX 3050, @@ -35,7 +40,7 @@ def _working_directory(directory: Path): os.chdir(previous) -def probe_devices(library_dir: Path) -> list[Device]: +def probe_devices_in_process(library_dir: Path) -> list[Device]: """Every device ggml can see, in ggml's own preference order. An empty list means ggml found nothing, which is not the same as the machine diff --git a/surfsense_local/backend/modules/llm/hardware/probe_script.py b/surfsense_local/backend/modules/llm/hardware/probe_script.py new file mode 100644 index 0000000000..3667476c8b --- /dev/null +++ b/surfsense_local/backend/modules/llm/hardware/probe_script.py @@ -0,0 +1,43 @@ +"""The probe, as a child process. One line per device on stdout. + +Run rather than imported. Everything it needs is the library directory, given as +its one argument, and the parent sets the working directory so ggml's backend +scan finds the libraries beside them. + +Kept to the standard library and this package's own loader: this runs before the +app is up, and a heavy import here would be paid on every probe. +""" + +import sys +import traceback +from pathlib import Path + +from modules.llm.hardware.device_lines import encode_line +from modules.llm.hardware.probe import probe_devices_in_process + + +def main(argv: list[str]) -> int: + """Print this machine's devices, or say why not on stderr. + + Exit 1 with a traceback rather than an empty listing: zero devices is a + meaningful answer that the parent treats as a real machine with no GPU, and + a crash must not be mistaken for one. + """ + if len(argv) != 1: + sys.stderr.write("usage: probe_script \n") + return 2 + try: + devices = probe_devices_in_process(Path(argv[0])) + except Exception: # the parent reads the traceback, then falls back + traceback.print_exc(file=sys.stderr) + return 1 + + # Written rather than printed: this is the protocol between two processes, + # and the parent parses it line by line. + for device in devices: + sys.stdout.write(encode_line(device) + "\n") + return 0 + + +if __name__ == "__main__": # pragma: no cover - exercised as a subprocess + sys.exit(main(sys.argv[1:])) diff --git a/surfsense_local/backend/modules/llm/hardware/probe_subprocess.py b/surfsense_local/backend/modules/llm/hardware/probe_subprocess.py new file mode 100644 index 0000000000..0d6bd23c86 --- /dev/null +++ b/surfsense_local/backend/modules/llm/hardware/probe_subprocess.py @@ -0,0 +1,122 @@ +"""Ask ggml what this machine has, from a process that can afford to die. + +Three reasons the probe is not run in the API process, all of them measured: + +- `ggml_backend_load_all` scans the **running executable's own directory**, so + the child is spawned with its working directory set to the library folder. + In process that meant a global `chdir` under a lock, in a server that is + serving other requests. +- On Metal the first device query compiles twenty shader libraries, once, at 19 + seconds. In a child that is 19 seconds of a process nobody is waiting on. +- A GPU driver that faults takes its process with it. A child is a probe that + failed; the API process is the application. + +Unsloth runs its own Vulkan probe this way and for the first of those reasons. + +The in-process path stays as a fallback, because a child that cannot be spawned +at all is still a question worth answering. +""" + +import logging +import os +import subprocess +import sys +from pathlib import Path + +from modules.llm.hardware.device_lines import decode_lines +from modules.llm.hardware.devices import Device +from modules.llm.hardware.probe import probe_devices_in_process + +logger = logging.getLogger(__name__) + +# The flag the frozen binary dispatches on, mirroring the one the retrieval +# check already uses. A frozen build has no interpreter to run `-m` with, so it +# re-executes itself. +PROBE_FLAG = "--probe-devices" + +# Generous, because it is bounding the Metal shader compile rather than the +# probe: 19 seconds measured cold, and a machine slower than that still deserves +# an answer rather than a fallback. +DEFAULT_TIMEOUT_SECONDS = 90.0 + +_BACKEND_ROOT = Path(__file__).resolve().parents[3] + + +def probe_devices( + library_dir: Path, *, timeout_seconds: float = DEFAULT_TIMEOUT_SECONDS +) -> list[Device]: + """Every device ggml can see, in ggml's own preference order. + + An empty list means ggml found nothing, which is not the same as the machine + having nothing. `gpu_status` is what tells those apart. + + Raises `OSError` when there is no staged runtime to probe, which is the + contract the catalog relies on to render without one. + """ + if not library_dir.is_dir(): + raise OSError(f"no staged runtime at {library_dir}") + + # Resolved before it is handed over, because the child is started *in* this + # directory. A relative path would be read again against the new working + # directory, find nothing, and fall back in process: the exact behaviour + # this module exists to avoid, and silent, because the fallback answers. + library_dir = library_dir.resolve() + + try: + finished = subprocess.run( + _command(library_dir), + cwd=library_dir, + env=_environment(), + capture_output=True, + text=True, + timeout=timeout_seconds, + check=False, + ) + except (OSError, subprocess.SubprocessError) as error: + logger.warning("device probe could not run as a child process: %s", error) + return probe_devices_in_process(library_dir) + + if finished.returncode != 0: + logger.warning( + "device probe exited %s, falling back in process: %s", + finished.returncode, + finished.stderr.strip()[-500:], + ) + return probe_devices_in_process(library_dir) + + return decode_lines(finished.stdout) + + +def _command(library_dir: Path) -> list[str]: + """How to start the child, frozen or not. + + Frozen there is no interpreter and no importable package, so the binary + re-executes itself behind a flag its entry point dispatches on. + """ + if getattr(sys, "frozen", False): + return [sys.executable, PROBE_FLAG, str(library_dir)] + return [ + sys.executable, + "-m", + "modules.llm.hardware.probe_script", + str(library_dir), + ] + + +def _environment() -> dict[str, str]: + """The child's environment, with this backend importable. + + The working directory is the library folder, which is what ggml's scan + needs and what `-m` would otherwise resolve imports against. + """ + environment = dict(os.environ) + if getattr(sys, "frozen", False): + return environment + + existing = environment.get("PYTHONPATH", "") + root = str(_BACKEND_ROOT) + if root not in existing.split(os.pathsep): + environment["PYTHONPATH"] = ( + f"{root}{os.pathsep}{existing}" if existing else root + ) + return environment diff --git a/surfsense_local/backend/tests/unit/llm/hardware/test_device_lines.py b/surfsense_local/backend/tests/unit/llm/hardware/test_device_lines.py new file mode 100644 index 0000000000..2507f69622 --- /dev/null +++ b/surfsense_local/backend/tests/unit/llm/hardware/test_device_lines.py @@ -0,0 +1,46 @@ +"""The protocol between the probe child and its parent.""" + +import pytest + +from modules.llm.hardware import Device, DeviceType, decode_lines, encode_line + +pytestmark = pytest.mark.unit + +GIB = 1024**3 + + +def test_a_device_survives_the_round_trip() -> None: + """The contract, stated once.""" + device = Device("Vulkan0", "NVIDIA GeForce RTX 3050", DeviceType.GPU, 6 * GIB, 5 * GIB) + + assert decode_lines(encode_line(device)) == [device] + + +def test_a_separator_inside_a_description_does_not_shift_the_fields() -> None: + """A description carries whatever a driver put in it, and the fields are + read positionally, so a stray tab would move memory into the type column.""" + device = Device("Vulkan0", "Weird\tVendor\tCard", DeviceType.GPU, 6 * GIB, 5 * GIB) + + decoded = decode_lines(encode_line(device)) + + assert decoded[0].total_bytes == 6 * GIB + assert decoded[0].type is DeviceType.GPU + + +def test_a_log_line_on_the_same_stream_is_skipped_not_fatal() -> None: + """ggml logs to stdout on some backends, and losing the whole listing over + one of its lines would turn a working machine into a GPU-less one.""" + device = Device("MTL0", "Apple M2", DeviceType.GPU, 8 * GIB, 5 * GIB) + noisy = "ggml_metal_init: found device\n" + encode_line(device) + "\nload: done\n" + + assert decode_lines(noisy) == [device] + + +def test_a_device_type_the_enum_does_not_know_is_not_a_crash() -> None: + """ggml has added a member twice and will again, and this runs at startup.""" + assert decode_lines(f"X\tY\t99\t{GIB}\t{GIB}")[0].type is DeviceType.UNKNOWN + + +def test_nothing_at_all_decodes_to_nothing() -> None: + """An empty listing is a real answer: a machine ggml found no devices on.""" + assert decode_lines("") == [] diff --git a/surfsense_local/backend/tests/unit/llm/hardware/test_probe.py b/surfsense_local/backend/tests/unit/llm/hardware/test_probe.py index 6038d40322..e76bd30489 100644 --- a/surfsense_local/backend/tests/unit/llm/hardware/test_probe.py +++ b/surfsense_local/backend/tests/unit/llm/hardware/test_probe.py @@ -1,14 +1,25 @@ -"""The ctypes probe, against the libraries that actually ship. +"""The in-process ctypes probe, against the libraries that actually ship. + +The fallback path, used when a child process cannot be spawned. What the app +calls is `probe_devices`, which runs this in a child. Skipped when llama.cpp has not been staged, because there is nothing honest to assert without it: a mocked ggml would only prove the mock works. """ +import logging +import os from pathlib import Path import pytest -from modules.llm.hardware import DeviceType, load, probe_devices, select_device +from modules.llm.hardware import ( + DeviceType, + load, + probe_devices, + probe_devices_in_process, + select_device, +) pytestmark = pytest.mark.unit @@ -21,7 +32,7 @@ @staged def test_the_probe_finds_the_machines_devices() -> None: """Every supported host has at least a CPU device, so an empty list is a bug.""" - devices = probe_devices(LIBRARY_DIR) + devices = probe_devices_in_process(LIBRARY_DIR) assert devices assert any(d.type is DeviceType.CPU for d in devices) @@ -32,7 +43,7 @@ def test_the_probe_leaves_the_working_directory_where_it_found_it() -> None: """It chdirs to satisfy ggml's backend scan, and this process has other work.""" before = Path.cwd() - probe_devices(LIBRARY_DIR) + probe_devices_in_process(LIBRARY_DIR) assert Path.cwd() == before @@ -40,14 +51,14 @@ def test_the_probe_leaves_the_working_directory_where_it_found_it() -> None: @staged def test_a_device_type_outside_the_enum_never_reaches_a_caller() -> None: """Accelerate lists on every Mac at a value a three-member enum would reject.""" - for device in probe_devices(LIBRARY_DIR): + for device in probe_devices_in_process(LIBRARY_DIR): assert isinstance(device.type, DeviceType) @staged def test_the_selected_device_is_never_larger_than_a_real_one() -> None: """The guard against summing devices, asserted on whatever this host reports.""" - devices = probe_devices(LIBRARY_DIR) + devices = probe_devices_in_process(LIBRARY_DIR) chosen = select_device(devices) if chosen is not None: @@ -63,3 +74,40 @@ def test_one_handle_resolves_every_symbol_off_windows() -> None: assert libraries.core.ggml_backend_load_all assert libraries.base.ggml_backend_dev_memory + + +@staged +def test_the_child_process_sees_what_this_process_sees(caplog) -> None: + """The two paths must agree, and the child must be the one that answered. + + Comparing the results alone proves nothing: the fallback returns the same + devices, so a child that never ran passes a comparison. It was a relative + library path that broke this, spawned into a working directory it was then + resolved against, and the answer stayed correct while the Metal compile + moved back into this process. So the absence of the fallback warning is the + assertion that matters. + """ + with caplog.at_level(logging.WARNING, logger="modules.llm.hardware.probe_subprocess"): + devices = probe_devices(LIBRARY_DIR) + + assert devices == probe_devices_in_process(LIBRARY_DIR) + assert caplog.records == [] + + +@staged +def test_a_relative_library_directory_still_reaches_the_child(caplog) -> None: + """The caller passes whatever the settings hold, and that can be relative.""" + relative = Path(os.path.relpath(LIBRARY_DIR, Path.cwd())) + + with caplog.at_level(logging.WARNING, logger="modules.llm.hardware.probe_subprocess"): + devices = probe_devices(relative) + + assert devices + assert caplog.records == [] + + +def test_a_missing_runtime_is_an_oserror_not_an_empty_listing(tmp_path: Path) -> None: + """The catalog renders without a staged runtime by catching this. An empty + list would instead read as a machine that genuinely has no devices.""" + with pytest.raises(OSError): + probe_devices(tmp_path / "nowhere") From 7ed8f21b9196594809bcbf8a816885f067da6841 Mon Sep 17 00:00:00 2001 From: Anish Sarkar <104695310+AnishSarkar22@users.noreply.github.com> Date: Mon, 21 Sep 2026 17:00:07 +0530 Subject: [PATCH 39/60] feat(hardware): report whether the runtime can reach the GPU at all --- .../modules/llm/hardware/gpu_status.py | 51 ++++++++++++ .../backend/modules/llm/hardware/inventory.py | 51 ++++++++++++ .../modules/llm/hardware/os_gpu/__init__.py | 29 +++++++ .../modules/llm/hardware/os_gpu/darwin.py | 19 +++++ .../modules/llm/hardware/os_gpu/linux.py | 44 +++++++++++ .../modules/llm/hardware/os_gpu/windows.py | 63 +++++++++++++++ .../unit/llm/hardware/test_gpu_status.py | 51 ++++++++++++ .../tests/unit/llm/hardware/test_inventory.py | 64 +++++++++++++++ .../tests/unit/llm/hardware/test_os_gpu.py | 78 +++++++++++++++++++ 9 files changed, 450 insertions(+) create mode 100644 surfsense_local/backend/modules/llm/hardware/gpu_status.py create mode 100644 surfsense_local/backend/modules/llm/hardware/inventory.py create mode 100644 surfsense_local/backend/modules/llm/hardware/os_gpu/__init__.py create mode 100644 surfsense_local/backend/modules/llm/hardware/os_gpu/darwin.py create mode 100644 surfsense_local/backend/modules/llm/hardware/os_gpu/linux.py create mode 100644 surfsense_local/backend/modules/llm/hardware/os_gpu/windows.py create mode 100644 surfsense_local/backend/tests/unit/llm/hardware/test_gpu_status.py create mode 100644 surfsense_local/backend/tests/unit/llm/hardware/test_inventory.py create mode 100644 surfsense_local/backend/tests/unit/llm/hardware/test_os_gpu.py diff --git a/surfsense_local/backend/modules/llm/hardware/gpu_status.py b/surfsense_local/backend/modules/llm/hardware/gpu_status.py new file mode 100644 index 0000000000..591dcd12f3 --- /dev/null +++ b/surfsense_local/backend/modules/llm/hardware/gpu_status.py @@ -0,0 +1,51 @@ +"""Whether this machine has a GPU the runtime can reach, and if not, why not. + +ggml answers an empty device list with exit 0 on two very different machines: a +laptop with no graphics card, and a workstation whose backend library did not +ship. Measured on Windows and Linux with a working card and `ggml-cuda.dll` +staged but no cudart beside it, `--list-devices` prints `(none)` and exits 0, +and `GGML_BACKEND_DEBUG=1` changes nothing. + +Badging the second as the first is the failure this module exists to stop. The +user sees every model marked as running on the processor, with no suggestion +that the card they bought is sitting idle because a file is missing. +""" + +from collections.abc import Sequence +from enum import StrEnum + +from modules.llm.hardware.devices import Device, DeviceType + + +class GpuStatus(StrEnum): + """What to say about this machine's graphics hardware. + + `PRESENT` means ggml listed a GPU, which is not the same as the budget being + priced against one: an integrated part counts here and is still skipped by + `select_device`, because it carves from the same memory the host is using. + This answers "is the runtime seeing the hardware", not "what will hold the + layers". + """ + + PRESENT = "present" + ABSENT = "absent" + BROKEN_INSTALL = "broken_install" + UNKNOWN = "unknown" + + +def classify(devices: Sequence[Device], os_reports_gpu: bool | None) -> GpuStatus: + """Reconcile what ggml found with what the operating system can see.""" + if any(device.type in (DeviceType.GPU, DeviceType.IGPU) for device in devices): + return GpuStatus.PRESENT + + if os_reports_gpu: + return GpuStatus.BROKEN_INSTALL + if os_reports_gpu is False: + return GpuStatus.ABSENT + + # The OS could not say. A CPU device means ggml at least loaded and looked, + # so no GPU is its genuine answer; nothing at all means the probe told us + # nothing and neither will we. + if any(device.type is DeviceType.CPU for device in devices): + return GpuStatus.ABSENT + return GpuStatus.UNKNOWN diff --git a/surfsense_local/backend/modules/llm/hardware/inventory.py b/surfsense_local/backend/modules/llm/hardware/inventory.py new file mode 100644 index 0000000000..02e41bd2ac --- /dev/null +++ b/surfsense_local/backend/modules/llm/hardware/inventory.py @@ -0,0 +1,51 @@ +"""What this machine has, taken once and answered with a diagnosis. + +The devices and the verdict about them travel together because they are read +together and are only meaningful together: an empty device list means nothing +until you know whether the operating system sees a card. +""" + +import logging +from collections.abc import Callable, Sequence +from dataclasses import dataclass +from pathlib import Path + +from modules.llm.hardware.devices import Device +from modules.llm.hardware.gpu_status import GpuStatus, classify +from modules.llm.hardware.os_gpu import os_reports_gpu +from modules.llm.hardware.probe_subprocess import probe_devices + +logger = logging.getLogger(__name__) + +Probe = Callable[[Path], Sequence[Device]] +OsGpu = Callable[[], bool | None] + + +@dataclass(frozen=True) +class SystemInventory: + devices: tuple[Device, ...] + gpu_status: GpuStatus + + +def system_inventory( + library_dir: Path, + *, + probe: Probe = probe_devices, + os_gpu: OsGpu = os_reports_gpu, +) -> SystemInventory: + """Probe this machine, and say what an empty answer means. + + A probe that raises is the strongest form of the same question the empty + list asks: ggml could not be loaded at all. If the operating system can see + a card, that is a broken install and saying so is the whole point. + + `probe` and `os_gpu` are arguments so a test can state a machine rather than + patch one into existence. Production passes neither. + """ + try: + devices = tuple(probe(library_dir)) + except OSError as error: + logger.warning("could not probe devices: %s", error) + devices = () + + return SystemInventory(devices=devices, gpu_status=classify(devices, os_gpu())) diff --git a/surfsense_local/backend/modules/llm/hardware/os_gpu/__init__.py b/surfsense_local/backend/modules/llm/hardware/os_gpu/__init__.py new file mode 100644 index 0000000000..0b774b67aa --- /dev/null +++ b/surfsense_local/backend/modules/llm/hardware/os_gpu/__init__.py @@ -0,0 +1,29 @@ +"""What the operating system says about this machine's graphics hardware. + +Asked for one reason: to tell a machine with no GPU apart from a machine whose +GPU the runtime cannot reach. Both look identical to ggml, which reports an +empty device list and exit 0 either way, so the OS is the second opinion. + +Every platform answers True, False, or None for "I cannot tell". None is a real +answer and is never collapsed into False, because False is what badges a machine +CPU only, and doing that to a broken install is the failure this exists to stop. +""" + +import sys + + +def os_reports_gpu() -> bool | None: + """True, False, or None when this platform cannot say.""" + if sys.platform == "darwin": + from modules.llm.hardware.os_gpu.darwin import reports_gpu + + return reports_gpu() + if sys.platform == "win32": + from modules.llm.hardware.os_gpu.windows import reports_gpu + + return reports_gpu() + if sys.platform.startswith("linux"): + from modules.llm.hardware.os_gpu.linux import reports_gpu + + return reports_gpu() + return None diff --git a/surfsense_local/backend/modules/llm/hardware/os_gpu/darwin.py b/surfsense_local/backend/modules/llm/hardware/os_gpu/darwin.py new file mode 100644 index 0000000000..fa8a4eb967 --- /dev/null +++ b/surfsense_local/backend/modules/llm/hardware/os_gpu/darwin.py @@ -0,0 +1,19 @@ +"""Whether macOS itself can see a graphics card. + +Apple Silicon always has one and Metal is part of the OS, so there is nothing to +query: a Mac that reports no Metal device has a staging problem, not a hardware +one. That is the whole value of the cross-check here. +""" + +import platform + + +def reports_gpu() -> bool | None: + """True on Apple Silicon, None anywhere else. + + None on Intel rather than False: the runtime we ship is arm64 only, so an + Intel Mac is a machine we cannot make a claim about either way. + """ + if platform.machine() == "arm64": + return True + return None diff --git a/surfsense_local/backend/modules/llm/hardware/os_gpu/linux.py b/surfsense_local/backend/modules/llm/hardware/os_gpu/linux.py new file mode 100644 index 0000000000..37c5a4b521 --- /dev/null +++ b/surfsense_local/backend/modules/llm/hardware/os_gpu/linux.py @@ -0,0 +1,44 @@ +"""Whether Linux itself can see a graphics card. + +Read from sysfs rather than by running a tool: `lspci` is not installed +everywhere, and every kernel with a DRM driver populates this directory. +""" + +from pathlib import Path + +SYSFS_ROOT = Path("/sys/class/drm") + +# PCI vendor ids that mean a virtual display rather than a card that can compute. +# A guest with one of these has no GPU to be missing, so reporting one would +# turn every VM into a broken install. +_VIRTUAL_VENDORS = frozenset( + { + "0x1234", # QEMU / Bochs + "0x1af4", # virtio + "0x15ad", # VMware + "0x80ee", # VirtualBox + "0x1b36", # QEMU QXL + } +) + + +def reports_gpu(sysfs_root: Path = SYSFS_ROOT) -> bool | None: + """True when a real card is present, False when none is, None when unknown. + + None rather than False if the directory is missing entirely: a kernel + without DRM tells us nothing about the hardware, and guessing False would + call a broken driver install a CPU only machine. + """ + if not sysfs_root.is_dir(): + return None + + for card in sorted(sysfs_root.glob("card[0-9]*")): + vendor = card / "device" / "vendor" + try: + value = vendor.read_text().strip().lower() + except OSError: + continue + if value not in _VIRTUAL_VENDORS: + return True + # Either no cards at all, or only virtual ones. Both mean no GPU to reach. + return False diff --git a/surfsense_local/backend/modules/llm/hardware/os_gpu/windows.py b/surfsense_local/backend/modules/llm/hardware/os_gpu/windows.py new file mode 100644 index 0000000000..06f544a502 --- /dev/null +++ b/surfsense_local/backend/modules/llm/hardware/os_gpu/windows.py @@ -0,0 +1,63 @@ +"""Whether Windows itself can see a graphics card. + +`Win32_VideoController` through PowerShell, because it needs no dependency and +is present on every supported build. This is the platform where the cross-check +earns its place: a missing backend DLL makes ggml report no devices, silently, +with exit 0, on a machine with a working card. +""" + +import subprocess + +_COMMAND = [ + "powershell", + "-NoProfile", + "-NonInteractive", + "-Command", + "Get-CimInstance Win32_VideoController | Select-Object -ExpandProperty Name", +] +_TIMEOUT_SECONDS = 15.0 + +# Adapters that are software, not hardware. The measured machine carried a +# Parsec virtual display beside its two real cards, and counting it would report +# a card on a machine that genuinely has none. +_VIRTUAL_MARKERS = ( + "parsec", + "virtual", + "remote desktop", + "basic display", + "basic render", + "citrix", + "vnc", + "idd", + "meta driver", +) + + +def parse_adapters(output: str) -> list[str]: + """The real adapters in a controller listing, virtual ones dropped.""" + adapters = [] + for line in output.splitlines(): + name = line.strip() + if not name: + continue + if any(marker in name.lower() for marker in _VIRTUAL_MARKERS): + continue + adapters.append(name) + return adapters + + +def reports_gpu() -> bool | None: + """True when a real adapter is listed, False when none is, None when unknown.""" + try: + finished = subprocess.run( + _COMMAND, + capture_output=True, + text=True, + timeout=_TIMEOUT_SECONDS, + check=False, + ) + except (OSError, subprocess.SubprocessError): + return None + if finished.returncode != 0: + return None + return bool(parse_adapters(finished.stdout)) diff --git a/surfsense_local/backend/tests/unit/llm/hardware/test_gpu_status.py b/surfsense_local/backend/tests/unit/llm/hardware/test_gpu_status.py new file mode 100644 index 0000000000..c3d1c8e380 --- /dev/null +++ b/surfsense_local/backend/tests/unit/llm/hardware/test_gpu_status.py @@ -0,0 +1,51 @@ +"""Telling a machine with no GPU apart from one whose GPU cannot be reached. + +ggml answers both with an empty device list and exit 0. Measured on Windows and +Linux with a working card and the backend library staged but its runtime +missing, `--list-devices` prints `(none)`, exits 0, and `GGML_BACKEND_DEBUG=1` +changes nothing. +""" + +import pytest + +from modules.llm.hardware import Device, DeviceType, GpuStatus, classify + +pytestmark = pytest.mark.unit + +GIB = 1024**3 +DISCRETE = Device("Vulkan0", "RTX 3050", DeviceType.GPU, 6 * GIB, 5 * GIB) +INTEGRATED = Device("Vulkan1", "Radeon", DeviceType.IGPU, 16 * GIB, 16 * GIB) +CPU = Device("CPU", "Ryzen 5", DeviceType.CPU, 32 * GIB, 22 * GIB) + + +def test_a_card_the_runtime_can_see_is_simply_present() -> None: + """The ordinary machine, and the only case with nothing to reconcile.""" + assert classify([DISCRETE, CPU], True) is GpuStatus.PRESENT + + +def test_no_card_and_the_os_agrees_is_a_cpu_only_machine() -> None: + """An older laptop. Nothing is wrong with it and nothing should say so.""" + assert classify([CPU], False) is GpuStatus.ABSENT + + +def test_no_card_while_the_os_sees_one_is_a_broken_install() -> None: + """The row this module exists for. Badging it CPU only tells a user their + card does not work, when a file is missing from our own package.""" + assert classify([CPU], True) is GpuStatus.BROKEN_INSTALL + + +def test_an_integrated_part_counts_as_the_runtime_seeing_hardware() -> None: + """It is skipped for pricing, because it carves from the host's memory, but + the runtime did reach it, so nothing is broken and nothing should be said.""" + assert classify([INTEGRATED, CPU], True) is GpuStatus.PRESENT + + +def test_an_os_that_cannot_say_still_trusts_a_loaded_runtime() -> None: + """ggml loaded and looked, and found only a processor. That is an answer.""" + assert classify([CPU], None) is GpuStatus.ABSENT + + +def test_nothing_from_either_side_is_admitted_rather_than_guessed() -> None: + """The probe failed and the OS declined. Calling that CPU only would be + inventing the one answer that reads as a verdict about the hardware.""" + assert classify([], None) is GpuStatus.UNKNOWN diff --git a/surfsense_local/backend/tests/unit/llm/hardware/test_inventory.py b/surfsense_local/backend/tests/unit/llm/hardware/test_inventory.py new file mode 100644 index 0000000000..6c11a4886e --- /dev/null +++ b/surfsense_local/backend/tests/unit/llm/hardware/test_inventory.py @@ -0,0 +1,64 @@ +"""Probing once, and saying what an empty answer means.""" + +from pathlib import Path + +import pytest + +from modules.llm.hardware import Device, DeviceType, GpuStatus, system_inventory + +pytestmark = pytest.mark.unit + +GIB = 1024**3 +CPU = Device("CPU", "Ryzen 5", DeviceType.CPU, 32 * GIB, 22 * GIB) +DISCRETE = Device("Vulkan0", "RTX 3050", DeviceType.GPU, 6 * GIB, 5 * GIB) +NOWHERE = Path("/nowhere") + + +def test_a_probe_that_finds_a_card_reports_it_present() -> None: + """The ordinary machine: devices and verdict travel together.""" + inventory = system_inventory( + NOWHERE, probe=lambda _: [DISCRETE, CPU], os_gpu=lambda: True + ) + + assert inventory.devices == (DISCRETE, CPU) + assert inventory.gpu_status is GpuStatus.PRESENT + + +def test_a_card_the_os_sees_and_the_runtime_does_not_is_reported_broken() -> None: + """Phase 8.3: this must never be badged as a machine without a GPU.""" + inventory = system_inventory(NOWHERE, probe=lambda _: [CPU], os_gpu=lambda: True) + + assert inventory.gpu_status is GpuStatus.BROKEN_INSTALL + + +def test_a_runtime_that_will_not_load_at_all_is_the_same_diagnosis() -> None: + """The strongest form of the same question. ggml could not be loaded, the + operating system can see a card, so the install is what is wrong.""" + + def raises(_: Path) -> list[Device]: + raise OSError("no staged runtime") + + inventory = system_inventory(NOWHERE, probe=raises, os_gpu=lambda: True) + + assert inventory.devices == () + assert inventory.gpu_status is GpuStatus.BROKEN_INSTALL + + +def test_a_missing_runtime_on_a_machine_with_no_card_is_not_called_broken() -> None: + """Nothing to reach and nothing reaching it. Two absences, not a fault.""" + + def raises(_: Path) -> list[Device]: + raise OSError("no staged runtime") + + inventory = system_inventory(NOWHERE, probe=raises, os_gpu=lambda: False) + + assert inventory.gpu_status is GpuStatus.ABSENT + + +def test_a_failed_probe_never_raises_out_of_the_inventory() -> None: + """The catalog renders without a staged runtime, priced against the host.""" + + def raises(_: Path) -> list[Device]: + raise OSError("no staged runtime") + + assert system_inventory(NOWHERE, probe=raises, os_gpu=lambda: None).devices == () diff --git a/surfsense_local/backend/tests/unit/llm/hardware/test_os_gpu.py b/surfsense_local/backend/tests/unit/llm/hardware/test_os_gpu.py new file mode 100644 index 0000000000..4239f41c21 --- /dev/null +++ b/surfsense_local/backend/tests/unit/llm/hardware/test_os_gpu.py @@ -0,0 +1,78 @@ +"""Asking the operating system whether a graphics card is there at all. + +Each platform's reader is tested directly rather than through the dispatcher, +because only one of the three can run on any given host and the dispatcher's +own job is one `sys.platform` comparison. +""" + +import pytest + +from modules.llm.hardware.os_gpu import linux, windows + +pytestmark = pytest.mark.unit + + +def card(root, index: int, vendor: str) -> None: + """One DRM card in a fake sysfs, as a kernel with a driver populates it.""" + device = root / f"card{index}" / "device" + device.mkdir(parents=True) + (device / "vendor").write_text(f"{vendor}\n") + + +def test_linux_reports_a_real_card(tmp_path) -> None: + """0x10de is NVIDIA. A card the kernel bound a driver to.""" + card(tmp_path, 0, "0x10de") + + assert linux.reports_gpu(tmp_path) is True + + +def test_linux_does_not_count_a_virtual_display(tmp_path) -> None: + """0x1234 is QEMU's display adapter. A VM has no card to be missing, so + counting it would report every guest as a broken install.""" + card(tmp_path, 0, "0x1234") + + assert linux.reports_gpu(tmp_path) is False + + +def test_linux_sees_a_real_card_beside_a_virtual_one(tmp_path) -> None: + """A passthrough guest: both are listed and only one can compute.""" + card(tmp_path, 0, "0x1234") + card(tmp_path, 1, "0x1002") + + assert linux.reports_gpu(tmp_path) is True + + +def test_linux_with_no_cards_at_all_says_so(tmp_path) -> None: + """A headless server. Nothing is wrong with it.""" + assert linux.reports_gpu(tmp_path) is False + + +def test_a_kernel_without_drm_admits_it_cannot_say(tmp_path) -> None: + """No directory means no information, and guessing False here would call a + broken driver install a machine without a card.""" + assert linux.reports_gpu(tmp_path / "absent") is None + + +def test_windows_lists_the_real_adapters() -> None: + """What `Win32_VideoController` prints, one name per line.""" + output = "NVIDIA GeForce RTX 3050\nAMD Radeon(TM) Graphics\n" + + assert windows.parse_adapters(output) == [ + "NVIDIA GeForce RTX 3050", + "AMD Radeon(TM) Graphics", + ] + + +def test_windows_drops_the_virtual_display_the_measured_machine_carried() -> None: + """The RTX 3050 box had a Parsec adapter beside its two real GPUs. Counting + it would report a card on a machine that has none.""" + output = "Parsec Virtual Display Adapter\nMicrosoft Basic Display Adapter\n" + + assert windows.parse_adapters(output) == [] + + +def test_windows_keeps_a_card_whose_name_merely_mentions_a_vendor() -> None: + """The filter is on what the adapter is, not on who made it.""" + output = "Parsec Virtual Display Adapter\nIntel(R) Arc(TM) A770 Graphics\n" + + assert windows.parse_adapters(output) == ["Intel(R) Arc(TM) A770 Graphics"] From af2ccacd0f8fafd4b3471e8780bee8695f06dbf5 Mon Sep 17 00:00:00 2001 From: Anish Sarkar <104695310+AnishSarkar22@users.noreply.github.com> Date: Mon, 21 Sep 2026 17:00:30 +0530 Subject: [PATCH 40/60] feat(hardware): budget unified memory as one pool and expose the fit target --- .../backend/modules/llm/hardware/__init__.py | 17 +++++- .../backend/modules/llm/hardware/budget.py | 60 +++++++++++-------- .../modules/llm/hardware/unified_pool.py | 28 +++++++++ .../tests/unit/llm/hardware/test_budget.py | 59 +++++++++++++++++- 4 files changed, 137 insertions(+), 27 deletions(-) create mode 100644 surfsense_local/backend/modules/llm/hardware/unified_pool.py diff --git a/surfsense_local/backend/modules/llm/hardware/__init__.py b/surfsense_local/backend/modules/llm/hardware/__init__.py index 8516c72408..a1255f7c0d 100644 --- a/surfsense_local/backend/modules/llm/hardware/__init__.py +++ b/surfsense_local/backend/modules/llm/hardware/__init__.py @@ -2,10 +2,16 @@ LLAMA_CPP_FIT_MARGIN_BYTES, BudgetMode, build_budget, + fit_target_mib, ) +from modules.llm.hardware.device_lines import decode_lines, encode_line from modules.llm.hardware.devices import Device, DeviceType +from modules.llm.hardware.gpu_status import GpuStatus, classify +from modules.llm.hardware.inventory import SystemInventory, system_inventory from modules.llm.hardware.libraries import GgmlLibraries, load -from modules.llm.hardware.probe import probe_devices +from modules.llm.hardware.os_gpu import os_reports_gpu +from modules.llm.hardware.probe import probe_devices_in_process +from modules.llm.hardware.probe_subprocess import probe_devices from modules.llm.hardware.selection import select_device from modules.llm.hardware.system_memory import available_bytes @@ -15,9 +21,18 @@ "Device", "DeviceType", "GgmlLibraries", + "GpuStatus", + "SystemInventory", "available_bytes", "build_budget", + "classify", + "decode_lines", + "encode_line", + "fit_target_mib", "load", + "os_reports_gpu", "probe_devices", + "probe_devices_in_process", "select_device", + "system_inventory", ] diff --git a/surfsense_local/backend/modules/llm/hardware/budget.py b/surfsense_local/backend/modules/llm/hardware/budget.py index 5c5f0b8dfb..83b3ad9ae8 100644 --- a/surfsense_local/backend/modules/llm/hardware/budget.py +++ b/surfsense_local/backend/modules/llm/hardware/budget.py @@ -14,6 +14,7 @@ from modules.llm.fit import HardwareBudget from modules.llm.hardware.devices import Device, DeviceType from modules.llm.hardware.selection import select_device +from modules.llm.hardware.unified_pool import unified_pool_bytes # llama.cpp's own `fit_params_target`, one GiB per device, read from # common/common.h at b11050 and observed on both Vulkan and Metal. Not an @@ -21,6 +22,20 @@ # can pin through --fit-target rather than predict. LLAMA_CPP_FIT_MARGIN_BYTES = 1024 * 1024 * 1024 +_MIB = 1024 * 1024 + + +def fit_target_mib(mmproj_bytes: int = 0) -> int: + """The margin to pass the fitter, in the MiB `--fit-target` expects. + + A vision projector is added to it rather than subtracted from the budget, + because `--fit` allocates the projector after it has finished placing layers + and does not count it while deciding. Asking the fitter to leave room for it + is what stops a model it called resident from failing to allocate. Ollama + compensates the same way, through `LLAMA_ARG_FIT_TARGET`. + """ + return (LLAMA_CPP_FIT_MARGIN_BYTES + max(0, mmproj_bytes) + _MIB - 1) // _MIB + class BudgetMode(StrEnum): """Which question the budget is answering.""" @@ -56,12 +71,28 @@ def build_budget( has_gpu=False, ) + if uma: + # One memory with two ceilings on it, which is not the same as two + # pools. The device figure bounds what Metal will allocate, and so what + # can stay resident. The host figure bounds what the machine can hold at + # all, because the CPU backend reads the same chips without Metal's + # working set ceiling. `refusal_bytes` takes the host one and never a + # sum of both, which is where the double count used to be. + return HardwareBudget( + device_free_bytes=unified_pool_bytes(device.free_bytes, ram), + device_total_bytes=device.total_bytes, + fit_reserve_bytes=LLAMA_CPP_FIT_MARGIN_BYTES, + ram_available_bytes=ram, + uma=True, + has_gpu=True, + ) + return HardwareBudget( - device_free_bytes=_device_free(device, uma, ram), + device_free_bytes=device.free_bytes, device_total_bytes=device.total_bytes, fit_reserve_bytes=LLAMA_CPP_FIT_MARGIN_BYTES, ram_available_bytes=ram, - uma=uma, + uma=False, has_gpu=True, ) @@ -70,35 +101,14 @@ def _is_unified(gpu: Device, devices: Sequence[Device]) -> bool: """A GPU sharing the CPU's description is sharing its memory. Apple Silicon reports the same part name for both, which is the cheap and - reliable signal. It decides badge copy, not arithmetic. + reliable signal. It decides badge copy and, since the two readings name one + memory, whether the budget is a pool or a pair. """ return any( d.type is DeviceType.CPU and d.description == gpu.description for d in devices ) -def _device_free(device: Device, uma: bool, host_memory: int) -> int: - """How much the device can actually be given. - - On a discrete card this is the card's own figure: dedicated VRAM is a - separate pool and its `free` really is live. - - On unified memory it is neither. Metal reports - `recommendedMaxWorkingSetSize` as both total and free, a static number that - does not move however busy the machine is, and the memory it describes is - the same memory the OS and every other app are using. So **both** limits - apply and the smaller governs: the working set is a ceiling Metal will not - allocate past, and host memory is what is actually there to give. - - Spending the working set as though it were live is what sized a 28,672 token - window on an 8 GB Mac with 2.3 GB reclaimable. The load paged and took 43 - seconds, which is long enough for title generation to time out at 30. - """ - if not uma: - return device.free_bytes - return min(device.free_bytes, host_memory) - - def _host_memory( devices: Sequence[Device], ram_available_bytes: int, diff --git a/surfsense_local/backend/modules/llm/hardware/unified_pool.py b/surfsense_local/backend/modules/llm/hardware/unified_pool.py new file mode 100644 index 0000000000..23b9eb8e08 --- /dev/null +++ b/surfsense_local/backend/modules/llm/hardware/unified_pool.py @@ -0,0 +1,28 @@ +"""One pool, on a machine where the GPU and the CPU share the same chips. + +Apple Silicon reports two views of one memory, and adding them is the mistake +this module exists to prevent: Metal states `recommendedMaxWorkingSetSize` as +both its total and its free, while the host states the same physical memory +again. Summed, an 8 GB Mac priced as though it had 10.3 GB. + +Two limits apply and the smaller governs. The working set is a ceiling Metal +will not allocate past, and it is exactly the figure llama.cpp's fitter +subtracts its own margin from, so discounting it again would refuse a model the +runtime demonstrably places. Host memory is what is actually there, and that is +the leg that lies: it is a snapshot, and loading weights takes seconds during +which another application can take memory. + +Measured: spending the working set as though it were live sized a 28,672 token +window on an 8 GB Mac with 2.3 GB reclaimable. The load paged and took 43 +seconds, long enough for title generation to time out at 30. +""" + +# What the pool leaves to the rest of the machine. Not a margin for the model, +# which llama.cpp's own fit reserve already covers, but for the seconds between +# reading this number and finishing the load. +UMA_HOST_FRACTION = 0.85 + + +def unified_pool_bytes(working_set_bytes: int, host_bytes: int) -> int: + """What a unified memory machine can actually give a model.""" + return min(working_set_bytes, int(UMA_HOST_FRACTION * host_bytes)) diff --git a/surfsense_local/backend/tests/unit/llm/hardware/test_budget.py b/surfsense_local/backend/tests/unit/llm/hardware/test_budget.py index fcf6bbb6cd..84d2122b2d 100644 --- a/surfsense_local/backend/tests/unit/llm/hardware/test_budget.py +++ b/surfsense_local/backend/tests/unit/llm/hardware/test_budget.py @@ -46,7 +46,11 @@ def test_a_machine_with_no_gpu_still_produces_a_budget() -> None: def test_the_ram_half_comes_from_the_caller_not_from_ggml() -> None: """ggml's CPU device restates the total on macOS and under WSL2, and that - number is what separates PARTIAL from TOO_BIG.""" + number is what separates PARTIAL from TOO_BIG. + + Here the CPU device claims 8192 MiB and the caller knows only 1500 MiB is + reclaimable. The caller wins. + """ budget = build_budget([METAL, CPU_RESTATED], ram_available_bytes=1500 * MIB) assert budget.ram_available_bytes == 1500 * MIB @@ -159,3 +163,56 @@ def test_a_discrete_card_is_untouched_by_the_mode() -> None: assert live.device_free_bytes == DISCRETE.free_bytes assert capacity.device_free_bytes == DISCRETE.free_bytes + + +def test_unified_memory_is_one_pool_never_a_sum() -> None: + """The measured bug: the same physical memory was counted twice. + + Metal reports `recommendedMaxWorkingSetSize` as both total and free, and the + host reports the same chips again. Adding them gave an 8 GB Mac a refusal + threshold of 10.3 GB, so a model that could not possibly load was offered. + """ + budget = build_budget( + [METAL, CPU_RESTATED], ram_available_bytes=2300 * MIB, mode=BudgetMode.CAPACITY + ) + + assert budget.uma + # The sum is what was wrong. Either ceiling alone is a real limit; added + # together they describe memory the machine does not have twice over. + assert budget.refusal_bytes < budget.usable_vram_bytes + budget.ram_available_bytes + assert budget.refusal_bytes == budget.ram_available_bytes + + +def test_the_unified_pool_keeps_a_margin_against_the_host_but_not_the_working_set() -> None: + """Two limits, one pool, and the fraction sits on the leg that lies. + + The working set is Apple's own ceiling and is exactly what llama.cpp + subtracts its margin from, so discounting it again would refuse a model the + fitter really does place. Host memory is a snapshot and a load takes + seconds, so that leg is discounted. + """ + budget = build_budget( + [METAL, CPU_RESTATED], ram_available_bytes=2300 * MIB, mode=BudgetMode.CAPACITY + ) + + # Residency is bounded by whichever ceiling is lower, with the margin on the + # host leg because that is the reading that moves while a load is running. + assert budget.device_free_bytes == min(5461 * MIB, int(0.85 * 6144 * MIB)) + assert budget.usable_vram_bytes == budget.device_free_bytes - 1024 * MIB + + +def test_a_machine_with_no_gpu_refuses_only_past_its_own_memory() -> None: + """There is nothing to spill from, so residency and physics are one number.""" + budget = build_budget([CPU_LIVE], ram_available_bytes=22750 * MIB) + + assert not budget.has_gpu + assert budget.resident_bytes == budget.ram_available_bytes + assert budget.refusal_bytes == budget.ram_available_bytes + + +def test_a_discrete_card_still_spills_into_host_memory() -> None: + """The one case where the two subtractions really are different pools.""" + budget = build_budget([DISCRETE, CPU_LIVE], ram_available_bytes=22750 * MIB) + + assert budget.resident_bytes == budget.usable_vram_bytes + assert budget.refusal_bytes == budget.usable_vram_bytes + budget.ram_available_bytes From 2494a2991bafe634cc7f29525e19738b18b65466 Mon Sep 17 00:00:00 2001 From: Anish Sarkar <104695310+AnishSarkar22@users.noreply.github.com> Date: Mon, 21 Sep 2026 17:00:43 +0530 Subject: [PATCH 41/60] feat(llamacpp): load the vision projector and pin the fitter's margin --- .../llm/providers/llamacpp/__init__.py | 3 + .../modules/llm/providers/llamacpp/preset.py | 18 +++++ .../llm/providers/llamacpp/projector.py | 45 ++++++++++++ .../llm/providers/llamacpp/test_preset.py | 37 ++++++++++ .../llm/providers/llamacpp/test_projector.py | 68 +++++++++++++++++++ 5 files changed, 171 insertions(+) create mode 100644 surfsense_local/backend/modules/llm/providers/llamacpp/projector.py create mode 100644 surfsense_local/backend/tests/unit/llm/providers/llamacpp/test_projector.py diff --git a/surfsense_local/backend/modules/llm/providers/llamacpp/__init__.py b/surfsense_local/backend/modules/llm/providers/llamacpp/__init__.py index 10982cd37e..40c507944f 100644 --- a/surfsense_local/backend/modules/llm/providers/llamacpp/__init__.py +++ b/surfsense_local/backend/modules/llm/providers/llamacpp/__init__.py @@ -11,6 +11,7 @@ render_presets, write_presets, ) +from modules.llm.providers.llamacpp.projector import is_projector, projector_for from modules.llm.providers.llamacpp.provider import PROVIDER, LlamaCppProvider from modules.llm.providers.llamacpp.router_client import RouterClient, RouterModel @@ -25,6 +26,8 @@ "RouterModel", "download_gguf", "for_template", + "is_projector", + "projector_for", "read_capabilities", "render_presets", "write_presets", diff --git a/surfsense_local/backend/modules/llm/providers/llamacpp/preset.py b/surfsense_local/backend/modules/llm/providers/llamacpp/preset.py index 3054a60692..a6612a9033 100644 --- a/surfsense_local/backend/modules/llm/providers/llamacpp/preset.py +++ b/surfsense_local/backend/modules/llm/providers/llamacpp/preset.py @@ -30,6 +30,10 @@ class ModelPreset: path: str n_ctx: int precision: KvPrecision + # The margin the fitter must leave, in MiB. No default: the badge was drawn + # against a specific number and the caller always knows which. + fit_target_mib: int + mmproj_path: str | None = None def render_presets(presets: list[ModelPreset]) -> str: @@ -50,7 +54,21 @@ def _section(preset: ModelPreset) -> str: # One user, one question. The default of four sizes the KV cache for # concurrency this app never uses. "parallel = 1", + # Pinned rather than inherited. The badge subtracted this margin, so + # passing it makes the two agree by construction instead of by assuming + # a default read from the source once. + f"fit-target = {preset.fit_target_mib}", + # Inert while `ctx-size` is set: llama.cpp only shrinks a context it + # chose itself. Written anyway, because it states the floor at the place + # the fitter would look for one, so a later change to how the window is + # set cannot quietly hand that floor back to llama.cpp's own 4096. + f"fit-ctx = {preset.n_ctx}", ] + if preset.mmproj_path is not None: + # `--fit` does not count the projector, so a vision model the fitter + # calls resident can still fail to allocate. The margin above carries + # its bytes; this is what tells the worker to load it at all. + lines.append(f"mmproj = {preset.mmproj_path}") if preset.precision is KvPrecision.Q8_0: # Both halves together, or the fused flash-attention kernel is skipped # and attention silently falls back to the CPU. diff --git a/surfsense_local/backend/modules/llm/providers/llamacpp/projector.py b/surfsense_local/backend/modules/llm/providers/llamacpp/projector.py new file mode 100644 index 0000000000..f278c1044f --- /dev/null +++ b/surfsense_local/backend/modules/llm/providers/llamacpp/projector.py @@ -0,0 +1,45 @@ +"""Pairing a model on disk with the vision projector that belongs to it. + +A vision model is two files. The weights answer questions and the projector +turns an image into something the weights can read, and llama.cpp loads the +second only when told to. The fitter does not count its memory either, so the +margin has to carry it. + +Both files sit in the models directory, so this is also what stops a projector +being offered as a model in its own right: it has a GGUF header and a size, and +nothing else about it says it cannot answer a question. +""" + +from collections.abc import Iterable +from pathlib import Path + +_MARKER = "mmproj" + + +def is_projector(path: Path) -> bool: + """Whether this file is a projector rather than a model. + + Named by convention rather than read from the header: every publisher marks + them this way, and the alternative is a header read per file on a path that + runs at startup. + """ + return _MARKER in path.stem.lower() + + +def projector_for(model_path: Path, named: Iterable[str] = ()) -> Path | None: + """The projector belonging to `model_path`, or None for a text model. + + `named` is what the manifest says this model's projector is called, which is + the exact answer for a curated entry. A searched model has no manifest row, + so the fallback is the one projector sitting beside it. One, not the first of + several: two projectors in a directory belong to two different models, and + guessing between them would attach the wrong one. + """ + directory = model_path.parent + for name in named: + candidate = directory / name + if candidate.exists(): + return candidate + + beside = [path for path in sorted(directory.glob("*.gguf")) if is_projector(path)] + return beside[0] if len(beside) == 1 else None diff --git a/surfsense_local/backend/tests/unit/llm/providers/llamacpp/test_preset.py b/surfsense_local/backend/tests/unit/llm/providers/llamacpp/test_preset.py index f09ba537b4..c3fcf1a0de 100644 --- a/surfsense_local/backend/tests/unit/llm/providers/llamacpp/test_preset.py +++ b/surfsense_local/backend/tests/unit/llm/providers/llamacpp/test_preset.py @@ -21,6 +21,7 @@ def a_preset(**overrides) -> ModelPreset: "path": "/models/Qwen3-8B-Q4_K_M.gguf", "n_ctx": 16384, "precision": KvPrecision.F16, + "fit_target_mib": 1024, } return ModelPreset(**{**fields, **overrides}) @@ -83,3 +84,39 @@ def test_no_explicit_layer_count_is_ever_written() -> None: assert "n-gpu-layers" not in ini assert "ngl" not in ini + + +def test_the_fitter_is_told_the_margin_we_priced_against() -> None: + """The badge subtracted llama.cpp's own default margin. Pinning it makes that + true by construction rather than by assuming a default we read once.""" + ini = render_presets([a_preset()]) + + assert "fit-target = 1024" in ini + + +def test_a_projector_raises_the_target_by_its_own_size() -> None: + """`--fit` does not count mmproj memory, so a vision model the fitter calls + resident can still fail to allocate. Ollama compensates the same way: the + projector's bytes are added to the margin the fitter must leave.""" + ini = render_presets( + [a_preset(fit_target_mib=1624, mmproj_path="/models/mmproj-F16.gguf")] + ) + + assert "fit-target = 1624" in ini + assert "mmproj = /models/mmproj-F16.gguf" in ini + + +def test_a_text_model_names_no_projector() -> None: + """Most models have none, and an empty value is not the same as absence.""" + assert "mmproj" not in render_presets([a_preset()]) + + +def test_the_context_floor_is_stated_for_the_fitter_too() -> None: + """Inert while `ctx-size` is set, because llama.cpp only reduces a context it + chose itself. Written anyway: it says what the floor is at the place the + fitter would read one, so a future change to how we set the window cannot + silently hand the floor back to llama.cpp's own 4096. + """ + ini = render_presets([a_preset(n_ctx=16384)]) + + assert "fit-ctx = 16384" in ini diff --git a/surfsense_local/backend/tests/unit/llm/providers/llamacpp/test_projector.py b/surfsense_local/backend/tests/unit/llm/providers/llamacpp/test_projector.py new file mode 100644 index 0000000000..2fb3b31145 --- /dev/null +++ b/surfsense_local/backend/tests/unit/llm/providers/llamacpp/test_projector.py @@ -0,0 +1,68 @@ +"""Pairing a model on disk with the projector that belongs to it.""" + +from pathlib import Path + +import pytest + +from modules.llm.providers.llamacpp import is_projector, projector_for + +pytestmark = pytest.mark.unit + + +def touch(path: Path) -> Path: + """A file on disk. Pairing reads names and sizes, never contents.""" + path.write_bytes(b"GGUF") + return path + + +def test_a_projector_is_recognised_however_it_is_named() -> None: + """Every publisher marks them the same way, which is why the name is enough + and a header read per file at startup is not needed.""" + assert is_projector(Path("mmproj-F16.gguf")) + assert is_projector(Path("Qwen3-VL-mmproj-BF16.gguf")) + assert not is_projector(Path("Qwen3-8B-Q4_K_M.gguf")) + + +def test_the_manifest_name_wins_when_there_is_one(tmp_path: Path) -> None: + """A curated entry states its projector exactly, so nothing is inferred.""" + model = touch(tmp_path / "Qwen3-VL-Q4_K_M.gguf") + named = touch(tmp_path / "mmproj-F16.gguf") + touch(tmp_path / "mmproj-Q8_0.gguf") + + assert projector_for(model, ["mmproj-F16.gguf"]) == named + + +def test_a_searched_model_takes_the_one_projector_beside_it(tmp_path: Path) -> None: + """No manifest row to consult, and exactly one candidate.""" + model = touch(tmp_path / "Some-VL-Q4_K_M.gguf") + projector = touch(tmp_path / "mmproj-F16.gguf") + + assert projector_for(model) == projector + + +def test_two_projectors_with_nothing_to_choose_between_them_pair_with_neither( + tmp_path: Path, +) -> None: + """They belong to two different models. Guessing attaches the wrong one, and + a wrong projector is worse than none: the model answers, badly, about images + it never really saw.""" + model = touch(tmp_path / "Some-VL-Q4_K_M.gguf") + touch(tmp_path / "mmproj-F16.gguf") + touch(tmp_path / "other-mmproj-F16.gguf") + + assert projector_for(model) is None + + +def test_a_text_model_alone_in_a_directory_has_none(tmp_path: Path) -> None: + """The common case, and it must not cost a reserve.""" + model = touch(tmp_path / "Qwen3-8B-Q4_K_M.gguf") + + assert projector_for(model) is None + + +def test_a_manifest_name_that_is_not_on_disk_falls_through(tmp_path: Path) -> None: + """The weights downloaded and the projector did not, so the model still runs + as text rather than naming a file the worker cannot open.""" + model = touch(tmp_path / "Qwen3-VL-Q4_K_M.gguf") + + assert projector_for(model, ["mmproj-F16.gguf"]) is None From e8962ae0ed973793d2287d846972c05b0bb87597 Mon Sep 17 00:00:00 2001 From: Anish Sarkar <104695310+AnishSarkar22@users.noreply.github.com> Date: Mon, 21 Sep 2026 17:00:49 +0530 Subject: [PATCH 42/60] fix(providers): budget the first token apart from the ones after it --- surfsense_local/backend/modules/chat/title.py | 34 ++++--- .../llm/providers/openai_compatible/chat.py | 36 ++++++- .../modules/llm/providers/stream_deadline.py | 70 +++++++++++++ .../llm/providers/test_stream_deadline.py | 97 +++++++++++++++++++ 4 files changed, 218 insertions(+), 19 deletions(-) create mode 100644 surfsense_local/backend/modules/llm/providers/stream_deadline.py create mode 100644 surfsense_local/backend/tests/unit/llm/providers/test_stream_deadline.py diff --git a/surfsense_local/backend/modules/chat/title.py b/surfsense_local/backend/modules/chat/title.py index fce3161a19..43da288d7f 100644 --- a/surfsense_local/backend/modules/chat/title.py +++ b/surfsense_local/backend/modules/chat/title.py @@ -1,5 +1,3 @@ -import asyncio - from modules.llm.providers.protocols import Generator from modules.llm.providers.types import Message @@ -17,7 +15,6 @@ Query: {user_query} Title:""" -TITLE_TIMEOUT_SECONDS = 30 TITLE_MAX_TOKENS = 12 TITLE_MAX_CHARACTERS = 100 @@ -25,22 +22,29 @@ async def generate_title( generator: Generator, model: str, user_query: str ) -> str | None: - """Generate and validate a short title without exposing raw model output.""" + """Generate and validate a short title without exposing raw model output. + + No timeout of its own. How long a model may take to answer is a fact about + the runtime, which this module cannot observe: a 30 second budget here + cancelled a title 9 seconds before a freshly downloaded model finished + loading. The provider applies one budget until the first token and a tighter + one between them. What belongs here is the shape of the answer: short, + single line, and cut off if the model runs on. + """ prompt = TITLE_PROMPT.replace( "{user_query}", user_query.strip()[:500] or "(message)" ) parts: list[str] = [] - async with asyncio.timeout(TITLE_TIMEOUT_SECONDS): - async for delta in generator.chat( - model, - [Message("user", prompt)], - max_tokens=TITLE_MAX_TOKENS, - temperature=0, - reasoning=False, - ): - parts.append(delta) - if sum(map(len, parts)) > TITLE_MAX_CHARACTERS + 2: - return None + async for delta in generator.chat( + model, + [Message("user", prompt)], + max_tokens=TITLE_MAX_TOKENS, + temperature=0, + reasoning=False, + ): + parts.append(delta) + if sum(map(len, parts)) > TITLE_MAX_CHARACTERS + 2: + return None return valid_title("".join(parts)) diff --git a/surfsense_local/backend/modules/llm/providers/openai_compatible/chat.py b/surfsense_local/backend/modules/llm/providers/openai_compatible/chat.py index fcd29a8d0b..8340116cb4 100644 --- a/surfsense_local/backend/modules/llm/providers/openai_compatible/chat.py +++ b/surfsense_local/backend/modules/llm/providers/openai_compatible/chat.py @@ -5,9 +5,27 @@ from modules.llm.connections.service import parse_models from modules.llm.profile import Fingerprint, from_remote +from modules.llm.providers.stream_deadline import with_deadlines from modules.llm.providers.types import Message, Model -TIMEOUT = httpx.Timeout(120.0, connect=5.0) +# Waiting for the first token is waiting for a model to load, which on a cold +# file is tens of seconds and on a large one more. Once tokens are flowing, a +# long gap is a fault rather than a slow machine, so the two are budgeted apart. +# The numbers live here because this is the seam that knows what it is talking +# to; a feature asking for a generation states what it wants, not how long the +# runtime may take to produce it. +FIRST_TOKEN_SECONDS = 300.0 +BETWEEN_TOKENS_SECONDS = 30.0 + +# A backstop, not the rule. httpx's read timeout is per read, so a value below +# the load budget would quietly become the real limit and surface as a network +# error instead of saying which budget expired. +TIMEOUT = httpx.Timeout(FIRST_TOKEN_SECONDS, connect=5.0) + +# Listing what an endpoint offers waits for no model, so it keeps the short +# budget: a mistyped address should fail while the person is still looking at +# the dialog. +LISTING_TIMEOUT = httpx.Timeout(120.0, connect=5.0) MAX_ERROR_CHARS = 400 @@ -32,10 +50,10 @@ def __init__( # that cannot name one gets today's behaviour: the flag is ignored. self._thinking_off = thinking_off - def _client(self) -> httpx.AsyncClient: + def _client(self, timeout: httpx.Timeout = LISTING_TIMEOUT) -> httpx.AsyncClient: headers = {"Authorization": f"Bearer {self._api_key}"} if self._api_key else {} return httpx.AsyncClient( - timeout=TIMEOUT, headers=headers, transport=self._transport + timeout=timeout, headers=headers, transport=self._transport ) async def health(self) -> bool: @@ -103,8 +121,18 @@ async def chat( # and a thinking model spends that cap before its first answer # token, so this is what keeps a short request from returning "". body.update(self._thinking_off) + async for delta in with_deadlines( + self._stream(body), + first_item_seconds=FIRST_TOKEN_SECONDS, + between_items_seconds=BETWEEN_TOKENS_SECONDS, + subject="the model", + ): + yield delta + + async def _stream(self, body: dict[str, object]) -> AsyncIterator[str]: + """The deltas as the endpoint sends them, with no waiting rule of its own.""" async with ( - self._client() as client, + self._client(TIMEOUT) as client, client.stream( "POST", f"{self._base_url}/chat/completions", json=body ) as reply, diff --git a/surfsense_local/backend/modules/llm/providers/stream_deadline.py b/surfsense_local/backend/modules/llm/providers/stream_deadline.py new file mode 100644 index 0000000000..06dab819c3 --- /dev/null +++ b/surfsense_local/backend/modules/llm/providers/stream_deadline.py @@ -0,0 +1,70 @@ +"""How long a stream may stay quiet, and when that stops being acceptable. + +One clock cannot answer two questions. Waiting for the first item is waiting +for work that has not started producing yet: for a generation that is the model +loading, and a cold 2.5 GB file on an 8 GB Mac measured 39 seconds. A gap +*between* items is not the same thing, and a stream that dies half way through +should be given up on in seconds. +""" + +import asyncio +import contextlib +from collections.abc import AsyncIterator + + +class StreamTimeoutError(TimeoutError): + """A stream stayed quiet past its budget. + + A `TimeoutError` so that callers written against `asyncio.timeout` keep + working, and carrying `first_item` because the two cases mean different + things: one is work that never started, the other is work that died. + """ + + def __init__(self, seconds: float, *, first_item: bool, subject: str) -> None: + self.first_item = first_item + self.seconds = seconds + super().__init__( + f"{subject} did not start within {seconds:g}s" + if first_item + else f"{subject} stopped for more than {seconds:g}s" + ) + + +async def with_deadlines( + stream: AsyncIterator[str], + *, + first_item_seconds: float, + between_items_seconds: float, + subject: str = "the stream", +) -> AsyncIterator[str]: + """Yield from `stream`, applying the start budget then the stall budget. + + The deadline restarts on every item, so a long stream is never cut off for + being long, only for going quiet. `subject` names what was being waited on, + so the failure reads as a sentence wherever it is logged. + """ + items = stream.__aiter__() + seconds = first_item_seconds + first_item = True + try: + while True: + try: + async with asyncio.timeout(seconds): + item = await anext(items) + except StopAsyncIteration: + return + except TimeoutError as expired: + raise StreamTimeoutError( + seconds, first_item=first_item, subject=subject + ) from expired + yield item + seconds = between_items_seconds + first_item = False + finally: + # The timeout cancelled a read on whatever the stream was holding: for + # HTTP, an open response. Leaving it to the garbage collector holds the + # connection, and with it the worker slot the next request needs. + aclose = getattr(items, "aclose", None) + if aclose is not None: + with contextlib.suppress(Exception): + await aclose() diff --git a/surfsense_local/backend/tests/unit/llm/providers/test_stream_deadline.py b/surfsense_local/backend/tests/unit/llm/providers/test_stream_deadline.py new file mode 100644 index 0000000000..1aa0b09a2f --- /dev/null +++ b/surfsense_local/backend/tests/unit/llm/providers/test_stream_deadline.py @@ -0,0 +1,97 @@ +"""Two clocks in front of a generation, not one. + +The measured failure: a freshly downloaded 2.5 GB model took 39 seconds to load +on an 8 GB Mac, and a single 30 second budget around the whole stream cancelled +the request 9 seconds before the first token arrived. Waiting for a load is not +evidence of a fault; silence part way through an answer is. +""" + +import asyncio +from collections.abc import AsyncIterator + +import pytest + +from modules.llm.providers.stream_deadline import StreamTimeoutError, with_deadlines + +pytestmark = pytest.mark.unit + + +async def _after(delays: list[float]) -> AsyncIterator[str]: + """One token per delay, each arriving that many seconds after the last.""" + for index, delay in enumerate(delays): + await asyncio.sleep(delay) + yield str(index) + + +async def _collect(stream: AsyncIterator[str]) -> list[str]: + return [delta async for delta in stream] + + +async def test_a_slow_load_is_waited_out_not_cancelled() -> None: + """The whole point. The first token may take far longer than any later one, + because that wait is the model loading, and its length is a property of the + machine rather than of the request.""" + tokens = await _collect( + with_deadlines( + _after([0.15, 0.0, 0.0]), + first_item_seconds=1.0, + between_items_seconds=0.05, + ) + ) + + assert tokens == ["0", "1", "2"] + + +async def test_going_quiet_mid_answer_is_cut_off_without_waiting_for_the_load_budget() -> None: + """The opposite case, and the reason the second clock is tight: once tokens + are flowing, a long gap really is a fault and should not inherit the + patience a cold load earns.""" + with pytest.raises(StreamTimeoutError) as raised: + await _collect( + with_deadlines( + _after([0.0, 5.0]), + first_item_seconds=5.0, + between_items_seconds=0.05, + ) + ) + + assert not raised.value.first_item + + +async def test_a_model_that_never_starts_answering_says_so() -> None: + """Told apart from the case above, because they mean different things to + whoever reads the log: one is a machine too slow or a load that failed, the + other is a stream that died.""" + with pytest.raises(StreamTimeoutError) as raised: + await _collect( + with_deadlines( + _after([5.0]), + first_item_seconds=0.05, + between_items_seconds=5.0, + ) + ) + + assert raised.value.first_item + + +async def test_the_underlying_stream_is_closed_when_a_budget_fires() -> None: + """A timeout abandons an open HTTP response, and leaving it to the garbage + collector holds the connection and the worker's slot.""" + closed = False + + async def source() -> AsyncIterator[str]: + nonlocal closed + try: + await asyncio.sleep(5.0) + yield "never" + finally: + closed = True + + with pytest.raises(StreamTimeoutError): + await _collect( + with_deadlines( + source(), first_item_seconds=0.05, between_items_seconds=5.0 + ) + ) + + assert closed From a9bb6c414b1d86ff15656cf4562cb5f7517b46c6 Mon Sep 17 00:00:00 2001 From: Anish Sarkar <104695310+AnishSarkar22@users.noreply.github.com> Date: Mon, 21 Sep 2026 17:00:59 +0530 Subject: [PATCH 43/60] refactor(llm): drop the unused ModelStore protocol and its llmfit fixtures --- .../backend/modules/llm/dependencies.py | 14 +----- .../llm/providers/llamacpp/provider.py | 10 ++-- .../modules/llm/providers/protocols.py | 23 +-------- .../backend/modules/llm/providers/types.py | 10 ---- .../backend/modules/llm/schemas.py | 15 ------ .../tests/integration/llm/test_egress.py | 44 ----------------- .../tests/integration/llm/test_routes.py | 48 ------------------- .../llm/providers/llamacpp/test_provider.py | 11 ++--- 8 files changed, 13 insertions(+), 162 deletions(-) diff --git a/surfsense_local/backend/modules/llm/dependencies.py b/surfsense_local/backend/modules/llm/dependencies.py index 09d631d7df..9c16929c28 100644 --- a/surfsense_local/backend/modules/llm/dependencies.py +++ b/surfsense_local/backend/modules/llm/dependencies.py @@ -3,7 +3,7 @@ from fastapi import Depends, HTTPException, status from modules.llm.providers import get_provider, llamacpp -from modules.llm.providers.protocols import Generator, ModelStore +from modules.llm.providers.protocols import Generator def get_provider_or_404(provider: str) -> Generator: @@ -17,18 +17,6 @@ def get_provider_or_404(provider: str) -> Generator: ProviderDep = Annotated[Generator, Depends(get_provider_or_404)] -def get_store_or_409(provider: ProviderDep) -> ModelStore: - """The provider as a local store, or a conflict for a remote API.""" - if not isinstance(provider, ModelStore): - raise HTTPException( - status.HTTP_409_CONFLICT, f"{provider.name} does not manage local models" - ) - return provider - - -StoreDep = Annotated[ModelStore, Depends(get_store_or_409)] - - def get_local_runtime() -> Generator: """The one runtime that holds models on this machine. diff --git a/surfsense_local/backend/modules/llm/providers/llamacpp/provider.py b/surfsense_local/backend/modules/llm/providers/llamacpp/provider.py index 38e7be94e9..c3972a8657 100644 --- a/surfsense_local/backend/modules/llm/providers/llamacpp/provider.py +++ b/surfsense_local/backend/modules/llm/providers/llamacpp/provider.py @@ -7,11 +7,11 @@ `/v1/chat/completions`, so the streaming, error handling and message shaping in `OpenAICompatibleChatProvider` already work against it. -Deliberately **not** a `ModelStore`. That protocol requires `pull()`, which made -sense when the runtime fetched its own weights from a name. Here SurfSense fetches the -GGUF itself, because that is the only place `egress.require()` can hold, and -because it buys resume, checksums and the header as the file lands. Downloading -is a catalog concern; this adapter answers questions and reports what is on disk. +Deliberately **no** `pull()`. Fetching its own weights from a name made sense +when the runtime owned the download. Here SurfSense fetches the GGUF itself, +because that is the only place `egress.require()` can hold, and because it buys +resume, checksums and the header as the file lands. Downloading is a catalog +concern; this adapter answers questions and reports what is on disk. """ import logging diff --git a/surfsense_local/backend/modules/llm/providers/protocols.py b/surfsense_local/backend/modules/llm/providers/protocols.py index 343afcd618..bfdb882c91 100644 --- a/surfsense_local/backend/modules/llm/providers/protocols.py +++ b/surfsense_local/backend/modules/llm/providers/protocols.py @@ -1,8 +1,8 @@ from collections.abc import AsyncIterator from dataclasses import dataclass -from typing import Protocol, runtime_checkable +from typing import Protocol -from modules.llm.providers.types import CatalogEntry, DownloadProgress, Message, Model +from modules.llm.providers.types import Message, Model class Generator(Protocol): @@ -62,22 +62,3 @@ class TextToSpeech(Protocol): def voices(self) -> list[Voice]: ... async def synthesize(self, turns: list[SpokenTurn]) -> SynthesizedAudio: ... - - -@runtime_checkable -class ModelStore(Protocol): - """Only runtimes that keep models on disk can fetch them. - - Checked with isinstance, so a remote API that cannot download simply does - not satisfy it and the download UI is hidden without naming a provider. - """ - - name: str - - async def models(self) -> list[Model]: ... - - def catalog(self) -> list[CatalogEntry]: ... - - def pull(self, name: str) -> AsyncIterator[DownloadProgress]: ... - - async def delete(self, name: str) -> None: ... diff --git a/surfsense_local/backend/modules/llm/providers/types.py b/surfsense_local/backend/modules/llm/providers/types.py index 31a4dcb248..2002ac9890 100644 --- a/surfsense_local/backend/modules/llm/providers/types.py +++ b/surfsense_local/backend/modules/llm/providers/types.py @@ -11,16 +11,6 @@ class Model: display_name: str | None = None -@dataclass(frozen=True) -class CatalogEntry: - """A model a provider offers to download, and its size.""" - - name: str - label: str - size_gb: float - installed: bool = False - - @dataclass(frozen=True) class Message: """One turn of a conversation handed to a generator.""" diff --git a/surfsense_local/backend/modules/llm/schemas.py b/surfsense_local/backend/modules/llm/schemas.py index 441180e92d..6a961106d6 100644 --- a/surfsense_local/backend/modules/llm/schemas.py +++ b/surfsense_local/backend/modules/llm/schemas.py @@ -33,21 +33,6 @@ class ModelDeleteRead(BaseModel): selection_cleared: bool -class CatalogEntryRead(BaseModel): - """A model on offer to download, with its size.""" - - name: str - label: str - size_gb: float - installed: bool - - -class PullRequest(BaseModel): - """The one model to fetch, by its provider name.""" - - name: str - - class ConnectionWrite(BaseModel): label: str = Field(min_length=1, max_length=100) provider: str = "openai_compatible" diff --git a/surfsense_local/backend/tests/integration/llm/test_egress.py b/surfsense_local/backend/tests/integration/llm/test_egress.py index d6ba17a554..4e78a1aa6e 100644 --- a/surfsense_local/backend/tests/integration/llm/test_egress.py +++ b/surfsense_local/backend/tests/integration/llm/test_egress.py @@ -1,15 +1,10 @@ """Nothing leaves the machine until the user allows that destination.""" -import json -from pathlib import Path - import pytest from httpx import AsyncClient from sqlalchemy import Engine -from modules.llm.catalog.dependencies import get_catalog_service from modules.llm.models import ProviderConnection -from shared.config import get_llm_settings from shared.db import create_session_factory pytestmark = pytest.mark.integration @@ -75,42 +70,3 @@ async def test_unknown_destination_is_rejected(client: AsyncClient) -> None: """Only destinations the app contacts can be toggled.""" reply = await client.put("/egress/keygen", json={"enabled": True}) assert reply.status_code == 422 - - -def _configure_llmfit(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: - system = {"system": {"available_ram_gb": 16, "total_ram_gb": 16}} - fit = { - "models": [ - { - "name": "Qwen/Qwen3-8B", - "provider": "Qwen", - "parameter_count": "8B", - "use_case": "chat", - "fit_level": "good", - "score": 85, - "runtime": "llamacpp", - "run_mode": "gpu", - "best_quant": "Q4_K_M", - "memory_required_gb": 6, - "memory_available_gb": 16, - "disk_size_gb": 5.2, - "effective_context_length": 8192, - "capability_ids": ["tool_use"], - "file": "Qwen3-8B-Q4_K_M.gguf", - "gguf_sources": [], - } - ] - } - executable = tmp_path / "llmfit" - executable.write_text( - "#!/usr/bin/env python3\n" - "import sys\n" - f"system = {json.dumps(system)!r}\n" - f"fit = {json.dumps(fit)!r}\n" - 'print("llmfit 1.1.11" if "--version" in sys.argv ' - 'else system if "system" in sys.argv else fit)\n' - ) - executable.chmod(0o755) - monkeypatch.setattr(get_llm_settings(), "llmfit_path", executable) - get_catalog_service.cache_clear() - diff --git a/surfsense_local/backend/tests/integration/llm/test_routes.py b/surfsense_local/backend/tests/integration/llm/test_routes.py index 3bcffdc12a..77955dec6c 100644 --- a/surfsense_local/backend/tests/integration/llm/test_routes.py +++ b/surfsense_local/backend/tests/integration/llm/test_routes.py @@ -1,6 +1,3 @@ -import json -from pathlib import Path - import pytest from httpx import AsyncClient from sqlalchemy import Engine @@ -11,7 +8,6 @@ from modules.llm.catalog.dependencies import get_catalog_service from modules.llm.providers.types import Message, Model from modules.workspaces.models import Workspace -from shared.config import get_llm_settings from shared.db import create_session_factory pytestmark = pytest.mark.integration @@ -259,47 +255,3 @@ def chat(self, model: str, messages: list[Message]): # pragma: no cover assert reply.status_code == 422 assert reply.json()["detail"] == "model does not support generation: embedder" assert (await client.get("/llm/selection/generation")).status_code == 404 - - -def _configure_llmfit( - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, - models: list[dict[str, object]] | None = None, -) -> None: - system = {"system": {"available_ram_gb": 16, "total_ram_gb": 16}} - fit = { - "models": models - or [ - { - "name": "Qwen/Qwen3-8B", - "provider": "Qwen", - "parameter_count": "8B", - "use_case": "chat", - "fit_level": "good", - "score": 85, - "runtime": "llamacpp", - "run_mode": "gpu", - "best_quant": "Q4_K_M", - "memory_required_gb": 6, - "memory_available_gb": 16, - "disk_size_gb": 5.2, - "effective_context_length": 8192, - "capability_ids": ["tool_use"], - "file": "Qwen3-8B-Q4_K_M.gguf", - "gguf_sources": [], - } - ] - } - executable = tmp_path / "llmfit" - executable.write_text( - "#!/usr/bin/env python3\n" - "import sys\n" - f"system = {json.dumps(system)!r}\n" - f"fit = {json.dumps(fit)!r}\n" - 'print("llmfit 1.1.11" if "--version" in sys.argv ' - 'else system if "system" in sys.argv else fit)\n' - ) - executable.chmod(0o755) - monkeypatch.setattr(get_llm_settings(), "llmfit_path", executable) - get_catalog_service.cache_clear() - diff --git a/surfsense_local/backend/tests/unit/llm/providers/llamacpp/test_provider.py b/surfsense_local/backend/tests/unit/llm/providers/llamacpp/test_provider.py index 11690d509b..22c5807fee 100644 --- a/surfsense_local/backend/tests/unit/llm/providers/llamacpp/test_provider.py +++ b/surfsense_local/backend/tests/unit/llm/providers/llamacpp/test_provider.py @@ -7,7 +7,6 @@ import pytest from modules.llm.providers.llamacpp import LlamaCppProvider -from modules.llm.providers.protocols import ModelStore from modules.llm.providers.types import Message from tests.unit.llm.providers.llamacpp.fake_router import FakeRouter @@ -24,18 +23,18 @@ def test_it_is_the_registry_entry_the_selection_column_will_hold() -> None: assert LlamaCppProvider("http://x").name == "llamacpp" -def test_it_answers_and_keeps_inventory_without_being_a_model_store() -> None: - """`ModelStore` requires `pull()`, and this runtime deliberately has none. +def test_it_answers_and_keeps_inventory_without_pulling_its_own_weights() -> None: + """This runtime deliberately has no `pull()`. The previous runtime pulled its own weights, so a name was enough. Here SurfSense fetches the GGUF in process, which is the only place `egress.require()` actually holds, and it also buys resume, checksums and the header as the file lands. - Downloads are a catalog concern, not a runtime one, so claiming ModelStore - would advertise a capability that belongs somewhere else. + Downloads are a catalog concern, not a runtime one, so a `pull` here would + advertise a capability that belongs somewhere else. """ provider = LlamaCppProvider("http://x") - assert not isinstance(provider, ModelStore) + assert not hasattr(provider, "pull") assert callable(provider.health) assert callable(provider.models) assert callable(provider.chat) From 99fd11a4851c0f8fbbb0ae608a08e42d0d62cb9e Mon Sep 17 00:00:00 2001 From: Anish Sarkar <104695310+AnishSarkar22@users.noreply.github.com> Date: Mon, 21 Sep 2026 17:01:21 +0530 Subject: [PATCH 44/60] feat(catalog): carry the widened shape at manifest schema 4 --- .../modules/llm/catalog/curated-models.json | 74 +++++++++++++++++-- .../modules/llm/catalog/dependencies.py | 3 +- .../backend/modules/llm/catalog/manifest.py | 27 ++++++- .../tests/unit/llm/catalog/test_manifest.py | 38 ++++++++-- 4 files changed, 126 insertions(+), 16 deletions(-) diff --git a/surfsense_local/backend/modules/llm/catalog/curated-models.json b/surfsense_local/backend/modules/llm/catalog/curated-models.json index 59ea76878e..fd308b6bbb 100644 --- a/surfsense_local/backend/modules/llm/catalog/curated-models.json +++ b/surfsense_local/backend/modules/llm/catalog/curated-models.json @@ -1,5 +1,5 @@ { - "schema_version": 3, + "schema_version": 4, "models": [ { "model_id": "Qwen/Qwen3-0.6B", @@ -15,7 +15,17 @@ "context_length": 40960, "n_vocab": 151936, "sliding_window": 0, - "expert_count": 0 + "expert_count": 0, + "embedding_length": 1024, + "feed_forward_length": 3072, + "expert_feed_forward_length": 0, + "expert_shared_feed_forward_length": 0, + "expert_used_count": 0, + "sliding_window_pattern": 0, + "sliding_window_layers": [], + "shared_kv_layers": 0, + "kv_lora_rank": 0, + "key_length_mla": 0 }, "capabilities": [], "decode_fraction": 1.0, @@ -46,7 +56,17 @@ "context_length": 40960, "n_vocab": 151936, "sliding_window": 0, - "expert_count": 0 + "expert_count": 0, + "embedding_length": 2048, + "feed_forward_length": 6144, + "expert_feed_forward_length": 0, + "expert_shared_feed_forward_length": 0, + "expert_used_count": 0, + "sliding_window_pattern": 0, + "sliding_window_layers": [], + "shared_kv_layers": 0, + "kv_lora_rank": 0, + "key_length_mla": 0 }, "capabilities": [], "decode_fraction": 1.0, @@ -77,7 +97,17 @@ "context_length": 40960, "n_vocab": 151936, "sliding_window": 0, - "expert_count": 0 + "expert_count": 0, + "embedding_length": 2560, + "feed_forward_length": 9728, + "expert_feed_forward_length": 0, + "expert_shared_feed_forward_length": 0, + "expert_used_count": 0, + "sliding_window_pattern": 0, + "sliding_window_layers": [], + "shared_kv_layers": 0, + "kv_lora_rank": 0, + "key_length_mla": 0 }, "capabilities": [], "decode_fraction": 1.0, @@ -108,7 +138,17 @@ "context_length": 40960, "n_vocab": 151936, "sliding_window": 0, - "expert_count": 0 + "expert_count": 0, + "embedding_length": 4096, + "feed_forward_length": 12288, + "expert_feed_forward_length": 0, + "expert_shared_feed_forward_length": 0, + "expert_used_count": 0, + "sliding_window_pattern": 0, + "sliding_window_layers": [], + "shared_kv_layers": 0, + "kv_lora_rank": 0, + "key_length_mla": 0 }, "capabilities": [], "decode_fraction": 1.0, @@ -139,7 +179,17 @@ "context_length": 40960, "n_vocab": 151936, "sliding_window": 0, - "expert_count": 0 + "expert_count": 0, + "embedding_length": 5120, + "feed_forward_length": 17408, + "expert_feed_forward_length": 0, + "expert_shared_feed_forward_length": 0, + "expert_used_count": 0, + "sliding_window_pattern": 0, + "sliding_window_layers": [], + "shared_kv_layers": 0, + "kv_lora_rank": 0, + "key_length_mla": 0 }, "capabilities": [], "decode_fraction": 1.0, @@ -170,7 +220,17 @@ "context_length": 40960, "n_vocab": 151936, "sliding_window": 0, - "expert_count": 0 + "expert_count": 0, + "embedding_length": 5120, + "feed_forward_length": 25600, + "expert_feed_forward_length": 0, + "expert_shared_feed_forward_length": 0, + "expert_used_count": 0, + "sliding_window_pattern": 0, + "sliding_window_layers": [], + "shared_kv_layers": 0, + "kv_lora_rank": 0, + "key_length_mla": 0 }, "capabilities": [], "decode_fraction": 1.0, diff --git a/surfsense_local/backend/modules/llm/catalog/dependencies.py b/surfsense_local/backend/modules/llm/catalog/dependencies.py index 8bce7e5d03..4235f49a1b 100644 --- a/surfsense_local/backend/modules/llm/catalog/dependencies.py +++ b/surfsense_local/backend/modules/llm/catalog/dependencies.py @@ -7,6 +7,7 @@ from fastapi import Depends from modules.llm.catalog.manifest import ( + SCHEMA_VERSION, CuratedModelsManifest, load_curated_models, ) @@ -22,7 +23,7 @@ def get_catalog_service() -> CatalogService: except (OSError, ValueError): # A broken manifest must not take the screen down: installed models and # a local .gguf import still work without it. - manifest = CuratedModelsManifest(schema_version=3, models=[]) + manifest = CuratedModelsManifest(schema_version=SCHEMA_VERSION, models=[]) models_dir = settings.llamacpp_models_dir or Path("models") return CatalogService( manifest, diff --git a/surfsense_local/backend/modules/llm/catalog/manifest.py b/surfsense_local/backend/modules/llm/catalog/manifest.py index 3c4e9f3473..4c5418c769 100644 --- a/surfsense_local/backend/modules/llm/catalog/manifest.py +++ b/surfsense_local/backend/modules/llm/catalog/manifest.py @@ -17,7 +17,7 @@ from modules.llm.fit import ModelShape -SCHEMA_VERSION = 3 +SCHEMA_VERSION = 4 class ModelShapeSpec(BaseModel): @@ -37,8 +37,25 @@ class ModelShapeSpec(BaseModel): value_length: int = Field(gt=0) context_length: int = Field(gt=0) n_vocab: int = Field(gt=0) + + # Required, unlike the optional fields on `ModelShape`. A committed entry is + # authored by a script that read a real header, so a missing width here is a + # manifest written by hand or by an older script, and the compute buffer + # would quietly price it as though the model had no layers to compute. The + # app refuses to start on that rather than shipping a confident wrong badge. + embedding_length: int = Field(gt=0) + feed_forward_length: int = Field(gt=0) + sliding_window: int = Field(default=0, ge=0) expert_count: int = Field(default=0, ge=0) + expert_feed_forward_length: int = Field(default=0, ge=0) + expert_shared_feed_forward_length: int = Field(default=0, ge=0) + expert_used_count: int = Field(default=0, ge=0) + sliding_window_pattern: int = Field(default=0, ge=0) + sliding_window_layers: list[bool] = Field(default_factory=list) + shared_kv_layers: int = Field(default=0, ge=0) + kv_lora_rank: int = Field(default=0, ge=0) + key_length_mla: int = Field(default=0, ge=0) class Variant(BaseModel): @@ -82,7 +99,7 @@ class CuratedModel(BaseModel): # A list from the start, with one thing in it. Costs nothing now and is the # only part of this schema that is expensive to add later, because changing - # it means schema_version 4 and re-authoring every entry. + # it means a schema version bump and re-authoring every entry. variants: list[Variant] = Field(min_length=1) @property @@ -92,7 +109,11 @@ def model_shape(self) -> ModelShape: This is what lets a curated row be priced on first paint with no network and nothing downloaded. """ - return ModelShape(**self.shape.model_dump()) + fields = self.shape.model_dump() + # JSON has no tuples and the shape is frozen, so the one list field is + # converted here rather than leaving a mutable member on a frozen value. + fields["sliding_window_layers"] = tuple(fields["sliding_window_layers"]) + return ModelShape(**fields) class CuratedModelsManifest(BaseModel): diff --git a/surfsense_local/backend/tests/unit/llm/catalog/test_manifest.py b/surfsense_local/backend/tests/unit/llm/catalog/test_manifest.py index 2739bbce77..97f5a599d8 100644 --- a/surfsense_local/backend/tests/unit/llm/catalog/test_manifest.py +++ b/surfsense_local/backend/tests/unit/llm/catalog/test_manifest.py @@ -1,4 +1,4 @@ -"""The curated manifest, schema 3. +"""The curated manifest, schema 4. Source, not build output: a person commits these numbers, so the schema is what catches a mistake before review does. @@ -31,6 +31,8 @@ def entry(**overrides) -> dict: "value_length": 128, "context_length": 40960, "n_vocab": 151936, + "embedding_length": 4096, + "feed_forward_length": 12288, }, "capabilities": [], "decode_fraction": 1.0, @@ -51,7 +53,7 @@ def entry(**overrides) -> dict: def manifest(models: list[dict] | None = None, **overrides) -> dict: """A whole manifest around one or more entries.""" - return {"schema_version": 3, "models": models or [entry()], **overrides} + return {"schema_version": SCHEMA_VERSION, "models": models or [entry()], **overrides} def test_a_well_formed_manifest_loads() -> None: @@ -62,10 +64,22 @@ def test_a_well_formed_manifest_loads() -> None: assert parsed.models[0].shape.block_count == 36 -def test_schema_two_is_refused_rather_than_read_as_three() -> None: - """Silently reading a v2 entry would drop every shape and price nothing.""" +def test_an_older_schema_is_refused_rather_than_read_as_this_one() -> None: + """Silently reading a v3 entry would leave the compute widths unset, and the + scratch term would price every model as though it had nothing to compute.""" with pytest.raises(ValidationError, match="unsupported curated-model schema"): - CuratedModelsManifest.model_validate(manifest() | {"schema_version": 2}) + CuratedModelsManifest.model_validate(manifest() | {"schema_version": 3}) + + +def test_a_shape_without_the_widths_a_compute_buffer_needs_is_refused() -> None: + """Optional on `ModelShape`, required here. A committed entry was authored + by a script that read a real header, so a missing width is a manifest edited + by hand, and the badge it produces would be confident and wrong.""" + shape = entry()["shape"] + del shape["embedding_length"] + + with pytest.raises(ValidationError, match="embedding_length"): + CuratedModelsManifest.model_validate(manifest(models=[entry(shape=shape)])) def test_an_entry_without_a_shape_is_refused() -> None: @@ -148,7 +162,21 @@ def test_the_shipped_manifest_is_valid_and_prices_offline() -> None: assert parsed.models for model in parsed.models: assert model.shape.block_count > 0 + # The widths the scratch estimate is sized from. A regenerated manifest + # that dropped them would price every row against one constant again. + assert model.shape.embedding_length > 0 + assert model.shape.feed_forward_length > 0 assert model.variants for variant in model.variants: assert variant.rank > 0 assert variant.quantization + + +def test_every_shipped_entry_reaches_the_estimator_as_a_shape() -> None: + """The property the catalog calls on first paint. A list field arriving from + JSON has to become a tuple, because the shape it feeds is frozen.""" + for model in load_curated_models().models: + shape = model.model_shape + + assert isinstance(shape.sliding_window_layers, tuple) + assert shape.embedding_length == model.shape.embedding_length From a3da20ded40e18a31f835ba8533427fb8c13bfed Mon Sep 17 00:00:00 2001 From: Anish Sarkar <104695310+AnishSarkar22@users.noreply.github.com> Date: Mon, 21 Sep 2026 17:01:34 +0530 Subject: [PATCH 45/60] fix(catalog): price rows at the precision the loader will choose --- .../modules/llm/catalog/recommendation.py | 11 +- .../backend/modules/llm/catalog/rows.py | 22 +++- .../catalog/test_badge_matches_the_load.py | 113 ++++++++++++++++++ .../unit/llm/catalog/test_recommendation.py | 48 ++++++++ .../tests/unit/llm/catalog/test_rows.py | 1 + 5 files changed, 191 insertions(+), 4 deletions(-) create mode 100644 surfsense_local/backend/tests/unit/llm/catalog/test_badge_matches_the_load.py diff --git a/surfsense_local/backend/modules/llm/catalog/recommendation.py b/surfsense_local/backend/modules/llm/catalog/recommendation.py index bf3ed2cbd1..a2c76a5703 100644 --- a/surfsense_local/backend/modules/llm/catalog/recommendation.py +++ b/surfsense_local/backend/modules/llm/catalog/recommendation.py @@ -17,7 +17,7 @@ from dataclasses import dataclass from modules.llm.catalog.manifest import CuratedModel, Variant -from modules.llm.fit import FitState, HardwareBudget, estimate +from modules.llm.fit import FitState, HardwareBudget, estimate, planned_precision from modules.llm.fit.estimate import FitVerdict # > ponytail: provisional, and possibly unnecessary. On the one machine measured @@ -47,7 +47,14 @@ def recommend( candidates: list[Recommendation] = [] for entry in curated: for variant in entry.variants: - verdict = estimate(entry.model_shape, variant.size_bytes, budget) + # At the cache the loader would choose, so the gate judges the + # configuration that will actually run rather than a slower one. + precision = planned_precision( + entry.model_shape, variant.size_bytes, budget + ) + verdict = estimate( + entry.model_shape, variant.size_bytes, budget, precision=precision + ) if verdict.state is FitState.TOO_BIG: continue if verdict.offload_fraction > MAX_OFFLOAD: diff --git a/surfsense_local/backend/modules/llm/catalog/rows.py b/surfsense_local/backend/modules/llm/catalog/rows.py index cd131b92ce..15fa10d45b 100644 --- a/surfsense_local/backend/modules/llm/catalog/rows.py +++ b/surfsense_local/backend/modules/llm/catalog/rows.py @@ -9,7 +9,16 @@ from dataclasses import dataclass from modules.llm.catalog.manifest import CuratedModel, Variant -from modules.llm.fit import Badge, FitState, FitVerdict, HardwareBudget, badge, estimate +from modules.llm.fit import ( + Badge, + FitState, + FitVerdict, + HardwareBudget, + ModelShape, + badge, + estimate, + planned_precision, +) _FIT_ORDER = {FitState.FITS: 0, FitState.PARTIAL: 1, FitState.TOO_BIG: 2} @@ -53,6 +62,12 @@ def curated_rows( ) +def _fit(shape: ModelShape, weights_bytes: int, budget: HardwareBudget) -> FitVerdict: + """One build's verdict, at the cache the loader would give it.""" + precision = planned_precision(shape, weights_bytes, budget) + return estimate(shape, weights_bytes, budget, precision=precision) + + def _row(model: CuratedModel, budget: HardwareBudget) -> CatalogRow: """The best build of one model: a row is a model, not a file. @@ -60,8 +75,11 @@ def _row(model: CuratedModel, budget: HardwareBudget) -> CatalogRow: choosing a model. It takes the best state among its builds, and the install action uses the build that produced it. """ + # Priced at the precision the loader will actually choose, not at the + # default. Pricing f16 here while `plan_load` picks q8_0 badged a model + # `Reduced speed` that the runtime then placed entirely on the device. priced = [ - (variant, estimate(model.model_shape, variant.size_bytes, budget)) + (variant, _fit(model.model_shape, variant.size_bytes, budget)) for variant in model.variants ] variant, fit = min( diff --git a/surfsense_local/backend/tests/unit/llm/catalog/test_badge_matches_the_load.py b/surfsense_local/backend/tests/unit/llm/catalog/test_badge_matches_the_load.py new file mode 100644 index 0000000000..125a664cd8 --- /dev/null +++ b/surfsense_local/backend/tests/unit/llm/catalog/test_badge_matches_the_load.py @@ -0,0 +1,113 @@ +"""The badge has to describe the load the app will actually perform. + +The screen said `Reduced speed` for Qwen3 4B on an 8 GB Mac while `plan_load` +placed it entirely on the GPU, because the row was priced with a full cache and +the loader chose a quantized one. Two modules, two answers, one model. + +Parametrised over every budget shape the app can produce rather than the Mac +alone, because the precision rule is pure arithmetic and must not start +depending on which machine it runs on. +""" + +import pytest + +from modules.llm.catalog import curated_rows, load_curated_models +from modules.llm.catalog.recommendation import recommend +from modules.llm.fit import HardwareBudget, plan_load + +pytestmark = pytest.mark.unit + +MIB = 1024**2 + +BUDGETS = { + # Windows or Linux with a card: two real pools, so spilling is a transfer. + "discrete-6gb": HardwareBudget(5234 * MIB, 6002 * MIB, 1024 * MIB, 22750 * MIB, False, True), + "discrete-24gb": HardwareBudget(23000 * MIB, 24576 * MIB, 1024 * MIB, 32000 * MIB, False, True), + # Apple Silicon: one memory with two ceilings on it. + "unified-8gb": HardwareBudget(5222 * MIB, 5461 * MIB, 1024 * MIB, 6144 * MIB, True, True), + "unified-16gb": HardwareBudget(10922 * MIB, 11468 * MIB, 1024 * MIB, 14336 * MIB, True, True), + # No card ggml can use, on any platform. Nothing to spill from. + "no-gpu-16gb": HardwareBudget(0, 0, 1024 * MIB, 14 * 1024 * MIB, False, False), + "no-gpu-4gb": HardwareBudget(0, 0, 1024 * MIB, 3 * 1024 * MIB, False, False), + # A card the runtime cannot reach: priced against the host, like no GPU. + "broken-install": HardwareBudget(0, 0, 1024 * MIB, 30 * 1024 * MIB, False, False), +} + +CURATED = load_curated_models().models + + +def _live(budget: HardwareBudget, factor: float) -> HardwareBudget: + """The same machine with more or less free right now than the shelf assumes.""" + return HardwareBudget( + device_free_bytes=int(budget.device_free_bytes * factor), + device_total_bytes=budget.device_total_bytes, + fit_reserve_bytes=budget.fit_reserve_bytes, + ram_available_bytes=int(budget.ram_available_bytes * factor), + uma=budget.uma, + has_gpu=budget.has_gpu, + ) + + +# None is the catalog's own call. The other two are what `reprice` passes: a +# busy machine, and an idle one with more free than the shelf was priced for. +LIVE_FACTORS = (None, 0.4, 1.6) + + +@pytest.mark.parametrize("budget_name", BUDGETS) +@pytest.mark.parametrize("factor", LIVE_FACTORS) +def test_every_row_is_badged_as_it_will_be_loaded(budget_name, factor) -> None: + """The invariant that was broken, on every machine shape. + + A row whose badge and load plan disagree is a promise the app does not keep, + in whichever direction: pessimistic sells a model short, optimistic tells + someone a model will be fast when it will not. + + Swept over what is free at load as well as over the machine, because + `reprice` passes both budgets and the widening search reads the live one. + An idle machine is the case that caught a second mismatch: widening against + a more generous `live` and reporting against capacity said PARTIAL for a row + the catalog had badged FITS. + """ + budget = BUDGETS[budget_name] + live = None if factor is None else _live(budget, factor) + + shapes = {model.model_id: model.model_shape for model in CURATED} + + for row in curated_rows(CURATED, budget): + plan = plan_load(shapes[row.model_id], row.variant.size_bytes, budget, live=live) + + assert row.fit.state is plan.verdict.state, ( + f"{row.label} badges {row.fit.state.value} and loads " + f"{plan.verdict.state.value} at {plan.precision.value}" + ) + + +@pytest.mark.parametrize("budget_name", BUDGETS) +def test_the_star_is_badged_as_it_will_be_loaded(budget_name) -> None: + """The one row a user is steered to, so the one that must not mislead.""" + budget = BUDGETS[budget_name] + pick = recommend(CURATED, budget) + if pick is None: + return + + plan = plan_load(pick.entry.model_shape, pick.variant.size_bytes, budget) + + assert pick.verdict.state is plan.verdict.state + + +def test_a_model_resident_only_with_a_quantized_cache_reads_full_speed() -> None: + """The measured case. Qwen3 4B on an 8 GB Mac needs 4857 MiB with a full + cache against 4198 resident, and 3777 with a quantized one, so it is placed + entirely on the GPU and the badge has to say so.""" + budget = BUDGETS["unified-8gb"] + rows = {row.label: row for row in curated_rows(CURATED, budget)} + + assert rows["Qwen3 4B"].badge.verdict == "Full speed" + + +def test_a_machine_with_no_gpu_is_never_told_about_a_graphics_card() -> None: + """Whatever the precision rule decides, there is no device to be too big for.""" + for name in ("no-gpu-16gb", "no-gpu-4gb", "broken-install"): + for row in curated_rows(CURATED, BUDGETS[name]): + assert "graphics card" not in row.badge.reason + assert row.badge.verdict in {"Works here", "Won't fit"} diff --git a/surfsense_local/backend/tests/unit/llm/catalog/test_recommendation.py b/surfsense_local/backend/tests/unit/llm/catalog/test_recommendation.py index 9487466c96..61a2ddad40 100644 --- a/surfsense_local/backend/tests/unit/llm/catalog/test_recommendation.py +++ b/surfsense_local/backend/tests/unit/llm/catalog/test_recommendation.py @@ -26,6 +26,8 @@ def qwen(block_count: int) -> dict: "value_length": 128, "context_length": 40960, "n_vocab": 151936, + "embedding_length": 4096, + "feed_forward_length": 12288, } @@ -118,3 +120,49 @@ def test_the_policy_ranges_over_builds_so_a_smaller_one_can_rescue_an_entry() -> assert pick is not None assert pick.variant.quantization == "Q4_K_M" + + +def test_a_machine_with_no_gpu_still_gets_a_recommendation() -> None: + """The measured bug: nothing was ever starred on a CPU only machine. + + Every row came back as a full spill, which the speed gate then refused, so + the one screen whose job is to choose a model chose nothing on the hardware + that most needs the help. + """ + no_gpu = HardwareBudget(0, 0, 1024 * MIB, 14 * 1024 * MIB, False, False) + + pick = recommend(CURATED, no_gpu) + + assert pick is not None + assert pick.entry.label == "Qwen3 8B" + + +def test_a_machine_with_no_gpu_is_still_refused_what_it_cannot_hold() -> None: + """The star moves down the ladder rather than off it. + + 2000 MiB holds the 0.6B, which needs 1436 with a quantized cache, and not + the 1.7B, which needs 2113 even with one. + """ + tiny = HardwareBudget(0, 0, 1024 * MIB, 2000 * MIB, False, False) + + pick = recommend(CURATED, tiny) + + assert pick is not None + assert pick.entry.label == "Qwen3 0.6B" + + +def test_a_quantized_cache_is_allowed_to_rescue_a_larger_model() -> None: + """The star has to name the model the loader will actually run. + + 2600 MiB does not hold the 1.7B with a full cache, at 2953 MiB, and does + hold it with a quantized one, at 2113. `plan_load` already chose the + quantized cache here, so pricing the star against the full one pointed at + the 0.6B while the app was perfectly able to run the rung above it. That + mattered most on the machines with the least to choose from. + """ + small = HardwareBudget(0, 0, 1024 * MIB, 2600 * MIB, False, False) + + pick = recommend(CURATED, small) + + assert pick is not None + assert pick.entry.label == "Qwen3 1.7B" diff --git a/surfsense_local/backend/tests/unit/llm/catalog/test_rows.py b/surfsense_local/backend/tests/unit/llm/catalog/test_rows.py index 2935a9edc6..b6d9eed818 100644 --- a/surfsense_local/backend/tests/unit/llm/catalog/test_rows.py +++ b/surfsense_local/backend/tests/unit/llm/catalog/test_rows.py @@ -22,6 +22,7 @@ def model(label: str, blocks: int, variants: list[dict]) -> CuratedModel: "architecture": "qwen3", "block_count": blocks, "head_count_kv": 8, "key_length": 128, "value_length": 128, "context_length": 40960, "n_vocab": 151936, + "embedding_length": 4096, "feed_forward_length": 12288, }, "variants": variants, }) From 56673be37473ad09a50a1b6c6c4ab84ea92358d2 Mon Sep 17 00:00:00 2001 From: Anish Sarkar <104695310+AnishSarkar22@users.noreply.github.com> Date: Mon, 21 Sep 2026 17:01:40 +0530 Subject: [PATCH 46/60] feat(catalog): serve the GPU diagnosis and warm the probe off the boot path --- surfsense_local/backend/api/main.py | 32 ++- .../backend/modules/llm/catalog/router.py | 5 +- .../backend/modules/llm/catalog/schemas.py | 8 + .../backend/modules/llm/catalog/service.py | 195 ++++++++++++++++-- .../integration/llm/test_catalog_routes.py | 26 +++ .../tests/unit/llm/catalog/test_reprice.py | 46 +++++ .../tests/unit/llm/catalog/test_service.py | 169 +++++++++++++++ 7 files changed, 448 insertions(+), 33 deletions(-) diff --git a/surfsense_local/backend/api/main.py b/surfsense_local/backend/api/main.py index 603fda3a09..719284b527 100644 --- a/surfsense_local/backend/api/main.py +++ b/surfsense_local/backend/api/main.py @@ -1,4 +1,5 @@ import logging +import threading from collections.abc import AsyncIterator from contextlib import asynccontextmanager @@ -63,23 +64,34 @@ async def lifespan(app: FastAPI) -> AsyncIterator[None]: ensure_default_workspace(session) session.commit() app.state.session_factory = session_factory - # The preset is derived from what is on disk, so rebuild it at boot - # rather than only after an install. Without this a model imported from - # disk, or installed before this existed, loads at llama.cpp's own - # default window instead of the one the fit calculation chose, and a - # stale section keeps advertising a model whose file is gone. - _rewrite_generation_preset() + # Off the startup path, on a thread. Both halves are slow for the same + # reason: the preset is priced against the devices, and taking the + # device probe costs about 19 seconds on a Mac the first time, while + # Metal compiles its shader libraries. Done here, /health would not + # answer until it finished. + # + # The preset itself is rebuilt at boot rather than only after an + # install, because without it a model imported from disk loads at + # llama.cpp's own default window instead of the one the fit calculation + # chose, and a stale section keeps advertising a model whose file is + # gone. + threading.Thread(target=_warm_catalog, name="catalog-warm", daemon=True).start() yield finally: engine.dispose() -def _rewrite_generation_preset() -> None: - """Best effort: a preset that cannot be written must not stop the API.""" +def _warm_catalog() -> None: + """Best effort: neither the probe nor the preset may stop the API. + + A daemon thread, so a probe wedged on a driver call cannot hold shutdown + open. A request arriving meanwhile waits on the service's own lock rather + than starting a second probe. + """ try: - get_catalog_service().reprice() + get_catalog_service().warm() except Exception: - logger.exception("could not rewrite the generation preset at startup") + logger.exception("could not warm the model catalog at startup") def create_app() -> FastAPI: diff --git a/surfsense_local/backend/modules/llm/catalog/router.py b/surfsense_local/backend/modules/llm/catalog/router.py index beff919273..27322d70ff 100644 --- a/surfsense_local/backend/modules/llm/catalog/router.py +++ b/surfsense_local/backend/modules/llm/catalog/router.py @@ -40,8 +40,10 @@ @router.get("/system", response_model=SystemRead, summary="This machine's memory") def read_system(service: CatalogServiceDep) -> dict: """No scan, no button, no network. The allocator's own view.""" + inventory = service.inventory() return { "budget": _budget(service.budget()), + "gpu_status": inventory.gpu_status.value, "devices": [ { "name": device.name, @@ -50,7 +52,7 @@ def read_system(service: CatalogServiceDep) -> dict: "total_bytes": device.total_bytes, "free_bytes": device.free_bytes, } - for device in service.devices() + for device in inventory.devices ], } @@ -62,6 +64,7 @@ def read_catalog(service: CatalogServiceDep, session: SessionDep) -> dict: catalog = service.catalog() return { "budget": _budget(catalog.budget), + "gpu_status": catalog.gpu_status.value, "curated": [ _row(row, catalog, selected, service) for row in catalog.curated ], diff --git a/surfsense_local/backend/modules/llm/catalog/schemas.py b/surfsense_local/backend/modules/llm/catalog/schemas.py index 95579de9fe..55908c7224 100644 --- a/surfsense_local/backend/modules/llm/catalog/schemas.py +++ b/surfsense_local/backend/modules/llm/catalog/schemas.py @@ -32,6 +32,10 @@ class BudgetRead(BaseModel): class SystemRead(BaseModel): budget: BudgetRead devices: list[DeviceRead] + # Beside the budget rather than in it. The budget is memory; this says + # whether the runtime can reach the hardware at all, and a machine whose + # card the runtime cannot see must never read as a machine without one. + gpu_status: str class BadgeRead(BaseModel): @@ -85,6 +89,7 @@ class InstalledRowRead(BaseModel): class CatalogRead(BaseModel): budget: BudgetRead + gpu_status: str curated: list[CatalogRowRead] installed: list[InstalledRowRead] recommended_model_id: str | None @@ -123,6 +128,9 @@ class RepoRead(BaseModel): architecture: str context_length: int supported: bool + # Installable without one, but it will answer badly in a chat, so this warns + # rather than blocks. Eligibility is not fit and neither is a warning. + chat_template: bool = True builds: list[BuildRead] # Eligibility is not fit: a model can be FITS and still refused here, and the # two must never render as one thing. diff --git a/surfsense_local/backend/modules/llm/catalog/service.py b/surfsense_local/backend/modules/llm/catalog/service.py index b98f07bfa5..9ab4ca5f6f 100644 --- a/surfsense_local/backend/modules/llm/catalog/service.py +++ b/surfsense_local/backend/modules/llm/catalog/service.py @@ -12,6 +12,7 @@ import asyncio import logging import secrets +import threading import time from collections.abc import AsyncIterator from dataclasses import dataclass @@ -29,26 +30,56 @@ list_builds, search_models, ) -from modules.llm.fit import HardwareBudget, badge, estimate, plan_load +from modules.llm.catalog.search.eligibility import RepoFacts, read_repo_facts +from modules.llm.fit import ( + FitState, + HardwareBudget, + ModelShape, + badge, + estimate, + plan_load, + planned_precision, +) from modules.llm.gguf import shape_from_file, shape_from_url from modules.llm.hardware import ( BudgetMode, Device, + GpuStatus, + SystemInventory, available_bytes, build_budget, + fit_target_mib, + os_reports_gpu, probe_devices, + system_inventory, ) +from modules.llm.hardware.inventory import OsGpu, Probe from modules.llm.providers.llamacpp import ( PRESET_FILE, PROVIDER, ModelPreset, RouterClient, download_gguf, + is_projector, + projector_for, write_presets, ) logger = logging.getLogger(__name__) +# Priced from the listing alone when a header cannot be read. Every +# architecture-derived term is zero, so `need` is the file size and nothing +# else, and the row is marked approximate so the screen can say so. +_SIZE_ONLY_SHAPE = ModelShape( + architecture="", + block_count=0, + head_count_kv=0, + key_length=0, + value_length=0, + context_length=0, + n_vocab=0, +) + @dataclass(frozen=True) class InstalledRow: @@ -73,6 +104,10 @@ class InstallPlan: class Catalog: budget: HardwareBudget devices: tuple[Device, ...] + # Whether the runtime can reach this machine's graphics hardware. Beside the + # budget rather than inside it: the budget is memory, this is a diagnosis, + # and a broken install must not read as a machine that has no card. + gpu_status: GpuStatus curated: tuple[CatalogRow, ...] installed: tuple[InstalledRow, ...] recommended_model_id: str | None @@ -87,12 +122,19 @@ def __init__( models_dir: Path, library_dir: Path, runtime_url: str = "http://127.0.0.1:8080", + *, + probe: Probe = probe_devices, + os_gpu: OsGpu = os_reports_gpu, ) -> None: self._runtime_url = runtime_url self._manifest = manifest self._models_dir = models_dir self._library_dir = library_dir - self._devices: tuple[Device, ...] | None = None + self._probe = probe + self._os_gpu = os_gpu + self._inventory: SystemInventory | None = None + # The startup warm and the first request race for this. + self._inventory_lock = threading.Lock() self._tickets = TicketStore() self._catalog_ids: dict[str, tuple[str, str]] = {} # One pull at a time: two concurrent downloads compete for the same disk @@ -138,6 +180,11 @@ def reprice(self) -> None: live = self.budget(BudgetMode.LIVE) presets = [] for path in sorted(self._models_dir.glob("*.gguf")): + # A projector is half of a vision model, not a model. It has a + # header and a size like any other file here, so without this it + # would be offered as something to chat with. + if is_projector(path): + continue try: shape = shape_from_file(path) except (OSError, ValueError): @@ -146,12 +193,33 @@ def reprice(self) -> None: # over a file nobody asked it to load. logger.warning("skipping unreadable model %s", path.name) continue - plan = plan_load(shape, path.stat().st_size, budget, live=live) + + projector = projector_for(path, self._projector_names(path)) + mmproj_bytes = projector.stat().st_size if projector else 0 + plan = plan_load( + shape, path.stat().st_size, budget, live=live, mmproj_bytes=mmproj_bytes + ) presets.append( - ModelPreset(path.stem, str(path), plan.n_ctx, plan.precision) + ModelPreset( + model_id=path.stem, + path=str(path), + n_ctx=plan.n_ctx, + precision=plan.precision, + fit_target_mib=fit_target_mib(mmproj_bytes), + mmproj_path=str(projector) if projector else None, + ) ) write_presets(self._models_dir / PRESET_FILE, presets) + def _projector_names(self, model_path: Path) -> list[str]: + """What the manifest calls this model's projector, if it is a curated one.""" + return [ + variant.mmproj + for model in self._manifest.models + for variant in model.variants + if variant.mmproj and variant.file == model_path.name + ] + def resolve_install(self, catalog_id: str) -> InstallPlan | None: """Turn an id from either tier into a build, or None if it has gone stale.""" pinned = self._catalog_ids.get(catalog_id) @@ -218,6 +286,8 @@ async def repo(self, repo: str) -> dict: listing. """ budget = self.budget() + approximate = False + chat_template = True async with httpx.AsyncClient(follow_redirects=True) as client: builds = await list_builds(client, repo) if not builds: @@ -226,16 +296,43 @@ async def repo(self, repo: str) -> dict: "architecture": "", "context_length": 0, "supported": False, + "chat_template": True, "builds": [], "ineligible_reason": "no single file build in this repo", } - url = f"https://huggingface.co/{repo}/resolve/main/{builds[0].file}" - shape = await shape_from_url(client, url) - supported = is_supported(shape.architecture) + # Hugging Face has already parsed the first GGUF's header, so the + # architecture is free. Reading it first means an architecture + # llama.cpp cannot run costs no range request against a file we were + # never going to install. + facts = await read_repo_facts(client, repo) + if facts is not None: + chat_template = facts.has_chat_template + if facts is not None and not is_supported(facts.architecture): + return self._ineligible(repo, facts, builds) + + try: + url = f"https://huggingface.co/{repo}/resolve/main/{builds[0].file}" + shape = await shape_from_url(client, url) + except (httpx.HTTPError, ValueError): + # The listing is enough to say how large each build is, and a + # size alone still orders the ladder. Refusing the whole repo + # over one unreadable header would hide builds the user can run. + logger.warning("could not read the header for %s", repo) + shape = _SIZE_ONLY_SHAPE + approximate = True + + supported = is_supported(shape.architecture) if not approximate else True rows = [] for build in builds: - fit = estimate(shape, build.size_bytes, budget) + # The same precision rule the curated rows and the loader use, so + # a searched build is described the way it will be loaded. + fit = estimate( + shape, + build.size_bytes, + budget, + precision=planned_precision(shape, build.size_bytes, budget), + ) text = badge(fit, budget) rows.append( { @@ -250,7 +347,7 @@ async def repo(self, repo: str) -> dict: "need_bytes": fit.need_bytes, "budget_bytes": fit.budget_bytes, "offload_fraction": fit.offload_fraction, - "approximate": False, + "approximate": approximate, }, "badge": {"verdict": text.verdict, "reason": text.reason}, "can_install": fit.can_install and supported, @@ -261,27 +358,79 @@ async def repo(self, repo: str) -> dict: "architecture": shape.architecture, "context_length": shape.context_length, "supported": supported, + "chat_template": chat_template, "builds": rows, "ineligible_reason": None if supported else f"llama.cpp cannot run the {shape.architecture} architecture", } - def devices(self) -> tuple[Device, ...]: - """Probe once and remember. + def _ineligible(self, repo: str, facts: RepoFacts, builds: list) -> dict: + """A repo llama.cpp cannot run, described from the listing alone. + + The builds are still listed, at their real sizes, because the screen is + answering "what is in here" as well as "can I run it", and an empty repo + reads as a broken page rather than an unsupported model. + """ + return { + "repo": repo, + "architecture": facts.architecture, + "context_length": facts.context_length, + "supported": False, + "chat_template": facts.has_chat_template, + "builds": [ + { + "catalog_id": "", + "file": build.file, + "quantization": build.quantization, + "size_bytes": build.size_bytes, + "fit": { + "state": FitState.TOO_BIG.value, + "need_bytes": 0, + "budget_bytes": 0, + "offload_fraction": 0.0, + "approximate": True, + }, + "badge": { + "verdict": "Cannot run", + "reason": f"llama.cpp has no support for {facts.architecture}", + }, + "can_install": False, + } + for build in builds + ], + "ineligible_reason": ( + f"llama.cpp cannot run the {facts.architecture} architecture" + ), + } + + def inventory(self) -> SystemInventory: + """Probe once and remember, across every thread that asks. The first call compiles Metal shaders and costs about 19 seconds on a - Mac; every call after is tens of milliseconds. Warm this in the - background at launch rather than on the path of the first render. + Mac; every call after is tens of milliseconds. The lock is what makes + the startup warm worth having: a request arriving during it waits for + that one probe instead of starting a second. """ - if self._devices is None: - try: - self._devices = tuple(probe_devices(self._library_dir)) - except OSError: - # No staged runtime. The catalog still renders, priced against - # the host, and installs are disabled higher up. - self._devices = () - return self._devices + with self._inventory_lock: + if self._inventory is None: + self._inventory = system_inventory( + self._library_dir, probe=self._probe, os_gpu=self._os_gpu + ) + return self._inventory + + def devices(self) -> tuple[Device, ...]: + """What ggml can see. Empty is a real answer; `gpu_status` says which.""" + return self.inventory().devices + + def warm(self) -> None: + """Take the probe and write the preset, off the path of the first render. + + Ordered: the preset is priced against the devices, so probing second + would mean probing twice. + """ + self.inventory() + self.reprice() def budget(self, mode: BudgetMode = BudgetMode.CAPACITY) -> HardwareBudget: """Capacity by default: a catalog is a shelf, not a launch decision. @@ -292,12 +441,14 @@ def budget(self, mode: BudgetMode = BudgetMode.CAPACITY) -> HardwareBudget: return build_budget(self.devices(), available_bytes(), mode=mode) def catalog(self) -> Catalog: + inventory = self.inventory() budget = self.budget() rows = curated_rows(self._manifest.models, budget) pick = recommend(self._manifest.models, budget) return Catalog( budget=budget, - devices=self.devices(), + devices=inventory.devices, + gpu_status=inventory.gpu_status, curated=tuple(rows), installed=tuple(self._installed()), recommended_model_id=pick.entry.model_id if pick else None, diff --git a/surfsense_local/backend/tests/integration/llm/test_catalog_routes.py b/surfsense_local/backend/tests/integration/llm/test_catalog_routes.py index 0857581316..22ff46375e 100644 --- a/surfsense_local/backend/tests/integration/llm/test_catalog_routes.py +++ b/surfsense_local/backend/tests/integration/llm/test_catalog_routes.py @@ -104,3 +104,29 @@ async def test_a_curated_row_can_be_installed_by_its_opaque_id( reply = await client.post("/llm/install", json={"catalog_id": catalog_id}) assert reply.status_code != 422 + + +async def test_the_system_route_says_whether_the_runtime_sees_the_hardware( + client: AsyncClient, +) -> None: + """An empty device list means nothing on its own: a machine with no card and + a machine whose card the runtime cannot reach both report one.""" + body = (await client.get("/llm/system")).json() + + assert body["gpu_status"] in {"present", "absent", "broken_install", "unknown"} + + +async def test_the_catalog_carries_the_same_diagnosis(client: AsyncClient) -> None: + """The screen that renders the badges is the one that has to explain them.""" + body = (await client.get("/llm/catalog")).json() + + assert body["gpu_status"] in {"present", "absent", "broken_install", "unknown"} + + +async def test_the_budget_never_reports_a_gpu_diagnosis(client: AsyncClient) -> None: + """The budget is memory. Phase 8.3 asks for a distinct state, not a flag + folded into the numbers, so a reader of one cannot mistake it for the other. + """ + body = (await client.get("/llm/system")).json() + + assert "gpu_status" not in body["budget"] diff --git a/surfsense_local/backend/tests/unit/llm/catalog/test_reprice.py b/surfsense_local/backend/tests/unit/llm/catalog/test_reprice.py index 951547ddef..8e60564915 100644 --- a/surfsense_local/backend/tests/unit/llm/catalog/test_reprice.py +++ b/surfsense_local/backend/tests/unit/llm/catalog/test_reprice.py @@ -149,3 +149,49 @@ async def test_waiting_for_the_runtime_gives_up_rather_than_hanging( reachable = await service.wait_until_servable("m", timeout=0.3, interval=0.05) assert reachable is False + + +def test_a_projector_is_never_offered_as_a_model(tmp_path: Path) -> None: + """Half of a vision model, not a model. It has a header and a size like any + other file here, so without a rule it gets its own section and the user is + offered something that cannot answer a question.""" + a_model(tmp_path / "Qwen3-VL-Q4_K_M.gguf") + a_model(tmp_path / "mmproj-F16.gguf") + service = CatalogService(load_curated_models(), tmp_path, tmp_path) + + service.reprice() + ini = (tmp_path / PRESET_FILE).read_text() + + assert "[Qwen3-VL-Q4_K_M]" in ini + assert "[mmproj-F16]" not in ini + + +def test_a_vision_model_names_its_projector_and_reserves_room_for_it( + tmp_path: Path, +) -> None: + """`--fit` allocates the projector after it has finished placing layers and + does not count it while deciding, so the margin has to carry its bytes.""" + a_model(tmp_path / "Qwen3-VL-Q4_K_M.gguf") + projector = tmp_path / "mmproj-F16.gguf" + a_model(projector) + padding = 600 * 1024**2 - projector.stat().st_size + projector.write_bytes(projector.read_bytes() + b"\0" * padding) + service = CatalogService(load_curated_models(), tmp_path, tmp_path) + + service.reprice() + ini = (tmp_path / PRESET_FILE).read_text() + + assert f"mmproj = {projector}" in ini + assert "fit-target = 1624" in ini + + +def test_a_text_model_leaves_the_margin_at_llama_cpps_own_default( + tmp_path: Path, +) -> None: + """One GiB, which is exactly what the badge subtracted.""" + a_model(tmp_path / "Qwen3-8B-Q4_K_M.gguf") + service = CatalogService(load_curated_models(), tmp_path, tmp_path) + + service.reprice() + + assert "fit-target = 1024" in (tmp_path / PRESET_FILE).read_text() diff --git a/surfsense_local/backend/tests/unit/llm/catalog/test_service.py b/surfsense_local/backend/tests/unit/llm/catalog/test_service.py index c10888d807..5e5a5392e0 100644 --- a/surfsense_local/backend/tests/unit/llm/catalog/test_service.py +++ b/surfsense_local/backend/tests/unit/llm/catalog/test_service.py @@ -1,10 +1,15 @@ """The screen's data, assembled without a network call.""" +import threading +import time from pathlib import Path +import httpx import pytest from modules.llm.catalog import CatalogService, load_curated_models +from modules.llm.hardware import GpuStatus +from tests.unit.llm.gguf.build import STRING, UINT32, array, gguf, kv pytestmark = pytest.mark.unit @@ -60,3 +65,167 @@ def test_the_probe_runs_once_however_often_the_catalog_is_asked(service) -> None first = service.devices() assert service.devices() is first + + +def test_the_machine_is_probed_once_however_many_threads_ask(tmp_path) -> None: + """The startup warm and a first request race for this. + + Without the lock both take the probe, which on a Mac means compiling Metal's + shader libraries twice, concurrently, at about 19 seconds each. + """ + calls = [] + + def slow_probe(_): + # Long enough that a second thread arriving unsynchronised would start + # its own probe before this one has recorded a result to reuse. + time.sleep(0.2) + calls.append(1) + return [] + + service = CatalogService( + load_curated_models(), + tmp_path, + tmp_path, + probe=slow_probe, + os_gpu=lambda: False, + ) + + threads = [threading.Thread(target=service.inventory) for _ in range(4)] + for thread in threads: + thread.start() + for thread in threads: + thread.join(timeout=5) + + assert len(calls) == 1 + + +def test_a_broken_install_is_never_reported_as_a_machine_without_a_card( + tmp_path, +) -> None: + """Phase 8.3's rule, at the seam the screen actually reads.""" + + def no_runtime(_): + raise OSError("no staged runtime") + + service = CatalogService( + load_curated_models(), tmp_path, tmp_path, probe=no_runtime, os_gpu=lambda: True + ) + + catalog = service.catalog() + + assert catalog.gpu_status is GpuStatus.BROKEN_INSTALL + assert catalog.devices == () + + +_REAL_ASYNC_CLIENT = httpx.AsyncClient + + +def _serve_repo(monkeypatch, transport: httpx.MockTransport) -> None: + """Point every client the service opens at a transport. + + The real class is captured at import, because replacing the name with a + lambda that then calls the name is a recursion rather than a fake. + """ + monkeypatch.setattr( + httpx, + "AsyncClient", + lambda **kwargs: _REAL_ASYNC_CLIENT(transport=transport), + ) + + +def _repo_transport(recorder: list[str], *, gguf: dict | None, header: bytes | None): + """Answers the three calls `repo()` makes, and records which were made.""" + + def handle(request: httpx.Request) -> httpx.Response: + recorder.append(str(request.url)) + if "/tree/main" in request.url.path: + return httpx.Response( + 200, + json=[ + {"path": "model-Q4_K_M.gguf", "size": 5_000_000_000}, + {"path": "model-Q8_0.gguf", "size": 8_000_000_000}, + ], + ) + if "/resolve/main/" in request.url.path: + if header is None: + return httpx.Response(500) + return httpx.Response(200, content=header) + return httpx.Response(200, json={"gguf": gguf} if gguf else {}) + + return httpx.MockTransport(handle) + + +async def test_an_architecture_llama_cpp_cannot_run_costs_no_header_read( + tmp_path, monkeypatch +) -> None: + """The listing already said so. Reading a range out of a multi gigabyte file + to re-learn it is bytes spent on a question that was already answered.""" + calls: list[str] = [] + transport = _repo_transport(calls, gguf={"architecture": "whisper"}, header=None) + _serve_repo(monkeypatch, transport) + service = CatalogService(load_curated_models(), tmp_path, tmp_path, probe=lambda _: []) + + body = await service.repo("some/whisper-gguf") + + assert body["supported"] is False + assert "whisper" in body["ineligible_reason"] + assert not any("/resolve/main/" in url for url in calls) + + +async def test_an_unsupported_repo_still_lists_what_is_in_it( + tmp_path, monkeypatch +) -> None: + """The screen answers "what is in here" as well as "can I run it", and an + empty repo reads as a broken page rather than an unsupported model.""" + transport = _repo_transport([], gguf={"architecture": "whisper"}, header=None) + _serve_repo(monkeypatch, transport) + service = CatalogService(load_curated_models(), tmp_path, tmp_path, probe=lambda _: []) + + body = await service.repo("some/whisper-gguf") + + assert len(body["builds"]) == 2 + assert all(not build["can_install"] for build in body["builds"]) + + +async def test_a_header_that_cannot_be_read_prices_from_size_and_says_so( + tmp_path, monkeypatch +) -> None: + """Refusing the whole repo over one unreadable header would hide builds the + user can run. The size alone still orders the ladder, and `approximate` is + what lets the screen show that the number is a rough one.""" + transport = _repo_transport([], gguf={"architecture": "qwen3"}, header=None) + _serve_repo(monkeypatch, transport) + service = CatalogService(load_curated_models(), tmp_path, tmp_path, probe=lambda _: []) + + body = await service.repo("unsloth/Qwen3-8B-GGUF") + + assert body["builds"] + assert all(build["fit"]["approximate"] for build in body["builds"]) + + +async def test_a_readable_header_is_never_called_approximate( + tmp_path, monkeypatch +) -> None: + """The flag means "priced from the file size alone", so a real read must + clear it or the screen hedges a number it did not need to hedge.""" + header = gguf( + [ + kv("general.architecture", STRING, "qwen3"), + kv("qwen3.block_count", UINT32, 36), + kv("qwen3.attention.head_count_kv", UINT32, 8), + kv("qwen3.attention.key_length", UINT32, 128), + kv("qwen3.attention.value_length", UINT32, 128), + kv("qwen3.context_length", UINT32, 40960), + kv("qwen3.embedding_length", UINT32, 4096), + kv("qwen3.feed_forward_length", UINT32, 12288), + array("tokenizer.ggml.tokens", STRING, ["a", "b"]), + ] + ) + transport = _repo_transport([], gguf={"architecture": "qwen3"}, header=header) + _serve_repo(monkeypatch, transport) + service = CatalogService(load_curated_models(), tmp_path, tmp_path, probe=lambda _: []) + + body = await service.repo("unsloth/Qwen3-8B-GGUF") + + assert body["supported"] is True + assert not any(build["fit"]["approximate"] for build in body["builds"]) From dd16963f40c744dd902234cd10a657c74fbc7bc7 Mon Sep 17 00:00:00 2001 From: Anish Sarkar <104695310+AnishSarkar22@users.noreply.github.com> Date: Mon, 21 Sep 2026 17:01:48 +0530 Subject: [PATCH 47/60] chore(scripts): author curated entries from the header's tensor table --- .../backend/scripts/curated/huggingface.py | 13 ++- .../backend/scripts/curated/projector.py | 39 ++++++++ .../backend/scripts/curated/tensor_bytes.py | 91 ++++++++++++++++++ .../backend/scripts/refresh_curated_models.py | 45 +++++---- .../backend/tests/unit/scripts/__init__.py | 0 .../backend/tests/unit/scripts/conftest.py | 15 +++ .../tests/unit/scripts/test_tensor_bytes.py | 95 +++++++++++++++++++ 7 files changed, 276 insertions(+), 22 deletions(-) create mode 100644 surfsense_local/backend/scripts/curated/projector.py create mode 100644 surfsense_local/backend/scripts/curated/tensor_bytes.py create mode 100644 surfsense_local/backend/tests/unit/scripts/__init__.py create mode 100644 surfsense_local/backend/tests/unit/scripts/conftest.py create mode 100644 surfsense_local/backend/tests/unit/scripts/test_tensor_bytes.py diff --git a/surfsense_local/backend/scripts/curated/huggingface.py b/surfsense_local/backend/scripts/curated/huggingface.py index d037106961..c373265d94 100644 --- a/surfsense_local/backend/scripts/curated/huggingface.py +++ b/surfsense_local/backend/scripts/curated/huggingface.py @@ -5,8 +5,7 @@ import httpx -from modules.llm.fit import ModelShape -from modules.llm.gguf import shape_from_url +from modules.llm.gguf import GgufHeader, header_from_url API = "https://huggingface.co/api/models" FILES = "https://huggingface.co/{repo}/resolve/main/{file}" @@ -38,7 +37,11 @@ def find_variant(client: httpx.Client, repo: str, quantization: str) -> tuple[st return chosen["path"], chosen["size"] -async def read_shape(repo: str, file: str) -> ModelShape: - """The architecture fields, over an HTTP range. No weights are downloaded.""" +async def read_header(repo: str, file: str) -> GgufHeader: + """The header, over an HTTP range. No weights are downloaded. + + The whole header rather than the shape: the tensor table is what says how + much of a mixture of experts is read per token. + """ async with httpx.AsyncClient(follow_redirects=True, timeout=120) as client: - return await shape_from_url(client, FILES.format(repo=repo, file=file)) + return await header_from_url(client, FILES.format(repo=repo, file=file)) diff --git a/surfsense_local/backend/scripts/curated/projector.py b/surfsense_local/backend/scripts/curated/projector.py new file mode 100644 index 0000000000..4feecf49c6 --- /dev/null +++ b/surfsense_local/backend/scripts/curated/projector.py @@ -0,0 +1,39 @@ +"""Find the vision projector that belongs with a build. + +Authoring time only. A vision model ships its projector as a separate file in +the same repo, and llama.cpp's fitter does not count it, so the manifest has to +name it for the launch path to reserve room. +""" + +import httpx + +API = "https://huggingface.co/api/models" + +# Highest fidelity first. A projector is small beside the weights, so the cost of +# the largest is a rounding error against the quality of what it feeds the model. +_PREFERRED = ("f16", "bf16", "f32", "q8_0") + + +def find_projector(client: httpx.Client, repo: str) -> tuple[str, int] | None: + """The projector file in `repo` and its size, or None for a text model.""" + reply = client.get(f"{API}/{repo}/tree/main", timeout=60) + reply.raise_for_status() + + candidates = [ + row + for row in reply.json() + if row["path"].endswith(".gguf") and "mmproj" in row["path"].lower() + ] + if not candidates: + return None + + def rank(row: dict) -> tuple[int, int]: + name = row["path"].lower() + precision = next( + (i for i, token in enumerate(_PREFERRED) if token in name), len(_PREFERRED) + ) + # Shortest name breaks a tie, the same rule find_variant uses. + return precision, len(row["path"]) + + chosen = min(candidates, key=rank) + return chosen["path"], chosen["size"] diff --git a/surfsense_local/backend/scripts/curated/tensor_bytes.py b/surfsense_local/backend/scripts/curated/tensor_bytes.py new file mode 100644 index 0000000000..f55de70675 --- /dev/null +++ b/surfsense_local/backend/scripts/curated/tensor_bytes.py @@ -0,0 +1,91 @@ +"""How much of a build is read to decode one token. + +Authoring time only. A dense model reads every weight per token and the answer +is 1.0; a mixture of experts reads its shared weights plus the few experts the +router picked, which is the whole reason such a model is worth shipping to a +machine that could not hold a dense one of the same size. + +Both halves come from llama.cpp rather than from conventions restated here: the +sizes from its quantization table, and the expert tensor names from its own +tensor name table. A rename upstream is then an import error rather than a +pattern that quietly stops matching and reports every mixture of experts as +dense. +""" + +from gguf.constants import ( + GGML_QUANT_SIZES, + MODEL_TENSOR, + TENSOR_NAMES, + GGMLQuantizationType, +) + +from modules.llm.gguf import GgufHeader, HeaderTensor + +# The expert weights a router reads per token, as llama.cpp defines that set. +# Its own fitter matches `blk.\d+.ffn_(up|down|gate_up|gate)_(ch|)exps` and calls +# that comment "matches all MoE tensors" (common/fit.cpp), which is the same +# seven members below and deliberately not `ffn_norm_exps`: a per expert norm is +# a vector beside these matrices, and the set that decides what llama.cpp moves +# to the host is the set that decides what it reads from the host. +_EXPERT_TENSORS = ( + MODEL_TENSOR.FFN_UP_EXP, + MODEL_TENSOR.FFN_DOWN_EXP, + MODEL_TENSOR.FFN_GATE_EXP, + MODEL_TENSOR.FFN_GATE_UP_EXP, + MODEL_TENSOR.FFN_UP_CHEXP, + MODEL_TENSOR.FFN_DOWN_CHEXP, + MODEL_TENSOR.FFN_GATE_CHEXP, +) + +# "blk.{bid}.ffn_up_exps" -> "ffn_up_exps". The part that names what a tensor is, +# with the block number and any "weight" suffix left to the comparison. +_EXPERT_KINDS = frozenset( + TENSOR_NAMES[tensor].rsplit(".", 1)[-1] for tensor in _EXPERT_TENSORS +) + + +def is_expert(name: str) -> bool: + """Whether a tensor is one the router reads only for the experts it picked. + + Compared segment by segment rather than by substring: `blk.0.ffn_up_exps` + and `blk.0.ffn_up_exps.weight` are the same tensor, and a name that merely + contains one of these words is not. + """ + return any(part in _EXPERT_KINDS for part in name.split(".")) + + +def tensor_bytes(tensor: HeaderTensor) -> int: + """Bytes one tensor occupies, or 0 for a type this build cannot size. + + Zero rather than an exception: an unknown id is one tensor we cannot price, + not a header we should refuse, and the caller is computing a ratio. + """ + try: + block_size, type_size = GGML_QUANT_SIZES[GGMLQuantizationType(tensor.ggml_type)] + except (ValueError, KeyError): + return 0 + elements = 1 + for dim in tensor.dims: + elements *= dim + return elements * type_size // block_size + + +def decode_fraction(header: GgufHeader, expert_used_count: int, expert_count: int) -> float: + """Fraction of the file's bytes read per decoded token. + + Dense models return exactly 1.0, including any model whose header does not + describe experts at all, so the field is safe to write for every entry. + """ + if expert_count <= 0 or expert_used_count <= 0 or expert_used_count >= expert_count: + return 1.0 + + total = 0 + read = 0.0 + active = expert_used_count / expert_count + for tensor in header.tensors: + size = tensor_bytes(tensor) + total += size + read += size * active if is_expert(tensor.name) else size + if total <= 0: + return 1.0 + return round(read / total, 4) diff --git a/surfsense_local/backend/scripts/refresh_curated_models.py b/surfsense_local/backend/scripts/refresh_curated_models.py index 40534e5644..a33e4b4cc2 100644 --- a/surfsense_local/backend/scripts/refresh_curated_models.py +++ b/surfsense_local/backend/scripts/refresh_curated_models.py @@ -26,12 +26,17 @@ import asyncio import json +from dataclasses import asdict from pathlib import Path import httpx -from curated.huggingface import find_variant, read_shape +from curated.huggingface import find_variant, read_header +from curated.projector import find_projector +from curated.tensor_bytes import decode_fraction from modules.llm.catalog import SCHEMA_VERSION, CuratedModelsManifest +from modules.llm.fit import ModelShape +from modules.llm.gguf import to_shape OUT = Path(__file__).resolve().parents[1] / "modules/llm/catalog/curated-models.json" RANK_BASIS = "llmfit-1.1.11" @@ -57,16 +62,32 @@ VALIDATED: set[str] = set() +def _shape_fields(shape: ModelShape) -> dict: + """Every field the estimator reads, taken from the dataclass itself. + + Listing them by hand here is what dropped two of them last time, and a + dropped width prices the compute buffer as though the model had no layers. + """ + fields = asdict(shape) + # JSON has no tuples. + fields["sliding_window_layers"] = list(fields["sliding_window_layers"]) + return fields + + async def build() -> dict: models = [] with httpx.Client(follow_redirects=True) as client: for model_id, family, label, params, repo, quant, rank in ENTRIES: file, size = find_variant(client, repo, quant) - shape = await read_shape(repo, file) + header = await read_header(repo, file) + shape = to_shape(header) + projector = find_projector(client, repo) + fraction = decode_fraction(header, shape.expert_used_count, shape.expert_count) print( f" {label:12s} {file:34s} {size / 1e9:5.2f} GB " f"{shape.block_count} blocks, {shape.head_count_kv} kv-heads, " - f"ctx {shape.context_length}" + f"ctx {shape.context_length}, reads {fraction:.2f} per token" + + (f", projector {projector[0]}" if projector else "") ) models.append( { @@ -74,26 +95,16 @@ async def build() -> dict: "family": family, "label": label, "parameter_count": params, - "shape": { - "architecture": shape.architecture, - "block_count": shape.block_count, - "head_count_kv": shape.head_count_kv, - "key_length": shape.key_length, - "value_length": shape.value_length, - "context_length": shape.context_length, - "n_vocab": shape.n_vocab, - "sliding_window": shape.sliding_window, - "expert_count": shape.expert_count, - }, - "capabilities": [], - "decode_fraction": 1.0, + "shape": _shape_fields(shape), + "capabilities": ["vision"] if projector else [], + "decode_fraction": fraction, "variants": [ { "repo": repo, "file": file, "quantization": quant, "size_bytes": size, - "mmproj": None, + "mmproj": projector[0] if projector else None, "rank": rank, "rank_basis": RANK_BASIS, "validated": file in VALIDATED, diff --git a/surfsense_local/backend/tests/unit/scripts/__init__.py b/surfsense_local/backend/tests/unit/scripts/__init__.py new file mode 100644 index 0000000000..e69de29bb2 diff --git a/surfsense_local/backend/tests/unit/scripts/conftest.py b/surfsense_local/backend/tests/unit/scripts/conftest.py new file mode 100644 index 0000000000..9e4c7d2094 --- /dev/null +++ b/surfsense_local/backend/tests/unit/scripts/conftest.py @@ -0,0 +1,15 @@ +"""Put the authoring helpers on the path the way running the script does. + +`uv run scripts/refresh_curated_models.py` puts `scripts/` on `sys.path`, which +is why that file imports `curated.x` rather than `scripts.curated.x`. Importing +these modules any other way here would test them under a name they are never +loaded with. +""" + +import sys +from pathlib import Path + +SCRIPTS = Path(__file__).resolve().parents[3] / "scripts" + +if str(SCRIPTS) not in sys.path: + sys.path.insert(0, str(SCRIPTS)) diff --git a/surfsense_local/backend/tests/unit/scripts/test_tensor_bytes.py b/surfsense_local/backend/tests/unit/scripts/test_tensor_bytes.py new file mode 100644 index 0000000000..22505d0999 --- /dev/null +++ b/surfsense_local/backend/tests/unit/scripts/test_tensor_bytes.py @@ -0,0 +1,95 @@ +"""How much of a build a router reads per decoded token. + +Authoring time code, but the number it writes goes into the shipped manifest and +feeds a speed estimate, so an expert tensor it fails to recognise reports a +mixture of experts as dense and over-states what the model costs to run. +""" + +import pytest +from curated.tensor_bytes import decode_fraction, is_expert, tensor_bytes +from gguf.constants import MODEL_TENSOR, TENSOR_NAMES + +from modules.llm.gguf import GgufHeader, HeaderTensor + +pytestmark = pytest.mark.unit + +F32, Q4_K = 0, 12 + + +def header(*tensors: HeaderTensor) -> GgufHeader: + """A header carrying nothing but a tensor table, which is all this reads.""" + return GgufHeader(metadata={}, tensors=tensors) + + +def test_the_expert_names_come_from_llama_cpps_own_table() -> None: + """Not restated here. A rename upstream has to break the import rather than + quietly stop matching, because a miss reads as a dense model.""" + assert is_expert(TENSOR_NAMES[MODEL_TENSOR.FFN_UP_EXP].format(bid=0)) + assert is_expert(TENSOR_NAMES[MODEL_TENSOR.FFN_DOWN_CHEXP].format(bid=7)) + + +def test_a_stored_tensor_carries_a_suffix_the_table_does_not() -> None: + """The table names `blk.{bid}.ffn_up_exps`; a file stores + `blk.0.ffn_up_exps.weight`. Comparing whole names would match neither.""" + assert is_expert("blk.0.ffn_up_exps.weight") + + +def test_the_per_expert_norm_is_not_counted_as_an_expert_weight() -> None: + """llama.cpp's own MoE pattern covers the seven matrices and not this, and + the set that decides what it moves to the host is the set that decides what + it reads back from the host.""" + assert not is_expert("blk.0.ffn_norm_exps.weight") + + +def test_an_ordinary_dense_tensor_is_not_mistaken_for_one() -> None: + """`ffn_up` is not `ffn_up_exps`, and a substring test would say it is.""" + assert not is_expert("blk.0.ffn_up.weight") + assert not is_expert("blk.0.attn_q.weight") + assert not is_expert("token_embd.weight") + + +def test_a_dense_model_reads_every_byte_it_holds() -> None: + """Exactly 1.0, so the field is safe to write for every entry.""" + dense = header( + HeaderTensor("blk.0.attn_q.weight", (4096, 4096), F32), + HeaderTensor("blk.0.ffn_up.weight", (4096, 11008), F32), + ) + + assert decode_fraction(dense, expert_used_count=0, expert_count=0) == 1.0 + + +def test_only_the_experts_a_router_picks_are_counted() -> None: + """Eight experts, two read. The shared weights are read whole, so the answer + is between the activated share and one rather than either of them.""" + shared = HeaderTensor("blk.0.attn_q.weight", (64, 64), F32) + experts = HeaderTensor("blk.0.ffn_up_exps.weight", (64, 64, 8), F32) + both = header(shared, experts) + + shared_bytes = tensor_bytes(shared) + expert_bytes = tensor_bytes(experts) + expected = (shared_bytes + expert_bytes * 0.25) / (shared_bytes + expert_bytes) + + assert decode_fraction(both, 2, 8) == round(expected, 4) + + +def test_a_model_that_routes_to_every_expert_reads_everything() -> None: + """Used equals total is dense in all but name, and dividing by it would + report a saving that is not there.""" + moe = header(HeaderTensor("blk.0.ffn_up_exps.weight", (64, 64, 8), F32)) + + assert decode_fraction(moe, 8, 8) == 1.0 + + +def test_a_quantized_expert_is_sized_by_its_own_block_layout() -> None: + """The ratio is over bytes, and experts are not always quantized like the + rest of the file, so a parameter count would answer a different question.""" + quantized = HeaderTensor("blk.0.ffn_up_exps.weight", (256, 256, 8), Q4_K) + full = HeaderTensor("blk.0.ffn_up_exps.weight", (256, 256, 8), F32) + + assert tensor_bytes(quantized) < tensor_bytes(full) + + +def test_a_tensor_type_this_build_cannot_size_is_skipped_not_fatal() -> None: + """ggml adds types, and one we cannot price is one tensor missing from a + ratio rather than a header to refuse.""" + assert tensor_bytes(HeaderTensor("blk.0.ffn_up_exps.weight", (64, 64), 9999)) == 0 From 1db86b19588f4f740e7e6e6626b8ca1b46edf7f8 Mon Sep 17 00:00:00 2001 From: Anish Sarkar <104695310+AnishSarkar22@users.noreply.github.com> Date: Mon, 21 Sep 2026 17:01:55 +0530 Subject: [PATCH 48/60] fix(bundling): ship the curated manifest from its catalog path --- surfsense_local/backend/bundling/api.spec | 4 +- .../tests/packaging/test_spec_data_files.py | 67 +++++++++++++++++++ 2 files changed, 69 insertions(+), 2 deletions(-) create mode 100644 surfsense_local/backend/tests/packaging/test_spec_data_files.py diff --git a/surfsense_local/backend/bundling/api.spec b/surfsense_local/backend/bundling/api.spec index c3963bde12..e8e840abb8 100644 --- a/surfsense_local/backend/bundling/api.spec +++ b/surfsense_local/backend/bundling/api.spec @@ -21,8 +21,8 @@ from common import BACKEND, database_inputs datas, binaries, hiddenimports = database_inputs() datas.append( ( - str(BACKEND / "modules" / "llm" / "recommendations" / "curated-models.json"), - "modules/llm/recommendations", + str(BACKEND / "modules" / "llm" / "catalog" / "curated-models.json"), + "modules/llm/catalog", ) ) # Read by path, so the analyser cannot see it. Without this every remote model diff --git a/surfsense_local/backend/tests/packaging/test_spec_data_files.py b/surfsense_local/backend/tests/packaging/test_spec_data_files.py new file mode 100644 index 0000000000..9e8557af4b --- /dev/null +++ b/surfsense_local/backend/tests/packaging/test_spec_data_files.py @@ -0,0 +1,67 @@ +"""Every data file the spec bundles has to exist on disk. + +PyInstaller is forgiving here: a `datas` entry naming a path that is not there +produces a warning in a long build log and a binary that is missing the file, +which then fails at runtime on a user's machine and nowhere else. The catalog +manifest reached exactly that state once, when the module it lived in was +renamed and the spec was not. + +A unit test cannot catch it, because nothing imports the spec. This reads it the +way PyInstaller does, as Python, with the globals it expects. +""" + +import ast +from pathlib import Path + +import pytest + +pytestmark = pytest.mark.packaging + +BACKEND = Path(__file__).resolve().parents[2] +SPECS = sorted((BACKEND / "bundling").glob("*.spec")) + + +def literal_data_paths(spec: Path) -> list[str]: + """The source paths in `datas.append((...))` calls that are plain strings. + + Only the literal ones: an entry built by `collect_data_files` is resolved by + PyInstaller at build time and is not ours to check. + """ + tree = ast.parse(spec.read_text()) + found: list[str] = [] + for node in ast.walk(tree): + if not isinstance(node, ast.Call) or not isinstance(node.func, ast.Attribute): + continue + if node.func.attr != "append" or not node.args: + continue + first = node.args[0] + if not isinstance(first, ast.Tuple) or not first.elts: + continue + source = first.elts[0] + # str(BACKEND / "a" / "b"): recover the parts of the division chain. + if isinstance(source, ast.Call) and getattr(source.func, "id", None) == "str": + parts = [] + current = source.args[0] + while isinstance(current, ast.BinOp) and isinstance(current.op, ast.Div): + if isinstance(current.right, ast.Constant): + parts.insert(0, str(current.right.value)) + current = current.left + if parts and getattr(current, "id", None) == "BACKEND": + found.append(str(Path(*parts))) + return found + + +@pytest.mark.parametrize("spec", SPECS, ids=lambda p: p.name) +def test_every_bundled_data_path_exists(spec: Path) -> None: + """A path here that is not on disk ships a binary missing that file.""" + missing = [rel for rel in literal_data_paths(spec) if not (BACKEND / rel).exists()] + + assert missing == [], f"{spec.name} bundles paths that do not exist: {missing}" + + +def test_the_curated_manifest_is_one_of_them() -> None: + """The file that was missed. Without it a frozen build has no curated list, + and the model screen is empty on the machine that most needs it.""" + bundled = {rel for spec in SPECS for rel in literal_data_paths(spec)} + + assert "modules/llm/catalog/curated-models.json" in bundled From 974c830c1522aff662c75262934a879a61ed0c91 Mon Sep 17 00:00:00 2001 From: Anish Sarkar <104695310+AnishSarkar22@users.noreply.github.com> Date: Mon, 21 Sep 2026 17:02:06 +0530 Subject: [PATCH 49/60] feat(frontend): surface the GPU diagnosis and the chat template warning --- .../src/features/model-catalog/api.ts | 24 +++++++- .../model-catalog/model-catalog-page.tsx | 17 +++++- .../model-catalog/model-catalog.test.tsx | 25 ++++++++ .../features/model-catalog/model-search.tsx | 59 +++++++++++-------- 4 files changed, 94 insertions(+), 31 deletions(-) diff --git a/surfsense_local/frontend/src/features/model-catalog/api.ts b/surfsense_local/frontend/src/features/model-catalog/api.ts index 36e61ea1ed..e886ae3dc6 100644 --- a/surfsense_local/frontend/src/features/model-catalog/api.ts +++ b/surfsense_local/frontend/src/features/model-catalog/api.ts @@ -68,12 +68,23 @@ export type Budget = { has_gpu: boolean } +/** + * Whether the runtime can reach this machine's graphics hardware. + * + * Kept apart from the budget on purpose. An empty device list means nothing on + * its own: a laptop with no card and a workstation whose backend library did + * not ship both report one, and calling the second a machine without a GPU + * tells a user their card is broken when a file is missing from ours. + */ +export type GpuStatus = "present" | "absent" | "broken_install" | "unknown" + /** * Curated plus installed. No `scanned` flag, because there is no scan: the * budget comes from the runtime's own allocator in about 180ms. */ export type ModelCatalog = { budget: Budget + gpu_status: GpuStatus curated: CatalogRow[] installed: InstalledRow[] recommended_model_id: string | null @@ -106,6 +117,11 @@ export type RepoDetail = { architecture: string context_length: number supported: boolean + /** + * Installable without one, but it will answer badly in a chat, so this warns + * rather than blocks. A warning is not a fit state and must not render as one. + */ + chat_template: boolean builds: RepoBuild[] /** Eligibility is not fit: a model can fit and still be refused here. */ ineligible_reason: string | null @@ -222,8 +238,12 @@ export function getModelCatalog(signal?: AbortSignal): Promise { return requestJson("/llm/catalog", { signal }) } -export function getSystem(signal?: AbortSignal): Promise<{ budget: Budget }> { - return requestJson<{ budget: Budget }>("/llm/system", { signal }) +export function getSystem( + signal?: AbortSignal +): Promise<{ budget: Budget; gpu_status: GpuStatus }> { + return requestJson<{ budget: Budget; gpu_status: GpuStatus }>("/llm/system", { + signal, + }) } export function searchModels( diff --git a/surfsense_local/frontend/src/features/model-catalog/model-catalog-page.tsx b/surfsense_local/frontend/src/features/model-catalog/model-catalog-page.tsx index 6439bbfca0..2cf93e76b5 100644 --- a/surfsense_local/frontend/src/features/model-catalog/model-catalog-page.tsx +++ b/surfsense_local/frontend/src/features/model-catalog/model-catalog-page.tsx @@ -17,7 +17,7 @@ import { ScrollShadow } from "@/components/ui/scroll-shadow" import { Spinner } from "@/components/ui/spinner" import { Separator } from "@/components/ui/separator" import { cn } from "@/lib/utils" -import type { Budget, CatalogRow, RepoBuild } from "./api" +import type { Budget, CatalogRow, GpuStatus, RepoBuild } from "./api" import { LocalImageModel } from "./local-image-model" import { ModelCard } from "./model-card" import { ModelFamilyGroup } from "./model-family-group" @@ -33,13 +33,24 @@ function messageFrom(error: unknown) { * What this machine has, in one line. No scan and no button: the figure comes * from the runtime's own allocator in about 180ms, so there is nothing to wait * for and nothing to trigger. + * + * The status is read before the budget, because a machine whose graphics card + * the runtime cannot reach is priced against its processor and would otherwise + * be described as a machine that has no card at all. That is a sentence about + * the user's hardware, and it would be wrong. */ -function hardwareSummary(budget: Budget | undefined) { +function hardwareSummary(budget: Budget | undefined, gpuStatus?: GpuStatus) { const gb = (bytes: number) => `${new Intl.NumberFormat(undefined, { maximumFractionDigits: 1 }).format(bytes / 1e9)} GB` if (!budget) { return ["Checking this computer"] } + if (gpuStatus === "broken_install") { + return [ + "Graphics card not detected by the runtime. Reinstall to fix", + `${gb(budget.ram_available_bytes)} memory`, + ] + } if (!budget.has_gpu) { return [ "Runs on your processor", @@ -241,7 +252,7 @@ export function ModelCatalogPage({

    - {hardwareSummary(data.budget).map((part, index) => ( + {hardwareSummary(data.budget, data.gpu_status).map((part, index) => ( {index > 0 ? ( = {}): ModelCatalog => ({ uma: true, has_gpu: true, }, + gpu_status: "present", curated: [row()], installed: [], recommended_model_id: null, @@ -115,6 +116,29 @@ describe("model catalog", () => { expect(screen.queryByRole("button", { name: /scan/i })).toBeNull() }) + it("says the card was not detected rather than calling the machine CPU only", async () => { + // A missing backend library makes the runtime report no devices, silently, + // with exit 0, on a machine with a working card. Priced against the + // processor it would otherwise read as a machine that has no card, which is + // a sentence about the user's hardware and it would be wrong. + vi.stubGlobal( + "fetch", + serving( + catalog({ + gpu_status: "broken_install", + budget: { ...catalog().budget, has_gpu: false }, + }) + ) + ) + + render() + + expect( + await screen.findByText(/Graphics card not detected by the runtime/) + ).toBeTruthy() + expect(screen.queryByText("Runs on your processor")).toBeNull() + }) + it("installs with only the opaque id", async () => { // The renderer never sends a repo, file, URL, path or quantization. const onSelected = vi.fn() @@ -293,6 +317,7 @@ describe("model catalog", () => { architecture: "qwen3", context_length: 40960, supported: true, + chat_template: true, ineligible_reason: null, builds: [ { diff --git a/surfsense_local/frontend/src/features/model-catalog/model-search.tsx b/surfsense_local/frontend/src/features/model-catalog/model-search.tsx index 1644b45f7d..04893f8edb 100644 --- a/surfsense_local/frontend/src/features/model-catalog/model-search.tsx +++ b/surfsense_local/frontend/src/features/model-catalog/model-search.tsx @@ -63,33 +63,40 @@ function RepoBuilds({ } return ( -

      - {detail.data.builds.map((build) => ( -
    • -
      -
      - {build.quantization} - - - {formatSize(build.size_bytes)} - -
      - -
      - -
      • - ))} -
    +
    +
    + {build.quantization} + + + {formatSize(build.size_bytes)} + +
    + +
    + + + ))} + + ) } From 445ac60e7810a80edbd5a2a4d92aac04c35e2e07 Mon Sep 17 00:00:00 2001 From: Anish Sarkar <104695310+AnishSarkar22@users.noreply.github.com> Date: Mon, 21 Sep 2026 17:02:39 +0530 Subject: [PATCH 50/60] feat(catalog): warn on a searched repo that ships no chat template --- .../modules/llm/catalog/search/__init__.py | 3 + .../modules/llm/catalog/search/eligibility.py | 61 +++++++++++++++++++ .../unit/llm/catalog/test_eligibility.py | 59 ++++++++++++++++++ 3 files changed, 123 insertions(+) create mode 100644 surfsense_local/backend/modules/llm/catalog/search/eligibility.py create mode 100644 surfsense_local/backend/tests/unit/llm/catalog/test_eligibility.py diff --git a/surfsense_local/backend/modules/llm/catalog/search/__init__.py b/surfsense_local/backend/modules/llm/catalog/search/__init__.py index a713982331..8c3b7711fc 100644 --- a/surfsense_local/backend/modules/llm/catalog/search/__init__.py +++ b/surfsense_local/backend/modules/llm/catalog/search/__init__.py @@ -6,16 +6,19 @@ list_builds, search_models, ) +from modules.llm.catalog.search.eligibility import RepoFacts, read_repo_facts from modules.llm.catalog.search.tickets import InstallTicket, TicketStore __all__ = [ "CACHE_SECONDS", "SUPPORTED", "InstallTicket", + "RepoFacts", "RepoFile", "SearchHit", "TicketStore", "is_supported", "list_builds", + "read_repo_facts", "search_models", ] diff --git a/surfsense_local/backend/modules/llm/catalog/search/eligibility.py b/surfsense_local/backend/modules/llm/catalog/search/eligibility.py new file mode 100644 index 0000000000..4efd220f2f --- /dev/null +++ b/surfsense_local/backend/modules/llm/catalog/search/eligibility.py @@ -0,0 +1,61 @@ +"""What Hugging Face already knows about a repo's GGUF files. + +The listing API parses the header of the first GGUF in a repo and serves the +result, which covers both eligibility questions without reading a byte of the +file: whether llama.cpp can run the architecture at all, and whether the model +carries a chat template. + +Worth asking first because the alternative is a range request against a +multi-gigabyte file, and for an architecture llama.cpp cannot run that read +answers a question we had already decided. + +It does not carry the KV shape, so it replaces nothing for a model we can run: +the header read still happens, it just no longer happens for models we cannot. +""" + +from dataclasses import dataclass + +import httpx + +API = "https://huggingface.co/api/models" +TIMEOUT = httpx.Timeout(15.0, connect=10.0) + + +@dataclass(frozen=True) +class RepoFacts: + """What the listing says about the first GGUF in a repo.""" + + architecture: str + context_length: int + has_chat_template: bool + + +async def read_repo_facts(client: httpx.AsyncClient, repo: str) -> RepoFacts | None: + """The parsed header Hugging Face already holds, or None when it holds none. + + None for a repo whose files it has not parsed, which is not a reason to + refuse the repo: the caller falls through to reading the header itself. + """ + try: + reply = await client.get( + f"{API}/{repo}", params={"expand[]": "gguf"}, timeout=TIMEOUT + ) + reply.raise_for_status() + payload = reply.json() + except (httpx.HTTPError, ValueError): + return None + + gguf = payload.get("gguf") if isinstance(payload, dict) else None + if not isinstance(gguf, dict): + return None + + architecture = gguf.get("architecture") + if not isinstance(architecture, str) or not architecture: + return None + + context = gguf.get("context_length") + return RepoFacts( + architecture=architecture, + context_length=int(context) if isinstance(context, int) else 0, + has_chat_template=bool(gguf.get("chat_template")), + ) diff --git a/surfsense_local/backend/tests/unit/llm/catalog/test_eligibility.py b/surfsense_local/backend/tests/unit/llm/catalog/test_eligibility.py new file mode 100644 index 0000000000..c29f15674e --- /dev/null +++ b/surfsense_local/backend/tests/unit/llm/catalog/test_eligibility.py @@ -0,0 +1,59 @@ +"""What Hugging Face already knows, asked before anything is downloaded. + +The listing API parses the first GGUF's header and serves the result, so the +architecture and the chat template are free. Reading them first means a model +llama.cpp cannot run costs no range request against a file nobody will install. +""" + +import httpx +import pytest + +from modules.llm.catalog.search import read_repo_facts + +pytestmark = pytest.mark.unit + +GGUF = { + "total": 8_190_735_360, + "architecture": "qwen3", + "context_length": 40960, + "chat_template": "{%- if tools %}...{%- endif %}", + "eos_token": "<|im_end|>", +} + + +def serving(payload: object, status: int = 200) -> httpx.MockTransport: + """A transport answering the listing call and nothing else.""" + return httpx.MockTransport(lambda request: httpx.Response(status, json=payload)) + + +async def test_the_parsed_header_the_listing_already_holds_is_read() -> None: + """Architecture, window and template, with no bytes of the model fetched.""" + async with httpx.AsyncClient(transport=serving({"gguf": GGUF})) as client: + facts = await read_repo_facts(client, "unsloth/Qwen3-8B-GGUF") + + assert facts is not None + assert facts.architecture == "qwen3" + assert facts.context_length == 40960 + assert facts.has_chat_template + + +async def test_a_model_with_no_chat_template_is_reported_as_such() -> None: + """Installable, but it will answer badly in a chat, so the screen warns.""" + payload = {"gguf": {**GGUF, "chat_template": ""}} + async with httpx.AsyncClient(transport=serving(payload)) as client: + facts = await read_repo_facts(client, "some/repo") + + assert facts is not None + assert not facts.has_chat_template + + +async def test_a_repo_hugging_face_has_not_parsed_is_not_a_refusal() -> None: + """None means "ask the file yourself", not "this repo is bad".""" + async with httpx.AsyncClient(transport=serving({})) as client: + assert await read_repo_facts(client, "some/repo") is None + + +async def test_a_listing_that_fails_is_not_a_refusal_either() -> None: + """The header read is still available, and it is the authoritative one.""" + async with httpx.AsyncClient(transport=serving({}, status=500)) as client: + assert await read_repo_facts(client, "some/repo") is None From a44c0b0d3f59ca7ccb273db09d3ef90b0e32a767 Mon Sep 17 00:00:00 2001 From: Anish Sarkar <104695310+AnishSarkar22@users.noreply.github.com> Date: Mon, 21 Sep 2026 17:02:44 +0530 Subject: [PATCH 51/60] docs(local): record the fit estimator's measurements in phase 7 --- .../api/07-llamacpp-runtime.md | 59 +++++++++++++++++++ 1 file changed, 59 insertions(+) diff --git a/plans/community-local/api/07-llamacpp-runtime.md b/plans/community-local/api/07-llamacpp-runtime.md index 0dbd375220..c891e95016 100644 --- a/plans/community-local/api/07-llamacpp-runtime.md +++ b/plans/community-local/api/07-llamacpp-runtime.md @@ -2384,6 +2384,65 @@ Device query versus advisor, same machine: | Context priced | 8,192 | 16,384 (our real turn) | | Qwen3 8B verdict | **Perfect** | **uses system RAM** | +## Follow-on work, with the measurement each one waits on + +The estimator's terms are implemented and its two verdict bugs are fixed. Three +things are deliberately left, each because it needs a number nobody has taken +rather than code nobody has written. + +### Measurement: a quantized cache against a small spill + +`plan_load` drops to `q8_0` whenever that turns a spill into residency. The +assumption underneath is that a resident model at `q8_0` beats a partly spilled +one at `f16`, and it has never been measured. Unsloth measured `q8_0` generation +at **35 % slower** and ships it off by default, which is evidence the assumption +can be wrong. + +One model, one window, three configurations, on the two machines already used +for the rest of this phase (RTX 3050 under Vulkan, M2 8 GB under Metal): + +| | cache | context | expected placement | +|---|---|---|---| +| A | `f16` | 16384 | spills; record `offloaded N/36` from the server log | +| B | `q8_0` with `flash-attn on` | 16384 | fully resident, 36/36 | +| C | `f16` | 8192 | fully resident, the control | + +Qwen3 4B Q4_K_M, `parallel 1`, `fit-target 1024`. Three prompts of roughly 2K, +8K and 14K grounding tokens, three runs each, median of +`prompt_tokens_seconds` and `predicted_tokens_seconds` from `/metrics`. + +**The rule, fixed before the run so the result cannot be argued with.** If B +decodes at 0.9x of A or better at the 8K prompt, the current preference stands. +If A matches or beats B, `plan_load` prefers a small `f16` spill over `q8_0` +below some layer count, and only then does the policy change. + +Note the app is prefill dominated, roughly 8,000 prompt tokens against 300 +decoded, so the prompt rate is the one that decides this and the decode rate is +the one every published `q8_0` benchmark reports. + +### Deferred: memory mapping policy + +Ollama disables mmap on Metal whenever a load is predicted to spill, and on +Windows with CUDA by default. We pass neither, and have measured neither. The +43 second paged load on an 8 GB Mac that motivated the unified pool rule is the +shape of failure this would address, so it is worth a measurement, but the pool +rule removed the case that produced it. + +### Deferred: a speed gate for the recommendation + +`MAX_OFFLOAD = 0.75` is a placeholder and its own comment says so. The offload +fraction is now counted in whole layers, which changes what the gate fires on, +so the first step is to sweep the curated ladder across the fixture budgets in +`test_properties.py` and record whether it ever fires at all. A gate that never +fires is not protection, it is untested code. + +The replacement is the roofline this document already describes, seeded from +Unsloth's measured constants: 5.5 ms per spilled GiB per decoded token on their +reference host, a cache spill 20x worse than a weight spill, prefill streamed at +55 GiB/s, and no penalty at all on unified memory. `decode_fraction` is now +computed from the tensor table by the authoring script, so the MoE half of that +model has its input. + ## References - [llama.cpp server README](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md) — router mode, `/models`, `/models/sse`, `/props` From 87a8194e1a1f10ab6ffa75e56d346a83a12c3eef Mon Sep 17 00:00:00 2001 From: Anish Sarkar <104695310+AnishSarkar22@users.noreply.github.com> Date: Mon, 21 Sep 2026 19:16:23 +0530 Subject: [PATCH 52/60] fix(chat): budget the prompt from the real context window, not a fixed guess Three failure modes existed even at the old 16384 floor: - A prompt that outgrew the window surfaced as "provider temporarily unavailable" instead of the true, retry-proof cause. Classify llama.cpp's own exceed_context_size_error as its own ChatErrorKind with copy that tells the user what actually happened. - The answer had no max_tokens, so a reply could run into the window's edge and stop mid-sentence with no error at all. Reserve tokens for it up front and cap the request. - History was trimmed against a hardcoded 3000-token constant that had no relationship to the model's actual n_ctx. Derive the budget from the real window instead, read once via a new context_tokens() on the Generator protocol (llama.cpp answers it from /props; a remote endpoint honestly returns None and keeps today's fixed fallback). CONTEXT_FLOOR_TOKENS drops from 16384 to 8192, and plan_load now widens across a fixed set of rungs (8192/16384/32768) instead of an unbounded binary search, so the catalog badge, the preset, and the actual load can only ever agree on one of a small, testable set of windows. 8192 is only safe because of the history-budget fix above: a constant sized for 16384 would have silently overflowed the narrower floor. --- .../backend/modules/chat/budget.py | 66 +++++++++++++++++++ .../backend/modules/chat/errors.py | 30 +++++++++ .../backend/modules/chat/history.py | 24 ++++--- .../backend/modules/chat/router.py | 25 ++++++- .../backend/modules/llm/fit/__init__.py | 8 ++- .../backend/modules/llm/fit/estimate.py | 24 +++++-- .../backend/modules/llm/fit/plan_load.py | 40 ++++++----- .../llm/providers/llamacpp/provider.py | 6 ++ .../llm/providers/openai_compatible/chat.py | 7 ++ .../modules/llm/providers/protocols.py | 9 +++ .../tests/integration/chat/conftest.py | 20 ++++++ .../tests/integration/chat/test_chat.py | 19 ++++++ .../backend/tests/unit/chat/test_budget.py | 41 ++++++++++++ .../backend/tests/unit/chat/test_errors.py | 41 ++++++++++++ .../backend/tests/unit/chat/test_history.py | 16 ++++- .../unit/llm/catalog/test_recommendation.py | 6 +- .../tests/unit/llm/fit/test_estimate.py | 4 +- .../tests/unit/llm/fit/test_plan_load.py | 37 ++++++++++- 18 files changed, 383 insertions(+), 40 deletions(-) create mode 100644 surfsense_local/backend/modules/chat/budget.py create mode 100644 surfsense_local/backend/tests/unit/chat/test_budget.py diff --git a/surfsense_local/backend/modules/chat/budget.py b/surfsense_local/backend/modules/chat/budget.py new file mode 100644 index 0000000000..367c2c2357 --- /dev/null +++ b/surfsense_local/backend/modules/chat/budget.py @@ -0,0 +1,66 @@ +"""How much of a model's context window each part of a turn may spend. + +Four named parts share one window: the system prompt, the retrieved excerpts, +the user's question, and the answer. The answer's share is reserved before +anything else claims it, because llama.cpp stops a reply wherever the window +runs out; spending the reserve on history is how a good answer gets cut off +mid-sentence with no error at all. What is left after the fixed parts and the +reserve is what history gets, so a narrower window means a shorter history +rather than a turn that silently exceeds the window (fix from +07-llamacpp-runtime.md's context-floor research: `HISTORY_BUDGET_TOKENS` used +to be a constant with no relationship to `n_ctx`, which was safe only because +the window was always the same 16384 number). + +An unknown window (a remote endpoint that does not report `n_ctx`) is not a +narrow one; it is an absent fact, and this module cannot tell whether the +fixed parts fit. The fallback there is today's number, unchanged, rather than +a guess. +""" + +# Kept back for the answer before anything else is priced. Never spent on +# history. +ANSWER_RESERVE_TOKENS = 1024 + +# What a rendered system prompt plus the grounding header cost, at the largest +# tier (frontier.md). Approximate: the exact figure is one `/apply-template` +# call away and this is the estimate until that lands. +SYSTEM_PROMPT_TOKENS = 400 + +# The retrieval cap already limits chunks to 480 tokens each at up to 5 hits +# (shared/search.py, worker/ingestion/chunking.py); priced here rather than +# measured so the budget holds even before retrieval runs. +EXCERPTS_TOKENS = 2400 + +# A pasted question can be long; capped so one turn cannot claim the whole +# window and leave nothing for history or the answer. MessageText enforces +# this at the wire (modules/chat/schemas.py). +QUESTION_TOKENS = 1024 + +# Today's number, kept as the fallback for a window this module cannot see: +# unchanged so a remote model with no reported `n_ctx` behaves exactly as it +# did before this budget existed. +DEFAULT_HISTORY_TOKENS = 3000 + + +def history_budget(n_ctx: int | None) -> int: + """Room left for prior turns, after the fixed parts and the answer reserve. + + Floors at zero rather than going negative: a window narrower than the + fixed parts plus the reserve keeps no history at all, which is correct, + not an error, on the smallest rung this app will ever run at. + """ + if n_ctx is None: + return DEFAULT_HISTORY_TOKENS + spent = SYSTEM_PROMPT_TOKENS + EXCERPTS_TOKENS + QUESTION_TOKENS + ANSWER_RESERVE_TOKENS + return max(0, n_ctx - spent) + + +def answer_max_tokens(n_ctx: int | None) -> int | None: + """`max_tokens` for the answer request, or None to leave the endpoint's + own default in place. + + None on an unknown window: capping a reply to this app's reserve on an + endpoint whose real window might be far larger would truncate answers for + a limit that was never really theirs. + """ + return ANSWER_RESERVE_TOKENS if n_ctx is not None else None diff --git a/surfsense_local/backend/modules/chat/errors.py b/surfsense_local/backend/modules/chat/errors.py index 7390ab6347..73f91d97b9 100644 --- a/surfsense_local/backend/modules/chat/errors.py +++ b/surfsense_local/backend/modules/chat/errors.py @@ -16,6 +16,7 @@ class ChatErrorKind(enum.StrEnum): PROVIDER_NOT_FOUND = "provider_not_found" PROVIDER_RATE_LIMITED = "provider_rate_limited" PROVIDER_UNAVAILABLE = "provider_unavailable" + CONTEXT_TOO_LONG = "context_too_long" NETWORK = "network" TIMEOUT = "timeout" UNKNOWN = "unknown" @@ -33,6 +34,10 @@ class ChatErrorKind(enum.StrEnum): ChatErrorKind.PROVIDER_UNAVAILABLE: ( "The model provider is temporarily unavailable. Try again shortly." ), + ChatErrorKind.CONTEXT_TOO_LONG: ( + "This conversation is too long for the model's context window. " + "Start a new chat or pick a model with a larger window." + ), ChatErrorKind.TIMEOUT: "The model took too long to respond. Try again.", ChatErrorKind.UNKNOWN: "Something went wrong generating a reply. Try again.", } @@ -49,6 +54,11 @@ class ChatErrorKind(enum.StrEnum): _AUTH_STATUS_CODES = {401, 403} +# llama.cpp's own name for the one 400 that retrying can never fix: the +# prompt does not get shorter on its own. Every other 400 keeps the generic +# PROVIDER_UNAVAILABLE bucket, because only this one shape names a fix. +_CONTEXT_TOO_LONG_ERROR_TYPE = "exceed_context_size_error" + def classify_chat_error(exc: Exception, provider: str) -> tuple[ChatErrorKind, str]: """Sort a generation failure into a kind, with the plain-language text to show. @@ -70,6 +80,8 @@ def classify_chat_error(exc: Exception, provider: str) -> tuple[ChatErrorKind, s kind = ChatErrorKind.PROVIDER_NOT_FOUND elif status_code == 429: kind = ChatErrorKind.PROVIDER_RATE_LIMITED + elif status_code == 400 and _is_context_too_long(exc.response): + kind = ChatErrorKind.CONTEXT_TOO_LONG else: kind = ChatErrorKind.PROVIDER_UNAVAILABLE return kind, _MESSAGES[kind] @@ -79,3 +91,21 @@ def classify_chat_error(exc: Exception, provider: str) -> tuple[ChatErrorKind, s message = _NETWORK_MESSAGES.get(provider, _DEFAULT_NETWORK_MESSAGE) return ChatErrorKind.NETWORK, message return ChatErrorKind.UNKNOWN, _MESSAGES[ChatErrorKind.UNKNOWN] + + +def _is_context_too_long(response: httpx.Response) -> bool: + """Whether a 400 is llama.cpp's `exceed_context_size_error`, read from the + body it already sent rather than parsed from the message text. + + The body was read once already, building the exception's own message + (`_error_message` in the chat provider), so this reads the cached content + rather than the network again. A response httpx has not read yet, or one + with no JSON body at all, is not this kind and falls through quietly. + """ + try: + payload = response.json() + except (httpx.HTTPError, ValueError): + return False + error = payload.get("error") if isinstance(payload, dict) else None + error_type = error.get("type") if isinstance(error, dict) else None + return error_type == _CONTEXT_TOO_LONG_ERROR_TYPE diff --git a/surfsense_local/backend/modules/chat/history.py b/surfsense_local/backend/modules/chat/history.py index 23ba1dc277..3866dc4c90 100644 --- a/surfsense_local/backend/modules/chat/history.py +++ b/surfsense_local/backend/modules/chat/history.py @@ -1,19 +1,25 @@ from collections.abc import Sequence +from modules.chat.budget import DEFAULT_HISTORY_TOKENS from modules.chat.models import ChatMessage from modules.llm.providers.types import Message -# Prior turns are trimmed to fit this budget; the system message and the new user -# turn are pinned on top of it. A fraction of a small model's context window. -HISTORY_BUDGET_TOKENS = 3000 - def build_messages( - system: str, history: Sequence[ChatMessage], user_text: str + system: str, + history: Sequence[ChatMessage], + user_text: str, + *, + history_budget: int = DEFAULT_HISTORY_TOKENS, ) -> list[Message]: - """Assemble `[system, *recent history within budget, user]` for the generator.""" + """Assemble `[system, *recent history within budget, user]` for the generator. + + `history_budget` defaults to today's fixed figure; a caller that knows the + model's real window computes a tighter one with `modules.chat.budget` so + the assembled prompt cannot outgrow it (see that module for why). + """ turns = [Message(role=str(row.role.value), content=message_text(row)) for row in history] - return [Message("system", system), *_within_budget(turns), Message("user", user_text)] + return [Message("system", system), *_within_budget(turns, history_budget), Message("user", user_text)] def message_text(row: ChatMessage) -> str: @@ -21,12 +27,12 @@ def message_text(row: ChatMessage) -> str: return row.content.get("text", "") -def _within_budget(turns: list[Message]) -> list[Message]: +def _within_budget(turns: list[Message], budget: int) -> list[Message]: kept: list[Message] = [] spent = 0 for turn in reversed(turns): spent += _tokens(turn.content) - if spent > HISTORY_BUDGET_TOKENS: + if spent > budget: break kept.append(turn) kept.reverse() diff --git a/surfsense_local/backend/modules/chat/router.py b/surfsense_local/backend/modules/chat/router.py index ede4c20d8a..ef2ed0c0c9 100644 --- a/surfsense_local/backend/modules/chat/router.py +++ b/surfsense_local/backend/modules/chat/router.py @@ -10,6 +10,7 @@ from sqlalchemy.orm import Session from api.dependencies import SessionDep, transact +from modules.chat.budget import answer_max_tokens, history_budget from modules.chat.dependencies import ThreadDep from modules.chat.errors import classify_chat_error from modules.chat.history import build_messages @@ -25,6 +26,7 @@ from modules.chat.title import generate_title from modules.documents.sources import load_selected_sources from modules.llm.activity import ModelBusyError, model_activity, model_key +from modules.llm.providers.protocols import Generator from modules.llm.resolution import ( ModelResolutionError, ResolvedGeneration, @@ -119,7 +121,10 @@ async def send_message( resolved.tier, len(citations), ) - messages = build_messages(context, history, payload.text) + n_ctx = await _context_tokens_or_none(generator, selected.name) + messages = build_messages( + context, history, payload.text, history_budget=history_budget(n_ctx) + ) activity_key = model_key(selected.provider, selected.name, selected.connection_id) try: @@ -174,7 +179,9 @@ async def stream() -> AsyncIterator[bytes]: assistant_completed_at: str | None = None try: try: - async for delta in generator.chat(selected.name, messages): + async for delta in generator.chat( + selected.name, messages, max_tokens=answer_max_tokens(n_ctx) + ): parts.append(delta) yield _frame({"type": "delta", "text": delta}) except Exception as exc: @@ -321,6 +328,20 @@ def _iso(instant: datetime) -> str: return instant.isoformat().replace("+00:00", "Z") +async def _context_tokens_or_none(generator: Generator, model: str) -> int | None: + """The model's window, or None on anything short of a clean answer. + + A transient failure to read it must not fail the turn: it only narrows how + the prompt is budgeted, and the unknown-window fallback in + `modules.chat.budget` is exactly today's behavior. + """ + try: + return await generator.context_tokens(model) + except Exception: + logger.warning("Could not read the context window for %s", model, exc_info=True) + return None + + def _frame(payload: dict) -> bytes: """One SSE data frame in the stream's accepted/delta/terminal protocol.""" return f"data: {json.dumps(payload)}\n\n".encode() diff --git a/surfsense_local/backend/modules/llm/fit/__init__.py b/surfsense_local/backend/modules/llm/fit/__init__.py index 29db150a7a..be0773269e 100644 --- a/surfsense_local/backend/modules/llm/fit/__init__.py +++ b/surfsense_local/backend/modules/llm/fit/__init__.py @@ -1,7 +1,12 @@ from modules.llm.fit.budget import HardwareBudget from modules.llm.fit.compute_buffers import compute_buffer_bytes from modules.llm.fit.copy import Badge, badge -from modules.llm.fit.estimate import CONTEXT_FLOOR_TOKENS, FitVerdict, estimate +from modules.llm.fit.estimate import ( + CONTEXT_FLOOR_TOKENS, + CONTEXT_RUNGS, + FitVerdict, + estimate, +) from modules.llm.fit.itemisation import NeedItems, itemise from modules.llm.fit.kv_cache import kv_cache_bytes from modules.llm.fit.offload import offload_fraction @@ -12,6 +17,7 @@ __all__ = [ "CONTEXT_FLOOR_TOKENS", + "CONTEXT_RUNGS", "Badge", "FitState", "FitVerdict", diff --git a/surfsense_local/backend/modules/llm/fit/estimate.py b/surfsense_local/backend/modules/llm/fit/estimate.py index e128c2b7f1..896ba147ee 100644 --- a/surfsense_local/backend/modules/llm/fit/estimate.py +++ b/surfsense_local/backend/modules/llm/fit/estimate.py @@ -23,7 +23,23 @@ from modules.llm.fit.states import FitState from modules.llm.fit.types import KvPrecision, ModelShape -CONTEXT_FLOOR_TOKENS = 16384 +# The window this app guarantees a grounded turn, and the one `TOO_BIG` is +# judged at: a model that will not fit here cannot be rescued by a smaller +# context, so the remedy to offer is a smaller build, not a narrower window. +# +# 8192, not llama.cpp's own 4096 reduction floor (`fit_params_min_ctx`, +# common/common.h): 4096 is too short once a system prompt, retrieved excerpts +# and a reply share the window, so the fit search would settle there rather +# than spill weights it could spill instead. Matches Unsloth Studio's own +# floor (`_FIT_MIN_CTX`), reasoned the same way. `modules.chat.budget` is what +# makes 8192 safe: the parts of a prompt are sized from `n_ctx` rather than +# a fixed history figure that assumed a wider window. +CONTEXT_FLOOR_TOKENS = 8192 + +# The fixed windows `plan_load` widens between. A bounded, testable set +# instead of a continuous search landing on a different number on every +# machine: the catalog and the preset only ever quote one of these three. +CONTEXT_RUNGS = (8192, 16384, 32768) @dataclass(frozen=True) @@ -51,9 +67,9 @@ def estimate( ) -> FitVerdict: """Price one build against one device. - `TOO_BIG` is evaluated at the context floor rather than the requested window: - a model that will not fit at 16K cannot be rescued by a smaller context, and - the remedy to offer is a smaller build. + `TOO_BIG` is evaluated at `CONTEXT_FLOOR_TOKENS` rather than the requested + window: a model that will not fit at the floor cannot be rescued by a + smaller context, and the remedy to offer is a smaller build. """ items = itemise(shape, weights_bytes, n_ctx, precision, mmproj_bytes) need = items.total diff --git a/surfsense_local/backend/modules/llm/fit/plan_load.py b/surfsense_local/backend/modules/llm/fit/plan_load.py index 58865c3b37..f3fa1aab86 100644 --- a/surfsense_local/backend/modules/llm/fit/plan_load.py +++ b/surfsense_local/backend/modules/llm/fit/plan_load.py @@ -14,7 +14,12 @@ from dataclasses import dataclass from modules.llm.fit.budget import HardwareBudget -from modules.llm.fit.estimate import CONTEXT_FLOOR_TOKENS, FitVerdict, estimate +from modules.llm.fit.estimate import ( + CONTEXT_FLOOR_TOKENS, + CONTEXT_RUNGS, + FitVerdict, + estimate, +) from modules.llm.fit.precision import FALLBACK, resident_precision from modules.llm.fit.states import FitState from modules.llm.fit.types import KvPrecision, ModelShape @@ -37,8 +42,8 @@ def plan_load( ) -> LoadPlan: """Choose the widest window that stays resident, at the cheapest precision. - The cap beats the floor. A model trained to 8192 tokens cannot be asked for - 16384 just because a grounded turn would like one: the floor describes what + The cap beats the floor. A model trained to 4096 tokens cannot be asked for + 8192 just because a grounded turn would like one: the floor describes what this app needs, not what a model can do. Two budgets, answering two questions. `budget` is capacity, and it decides @@ -124,26 +129,31 @@ def _widest_resident( floor: int, mmproj_bytes: int = 0, ) -> int: - """Binary search the window, in whole thousands of tokens.""" - low, high = floor, ceiling - while low < high: - mid = min(ceiling, (low + high + 1024) // 2) - mid -= mid % 1024 - if mid <= low: - break + """Largest rung that stays resident, never a number between two rungs. + + A fixed, small set of candidates instead of a continuous search: the + catalog badge, the preset and this plan can only ever quote one of + `CONTEXT_RUNGS`, so the whole surface of windows a person can see is + something a test can enumerate. `ceiling` (the model's own trained + length, or a caller's cap) is tried too when it falls between two rungs, + so a model is not held below what it was actually trained for; `floor` + is always a candidate so the search never returns nothing. + """ + candidates = {n for n in CONTEXT_RUNGS if floor <= n <= ceiling} + candidates.add(floor) + candidates.add(ceiling) + for n_ctx in sorted(candidates, reverse=True): fits = ( estimate( shape, weights_bytes, budget, - n_ctx=mid, + n_ctx=n_ctx, precision=precision, mmproj_bytes=mmproj_bytes, ).state is FitState.FITS ) if fits: - low = mid - else: - high = mid - 1024 - return max(floor, min(low, ceiling)) + return n_ctx + return floor diff --git a/surfsense_local/backend/modules/llm/providers/llamacpp/provider.py b/surfsense_local/backend/modules/llm/providers/llamacpp/provider.py index c3972a8657..114e94471a 100644 --- a/surfsense_local/backend/modules/llm/providers/llamacpp/provider.py +++ b/surfsense_local/backend/modules/llm/providers/llamacpp/provider.py @@ -65,6 +65,12 @@ async def capabilities(self, model: str) -> Capabilities: model, await self._router.raw_models(), await self._router.props(model) ) + async def context_tokens(self, model: str) -> int | None: + """The window this model is loaded with, from the same `/props` read + capabilities already makes. Not the window we requested: what the + fitter actually allocated, in case it differs.""" + return (await self.capabilities(model)).context_tokens + async def models(self) -> list[Model]: """Everything in the models directory, resident or not. diff --git a/surfsense_local/backend/modules/llm/providers/openai_compatible/chat.py b/surfsense_local/backend/modules/llm/providers/openai_compatible/chat.py index 8340116cb4..ffe76383f9 100644 --- a/surfsense_local/backend/modules/llm/providers/openai_compatible/chat.py +++ b/surfsense_local/backend/modules/llm/providers/openai_compatible/chat.py @@ -78,6 +78,13 @@ async def models(self) -> list[Model]: for model in discovered ] + async def context_tokens(self, model: str) -> int | None: + """Unknown. The `/models` listing this class speaks does not carry a + window (that is llama.cpp's `/props`, not the OpenAI shape), so a + caller that budgets a prompt from this must fall back to a fixed + figure rather than assume this endpoint's real window is small.""" + return None + async def inspect(self, name: str) -> Fingerprint: """What this endpoint's listing reveals about one model, for prompt tiering.""" async with self._client() as client: diff --git a/surfsense_local/backend/modules/llm/providers/protocols.py b/surfsense_local/backend/modules/llm/providers/protocols.py index bfdb882c91..d5f2f2a940 100644 --- a/surfsense_local/backend/modules/llm/providers/protocols.py +++ b/surfsense_local/backend/modules/llm/providers/protocols.py @@ -14,6 +14,15 @@ async def health(self) -> bool: ... async def models(self) -> list[Model]: ... + async def context_tokens(self, model: str) -> int | None: + """The window this model was loaded with, or None when it is not known. + + None is a fact, not a zero: a remote endpoint this app does not run + rarely states its window at all, and callers that budget a prompt from + this must treat that as "unknown" rather than "narrow". + """ + ... + def chat( self, model: str, diff --git a/surfsense_local/backend/tests/integration/chat/conftest.py b/surfsense_local/backend/tests/integration/chat/conftest.py index a979ce03df..1461a23cfe 100644 --- a/surfsense_local/backend/tests/integration/chat/conftest.py +++ b/surfsense_local/backend/tests/integration/chat/conftest.py @@ -13,6 +13,11 @@ # Each chat request the stub received, so a test can assert what the route sent. _REQUESTS: list[dict] = [] +# The context window /props reports, or None to omit it (an older build / +# a model report a test does not care about). Set per test before the fixture +# starts the server. +_PROPS_N_CTX: int | None = None + class StubRouterChat(BaseHTTPRequestHandler): """The router's OpenAI chat endpoint, streaming its reply as SSE. @@ -33,6 +38,13 @@ def do_GET(self) -> None: } ).encode() ) + elif self.path.startswith("/props"): + settings = ( + {"n_ctx": _PROPS_N_CTX} if _PROPS_N_CTX is not None else {} + ) + self._send( + json.dumps({"default_generation_settings": settings}).encode() + ) else: self.send_error(404) @@ -71,7 +83,9 @@ def log_message(self, *args: object) -> None: @pytest.fixture def llamacpp_server(monkeypatch: pytest.MonkeyPatch) -> Iterator[list[dict]]: """A real llama-server stand-in on a real port; yields the requests it sees.""" + global _PROPS_N_CTX _REQUESTS.clear() + _PROPS_N_CTX = None server = ThreadingHTTPServer(("127.0.0.1", 0), StubRouterChat) threading.Thread(target=server.serve_forever, daemon=True).start() url = f"http://127.0.0.1:{server.server_port}" @@ -110,6 +124,12 @@ def log_message(self, *args: object) -> None: """Keep the request log out of the test output.""" +def set_props_n_ctx(n_ctx: int) -> None: + """Make the next `llamacpp_server` request report this context window.""" + global _PROPS_N_CTX + _PROPS_N_CTX = n_ctx + + @pytest.fixture def llamacpp_server_unauthorized(monkeypatch: pytest.MonkeyPatch) -> Iterator[None]: """A chat stand-in that fails every request, for exercising error handling.""" diff --git a/surfsense_local/backend/tests/integration/chat/test_chat.py b/surfsense_local/backend/tests/integration/chat/test_chat.py index 2941675977..d6c0f09687 100644 --- a/surfsense_local/backend/tests/integration/chat/test_chat.py +++ b/surfsense_local/backend/tests/integration/chat/test_chat.py @@ -6,10 +6,12 @@ from httpx import AsyncClient from sqlalchemy import Engine +from modules.chat.budget import ANSWER_RESERVE_TOKENS from modules.documents.models import Document, DocumentStatus, DocumentType from modules.llm.models import ModelRole, SelectedModel from modules.workspaces.models import Workspace from shared.db import create_session_factory +from tests.integration.chat.conftest import set_props_n_ctx from worker.ingestion import run pytestmark = pytest.mark.integration @@ -346,3 +348,20 @@ async def test_missing_embedding_assets_are_an_actionable_503( "local embedding model is not installed; " "run `uv run scripts/fetch_embedding_model.py`" ) + + +async def test_the_answer_reserves_room_instead_of_taking_the_whole_window( + client: AsyncClient, engine: Engine, real_model: object, llamacpp_server: list[dict] +) -> None: + """A model whose window is known gets a capped answer request, so a long + reply stops cleanly instead of being cut off mid-sentence by the window + running out with no `max_tokens` set at all.""" + set_props_n_ctx(16384) + workspace_id, _ = _seed(engine) + thread_id = await _open_thread(client, workspace_id) + + await _send(client, thread_id, "what happened to revenue?") + + answer_requests = [r for r in llamacpp_server if r.get("max_tokens") != 12] + assert answer_requests + assert answer_requests[-1]["max_tokens"] == ANSWER_RESERVE_TOKENS diff --git a/surfsense_local/backend/tests/unit/chat/test_budget.py b/surfsense_local/backend/tests/unit/chat/test_budget.py new file mode 100644 index 0000000000..cb25dc95bf --- /dev/null +++ b/surfsense_local/backend/tests/unit/chat/test_budget.py @@ -0,0 +1,41 @@ +"""How a turn's fixed parts, the answer, and history share one context window.""" + +import pytest + +from modules.chat.budget import ( + ANSWER_RESERVE_TOKENS, + DEFAULT_HISTORY_TOKENS, + answer_max_tokens, + history_budget, +) + +pytestmark = pytest.mark.unit + + +def test_an_unknown_window_keeps_todays_fixed_budget() -> None: + """A remote endpoint that reports no window is not a smaller window; it is + an absent fact. Nothing here can tell the fixed parts fit, so the fallback + is today's number rather than a guess that could be too generous.""" + assert history_budget(None) == DEFAULT_HISTORY_TOKENS + + +def test_a_known_window_leaves_less_room_the_smaller_it_is() -> None: + """The whole point: history is spent out of what is actually left, not a + constant that assumes a 16384 window it may not have.""" + narrow = history_budget(8192) + wide = history_budget(16384) + + assert 0 <= narrow < wide + + +def test_the_fixed_parts_and_the_answer_never_starve_a_tight_window() -> None: + """A window smaller than the fixed parts plus the reserve leaves zero for + history rather than a negative number that would corrupt a slice.""" + assert history_budget(1) == 0 + + +def test_the_answer_reserve_is_capped_only_when_the_window_is_known() -> None: + """Sending a fixed cap to an endpoint whose real window might be far + larger would truncate replies for a limit that does not apply there.""" + assert answer_max_tokens(None) is None + assert answer_max_tokens(16384) == ANSWER_RESERVE_TOKENS diff --git a/surfsense_local/backend/tests/unit/chat/test_errors.py b/surfsense_local/backend/tests/unit/chat/test_errors.py index dead37be5e..a18b33c8d0 100644 --- a/surfsense_local/backend/tests/unit/chat/test_errors.py +++ b/surfsense_local/backend/tests/unit/chat/test_errors.py @@ -37,3 +37,44 @@ def test_transport_and_status_failures_are_unchanged() -> None: is ChatErrorKind.NETWORK ) assert classify_chat_error(RuntimeError("?"), "x")[0] is ChatErrorKind.UNKNOWN + + +def _response_with_body(status: int, body: dict) -> httpx.Response: + request = httpx.Request("POST", "http://x/v1/chat/completions") + return httpx.Response(status, request=request, json=body) + + +def test_a_prompt_too_long_for_the_window_names_the_real_problem() -> None: + """llama.cpp's own error, told apart from an ordinary 400. + + Bucketed with every other status under PROVIDER_UNAVAILABLE, "try again + shortly" tells someone to wait out a problem retrying will never fix: the + prompt does not get shorter on its own. + """ + request = httpx.Request("POST", "http://x/v1/chat/completions") + response = _response_with_body( + 400, + { + "error": { + "code": 400, + "type": "exceed_context_size_error", + "message": "the request exceeds the available context size", + } + }, + ) + exc = httpx.HTTPStatusError("400", request=request, response=response) + + kind, message = classify_chat_error(exc, "llamacpp") + + assert kind is ChatErrorKind.CONTEXT_TOO_LONG + assert "long" in message.casefold() or "context" in message.casefold() + + +def test_an_ordinary_400_is_still_provider_unavailable() -> None: + """The new branch reads the body; a 400 with no llama.cpp shape falls + through to the existing bucket rather than raising on a missing field.""" + request = httpx.Request("POST", "http://x/v1/chat/completions") + response = httpx.Response(400, request=request, content=b"not json") + exc = httpx.HTTPStatusError("400", request=request, response=response) + + assert classify_chat_error(exc, "llamacpp")[0] is ChatErrorKind.PROVIDER_UNAVAILABLE diff --git a/surfsense_local/backend/tests/unit/chat/test_history.py b/surfsense_local/backend/tests/unit/chat/test_history.py index f2f962b684..2bb55c62c0 100644 --- a/surfsense_local/backend/tests/unit/chat/test_history.py +++ b/surfsense_local/backend/tests/unit/chat/test_history.py @@ -2,7 +2,8 @@ import pytest -from modules.chat.history import HISTORY_BUDGET_TOKENS, build_messages +from modules.chat.budget import DEFAULT_HISTORY_TOKENS +from modules.chat.history import build_messages from modules.chat.models import ChatMessage, MessageRole pytestmark = pytest.mark.unit @@ -27,10 +28,21 @@ def test_the_system_and_new_turn_bracket_the_history() -> None: def test_the_oldest_turns_over_budget_are_dropped() -> None: """Turns are kept newest-first until the budget is spent, in original order.""" - half = HISTORY_BUDGET_TOKENS // 2 + half = DEFAULT_HISTORY_TOKENS // 2 history = [_turn("x", half), _turn("y", half), _turn("z", half)] kept = build_messages("SYSTEM", history, "ask")[1:-1] # Three half-budget turns cannot all fit; the oldest falls away, order holds. assert [message.content[0] for message in kept] == ["y", "z"] + + +def test_a_caller_supplied_budget_overrides_the_default() -> None: + """A model with a known, narrower window keeps less history than the + default figure would allow, without waiting for the default to change.""" + history = [_turn("a", 50), _turn("b", 50)] + + kept_default = build_messages("SYSTEM", history, "ask")[1:-1] + kept_tight = build_messages("SYSTEM", history, "ask", history_budget=50)[1:-1] + + assert len(kept_tight) < len(kept_default) diff --git a/surfsense_local/backend/tests/unit/llm/catalog/test_recommendation.py b/surfsense_local/backend/tests/unit/llm/catalog/test_recommendation.py index 61a2ddad40..7857886248 100644 --- a/surfsense_local/backend/tests/unit/llm/catalog/test_recommendation.py +++ b/surfsense_local/backend/tests/unit/llm/catalog/test_recommendation.py @@ -140,10 +140,10 @@ def test_a_machine_with_no_gpu_still_gets_a_recommendation() -> None: def test_a_machine_with_no_gpu_is_still_refused_what_it_cannot_hold() -> None: """The star moves down the ladder rather than off it. - 2000 MiB holds the 0.6B, which needs 1436 with a quantized cache, and not - the 1.7B, which needs 2113 even with one. + 1500 MiB holds the 0.6B, which needs 1013 with a quantized cache, and not + the 1.7B, which needs 1690 even with one. """ - tiny = HardwareBudget(0, 0, 1024 * MIB, 2000 * MIB, False, False) + tiny = HardwareBudget(0, 0, 1024 * MIB, 1500 * MIB, False, False) pick = recommend(CURATED, tiny) diff --git a/surfsense_local/backend/tests/unit/llm/fit/test_estimate.py b/surfsense_local/backend/tests/unit/llm/fit/test_estimate.py index 40b3e4d82c..e4be7f9b9a 100644 --- a/surfsense_local/backend/tests/unit/llm/fit/test_estimate.py +++ b/surfsense_local/backend/tests/unit/llm/fit/test_estimate.py @@ -191,7 +191,7 @@ def test_unified_memory_refuses_past_the_pool_not_past_the_pool_twice_over() -> set and the host reading, which describe the same chips, were added. A model between the pool and that sum was offered to a machine that cannot hold it. """ - need = estimate(QWEN3_1_7B, 5000 * MIB, M2).need_bytes + need = estimate(QWEN3_1_7B, 5200 * MIB, M2).need_bytes assert need > M2.device_free_bytes - assert estimate(QWEN3_1_7B, 5000 * MIB, M2).state is FitState.TOO_BIG + assert estimate(QWEN3_1_7B, 5200 * MIB, M2).state is FitState.TOO_BIG diff --git a/surfsense_local/backend/tests/unit/llm/fit/test_plan_load.py b/surfsense_local/backend/tests/unit/llm/fit/test_plan_load.py index 358d2be142..508f790a70 100644 --- a/surfsense_local/backend/tests/unit/llm/fit/test_plan_load.py +++ b/surfsense_local/backend/tests/unit/llm/fit/test_plan_load.py @@ -8,6 +8,7 @@ from modules.llm.fit import ( CONTEXT_FLOOR_TOKENS, + CONTEXT_RUNGS, FitState, HardwareBudget, KvPrecision, @@ -47,7 +48,7 @@ def test_precision_drops_only_when_it_buys_residency() -> None: Quality is lossless either way, so the cost is a flash-attention dependency we take on only where it converts a spill into residency. """ - plan = plan_load(QWEN3_1_7B, 2600 * MIB, budget(5460)) + plan = plan_load(QWEN3_1_7B, 3500 * MIB, budget(5460)) assert plan.precision is KvPrecision.Q8_0 assert plan.verdict.state is FitState.FITS @@ -61,7 +62,7 @@ def test_the_window_never_drops_below_the_floor() -> None: """ # Too large to stay resident at either precision, small enough that physics # does not refuse it, so the floor is what holds the window up. - plan = plan_load(QWEN3_1_7B, 3500 * MIB, budget(5460)) + plan = plan_load(QWEN3_1_7B, 4000 * MIB, budget(5460)) assert plan.n_ctx == CONTEXT_FLOOR_TOKENS assert plan.verdict.state is FitState.PARTIAL @@ -124,3 +125,35 @@ def test_widening_past_the_floor_respects_what_is_actually_free() -> None: assert roomy.n_ctx > constrained.n_ctx assert constrained.n_ctx == CONTEXT_FLOOR_TOKENS + + +def test_the_window_widens_to_a_fixed_rung_not_an_arbitrary_number() -> None: + """A bounded, testable set of windows instead of a different number on + every machine: enough room for the next rung and not the one above it + lands exactly on that rung, never a value in between.""" + plan = plan_load(QWEN3_1_7B, 1050 * MIB, budget(5800)) + + assert plan.n_ctx == 16384 + assert plan.n_ctx in CONTEXT_RUNGS + + +def test_a_machine_roomy_enough_for_the_top_rung_can_still_take_the_full_window() -> None: + """The rungs bound the search, not the answer: a machine with room for + everything up to the top rung but not the model's own trained length still + gets the top rung rather than being pushed up to a window that would spill.""" + plan = plan_load(QWEN3_1_7B, 1050 * MIB, budget(6500)) + + assert plan.n_ctx == 32768 + assert plan.n_ctx < QWEN3_1_7B.context_length + + +def test_a_machine_too_tight_for_the_next_rung_stays_on_the_floor_rung() -> None: + """Between two rungs, the search does not settle for a number in between: + it takes the lower rung whole, leaving the memory the finer-grained search + used to spend on a window that fit nowhere else.""" + # Enough for 8192 (the floor rung) at either precision, not enough for the + # next rung up (16384) at either: the old continuous search would have + # settled on some number strictly between the two. + plan = plan_load(QWEN3_1_7B, 2900 * MIB, budget(5460)) + + assert plan.n_ctx == CONTEXT_FLOOR_TOKENS From e4edfab0955790b8d70162ceebf358b282558977 Mon Sep 17 00:00:00 2001 From: Anish Sarkar <104695310+AnishSarkar22@users.noreply.github.com> Date: Mon, 21 Sep 2026 19:43:38 +0530 Subject: [PATCH 53/60] perf(llm): price history against the model's own tokenizer, cache /props per load Two related fixes to the local runtime seam: - Generator.token_count() reads llama.cpp's /tokenize endpoint, so history trimming uses the model's real tokenizer instead of a ~4 chars/token heuristic that could under-count dense text by 15-20%. A remote endpoint has no equivalent, so it keeps the heuristic exactly as before. - LlamaCppProvider.capabilities() now caches its /props read per model instead of fetching it fresh on every message. context_tokens() and chat()'s own template shaping were each issuing an independent /props round trip on every single turn; measured live, that showed up as real, avoidable latency layered onto an already slow load. The cache is cleared the one time it can go stale: _ensure_loaded() issuing a fresh load. --- .../backend/modules/chat/history.py | 33 ++++++++++-- .../backend/modules/chat/router.py | 25 +++++++-- .../llm/providers/llamacpp/provider.py | 35 ++++++++++-- .../llm/providers/llamacpp/router_client.py | 16 ++++++ .../llm/providers/openai_compatible/chat.py | 6 +++ .../modules/llm/providers/protocols.py | 10 ++++ .../tests/integration/chat/conftest.py | 25 ++++++++- .../tests/integration/chat/test_chat.py | 23 +++++++- .../backend/tests/unit/chat/test_history.py | 54 ++++++++++++++++--- .../llm/providers/llamacpp/fake_router.py | 14 +++++ .../llm/providers/llamacpp/test_provider.py | 39 ++++++++++++++ .../providers/llamacpp/test_router_client.py | 25 +++++++++ 12 files changed, 284 insertions(+), 21 deletions(-) diff --git a/surfsense_local/backend/modules/chat/history.py b/surfsense_local/backend/modules/chat/history.py index 3866dc4c90..d0a2119581 100644 --- a/surfsense_local/backend/modules/chat/history.py +++ b/surfsense_local/backend/modules/chat/history.py @@ -1,25 +1,36 @@ -from collections.abc import Sequence +from collections.abc import Awaitable, Callable, Sequence from modules.chat.budget import DEFAULT_HISTORY_TOKENS from modules.chat.models import ChatMessage from modules.llm.providers.types import Message +# A turn's exact cost by the model's own tokenizer, or None when it could not +# be counted (no endpoint, a transient failure). None falls back to the +# heuristic for that turn rather than being read as zero tokens. +TokenCounter = Callable[[str], Awaitable[int | None]] -def build_messages( + +async def build_messages( system: str, history: Sequence[ChatMessage], user_text: str, *, history_budget: int = DEFAULT_HISTORY_TOKENS, + token_count: TokenCounter | None = None, ) -> list[Message]: """Assemble `[system, *recent history within budget, user]` for the generator. `history_budget` defaults to today's fixed figure; a caller that knows the model's real window computes a tighter one with `modules.chat.budget` so the assembled prompt cannot outgrow it (see that module for why). + + `token_count` prices each turn exactly, by the model's own tokenizer, + instead of the `~4 chars/token` estimate: only the local runtime can + answer it, so a caller without one gets exactly today's heuristic trim. """ turns = [Message(role=str(row.role.value), content=message_text(row)) for row in history] - return [Message("system", system), *_within_budget(turns, history_budget), Message("user", user_text)] + kept = await _within_budget(turns, history_budget, token_count) + return [Message("system", system), *kept, Message("user", user_text)] def message_text(row: ChatMessage) -> str: @@ -27,11 +38,13 @@ def message_text(row: ChatMessage) -> str: return row.content.get("text", "") -def _within_budget(turns: list[Message], budget: int) -> list[Message]: +async def _within_budget( + turns: list[Message], budget: int, token_count: TokenCounter | None +) -> list[Message]: kept: list[Message] = [] spent = 0 for turn in reversed(turns): - spent += _tokens(turn.content) + spent += await _cost(turn.content, token_count) if spent > budget: break kept.append(turn) @@ -39,6 +52,16 @@ def _within_budget(turns: list[Message], budget: int) -> list[Message]: return kept +async def _cost(text: str, token_count: TokenCounter | None) -> int: + """A turn's price: exact when a counter is given and answers, the + heuristic otherwise. Never mixes the two within one turn.""" + if token_count is not None: + exact = await token_count(text) + if exact is not None: + return exact + return _tokens(text) + + def _tokens(text: str) -> int: # ponytail: ~4 chars per token dodges loading the model's tokenizer. Ceiling: # fine for a soft trim; swap in the real count if the runtime starts truncating. diff --git a/surfsense_local/backend/modules/chat/router.py b/surfsense_local/backend/modules/chat/router.py index ef2ed0c0c9..53fb078283 100644 --- a/surfsense_local/backend/modules/chat/router.py +++ b/surfsense_local/backend/modules/chat/router.py @@ -13,7 +13,7 @@ from modules.chat.budget import answer_max_tokens, history_budget from modules.chat.dependencies import ThreadDep from modules.chat.errors import classify_chat_error -from modules.chat.history import build_messages +from modules.chat.history import TokenCounter, build_messages from modules.chat.models import ChatMessage, ChatThread, MessageRole from modules.chat.prompt import build_context, resolve_citations from modules.chat.schemas import ( @@ -122,8 +122,12 @@ async def send_message( len(citations), ) n_ctx = await _context_tokens_or_none(generator, selected.name) - messages = build_messages( - context, history, payload.text, history_budget=history_budget(n_ctx) + messages = await build_messages( + context, + history, + payload.text, + history_budget=history_budget(n_ctx), + token_count=_token_counter(generator, selected.name), ) activity_key = model_key(selected.provider, selected.name, selected.connection_id) @@ -342,6 +346,21 @@ async def _context_tokens_or_none(generator: Generator, model: str) -> int | Non return None +def _token_counter(generator: Generator, model: str) -> TokenCounter: + """A per-turn counter bound to this turn's model, defensive the same way + `_context_tokens_or_none` is: a failure here only widens the heuristic + `build_messages` already falls back to for that one turn, never the turn.""" + + async def count(text: str) -> int | None: + try: + return await generator.token_count(model, text) + except Exception: + logger.warning("Could not count tokens for %s", model, exc_info=True) + return None + + return count + + def _frame(payload: dict) -> bytes: """One SSE data frame in the stream's accepted/delta/terminal protocol.""" return f"data: {json.dumps(payload)}\n\n".encode() diff --git a/surfsense_local/backend/modules/llm/providers/llamacpp/provider.py b/surfsense_local/backend/modules/llm/providers/llamacpp/provider.py index 114e94471a..3fa9e7bf5c 100644 --- a/surfsense_local/backend/modules/llm/providers/llamacpp/provider.py +++ b/surfsense_local/backend/modules/llm/providers/llamacpp/provider.py @@ -48,6 +48,12 @@ def __init__( self._base_url = base_url.rstrip("/") self._models_dir = models_dir self._router = RouterClient(self._base_url, transport=transport) + # `/props` describes a resident model and nothing about it changes + # between turns, so it is read once per load rather than once per + # message. `_ensure_loaded` is the only place that clears an entry, + # because a fresh load is the only event that can make the old answer + # wrong (the idle timer evicted it, or a reprice changed the preset). + self._capabilities_cache: dict[str, Capabilities] = {} self._chat = OpenAICompatibleChatProvider( f"{self._base_url}/v1", transport=transport, thinking_off=THINKING_OFF ) @@ -58,12 +64,22 @@ async def health(self) -> bool: async def capabilities(self, model: str) -> Capabilities: """What this model accepts and what its template can express. - Read per model rather than cached across them: two files in the same - directory can carry entirely different templates. + Cached per model rather than fetched per message: two files in the + same directory can carry entirely different templates, so the cache + is keyed by model id, never assumed to hold across them. What it does + hold across is repeated calls for the *same* resident model within one + turn, or between turns, since nothing about a loaded model's template + or window changes on its own. `_ensure_loaded` clears an entry the one + time that stops being true: a fresh load. """ - return read_capabilities( + cached = self._capabilities_cache.get(model) + if cached is not None: + return cached + caps = read_capabilities( model, await self._router.raw_models(), await self._router.props(model) ) + self._capabilities_cache[model] = caps + return caps async def context_tokens(self, model: str) -> int | None: """The window this model is loaded with, from the same `/props` read @@ -71,6 +87,18 @@ async def context_tokens(self, model: str) -> int | None: fitter actually allocated, in case it differs.""" return (await self.capabilities(model)).context_tokens + async def token_count(self, model: str, text: str) -> int | None: + """The exact cost of this text, by the router's own tokenizer. + + None on anything short of a clean count, same as `context_tokens`: a + caller pricing history from this must fall back to an estimate rather + than trust a failure as a token count. + """ + try: + return await self._router.tokenize(model, text) + except httpx.HTTPError: + return None + async def models(self) -> list[Model]: """Everything in the models directory, resident or not. @@ -153,4 +181,5 @@ async def _ensure_loaded(self, model: str) -> None: """ resident = {m.id for m in await self._router.models() if m.loaded} if model not in resident: + self._capabilities_cache.pop(model, None) await self._router.load(model) diff --git a/surfsense_local/backend/modules/llm/providers/llamacpp/router_client.py b/surfsense_local/backend/modules/llm/providers/llamacpp/router_client.py index a358e6f3e2..6716e1c245 100644 --- a/surfsense_local/backend/modules/llm/providers/llamacpp/router_client.py +++ b/surfsense_local/backend/modules/llm/providers/llamacpp/router_client.py @@ -92,6 +92,22 @@ async def props(self, model_id: str) -> dict: # generously from an empty payload rather than failing the turn. return {} + async def tokenize(self, model_id: str, text: str) -> int: + """How many tokens this text costs, by the model's own tokenizer. + + Proxied to the loaded worker the same way `/props` is (`model` in the + body), so it autoloads the model like any other request the router + forwards. Raises rather than returning 0 on failure: a caller pricing a + prompt from this must know the count did not come back, not mistake an + error for an empty turn. + """ + async with self._client() as client: + reply = await client.post( + "/tokenize", json={"model": model_id, "content": text, "add_special": False} + ) + reply.raise_for_status() + return len(reply.json().get("tokens", [])) + async def load(self, model_id: str) -> None: """Bring a model into memory. diff --git a/surfsense_local/backend/modules/llm/providers/openai_compatible/chat.py b/surfsense_local/backend/modules/llm/providers/openai_compatible/chat.py index ffe76383f9..90e7253616 100644 --- a/surfsense_local/backend/modules/llm/providers/openai_compatible/chat.py +++ b/surfsense_local/backend/modules/llm/providers/openai_compatible/chat.py @@ -85,6 +85,12 @@ async def context_tokens(self, model: str) -> int | None: figure rather than assume this endpoint's real window is small.""" return None + async def token_count(self, model: str, text: str) -> int | None: + """Unknown, for the same reason as `context_tokens`: the OpenAI shape + this class speaks has no tokenize endpoint, and every backend behind + it uses its own encoding, so a caller here must keep its estimate.""" + return None + async def inspect(self, name: str) -> Fingerprint: """What this endpoint's listing reveals about one model, for prompt tiering.""" async with self._client() as client: diff --git a/surfsense_local/backend/modules/llm/providers/protocols.py b/surfsense_local/backend/modules/llm/providers/protocols.py index d5f2f2a940..eef8f1bc4e 100644 --- a/surfsense_local/backend/modules/llm/providers/protocols.py +++ b/surfsense_local/backend/modules/llm/providers/protocols.py @@ -23,6 +23,16 @@ async def context_tokens(self, model: str) -> int | None: """ ... + async def token_count(self, model: str, text: str) -> int | None: + """This text's exact cost by the model's own tokenizer, or None. + + None on anything short of a clean count: no such endpoint, a transient + failure, or a model this generator does not run. A caller pricing a + prompt from this must fall back to an estimate rather than treat None + as zero tokens. + """ + ... + def chat( self, model: str, diff --git a/surfsense_local/backend/tests/integration/chat/conftest.py b/surfsense_local/backend/tests/integration/chat/conftest.py index 1461a23cfe..da1cf615ef 100644 --- a/surfsense_local/backend/tests/integration/chat/conftest.py +++ b/surfsense_local/backend/tests/integration/chat/conftest.py @@ -18,6 +18,10 @@ # starts the server. _PROPS_N_CTX: int | None = None +# Tokens per word /tokenize reports, or None to answer 404 (an older build). +# Set per test before the fixture starts the server. +_TOKENS_PER_WORD: int | None = None + class StubRouterChat(BaseHTTPRequestHandler): """The router's OpenAI chat endpoint, streaming its reply as SSE. @@ -50,12 +54,21 @@ def do_GET(self) -> None: def do_POST(self) -> None: raw = self.rfile.read(int(self.headers["Content-Length"])) - if self.path not in ("/v1/chat/completions", "/models/load"): + if self.path not in ("/v1/chat/completions", "/models/load", "/tokenize"): self.send_error(404) return if self.path == "/models/load": self._send(b'{"success": true}') return + if self.path == "/tokenize": + if _TOKENS_PER_WORD is None: + self.send_error(404) + return + words = json.loads(raw).get("content", "").split() + self._send( + json.dumps({"tokens": list(range(len(words) * _TOKENS_PER_WORD))}).encode() + ) + return request = json.loads(raw) _REQUESTS.append(request) @@ -83,9 +96,10 @@ def log_message(self, *args: object) -> None: @pytest.fixture def llamacpp_server(monkeypatch: pytest.MonkeyPatch) -> Iterator[list[dict]]: """A real llama-server stand-in on a real port; yields the requests it sees.""" - global _PROPS_N_CTX + global _PROPS_N_CTX, _TOKENS_PER_WORD _REQUESTS.clear() _PROPS_N_CTX = None + _TOKENS_PER_WORD = None server = ThreadingHTTPServer(("127.0.0.1", 0), StubRouterChat) threading.Thread(target=server.serve_forever, daemon=True).start() url = f"http://127.0.0.1:{server.server_port}" @@ -130,6 +144,13 @@ def set_props_n_ctx(n_ctx: int) -> None: _PROPS_N_CTX = n_ctx +def set_tokens_per_word(tokens_per_word: int) -> None: + """Make the next `llamacpp_server` request answer `/tokenize` exactly, + at this many tokens per word, instead of 404ing like an older build.""" + global _TOKENS_PER_WORD + _TOKENS_PER_WORD = tokens_per_word + + @pytest.fixture def llamacpp_server_unauthorized(monkeypatch: pytest.MonkeyPatch) -> Iterator[None]: """A chat stand-in that fails every request, for exercising error handling.""" diff --git a/surfsense_local/backend/tests/integration/chat/test_chat.py b/surfsense_local/backend/tests/integration/chat/test_chat.py index d6c0f09687..21b1238032 100644 --- a/surfsense_local/backend/tests/integration/chat/test_chat.py +++ b/surfsense_local/backend/tests/integration/chat/test_chat.py @@ -11,7 +11,7 @@ from modules.llm.models import ModelRole, SelectedModel from modules.workspaces.models import Workspace from shared.db import create_session_factory -from tests.integration.chat.conftest import set_props_n_ctx +from tests.integration.chat.conftest import set_props_n_ctx, set_tokens_per_word from worker.ingestion import run pytestmark = pytest.mark.integration @@ -365,3 +365,24 @@ async def test_the_answer_reserves_room_instead_of_taking_the_whole_window( answer_requests = [r for r in llamacpp_server if r.get("max_tokens") != 12] assert answer_requests assert answer_requests[-1]["max_tokens"] == ANSWER_RESERVE_TOKENS + + +async def test_history_is_trimmed_by_the_models_own_tokenizer_when_it_answers( + client: AsyncClient, engine: Engine, real_model: object, llamacpp_server: list[dict] +) -> None: + """The exact count reaches the wire, not just the unit tests: an early + turn a permissive heuristic would have kept is dropped once the stub's + `/tokenize` prices it heavily, and the turn nearer the question survives.""" + set_tokens_per_word(1000) # any one turn alone blows the whole history budget + workspace_id, _ = _seed(engine) + thread_id = await _open_thread(client, workspace_id) + + await _send(client, thread_id, "first turn establishing some history") + await _send(client, thread_id, "what happened to revenue?") + + answer_requests = [r for r in llamacpp_server if r.get("max_tokens") != 12] + assert answer_requests + roles = [m["role"] for m in answer_requests[-1]["messages"]] + # System and the new question always survive; the expensive first turn + # does not, because the exact count priced it out of the budget. + assert roles == ["system", "user"] diff --git a/surfsense_local/backend/tests/unit/chat/test_history.py b/surfsense_local/backend/tests/unit/chat/test_history.py index 2bb55c62c0..208598643a 100644 --- a/surfsense_local/backend/tests/unit/chat/test_history.py +++ b/surfsense_local/backend/tests/unit/chat/test_history.py @@ -16,9 +16,9 @@ def _turn(marker: str, tokens: int) -> ChatMessage: ) -def test_the_system_and_new_turn_bracket_the_history() -> None: +async def test_the_system_and_new_turn_bracket_the_history() -> None: """Whatever the window keeps, the system leads and the new user turn trails.""" - messages = build_messages("SYSTEM", [_turn("a", 10)], "ask") + messages = await build_messages("SYSTEM", [_turn("a", 10)], "ask") assert messages[0].role == "system" assert messages[0].content == "SYSTEM" @@ -26,23 +26,63 @@ def test_the_system_and_new_turn_bracket_the_history() -> None: assert messages[-1].content == "ask" -def test_the_oldest_turns_over_budget_are_dropped() -> None: +async def test_the_oldest_turns_over_budget_are_dropped() -> None: """Turns are kept newest-first until the budget is spent, in original order.""" half = DEFAULT_HISTORY_TOKENS // 2 history = [_turn("x", half), _turn("y", half), _turn("z", half)] - kept = build_messages("SYSTEM", history, "ask")[1:-1] + kept = (await build_messages("SYSTEM", history, "ask"))[1:-1] # Three half-budget turns cannot all fit; the oldest falls away, order holds. assert [message.content[0] for message in kept] == ["y", "z"] -def test_a_caller_supplied_budget_overrides_the_default() -> None: +async def test_a_caller_supplied_budget_overrides_the_default() -> None: """A model with a known, narrower window keeps less history than the default figure would allow, without waiting for the default to change.""" history = [_turn("a", 50), _turn("b", 50)] - kept_default = build_messages("SYSTEM", history, "ask")[1:-1] - kept_tight = build_messages("SYSTEM", history, "ask", history_budget=50)[1:-1] + kept_default = (await build_messages("SYSTEM", history, "ask"))[1:-1] + kept_tight = ( + await build_messages("SYSTEM", history, "ask", history_budget=50) + )[1:-1] assert len(kept_tight) < len(kept_default) + + +async def test_an_exact_counter_is_preferred_over_the_heuristic() -> None: + """The whole point of wiring a counter through: a turn the heuristic would + keep because it under-counted a dense, short string is trimmed correctly + once the model's own tokenizer prices it.""" + # Four characters, which the heuristic prices at 1 token; the fake + # tokenizer says this string is actually 40, well over a 10 token budget. + history = [_turn("q", 1)] + + async def exact(text: str) -> int: + return 40 + + kept = ( + await build_messages("SYSTEM", history, "ask", history_budget=10, token_count=exact) + )[1:-1] + + assert kept == [] + + +async def test_a_counter_that_fails_falls_back_to_the_heuristic_for_that_turn() -> None: + """None is a real answer (no endpoint, a transient failure), not a zero: + the turn is still priced, just approximately, rather than being kept for + free or dropped as though it cost nothing.""" + history = [_turn("q", DEFAULT_HISTORY_TOKENS)] + + async def unavailable(text: str) -> int | None: + return None + + kept = ( + await build_messages( + "SYSTEM", history, "ask", history_budget=10, token_count=unavailable + ) + )[1:-1] + + # The heuristic prices this turn at DEFAULT_HISTORY_TOKENS, over a budget + # of 10, so it is still dropped: the fallback ran, not a free pass. + assert kept == [] diff --git a/surfsense_local/backend/tests/unit/llm/providers/llamacpp/fake_router.py b/surfsense_local/backend/tests/unit/llm/providers/llamacpp/fake_router.py index 2ece1c347b..ea93edee3a 100644 --- a/surfsense_local/backend/tests/unit/llm/providers/llamacpp/fake_router.py +++ b/surfsense_local/backend/tests/unit/llm/providers/llamacpp/fake_router.py @@ -29,6 +29,13 @@ def __init__(self, models: list[str] | None = None) -> None: # `reasoning_content`, and a short `max_tokens` is spent before a # single `content` token exists. self.thinks = False + # What `/tokenize` reports for one content-encoded token, so a test + # can make the fake count deterministically instead of running a + # real tokenizer. + self.tokens_per_word = 1 + # How many times `/props` was actually asked, so a test can tell a + # cached read apart from a fresh round trip. + self.props_calls = 0 def transport(self) -> httpx.MockTransport: return httpx.MockTransport(self._handle) @@ -38,6 +45,7 @@ def _handle(self, request: httpx.Request) -> httpx.Response: if path == "/health": return httpx.Response(200, json={"status": "ok"}) if path == "/props": + self.props_calls += 1 return httpx.Response( 200, json={ @@ -80,6 +88,12 @@ def _handle(self, request: httpx.Request) -> httpx.Response: if path == "/models/unload": self.loaded.discard(json.loads(request.content)["model"]) return httpx.Response(200, json={"success": True}) + if path == "/tokenize": + body = json.loads(request.content) + words = body.get("content", "").split() + return httpx.Response( + 200, json={"tokens": list(range(len(words) * self.tokens_per_word))} + ) if path == "/v1/chat/completions": body = json.loads(request.content) self.chat_bodies.append(body) diff --git a/surfsense_local/backend/tests/unit/llm/providers/llamacpp/test_provider.py b/surfsense_local/backend/tests/unit/llm/providers/llamacpp/test_provider.py index 22c5807fee..c212dea2af 100644 --- a/surfsense_local/backend/tests/unit/llm/providers/llamacpp/test_provider.py +++ b/surfsense_local/backend/tests/unit/llm/providers/llamacpp/test_provider.py @@ -154,3 +154,42 @@ async def test_deleting_something_absent_says_so(tmp_path) -> None: with pytest.raises(FileNotFoundError): await provider.delete("never-installed") + + +@pytest.mark.asyncio +async def test_capabilities_are_cached_for_a_resident_model() -> None: + """`/props` describes a resident model, and nothing about it changes + between turns: asking twice should cost one round trip, not two. + + Measured live: `context_tokens()` and `chat()`'s own template shaping each + read `/props` independently on every single message, which is two calls + where one answer would do, and which showed up as real, avoidable latency + on an already slow load. + """ + fake = FakeRouter(["qwen3"]) + fake.loaded.add("qwen3") + provider = provider_for(fake) + + await provider.capabilities("qwen3") + await provider.capabilities("qwen3") + + assert fake.props_calls == 1 + + +@pytest.mark.asyncio +async def test_a_fresh_load_invalidates_the_cached_capabilities() -> None: + """The one event that can actually change what `/props` would say: the + idle timeout evicts a model, and the next turn reloads it, possibly with + a rewritten preset. The cache must not go on quoting the old answer.""" + fake = FakeRouter(["qwen3"]) + provider = provider_for(fake) + + async for _ in provider.chat("qwen3", [Message("user", "hi")]): + pass + assert fake.props_calls == 1 + + fake.loaded.discard("qwen3") # the router's own idle timer evicted it + + async for _ in provider.chat("qwen3", [Message("user", "hi")]): + pass + assert fake.props_calls == 2 diff --git a/surfsense_local/backend/tests/unit/llm/providers/llamacpp/test_router_client.py b/surfsense_local/backend/tests/unit/llm/providers/llamacpp/test_router_client.py index 7606186674..8c9fb5ae40 100644 --- a/surfsense_local/backend/tests/unit/llm/providers/llamacpp/test_router_client.py +++ b/surfsense_local/backend/tests/unit/llm/providers/llamacpp/test_router_client.py @@ -86,3 +86,28 @@ def refuse(request: httpx.Request) -> httpx.Response: client = RouterClient("http://127.0.0.1:1", transport=httpx.MockTransport(refuse)) assert await client.health() is False + + +@pytest.mark.asyncio +async def test_tokenize_counts_against_the_models_own_tokenizer() -> None: + """The exact count the router's tokenizer gives, not a heuristic.""" + fake = FakeRouter(["qwen3"]) + fake.tokens_per_word = 3 + + count = await client_for(fake).tokenize("qwen3", "three little words") + + assert count == 9 + + +@pytest.mark.asyncio +async def test_tokenize_raises_on_a_router_that_does_not_answer() -> None: + """An older build, or a transient failure: the caller decides the fallback, + this does not silently return zero.""" + + def _404(request: httpx.Request) -> httpx.Response: + return httpx.Response(404, json={"error": "not found"}) + + client = RouterClient("http://127.0.0.1:1234", transport=httpx.MockTransport(_404)) + + with pytest.raises(httpx.HTTPError): + await client.tokenize("qwen3", "hi") From df00e86a3a55b67eb1a47ef7cb73b347d3eb345c Mon Sep 17 00:00:00 2001 From: Anish Sarkar <104695310+AnishSarkar22@users.noreply.github.com> Date: Mon, 21 Sep 2026 19:43:38 +0530 Subject: [PATCH 54/60] fix(llm): make the recommendation star and the badge read one classification A build's badge and its star eligibility were computed by two independent thresholds over the same offload_fraction: copy.py drew lines at 0.25/0.5 to choose wording, recommendation.py drew a separate line at 0.75 (MAX_OFFLOAD) to choose whether to star it. Nothing kept them in step, so a build could be starred while its own badge read "Too big for the GPU, so part runs on the CPU" or worse. New modules/llm/fit/speed.py classifies a verdict into one SpeedTier once; copy.py renders wording from the tier instead of its own bands, and recommendation.py's eligibility is RECOMMENDABLE_TIERS (FULL and LIGHT_SPILL) on that same tier. MAX_OFFLOAD is gone. A recommended build can now only ever carry "Full speed" or "Most of it still fits" wording, by construction. test_badge_matches_the_load.py gains a property test sweeping every curated model against every budget shape, asserting a starred row's badge is never spill wording. Three test_recommendation.py fixtures tuned to the old 0.75 cutoff are recomputed against the tighter one, one of them (the module's founding case) with numbers that had gone stale independent of this change. --- .../backend/modules/llm/catalog/__init__.py | 3 +- .../modules/llm/catalog/recommendation.py | 25 ++++---- .../backend/modules/llm/fit/__init__.py | 4 ++ .../backend/modules/llm/fit/copy.py | 17 ++--- .../backend/modules/llm/fit/speed.py | 63 +++++++++++++++++++ .../catalog/test_badge_matches_the_load.py | 20 ++++++ .../unit/llm/catalog/test_recommendation.py | 22 ++++--- .../backend/tests/unit/llm/fit/test_speed.py | 56 +++++++++++++++++ 8 files changed, 181 insertions(+), 29 deletions(-) create mode 100644 surfsense_local/backend/modules/llm/fit/speed.py create mode 100644 surfsense_local/backend/tests/unit/llm/fit/test_speed.py diff --git a/surfsense_local/backend/modules/llm/catalog/__init__.py b/surfsense_local/backend/modules/llm/catalog/__init__.py index 90cd017c9d..8f82759689 100644 --- a/surfsense_local/backend/modules/llm/catalog/__init__.py +++ b/surfsense_local/backend/modules/llm/catalog/__init__.py @@ -6,12 +6,11 @@ Variant, load_curated_models, ) -from modules.llm.catalog.recommendation import MAX_OFFLOAD, Recommendation, recommend +from modules.llm.catalog.recommendation import Recommendation, recommend from modules.llm.catalog.rows import CatalogRow, curated_rows from modules.llm.catalog.service import Catalog, CatalogService, InstalledRow __all__ = [ - "MAX_OFFLOAD", "SCHEMA_VERSION", "Catalog", "CatalogRow", diff --git a/surfsense_local/backend/modules/llm/catalog/recommendation.py b/surfsense_local/backend/modules/llm/catalog/recommendation.py index a2c76a5703..c891a898ab 100644 --- a/surfsense_local/backend/modules/llm/catalog/recommendation.py +++ b/surfsense_local/backend/modules/llm/catalog/recommendation.py @@ -6,6 +6,13 @@ stars a 1.7B on the same machine. That rule is not conservative, it is wrong, because it bans a configuration that demonstrably works. +Eligibility is `speed.RECOMMENDABLE_TIERS`, not a threshold owned here. This +module used to draw its own line on `offload_fraction` (0.75) while `copy.py` +drew a different one (0.25, 0.5) to choose its wording, and nothing kept the +two in step: a build could be starred while its own badge said "expect it to +be slow". Reading the same tier both modules now share is what makes that +combination unrepresentable rather than merely untested. + The policy ranges over builds rather than models, because a build is what the user installs and what `rank` describes. With one variant per entry the cross-product is the entry list and the behaviour is identical, but writing it @@ -17,16 +24,9 @@ from dataclasses import dataclass from modules.llm.catalog.manifest import CuratedModel, Variant -from modules.llm.fit import FitState, HardwareBudget, estimate, planned_precision +from modules.llm.fit import HardwareBudget, estimate, planned_precision from modules.llm.fit.estimate import FitVerdict - -# > ponytail: provisional, and possibly unnecessary. On the one machine measured -# > every offload fraction stayed usable: fully on the CPU the model still -# > decoded at 13 t/s, above reading pace, and prefill held at half device speed. -# > No threshold would have fired anywhere in the curated range. Check whether -# > this gate ever triggers before treating the number as load bearing; a -# > mechanism that never fires reads as protection that was never tested. -MAX_OFFLOAD = 0.75 +from modules.llm.fit.speed import RECOMMENDABLE_TIERS, speed_tier @dataclass(frozen=True) @@ -55,9 +55,10 @@ def recommend( verdict = estimate( entry.model_shape, variant.size_bytes, budget, precision=precision ) - if verdict.state is FitState.TOO_BIG: - continue - if verdict.offload_fraction > MAX_OFFLOAD: + # TOO_BIG's own tier is never in RECOMMENDABLE_TIERS, so this is + # the one check: physics and speed are both judged by the same + # classification, not a state check plus a separate threshold. + if speed_tier(verdict) not in RECOMMENDABLE_TIERS: continue candidates.append(Recommendation(entry, variant, verdict)) diff --git a/surfsense_local/backend/modules/llm/fit/__init__.py b/surfsense_local/backend/modules/llm/fit/__init__.py index be0773269e..7bafadb5aa 100644 --- a/surfsense_local/backend/modules/llm/fit/__init__.py +++ b/surfsense_local/backend/modules/llm/fit/__init__.py @@ -12,12 +12,14 @@ from modules.llm.fit.offload import offload_fraction from modules.llm.fit.plan_load import LoadPlan, plan_load from modules.llm.fit.precision import planned_precision, resident_precision +from modules.llm.fit.speed import RECOMMENDABLE_TIERS, SpeedTier, speed_tier from modules.llm.fit.states import FitState from modules.llm.fit.types import KvPrecision, ModelShape __all__ = [ "CONTEXT_FLOOR_TOKENS", "CONTEXT_RUNGS", + "RECOMMENDABLE_TIERS", "Badge", "FitState", "FitVerdict", @@ -26,6 +28,7 @@ "LoadPlan", "ModelShape", "NeedItems", + "SpeedTier", "badge", "compute_buffer_bytes", "estimate", @@ -35,4 +38,5 @@ "plan_load", "planned_precision", "resident_precision", + "speed_tier", ] diff --git a/surfsense_local/backend/modules/llm/fit/copy.py b/surfsense_local/backend/modules/llm/fit/copy.py index 6814db4a0c..d39daf30ab 100644 --- a/surfsense_local/backend/modules/llm/fit/copy.py +++ b/surfsense_local/backend/modules/llm/fit/copy.py @@ -8,6 +8,11 @@ and never on the runtime's name. One label set is wrong on two of the three targets. +Which band a verdict falls in is `speed.speed_tier`'s job, not this module's: +that is the one place `offload_fraction` becomes a decision, so a recommended +build and its badge read off the same classification rather than two +independently thresholded ones. + **No em dashes and no hyphens in anything here.** Commas, full stops or parentheses. That is a standing rule for every user facing string in this phase. """ @@ -16,14 +21,9 @@ from modules.llm.fit.budget import HardwareBudget from modules.llm.fit.estimate import FitVerdict +from modules.llm.fit.speed import SpeedTier, speed_tier from modules.llm.fit.states import FitState -# Three bands, because the middle state spans barely noticeable to unusable and -# one sentence is wrong at both ends. Below the first, most of the model is still -# resident; above the second, most of it is not. -_SLIGHT_SPILL = 0.25 -_HEAVY_SPILL = 0.5 - @dataclass(frozen=True) class Badge: @@ -47,9 +47,10 @@ def badge(fit: FitVerdict, budget: HardwareBudget) -> Badge: def _spill(fit: FitVerdict, budget: HardwareBudget) -> str: where = "the GPU" if budget.uma else "the graphics card" elsewhere = "the CPU" if budget.uma else "the processor" - if fit.offload_fraction <= _SLIGHT_SPILL: + tier = speed_tier(fit) + if tier is SpeedTier.LIGHT_SPILL: return f"A little too big for {where}. Most of it still fits." - if fit.offload_fraction < _HEAVY_SPILL: + if tier is SpeedTier.MODERATE_SPILL: return f"Too big for {where}, so part runs on {elsewhere}." return f"Well over {where}'s memory. Expect it to be slow." diff --git a/surfsense_local/backend/modules/llm/fit/speed.py b/surfsense_local/backend/modules/llm/fit/speed.py new file mode 100644 index 0000000000..ee4f1c3893 --- /dev/null +++ b/surfsense_local/backend/modules/llm/fit/speed.py @@ -0,0 +1,63 @@ +"""The one place `offload_fraction` becomes a decision. + +A badge and a recommendation both start from the same `FitVerdict`, and both +used to threshold its raw `offload_fraction` independently: the badge drew +lines at a quarter and a half to choose its wording, the recommendation drew +a separate line at three quarters to choose whether to star a build. Nothing +connected the two, so a build could land in a badge's "expect it to be slow" +band while still being the one row the app steered someone toward. + +Collapsing both into one ordered tier removes the second line rather than +moving it: a badge's wording and a recommendation's eligibility are now both +read off the same value, so they cannot disagree about the same verdict. Once +this exists, "should this be recommended" is answered without reference to +`offload_fraction` at all. +""" + +from enum import StrEnum + +from modules.llm.fit.estimate import FitVerdict +from modules.llm.fit.states import FitState + +# The same quarter and half `copy.py` used to threshold on its own, moved here +# so nothing else can draw them differently. Below the first, most of the +# model is still resident; above the second, most of it is not. +_LIGHT_SPILL_CEILING = 0.25 +_MODERATE_SPILL_CEILING = 0.5 + + +class SpeedTier(StrEnum): + """How a build will actually feel to use, worst to best. + + Ordered so a caller can compare tiers, though nothing here does yet; the + order exists because "how good does this have to be to recommend" is a + threshold on this scale, not a fact about any one member of it. + """ + + TOO_BIG = "too_big" + HEAVY_SPILL = "heavy_spill" + MODERATE_SPILL = "moderate_spill" + LIGHT_SPILL = "light_spill" + FULL = "full" + + +def speed_tier(fit: FitVerdict) -> SpeedTier: + """Classify a verdict once, for every caller that used to classify it + itself.""" + if fit.state is FitState.TOO_BIG: + return SpeedTier.TOO_BIG + if fit.state is not FitState.PARTIAL: + return SpeedTier.FULL + if fit.offload_fraction <= _LIGHT_SPILL_CEILING: + return SpeedTier.LIGHT_SPILL + if fit.offload_fraction < _MODERATE_SPILL_CEILING: + return SpeedTier.MODERATE_SPILL + return SpeedTier.HEAVY_SPILL + + +# The tiers a build may be starred at. Anything `copy.py` would describe with +# spill wording heavier than "most of it still fits" is excluded, by +# construction rather than by a second, independently tuned number: the +# recommendation policy no longer has a threshold of its own to drift out of +# step with the badge's. +RECOMMENDABLE_TIERS = frozenset({SpeedTier.FULL, SpeedTier.LIGHT_SPILL}) diff --git a/surfsense_local/backend/tests/unit/llm/catalog/test_badge_matches_the_load.py b/surfsense_local/backend/tests/unit/llm/catalog/test_badge_matches_the_load.py index 125a664cd8..899c5c7540 100644 --- a/surfsense_local/backend/tests/unit/llm/catalog/test_badge_matches_the_load.py +++ b/surfsense_local/backend/tests/unit/llm/catalog/test_badge_matches_the_load.py @@ -111,3 +111,23 @@ def test_a_machine_with_no_gpu_is_never_told_about_a_graphics_card() -> None: for row in curated_rows(CURATED, BUDGETS[name]): assert "graphics card" not in row.badge.reason assert row.badge.verdict in {"Works here", "Won't fit"} + + +@pytest.mark.parametrize("budget_name", BUDGETS) +def test_a_recommended_row_never_carries_spill_wording(budget_name) -> None: + """The invariant the badge and the star share now: a starred build's own + badge can never be the wording that tells someone to expect it to be slow. + + Swept over every curated model and every budget shape rather than one + fixture, because this has to hold for whichever build ends up starred on + whichever machine, not just the one screenshot that found the gap. + """ + budget = BUDGETS[budget_name] + rows = {row.model_id: row for row in curated_rows(CURATED, budget)} + pick = recommend(CURATED, budget) + if pick is None: + return + + starred = rows[pick.entry.model_id] + + assert starred.badge.verdict in {"Full speed", "Works here"} diff --git a/surfsense_local/backend/tests/unit/llm/catalog/test_recommendation.py b/surfsense_local/backend/tests/unit/llm/catalog/test_recommendation.py index 7857886248..6870a5bb5b 100644 --- a/surfsense_local/backend/tests/unit/llm/catalog/test_recommendation.py +++ b/surfsense_local/backend/tests/unit/llm/catalog/test_recommendation.py @@ -70,20 +70,24 @@ def model(model_id: str, label: str, blocks: int, variants: list[dict]) -> Curat def test_a_spilling_model_can_be_recommended_over_a_resident_one() -> None: - """The measured contradiction. A FITS-only rule fails this, which is the point. + """A FITS-only rule fails this, which is the point. - On this card the 8B spills about a fifth and runs without noticeable lag, - while a residency-only rule stars the 1.7B. + A small discrete card, mostly free: the 4B spills a fifth of a layer or so + (0.22) rather than fitting whole, and still outranks the fully resident + 1.7B and 0.6B by build rank. A residency-only rule would star the 1.7B + instead, which is not conservative, it is wrong: the 4B demonstrably runs. """ - pick = recommend(CURATED, RTX_3050) + small_discrete = HardwareBudget(4000 * MIB, 4300 * MIB, 1024 * MIB, 16000 * MIB, False, True) + + pick = recommend(CURATED, small_discrete) assert pick is not None - assert pick.entry.label == "Qwen3 8B" + assert pick.entry.label == "Qwen3 4B" def test_physics_still_refuses_whatever_the_rank() -> None: """The ceiling relaxes residency, never physics.""" - tiny = HardwareBudget(2000 * MIB, 2000 * MIB, 1024 * MIB, 3000 * MIB, False, True) + tiny = HardwareBudget(2200 * MIB, 2200 * MIB, 1024 * MIB, 3200 * MIB, False, True) pick = recommend(CURATED, tiny) @@ -115,8 +119,12 @@ def test_the_policy_ranges_over_builds_so_a_smaller_one_can_rescue_an_entry() -> build("u/q", "8b-q4.gguf", "Q4_K_M", int(5.03 * GB), 78), ]) ] + # A roomier discrete card than RTX_3050: the Q4 build spills lightly + # (0.19) here rather than the third RTX_3050 would cost it, which is + # enough to demonstrate the point without also needing full residency. + roomier_discrete = HardwareBudget(6000 * MIB, 6800 * MIB, 1024 * MIB, 22750 * MIB, False, True) - pick = recommend(two_builds, RTX_3050) + pick = recommend(two_builds, roomier_discrete) assert pick is not None assert pick.variant.quantization == "Q4_K_M" diff --git a/surfsense_local/backend/tests/unit/llm/fit/test_speed.py b/surfsense_local/backend/tests/unit/llm/fit/test_speed.py new file mode 100644 index 0000000000..2d7d9036df --- /dev/null +++ b/surfsense_local/backend/tests/unit/llm/fit/test_speed.py @@ -0,0 +1,56 @@ +"""The one place `offload_fraction` becomes a decision. + +Everything that turns a verdict into a person-facing choice (what the badge +says, whether the star shows) reads this instead of thresholding the raw +fraction itself, so two callers reading the same verdict cannot land on two +different, contradicting answers. +""" + +import pytest + +from modules.llm.fit.estimate import FitVerdict +from modules.llm.fit.speed import RECOMMENDABLE_TIERS, SpeedTier, speed_tier +from modules.llm.fit.states import FitState + +pytestmark = pytest.mark.unit + + +def verdict(state: FitState, fraction: float = 0.0) -> FitVerdict: + """A verdict needing 21 GB against 13.6 GB available.""" + return FitVerdict(state, 21 * 1000**3, 13_600_000_000, fraction) + + +def test_too_big_is_its_own_tier() -> None: + """Physics refusing a build is a different fact from it merely spilling.""" + assert speed_tier(verdict(FitState.TOO_BIG)) is SpeedTier.TOO_BIG + + +def test_full_residency_is_the_top_tier() -> None: + """Nothing spilled, so nothing here is a compromise.""" + assert speed_tier(verdict(FitState.FITS)) is SpeedTier.FULL + + +@pytest.mark.parametrize( + ("fraction", "tier"), + [ + (0.0, SpeedTier.LIGHT_SPILL), + (0.1, SpeedTier.LIGHT_SPILL), + (0.25, SpeedTier.LIGHT_SPILL), + (0.26, SpeedTier.MODERATE_SPILL), + (0.35, SpeedTier.MODERATE_SPILL), + (0.5, SpeedTier.HEAVY_SPILL), + (0.9, SpeedTier.HEAVY_SPILL), + ], +) +def test_partial_splits_into_the_same_three_bands_the_badge_used_to_own( + fraction: float, tier: SpeedTier +) -> None: + """The exact boundaries `copy.py` used to threshold on its own: a quarter + and a half. Moved here so nothing else can draw them differently.""" + assert speed_tier(verdict(FitState.PARTIAL, fraction)) is tier + + +def test_only_full_and_light_spill_are_recommendable() -> None: + """The invariant this module exists for: a tier the star is allowed to use + can never be one `copy.py` would describe as a compromise.""" + assert {SpeedTier.FULL, SpeedTier.LIGHT_SPILL} == RECOMMENDABLE_TIERS From 8845e626aa3aaea7b235a9f8f7a1f75794a43677 Mon Sep 17 00:00:00 2001 From: Anish Sarkar <104695310+AnishSarkar22@users.noreply.github.com> Date: Mon, 21 Sep 2026 20:14:43 +0530 Subject: [PATCH 55/60] feat(plans): update teh llamacpp-runtime plan to match the current path --- .../api/07-llamacpp-runtime.md | 3741 ++++++++--------- 1 file changed, 1639 insertions(+), 2102 deletions(-) diff --git a/plans/community-local/api/07-llamacpp-runtime.md b/plans/community-local/api/07-llamacpp-runtime.md index c891e95016..2971caa4c9 100644 --- a/plans/community-local/api/07-llamacpp-runtime.md +++ b/plans/community-local/api/07-llamacpp-runtime.md @@ -1,752 +1,695 @@ -# API — Phase 7: Replace Ollama with llama.cpp +# API — Phase 7: the llama.cpp runtime -> Owns: `modules/llm/providers/`, the runtime sidecar, the model catalog, and -> packaging for all three targets. Supersedes the Ollama runtime decision in -> [`../00-umbrella-plan.md`](../00-umbrella-plan.md) and the `hf.co/` fallback in -> [`05d-llmfit-catalog-expansion.md`](05d-llmfit-catalog-expansion.md). Extends +> **Status: built.** This document describes what is implemented on +> `feat/llama-cpp`, and the reasoning behind each decision, so a reader can +> change the code without rediscovering why it is shaped this way. +> +> Owns `modules/llm/` (providers, fit, gguf, hardware, catalog), the +> `llama-server` sidecar, the curated manifest, and packaging for all three +> targets. Supersedes the Ollama runtime decision in +> [`../00-umbrella-plan.md`](../00-umbrella-plan.md), the `hf.co/` fallback in +> [`05d-llmfit-catalog-expansion.md`](05d-llmfit-catalog-expansion.md), and the +> recommendation half of > [`05a-model-recommendations.md`](05a-model-recommendations.md), which is -> reduced to what survives here: prompt tiers, fingerprinting and onboarding. +> reduced to prompt tiers, fingerprinting and onboarding. +> +> Ollama and llmfit are gone from the product. Ingestion, the 384-dimensional +> bge-small index, retrieval, Kokoro, local image generation and the chat SSE +> protocol are unchanged. ## Goal One local runtime: `llama-server` in router mode. Any GGUF model the user wants, -not a curated subset. Recommendations that come from the machine's own allocator -rather than an advisor's estimate. - -Ingestion, the 384-dimensional bge-small index, retrieval, and chat SSE do not -change. +not a curated subset. A fit verdict that comes from the allocator's own view of +the machine rather than from an advisor's database. -## Why +## Why llama.cpp replaced Ollama -Ollama's library holds 240 models. A measured scan on this branch resolved 138 -of 9,590 llmfit rows to an installable Ollama artifact — 1.4%. The -`ollama_name` gate in `OllamaRuntime.resolve()` drops every model Ollama's own -registry does not carry, including the ~1,500 per scan that llama.cpp could run -directly from a Hugging Face GGUF. +Ollama's library held 240 models. A measured scan on this branch resolved 138 of +9,590 llmfit rows to an installable Ollama artifact, 1.4%. The `ollama_name` +gate in `OllamaRuntime.resolve()` dropped every model Ollama's registry did not +carry, including roughly 1,500 per scan that llama.cpp could run directly from a +Hugging Face GGUF. -llama.cpp runs anything in GGUF: **204,797 repos on Hugging Face**, bounded only -by its 152 supported architectures. The runtime also ships smaller (11–31 MB -versus Ollama's 501 MB staged payload), fits models to the machine itself -(`--fit`, default on), and exposes multimodal and structured-output contracts -Ollama's native API does not. +llama.cpp runs anything in GGUF: 204,797 repos on Hugging Face, bounded only by +the architectures it supports. It also ships smaller (11 to 31 MB against +Ollama's 501 MB staged payload), fits models to the machine itself (`--fit`, +default on), and exposes multimodal and structured-output contracts Ollama's +native API did not. ## Decisions | Decision | Choice | Why | |---|---|---| -| **Runtime** | `llama-server` router mode, single sidecar | `--models-dir`, `--models-max 1`, LRU eviction, one PID for the supervisor to reap. Maps 1:1 onto every Ollama call the adapter makes. | -| **MLX** | **Not shipped.** Revisit after launch | Ollama switched its Apple Silicon engine to MLX in Mar 2026, so the swap costs Mac speed on models under ~14B. MLX's format covers 23,985 HF repos against GGUF's 204,797 — one format and the full catalog wins. Mac users who want MLX point a connection at LM Studio; see **The Mac path**. | -| **llmfit** | **Authoring-time only. Not shipped, and not in CI.** | A person runs `scripts/refresh_curated_models.py` when adding or changing a curated entry — a few times a year — and commits the numbers. llmfit is not a gate, not a hardware source, not in any request path, and not in the build. Removes 27 MB, a pinned version, and five failure modes per installer. | -| **Gating** | Only physics refuses | `can_install = state != TOO_BIG`, and `TOO_BIG` *is* the physics refusal — `need` at the 16K floor exceeding usable VRAM **plus** RAM. No other fit level gates anything; `PARTIAL` installs like `FITS`. Eligibility (architecture, chat template, gated repo) blocks separately and is not a fit state. See **Fit states**. | -| **Catalog** | Two tiers — curated manifest, and Hugging Face search | Curated is the offline product and ships frozen. Search is a network feature that is simply absent airgapped. | -| **Fit estimate** | Our own, from the GGUF header | llmfit cannot score a model outside its database (`llmfit plan` states the precondition). Search needs an estimate anyway; once it exists, llmfit's is redundant *and* less accurate here. | -| **`rank`** | **Curated entries only. A searched model never carries one, from llmfit or anywhere else** | A preference order over models we tested, **for this app's job** — answering from the user's documents with citations that resolve — not general capability. An integer typed by a person, with llmfit proposing. Called `rank`, not `quality`, because it is only ever a sort key: nothing reads its magnitude, so it must not imply a measurement we do not have. Attaching one to an arbitrary repo attaches a base model's score to a derivative that behaves differently: of the top 100 GGUF repos by downloads, 38 match an llmfit entry, and those matches include `Huihui-Qwen3.8-27B-abliterated-GGUF` and `Qwen3.8-27B-Uncensored-GGUF` resolving to the base model's score. A wrong number people trust is worse than a blank they investigate. Search rows are **described, not judged**. | -| **Search order** | `sort=downloads`, descending | The only HF sort that behaves as a default. Presented as a popularity fact, never as an endorsement. See **Search ordering**. | -| **Fit badge** | **Every row, both tiers** | Fit is subtraction, not judgement — the opposite of `rank`. Coarse from file size in the search list, exact from the header on open, exact from `shape` offline for curated. Three states, no `unknown`. | -| **Recommendation** | Highest `rank` among entries predicted **fast enough**, not among entries that fit entirely | Residency is a mechanism; speed is the goal, and on small cards they disagree. Measured on an RTX 3050: Qwen3 4B spills ~20% and Qwen3 8B ~28%, and a user runs the 8B without noticing. A `FITS`-only rule stars a 1.7B on that machine while the owner is happily using the 8B. v1 approximates the gate with an offload-fraction ceiling; the roofline and its self-calibration follow. See **Recommendation policy**. | -| **Curated manifest** | **Stays, schema 3, authored by hand** | It is the whole product airgapped, the first-run default, and the only tier the recommendation reads. `scripts/refresh_curated_models.py` writes it; a person commits it; **llmfit never runs in CI**. | -| **Why `rank` can be authored at all** | Quality is a function of *(model, quantization)*, not of hardware | Measured twice: across 2,424 models at a fixed quant, quality differs 0/2424; and on one model swept 12G to 48G, quality does not move once `best_quant` stops moving. Hardware reaches quality through exactly one channel, the quantization, and pinning the file closes it. So the number is a property of a file, computable once and shipped as data, and a scan on the user's machine has nothing to discover. This is the load-bearing fact under the whole authoring design. See **Appendix**. | -| **Which llmfit field** | `score_components.quality` only. **`score` is discarded** | `score` blends fit, speed and context into quality, so three of its four terms describe a machine. In the same sweep it runs 65.1, 63.8, 63.6, 66.2, 63.6, 67.4 — non-monotonic and hardware-dependent. The composite is a statement about a laptop; the component is a statement about a file. Same reason `fit_level`, `memory_required_gb`, `estimated_tps` and `best_quant` are all read and thrown away. | -| **Scoring quantization** | The pinned one, found by a ladder sweep. A miss is a hard failure | llmfit cannot be asked for a score at a named quantization, and quality tracks `best_quant` exactly, so the budget decides which file gets graded. Qwen3 8B is **78** at the pinned Q4_K_M and **83** at Q8_0. See **Scoring at the pinned quantization**. | -| **Manifest validation** | Existing unit tests. **No new CI** | Pydantic (`extra="forbid"`, `model_validator`) plus `test_curated_models.py` already reject malformed manifests. A dead pin is a runtime failure, gracefully handled — CI cannot prevent a repo disappearing after the build anyway. `validated: true` is the real check, and it is a human one. | -| **Hardware budget** | `ggml_backend_dev_memory()` via `ctypes`, from the shipped libs | The allocator's own view. Falls back to `llama-server --list-devices`, then OS APIs. | -| **Downloads** | **SurfSense fetches the GGUF**, not `POST /models` | `llama-server` is a second process we do not proxy; an in-process fetch is the only place `egress.require()` actually holds. Also buys resume, checksums, and the header as the file lands. | -| **Quantization** | **One pinned file per entry in v1** | A pinned file is what "tested by SurfSense" can honestly claim. Per-machine selection is possible at no extra cost (one header read prices every quant), so a second variant on the largest entries is a cheap follow-on, not v1. See **The quantization ladder**. | -| **KV precision** | **`f16` when it fits, `q8_0` when it buys residency.** Symmetric `-ctk`/`-ctv` always | Measured: `f16` KV at a 16K window **cannot allocate** on a 6 GB card (`ErrorOutOfDeviceMemory`), while `q8_0` at the same depth runs. Quality is lossless either way. So the choice is not a global default but an output of the fit calculation — pay the precision cost only where it converts a spill into residency. Requires flash attention; see **Failure behavior**. **Conditional on per-model load args working** — see 7.3; if they do not, this decision degrades to one global choice. | -| **Context** | **Fixed at load. Floor 16K, capped at the model's own `context_length`** | llama.cpp fixes context at load, so `num_ctx()`'s per-request sizing has no equivalent. The floor is 3K history plus ~8K grounding plus a reply. Growing on occupancy is additive later; it needs a mid-conversation reload and two constants with nothing measured behind them. | -| **GPU backend** | **Vulkan on every platform off Apple Silicon. No CUDA** | Measured on an RTX 3050 at `b11050`: CUDA leads Vulkan **9.1%** on `pp512`, **10.2%** on `pp8192`, **2.1%** on decode — 0.66 s on an 11 s turn, for 685 MB. Vulkan covers NVIDIA, AMD and Intel from one 31 MB archive, its loader ships with Windows, and it is what the app already does today (`pruneCudaRunners()`). CUDA is specified as an optional later addition in [`08-cuda-backend.md`](08-cuda-backend.md), which needs **no code change** — ggml selects it by the files present. | -| **Device selection** | First device with `type == GPU`. **Never sum** | The same physical card appears once per loaded backend, and an integrated GPU can advertise more memory than a discrete one (16198 MiB against 6002 MiB, measured). ggml's ordering already expresses backend preference, so this **is** backend selection. Specified in **7.2**. | -| **Backend selection** | Runtime, by ggml | Releases are built `GGML_BACKEND_DL=ON`. A failed `dlopen` is skipped silently, so a missing Vulkan loader degrades to CPU rather than failing. | -| **Prompt tier fallback** | Keys on **loopback**, not provider name | `provider == "ollama" → COMPACT` breaks the moment a Mac user runs a 4B through LM Studio. `host_destination()` already computes loopback. | +| **Runtime** | `llama-server` router mode, one sidecar | `--models-dir`, `--models-max 1`, one PID for the supervisor to reap, and a process that costs no device memory until a model loads. | +| **Per-model flags** | A preset INI, read once at router startup | `POST /models/load` accepts an `args` field and ignores it, measured. `--models-preset` is the mechanism that works. | +| **Layer placement** | `--fit` owns it. We never set `n-gpu-layers` | Setting it by hand aborts the fitter, after which the model loads entirely on the CPU with exit 0 and no error. | +| **GPU backend** | Vulkan on every platform off Apple Silicon. No CUDA payload | Measured on an RTX 3050 at `b11050`: CUDA leads Vulkan 9.1% on `pp512`, 10.2% on `pp8192`, 2.1% on decode. That is 0.66 s on an 11 s turn, for 685 MB. Vulkan covers NVIDIA, AMD and Intel from one 31 MB archive and its loader ships with Windows. CUDA is specified as an optional later addition in [`08-cuda-backend.md`](08-cuda-backend.md) and needs no code change, because ggml selects a backend by the files present. | +| **MLX** | Not shipped. Revisit after launch | MLX's format covers 23,985 HF repos against GGUF's 204,797. One format and the full catalog wins. Mac users who want MLX point a connection at LM Studio; see **The Mac path**. | +| **llmfit** | Authoring-time only. Not shipped, not in CI, not in any request path | A person runs `scripts/refresh_curated_models.py` when adding or changing a curated entry, a few times a year, and commits the numbers. | +| **GGUF parser** | llama.cpp's own `gguf` package, pinned at `0.19.0` | The search tier parses bytes from arbitrary repositories, and upstream's parser is the one that receives hardening. A 150-line adapter handles the one thing it cannot do, which is read a header out of a byte prefix. | +| **Hardware budget** | `ggml_backend_dev_memory()` through `ctypes`, in a child process | The allocator's own view. The child exists because the backend scan keys on the running executable's directory, because a driver fault would otherwise take the API with it, and because the first Metal device query compiles 20 shader libraries in 19 seconds. | +| **Device selection** | First device typed `GPU`. Never a sum | The same physical card appears once per loaded backend, and an integrated GPU can advertise more memory than a discrete one, 16198 MiB against 6002 MiB measured. ggml already orders backends by preference, so taking the first is also backend selection. | +| **Two memory questions** | `resident_bytes` and `refusal_bytes`, never one `overhead` term | They sit on opposite sides of the comparison. Collapsing them predicted residency for a configuration that measurably spilled, wrong by 922 MiB on the first attempt. | +| **Unified memory** | One pool: `min(working set, 0.85 x host)`. Never a sum | Apple Silicon reports two views of one memory. Added together, an 8 GB Mac priced as though it had 10.3 GB. | +| **Gating** | Only physics refuses | `can_install` is `state is not TOO_BIG`. `PARTIAL` installs exactly like `FITS`. Eligibility (architecture, chat template, gated repo) blocks separately and is not a fit state. | +| **Context** | Fixed at load. Floor 8192, rungs 8192 / 16384 / 32768, capped at the model's own `context_length` | llama.cpp fixes context at load, so per-request sizing has no equivalent. The floor is safe because `modules/chat/budget` sizes every part of a prompt from `n_ctx` rather than from a fixed history figure. | +| **KV precision** | `f16` when it fits, `q8_0` only when it converts a spill into residency | Measured: `f16` KV at a 16K window cannot allocate on a 6 GB card (`ErrorOutOfDeviceMemory`) while `q8_0` at the same depth runs. Quality is identical either way, but a quantized cache needs a working flash-attention kernel and `f16` does not, so the dependency is taken only where it pays. One rule, in `fit/precision.py`, read by the badge and by the loader. | +| **Catalog** | Two tiers: a curated manifest and Hugging Face search | Curated is the offline product and ships frozen. Search is a network feature and is simply absent airgapped. | +| **Fit badge** | Every row, both tiers | Fit is subtraction, not judgement. Exact from committed `shape` for curated, exact from the header for an opened search result, approximate from file size when a header cannot be read. | +| **`rank`** | Curated entries only, inside the variant, never on the wire | A preference order over models we tested for this app's job, answering from documents with citations that resolve. It is only ever a sort key, so it must not imply a measurement we do not have. A searched model never carries one. | +| **Recommendation** | Highest `rank` among builds whose speed tier is recommendable | Residency is a mechanism and speed is the goal, and on small cards they disagree. Both the badge's wording and the star's eligibility read one classification, so they cannot contradict each other. | +| **Downloads** | SurfSense fetches the GGUF, not `POST /models` | `llama-server` is a second process we do not proxy, so an in-process fetch is the only place `egress.require()` can hold. It also buys resume, checksums, and the header as the file lands. | +| **Install ids** | Opaque, server-minted, for both tiers | The renderer sends a `catalog_id` and nothing else. Letting it post a repo and file would hand the frontend the ability to name an arbitrary download. | +| **Egress** | `model_download` and `model_search`, both `huggingface.co` | Same host, different consent. A user who allowed model downloads allowed fetching a file they named; search sends text they are typing. | +| **Prompt tier fallback** | Keys on loopback, not on a provider name | `provider == "ollama"` broke the moment a Mac user ran a 4B through LM Studio. `host_destination()` already computes loopback. | ## Boundaries ```text - hardware budget ── ggml_backend_dev_memory() ← the allocator, not an advisor - │ - model facts ───── GGUF header (local file, or HTTP Range on huggingface.co) - │ - ▼ - FitEstimator ── advisory only - │ need = weights + KV(window) + compute_buffers - │ usable = device_free − fit_reserve - │ → FITS | PARTIAL | TOO_BIG (TOO_BIG = the physics refusal) - ▼ - CatalogService - ├── curated manifest: shape derived + rank by hand, ships frozen - └── search HF /api/models?filter=gguf + /tree/main - │ - ▼ - LlamaCppProvider ── Generator + ModelStore + LocalRuntime - │ - ▼ - llama-server router ── --fit places layers; the real answer + hardware ──── probe_devices() child process, ggml's own device list + │ │ + │ └─ gpu_status: present | absent | broken_install | unknown + │ + budget ────── build_budget(devices, ram) one device, never a sum + │ resident_bytes what must hold the model for full speed + │ refusal_bytes what physics allows at all + │ + model ─────── GGUF header, local file or HTTP Range on huggingface.co + │ ModelShape: 19 fields, quantization independent + ▼ + estimate ──── itemise() -> weights + mmproj + KV(window) + compute + │ FITS | PARTIAL | TOO_BIG, plus an offload fraction in layers + │ │ + │ └─ speed_tier() ─┬─ copy.badge() what the row says + │ └─ RECOMMENDABLE_TIERS whether it stars + ▼ + CatalogService ── curated (manifest, offline) + search (HF, egress gated) + │ plan_load() -> window and cache precision + │ reprice() -> models.ini + ▼ + llama-server router ── --fit places the layers. That is the real answer. ``` -The estimator is advisory at every stage above the runtime. `--fit`'s allocation -is authoritative at launch, and generation is ground truth after it. Unknown -shapes round **up**; never underestimate memory. +The estimate is advisory at every stage above the runtime. `--fit`'s allocation +is authoritative at load, and generation is ground truth after that. Unknown +shapes round up; never underestimate memory. An over-estimate costs a +pessimistic badge on a model that installs anyway. An under-estimate ships a +confident badge about a model that spills, which is the failure this estimator +exists to delete. -## The quantization ladder +## The Mac path -Measured: architecture fields are identical across quantizations of the same -model (`Q4_K_M`, `Q8_0`, `f16` of SmolLM2-135M all report `layers=30 -kv_heads=3 emb=576 ctx=8192`; only `general.file_type` differs). KV cost is -architecture-derived and quant-independent. So **one header read prices every -quantization of a model**, and the weights bytes come free from `tree/main`. +Dropping Ollama costs Apple Silicon speed on models under roughly 14B. Accepted, +because the escape hatch already ships and does not leave the machine: -Per-machine quant selection is therefore possible at no extra cost. We pin -anyway, for three reasons that are policy rather than capability: the -"tested by SurfSense" claim covers files we ran, support needs reproducible -bytes, and an unconstrained fit-maximiser hands small machines a Q2_K build -that answers noticeably worse (llmfit currently recommends exactly that for -Qwen3-14B on an 8 GB Mac). +- `host_destination()` returns `None` for loopback, so `require()` no-ops and no + egress prompt appears. +- `api_key_ciphertext` is nullable and `_headers(None)` returns `{}`, so keyless + local endpoints work end to end. +- `connection-form.tsx` ships `LM Studio (local)` and `Ollama (local)` presets. -**The two Qwen2.5-Coder entries are dropped.** SurfSense has no coding job — the -Studio formats are summary, flashcards, mindmap, quiz, podcast, office, web and -visuals, and chat is document Q&A with citations. A coding model was taking two -of eight slots in a list whose real gaps are a rung for 48-64 GB machines and a -vision entry. Anyone who wants one searches for it, gets a fit badge, and -installs it — that is what the search tier is for. +LM Studio runs MLX on Apple Silicon. A Mac user who wants it installs LM Studio +and picks the preset. **The Ollama preset is kept deliberately**: it is the +user-managed MLX path now, not dead code, and it is one of the few places the +string `ollama` legitimately survives. + +## The runtime + +### Router mode, and why the preset file exists + +`llama-server` starts in one of two shapes. Given `-m ` it loads that one +model and serves it. Given `--models-dir ` and no `-m` it starts as a +**router**: it holds no model, lists every GGUF in the directory as `unloaded`, +and spawns a child worker per model when one is loaded. `GET /props` reporting +`"role": "router"` is the check that the sidecar came up the way we asked; both +shapes answer `/health` identically. + +Router mode is what lets the sidecar boot before any model is installed, which +matters on a clean machine where the models directory is empty. Verified at +`b11050` on Windows and Linux: `/health` returns `{"status":"ok"}` and `/models` +returns `{"data":[],"object":"list"}` against an empty directory. + +The cost is that per-model arguments cannot be sent at load time. Measured on +macOS at `b11050`, every payload shape is accepted and none of them does +anything: `{"args": ["-c","4096"]}`, `{"args": ["--ctx-size","4096","-fa","on"]}` +and `{"preset": "..."}` all return `200 {"success":true}` while the spawned +worker's argv stays byte identical. The router reports that argv under +`GET /models` as `status.args`, which is the only reliable way to see what it +actually did. + +`--models-preset PATH` is the mechanism that works, and **the file is read once +at router startup**. Appending a section while the router runs does not surface +the model. Measured: a model dropped into a running router's directory is still +invisible 26 seconds later and appears immediately after a restart. So +installing a model, or changing its load plan, means rewriting the file and +restarting the sidecar, which costs 0.15 s because an idle router holds no model +memory. + +### Sidecar flags + +From `electron/src/main/sidecars/llamacpp.ts`: -The shipped six, measured at Q4_K_M and a 16K window: +```text +--models-dir /models +--host 127.0.0.1 --port +--models-max 1 +--sleep-idle-seconds 300 +--no-ui +--jinja +--reasoning-format deepseek +--models-preset /models/models.ini (only once the file exists) +``` -| model | file GB | layers | kv_heads | needs @16K | -|---|---|---|---|---| -| Qwen3 0.6B | 0.40 | 28 | 8 | 2,354 MiB | -| Qwen3 1.7B | 1.11 | 28 | 8 | 3,032 | -| Qwen3 4B | 2.50 | 36 | 8 | 4,630 | -| Qwen3 8B | 5.03 | 36 | 8 | 7,043 | -| Qwen3 14B | 9.00 | 40 | 8 | 10,969 | -| Qwen3 32B | 19.76 | 64 | 8 | 22,047 | - -> **This column excludes `compute_buffers` and predates the fit reserve**, so it -> under-states `need` by roughly 170 MiB at 16K and prices against raw free -> memory. `GPU` / `RAM` in the map below is therefore a simplification — the real -> output is an offload fraction, and the map shows it where measured. -> -> **The one row since measured came out a state worse than predicted, and it is -> the row that rewrote the recommendation policy.** Qwen3 4B on an RTX 3050 needs -> 4,858 MiB against `5,234 − 1,028 = 4,206 MiB` usable: `PARTIAL` at `f ≈ 0.20`, -> not `FITS` with 490 MiB spare. llama.cpp spilled 602 MiB of weights and 320 MiB -> of KV. Under a `FITS`-only rule that machine's ★ fell to **Qwen3 1.7B** — while -> its owner runs **Qwen3 8B** (`f ≈ 0.28`, computed) without noticeable lag. That -> contradiction is what moved the gate from residency to speed; see -> **Recommendation policy**. -> -> The other five rows are unverified predictions and are **not** re-derived here: -> applying a reserve measured once on a 6 GB Windows card to an M4 Max would be -> inventing numbers, which is the failure this phase exists to avoid. Re-derive -> each row when its hardware is measured. +with `cwd` set to the binaries directory and `LLAMA_CACHE` pointed at the models +directory. + +- `--models-max 1` because this app asks one question at a time, and a second + resident model is memory taken from the one being used. +- `--sleep-idle-seconds 300` is not optional. Measured at `b11050`, a model + without it self-evicted after roughly 30 s idle, which turns the second + question of a conversation into a reload. +- `--no-ui` because llama-server ships its own web UI, which we neither need nor + want exposed. +- `--reasoning-format deepseek` routes `` blocks to + `message.reasoning_content`. Without it a thinking model's trace enters the + answer and citation rewriting corrupts `[n]` tokens that appeared inside the + reasoning. +- `cwd` is the binaries directory because ggml scans the running executable's own + directory for backend libraries. Elsewhere it reports no devices, silently, + and every model runs on the CPU. +- The preset flag is conditional because the router exits non-zero on a missing + preset file, and on a clean install nothing has written one yet. + +`watchGenerationPreset()` in `electron/src/main/index.ts` polls the preset's size +and mtime every 5 s and restarts the sidecar on a change. Same shape as +`watchImageModel()` for sd-server, and for the same reason: the API is the +authority, and a change is user-initiated and rare. + +### Never set a layer count -What each machine gets: +```text +common_fit_params: failed to fit params to free device memory: + n_gpu_layers already set by user to 99, abort +``` + +Measured at `b11050`: with `-ngl 99` the model then loaded **entirely on the +CPU**, 478 MiB of VRAM touched on a machine with a working RTX 3050, no error, +exit 0. It looks like it worked. + +This is the easiest silent failure in the phase to introduce by accident, since +`-ngl 99` reads as "use the GPU harder". The preset therefore writes no layer +count at any time, and `preset.py`'s docstring says so where someone would go to +add one. + +What `--fit` does instead, measured at `-c 40960` against a 5,460 MiB working +set, is search rather than solve, reloading at each step: ```text -machine 0.6B 1.7B 4B 8B 14B 32B → recommended -M2 8 GB GPU GPU GPU RAM RAM — → Qwen3 4B (831 MiB spare) -RTX 3050 6 GB GPU GPU ~20% ~28% RAM RAM → Qwen3 8B (f ≈ 0.28) -M4 16 GB GPU GPU GPU GPU RAM RAM → Qwen3 8B (3,857 MiB spare) -RTX 4070 12 GB GPU GPU GPU GPU GPU RAM → Qwen3 14B (31 MiB spare) -RTX 4090 24 GB GPU GPU GPU GPU GPU GPU → Qwen3 32B (953 MiB spare) -M4 Max 64 GB GPU GPU GPU GPU GPU GPU → Qwen3 32B (26,000 MiB spare) +29/29 layers → 0/29 → 29/29 → 22/29 → 23/29 settles at 23/29 ``` -Every machine gets a GPU-resident pick, so six rungs strand nobody. Three gaps: - -- **The top collapses.** 24 GB and 64 GB — 2.7× apart — get the same file, and - the larger wastes 26 GB. A higher quant does not fix this; 32B at Q8 is ~35 GB - and still leaves room. **The top needs a bigger model.** -- **A gap at 16 GB.** 14B misses by 69 MiB (0.6%), so a 16 GB Mac drops to 8B - with 3.9 GB spare. -- **The 4070 row is decided by an uncalibrated constant.** 31 MiB of margin at - 1,024 MiB overhead; at Hermes' 2 GiB margin floor it does not fit and that - machine drops a rung. - -**Manifest growth is follow-on work, not phase 7.** v1 ships these six re-pinned -to GGUF; the runtime swap is the deliverable and the search tier covers the gaps -meanwhile. In priority order, once it lands: - -1. **A rung above 32B** — an 80B-A3B-class MoE. The only thing that serves - 48–64 GB machines. An MoE is the right shape for them: 80B of memory but ~3B - read per token, so it is the only way to spend that memory without the machine - crawling. Needs the `decode_fraction` field, which is computed from the header - (see **Authoring**) — without it the speed prediction is wrong by roughly 10×. -2. **A second variant on 8B and 32B** (`Q6_K`), filling the 16 GB gap and giving - the 4090 somewhere to go. Two extra pins, not fifty. -3. **A vision entry.** All six are text-only, in an app with a PDF pipeline. - Vision is a **capability**, not a separate list — see **Ranking**. -4. **Calibrate the overhead constant** before shipping. It decides the 4070 row. +Two behaviours matter. It never touched the context, because `-c` was set +explicitly and the fitter treats an explicit value as fixed. And if `-c` is left +unset it reduces context on its own as far as `fit_params_min_ctx`, 4096, which +is below our floor and happens silently. So `ctx-size` is not optional, and not +only because we want a particular window: leaving it out hands the floor to +llama.cpp. + +### The preset file + +`modules/llm/providers/llamacpp/preset.py` renders one section per model, keyed +by the id the router reports, which is the filename stem: + +```ini +[Qwen3-8B-Q4_K_M] +model = /Users/…/models/Qwen3-8B-Q4_K_M.gguf +ctx-size = 16384 +parallel = 1 +fit-target = 1024 +fit-ctx = 16384 +mmproj = /Users/…/models/mmproj-F16.gguf ; only with a projector +cache-type-k = q8_0 ; only at q8_0 +cache-type-v = q8_0 +flash-attn = on +``` -## The Mac path +- `parallel = 1`. llama-server defaults to four slots, which sizes the KV cache + for concurrency this app never uses. +- `fit-target` is pinned rather than inherited. The badge subtracted a specific + margin, so passing it makes the two agree by construction instead of by + assuming a default read from the source once. A vision projector's bytes are + **added** to it, because `--fit` allocates the projector after it has finished + placing layers and does not count it while deciding + ([llama.cpp#19980](https://github.com/ggml-org/llama.cpp/issues/19980)). +- `fit-ctx` is inert while `ctx-size` is set, since llama.cpp only shrinks a + context it chose itself. It is written anyway because it states the floor at + the place the fitter would look for one, so a later change to how the window is + set cannot quietly hand that floor back to llama.cpp's 4096. +- The `q8_0` lines are written as a group. Set one cache type and not the other + and the fused flash-attention kernel is skipped, after which attention falls + back to the CPU silently. + +`write_presets()` writes to a sibling `.tmp` and renames, so a half-written INI +never loads. + +### From download to answerable + +`CatalogService.reprice()` writes the preset for everything now on disk, priced +against two budgets: capacity decides the verdict so the plan agrees with the +badge the catalog already showed, and live caps how far the window widens past +the floor, because that part is opportunistic and on unified memory it comes out +of the same pool the OS is using. + +It skips projector files, because a projector is half of a vision model and has a +header and a size like any other file in the directory. It skips unreadable files +with a warning, because a truncated or foreign file costs that one model, while +failing would leave the runtime dead over a file nobody asked it to load. + +The install stream then waits: `wait_until_servable()` polls for up to 30 s at +0.5 s intervals, because the router only learns about a new model by restarting, +and reporting the install complete before then tells the user a model is ready +while a chat returns `model '' not found`, measured as a 400. A timeout +returns `False` rather than raising. The download did succeed and the file is on +disk, so reporting a failed install would be the wrong thing to say. + +### Turning thinking off + +A thinking model emits its whole trace before its first answer token, so a call +with a small `max_tokens` returns nothing. Measured against Qwen3 1.7B at +`b11050`: a 12-token title request came back with `content: ''`, +`finish_reason: length`, and a full `reasoning_content`. + +`thinking.py` sends two mechanisms per call, because each covers the other's +blind spot and both were measured to work: -Dropping Ollama costs Apple Silicon speed on models under ~14B. Accepted, -because the escape hatch already ships and does not leave the machine: +```python +THINKING_OFF = { + "thinking_budget_tokens": 0, + "chat_template_kwargs": {"enable_thinking": False}, +} +``` -- `host_destination()` returns `None` for loopback — *"nothing leaves the - machine"* — so `require()` no-ops and no egress prompt appears. -- `api_key_ciphertext` is nullable and `_headers(None)` returns `{}` — keyless - local endpoints work end to end. -- `connection-form.tsx` already ships `LM Studio (local) → http://localhost:1234/v1` - and `Ollama (local) → http://localhost:11434/v1`. +`chat_template_kwargs` is the documented one and is inert on a model whose +template never reads `enable_thinking`. `thinking_budget_tokens` is llama.cpp's +own end-of-thinking injection, so it holds whatever the template does. -LM Studio runs MLX on Apple Silicon. A Mac user who wants it installs LM Studio -and picks the preset. **Keep the Ollama preset** — it is the user-managed MLX -path now, not dead code. +Two rejected alternatives, both measured. `reasoning_budget` as a request field +is accepted and ignored, which is the failure mode worth naming because it looks +like it worked. `--reasoning-budget 0` on the command line applies to the whole +router, and worse, setting it at all makes the server ignore +`thinking_budget_tokens`, whose handler runs only while the flag is at its `-1` +default. The sidecar must never pass it, and a test in the Electron tree holds +that. + +### The provider -## Runtime contract +`LlamaCppProvider` satisfies the same `Generator` protocol the previous runtime +did, so resolution, selection and the catalog never learned that the runtime +changed. -| Ollama today | llama-server router | +| Ollama call | llama-server | |---|---| | `GET /` | `GET /health` | -| `GET /api/tags` | `GET /models` (+ `architecture.input_modalities`) | -| `POST /api/show` | `GET /props?model=…` — `chat_template_caps`, `modalities`, `n_ctx` | +| `GET /api/tags` | `GET /models`, plus `architecture.input_modalities` | +| `POST /api/show` | `GET /props?model=…`: `chat_template_caps`, `n_ctx` | | `POST /api/pull` | we fetch `resolve/main/{file}` ourselves | -| blob cleanup on cancel | **deleted** — `cleanup_cancelled_download()`, ~40 lines | -| `DELETE /api/delete` | `DELETE /models?model=…` | -| `POST /api/chat` | `POST /v1/chat/completions` — compose `OpenAICompatibleChatProvider` | - -Sidecar flags: `--models-dir /models --port --models-max 1 ---sleep-idle-seconds 300 --no-ui --jinja --reasoning-format deepseek`. - -Per-model at load, decided by the fit calculation rather than fixed here: -`-c `, `-fa on` and, when `q8_0` is chosen, `-ctk q8_0 -ctv q8_0`. -**Set both cache types or neither** — symmetric quantization enables the fused -flash-attention kernel, while a mismatched pair falls back to an unoptimised -path that exists for correctness only. - -> **All four of those flags ride the same unverified mechanism.** The ponytail -> in 7.3 records that `POST /models/load` ignored `{"args": ["-c", "16384"]}`. -> It is written up as a context problem because context is what was measured, -> but nothing distinguishes `-c` from `-fa` or `-ctk`/`-ctv`: they are the same -> field on the same call. If the router turns out to honour flags only at -> startup, then **KV precision cannot be per model either**, and the Decisions -> row above degrades from "an output of the fit calculation" to one global -> choice made for whichever model is largest. Resolve the mechanism once and -> both decisions follow; do not resolve it for context alone and assume the -> rest. - -`--sleep-idle-seconds` is not optional: measured at `b11050`, a model without it -self-evicted after roughly 30 s idle, which turns the second question of a -conversation into a reload. - -> **Never pass an explicit `-ngl`.** `--fit` owns layer placement, and setting -> `n_gpu_layers` by hand **disables it**: -> -> ``` -> common_fit_params: failed to fit params to free device memory: -> n_gpu_layers already set by user to 99, abort -> ``` -> -> Measured at `b11050`: with `-ngl 99` the model then loaded **entirely on the -> CPU** — 478 MiB of VRAM touched on a machine with a working RTX 3050, no error, -> exit 0. It looks like it worked. This is the same silent-GPU-blindness class as -> the probe search path (7.2) and the missing-backend case in **Failure -> behavior**, and it is the easiest of the three to introduce by accident, since -> `-ngl 99` reads as "use the GPU harder". - -`--no-ui` because llama-server ships its own web UI. `--reasoning-format deepseek` -routes `` blocks to `message.reasoning_content`; without it a thinking -model's trace enters `parts[]` and `resolve_citations()` rewrites `[n]` tokens -that appeared inside the reasoning. - -## Contracts - -> **Provenance.** The shapes and runtime calls below are grounded: the -> llama-server README at `b11043`, `nm` on the shipped libraries plus a live -> `ctypes` call, real GGUF headers read over HTTP Range, and live Hugging Face -> responses. **The HTTP routes below are proposed, not reported.** They follow -> this codebase's conventions: the `/llm` prefix, `*Read` Pydantic response -> models in `modules/llm/schemas.py`, and the NDJSON `{"type": …}` stream frame -> that `_event()` already emits. -> -> **Three of the six are not new paths, they are replacements at the same path.** -> `GET /llm/system`, `GET /llm/catalog` and `POST /llm/install` exist today, in -> `modules/llm/recommendations/router.py`, serving `RecommendationSystemRead`, -> `RecommendationCatalogRead` (with the `scanned` flag the renderer reads) and -> the same NDJSON install stream. So this is an **in-place breaking reshape, not -> an addition**: route, response schema and frontend have to move in one commit, -> because there is no additive path that does not serve two incompatible shapes -> at once. Only the two `/llm/search` routes and `DELETE /llm/models/…` are -> genuinely new surface. - -### Domain shapes +| blob cleanup on cancel | deleted, about 40 lines | +| `DELETE /api/delete` | unlink the file ourselves | +| `POST /api/chat` | `POST /v1/chat/completions` | +| nothing equivalent | `POST /tokenize` | + +Three of those deserve their reasoning written down. + +**Chat is composed, not reimplemented.** llama-server speaks OpenAI on +`/v1/chat/completions`, so the streaming, error handling and message shaping in +`OpenAICompatibleChatProvider` work against it unchanged. The one addition is a +fallback: [llama.cpp#29006](https://github.com/ggml-org/llama.cpp/issues/29006) +returns 400 for `json_schema` on some templates, and losing a whole Studio format +to a template quirk is worse than falling back to an unconstrained answer the +parser can still repair. + +**There is no `pull()`, deliberately.** Fetching weights from a name made sense +when the runtime owned the download. Here SurfSense fetches the GGUF itself, +because that is the only place `egress.require()` can hold. + +**Delete unlinks the file rather than calling `DELETE /models`.** The router only +removes what it downloaded into its own cache and refuses everything else: +measured, `model name=… is not removable (not from cache)`, a 500, with the file +left on disk. Everything SurfSense installs lands in `--models-dir`, so that call +can never succeed for us. + +`/props` is read once per load rather than once per message, cached by model id +and cleared only by `_ensure_loaded()` issuing a fresh load, which is the one +event that can make the old answer wrong. Before that cache, `context_tokens()` +and `chat()`'s own template shaping each issued an independent round trip on +every turn. + +## Reading a model + +### The parser + +`gguf==0.19.0`, llama.cpp's own `gguf-py` package, pinned exactly rather than +ranged. Two reasons it is upstream's parser and not one of ours: the search tier +parses bytes from arbitrary Hugging Face repositories, which is the one path +where a malformed file is adversarial rather than unlucky; and metadata key names +come from `gguf.constants.Keys`, so a rename upstream is an import error rather +than a field that silently reads zero. + +`scripts/curated/tensor_bytes.py` takes the same view for sizes and tensor names, +reading `GGML_QUANT_SIZES` and `TENSOR_NAMES` rather than restating either. + +### Reading a header out of a prefix + +`GGUFReader` memory-maps a **path** and its constructor always walks the tensor +table, slicing each tensor's data at an offset inside the weights. A prefix has +no weights, and numpy shortens a slice that runs past the end rather than +raising, so the shortfall would surface later as a confident wrong shape instead +of a retry. + +`modules/llm/gguf/header_prefix.py` is the adapter, and it overrides exactly two +private methods: + +- `_get` bounds-checks every read and raises `TruncatedHeaderError` when the + prefix is short. +- `_build_tensors` keeps names, dimensions and raw type ids without touching + data. + +`TruncatedHeaderError` subclasses `ValueError` on purpose. A caller that can +widen the read catches it by name and retries; a caller that merely needs to skip +an unusable file catches `ValueError` and gets both this and a file that was +never a GGUF. + +Because both methods are private upstream, the version is pinned exactly and +`test_header_prefix.py` asserts `gguf.__version__` beside the two behaviours, so +a bump cannot pass silently. + +The reader also closes its memory map explicitly. Windows refuses to unlink a +mapped file, and this runs once per searched model. + +### Where header bytes come from + +`modules/llm/gguf/source.py`. Hugging Face serves `Range` requests on model +files, so a model can be priced before a single weight is downloaded, which is +what puts a real fit badge on a search result. Verified: HF returns `206` with +`accept-ranges: bytes`. + +The widening is the point of the module. A 135M model's metadata ends at 1.77 MB; +Qwen3-Coder-30B-A3B's runs to 5.94 MB with 579 tensor entries after it. Header +size scales with **vocabulary**, not with model size, so no single prefix suits +every model: ```python -@dataclass(frozen=True) -class GgufArtifact: - """One downloadable build. `file` is required: a repo holds ~20 quants, and - repo + quantization does not name one (Q4_K_M vs UD-Q4_K_M vs a split set).""" - repo: str # "unsloth/Qwen3-8B-GGUF" - file: str # "Qwen3-8B-Q4_K_M.gguf" - quantization: str # "Q4_K_M" - size_bytes: int - mmproj: str | None = None # vision projector, fetched alongside - # A curated manifest variant is this plus judgement (`rank`, `rank_basis`, - # `validated`); a searched build is this alone. The runtime never sees the - # judgement half — see 7.4. +INITIAL_BYTES = 8 * 1024 * 1024 +WIDENED_BYTES = 24 * 1024 * 1024 +``` +A short read is a retry, once, and then a refusal. -@dataclass(frozen=True) -class ModelShape: - """Estimator inputs. Committed to the manifest for curated entries, read from - the header for everything else. Names mirror the GGUF metadata keys.""" - architecture: str # general.architecture - block_count: int # .block_count - head_count_kv: int # .attention.head_count_kv - key_length: int # .attention.key_length - value_length: int # .attention.value_length - context_length: int # .context_length - n_vocab: int - sliding_window: int = 0 # .attention.sliding_window - expert_count: int = 0 # .expert_count; > 0 means MoE +### `ModelShape` +Nineteen fields, named after the GGUF metadata keys they come from. +Quantization-independent: measured, the architecture fields are identical across +`Q4_K_M`, `Q8_0` and `f16` of the same model, so **one header read prices every +build of it**. Seven are required; the rest default to zero, and each one only +ever makes an estimate sharper, with absence being the conservative answer rather +than a zero cost. -@dataclass(frozen=True) -class HardwareBudget: - """One device, never a sum across devices. See 7.2 — device selection.""" - usable_vram_bytes: int # device_free − fit_reserve. NOT raw free: llama.cpp's - # own fitter refuses to use the last slice, measured - # at 1,028 MiB of 5,234 free on an RTX 3050. - total_device_bytes: int - fit_reserve_bytes: int # what was subtracted, carried so the badge can explain - ram_available_bytes: int # from OS APIs, not the ggml CPU device (see 7.2) - uma: bool # unified memory: selects badge copy and headroom - has_gpu: bool # False when --list-devices prints "(none)" +| Group | Fields | +|---|---| +| Required | `architecture`, `block_count`, `head_count_kv`, `key_length`, `value_length`, `context_length`, `n_vocab` | +| Graph width | `embedding_length`, `feed_forward_length`, `expert_feed_forward_length`, `expert_shared_feed_forward_length`, `expert_used_count`, `expert_count` | +| Sliding window | `sliding_window`, `sliding_window_pattern`, `sliding_window_layers`, `shared_kv_layers` | +| Latent attention | `kv_lora_rank`, `key_length_mla` | +Two reading rules in `shape.py` are worth knowing. A field that is scalar in most +models and per-layer in some is reduced with `_widest()`, because the widest +layer is what the cache must be sized for and taking the first element would +under-price a hybrid. And older headers that omit the per-head lengths have them +implied from `embedding_length // head_count`. -class FitState(StrEnum): - FITS = "fits" - PARTIAL = "partial" - TOO_BIG = "too_big" +## Reading the machine +### The probe runs out of process -@dataclass(frozen=True) -class FitVerdict: - state: FitState - need_bytes: int # weights + KV(window) + compute_buffers - budget_bytes: int # usable_vram, or usable_vram + ram for TOO_BIG - offload_fraction: float # 0.0 fully resident … 1.0 all on the CPU. - # The recommendation gates on this, and the - # badge's reason line is graded by it. Do not - # collapse it into `state` — three buckets lose - # the difference between 5% and 70% spilled. - approximate: bool = False # priced from file size alone, header not read yet +`probe_subprocess.probe_devices()` spawns a child, and the in-process path +survives only as a fallback. Three measured reasons, each independently +sufficient: +**The backend scan keys on the running executable's directory.** +`ggml_backend_load_all()` discovers backends by scanning the directory of the +running executable, not the directory the libraries were loaded from. Verified on +a Windows machine with a working RTX 3050, same minute, same process otherwise: -@dataclass(frozen=True) -class ContentPart: - kind: Literal["text", "image"] - text: str | None = None - path: str | None = None # local file path; image_url.url accepts one +```text +python wprobe.py → device count 0 card invisible +cd && python wprobe.py → device count 2 Vulkan0 + CPU ``` -`Capabilities` is defined in **7.7**. `Message.content` widens to -`str | tuple[ContentPart, ...]` there and nowhere else. +Loading the libraries by absolute path does not help, and `GGML_BACKEND_PATH` is +not an escape hatch because it expects a file: passing a directory logs +`load_backend: failed to load ` and leaves the count at zero. In process +this meant a process-global `chdir` under a lock, inside a server that is +handling other requests. In a child it is just the working directory. -### HTTP routes — proposed +**The first Metal device query compiles 20 shader libraries.** Measured on an M2: +`ggml_backend_load_all()` is 2.3 ms, and the compile happens inside the first +`ggml_backend_dev_count()`, at 19.0 s cold and 43 to 49 ms warm. A background +warm that calls `load_all()` and stops therefore warms nothing. The warm has to +enumerate the devices, which is what `CatalogService.warm()` does on a daemon +thread at lifespan start, so `/health` answers within a second of boot. -**Two catalog endpoints, deliberately.** Curated plus installed renders offline -and instantly; search needs `huggingface.co` and is paged. One response covering -both would either block on the network or return partial results behind a flag, -which is the `scanned` flag this phase deletes. +**A GPU driver that faults takes its process with it.** A child is a probe that +failed. The API process is the application. -| Route | Returns | Network | -|---|---|---| -| `GET /llm/system` | `HardwareBudget` plus device list | none | -| `GET /llm/catalog` | `{curated: [...], installed: [...]}`, every row badged | **none** | -| `GET /llm/search?q=&limit=&cursor=` | HF hits, coarse badge from file size | huggingface.co | -| `GET /llm/search/{repo:path}` | quant list with exact sizes, exact badge after the header read | huggingface.co | -| `POST /llm/install` | NDJSON progress, existing `_event()` frame shape | huggingface.co | -| `DELETE /llm/models/{model_id:path}` | `ModelDeleteRead`, `selection_cleared` | none | - -`GET /llm/search/{repo}` is where the 2–4 MB header read happens, so the trigger -is **opening a search result**, not hovering or typing. The list-level badge -stays `approximate` until then. - -**Installing a searched build needs an id the manifest cannot supply.** -`InstallRequest` keys on `catalog_id`, and [`../frontend/05-install-ux.md`](../frontend/05-install-ux.md) -requires that the renderer send *only* that — no repo, file, artifact URL, local -path or quantization. A curated row's id comes from the manifest; a search hit -has no manifest entry, so there is nothing to key on. Closing that by letting the -renderer post a repo and file would hand the frontend the ability to name an -arbitrary download, which is the exact capability the install contract exists to -withhold. +The child protocol is one tab-separated line per device on stdout: -So `GET /llm/search/{repo}` **mints a server-side install ticket per quant** and -returns it as that row's `catalog_id`, held with the same 300 s TTL as the search -cache. The install route then resolves tickets and manifest ids through one -lookup and cannot tell them apart. An expired ticket reuses the existing -`422 catalog id is stale or unknown; refresh the catalog` rather than inventing a -second staleness error, since it is the same failure and already has copy. +```text +namedescriptiontype_inttotal_bytesfree_bytes +Vulkan0 NVIDIA GeForce RTX 3050 1 6293553152 5487271936 +``` -Deleted with the Ollama adapter: `GET /llm/providers/{provider}/catalog` and -`POST /llm/providers/{provider}/pull`. Both are Ollama-shaped, and their -replacements are the rows above. +Deliberately plain, because ggml logs to stdout on some backends and the child's +output is shared with whatever it decides to print on the way past. A line that +does not parse is skipped rather than taking the listing with it, since it is far +more likely to be a log line than a device. A non-zero exit or a timeout (90 s, +bounding the Metal compile) logs a warning and falls back in process. -Unchanged: `GET /llm/providers`, `GET|PUT /llm/selection/{role}`, -`GET|POST /llm/onboarding`, and everything under `/llm/connections`. +The frozen binary re-executes itself behind `--probe-devices`, dispatched in +`main.py` beside the existing `--check-retrieval-runtime`. In development the +child is `python -m modules.llm.hardware.probe_script` with the backend root on +`PYTHONPATH`. The library directory is resolved before it is handed over, because +the child is started *in* that directory and a relative path would be read again +against it, find nothing, and fall back in process: the exact behaviour the +module exists to avoid, and silent, because the fallback answers. -### Identifiers +### Two library handles on Windows, one elsewhere -| | Value | +The exported symbols are split, and the intuitive single handle fails: + +| Library | Exports | |---|---| -| Registry key and `selected_models.provider` | `"llamacpp"` | -| Alembic revision for 7.6 | `0012` (head is `0011_document_cancelled_status`) | -| Egress destinations | `model_download`, `model_search` (both `huggingface.co`) | +| `ggml.dll` / `libggml.so` / `libggml.dylib` | `ggml_backend_load_all`, `dev_count`, `dev_get` | +| `ggml-base.*` | `dev_name`, `dev_description`, `dev_type`, `dev_memory` | -`"ollama"` appears at **115 lines across 24 non-test Python files**; each becomes -`"llamacpp"` or is deleted with the adapter. Counted across the whole tree it is -**421 lines in 66 files**, which is the figure 7.6 works from. +ELF and Mach-O resolve the second through their own dependency records, so one +`CDLL("libggml.so")` works, while `CDLL("libggml-base.so")` raises `undefined +symbol: ggml_backend_load_all`. **Windows needs both handles**: PE exports do not +chain, and `ggml.dll` alone raises `AttributeError: function +'ggml_backend_dev_name' not found`. A regression here is an `AttributeError` at +startup rather than a wrong answer, which is why it carries an explicit test. -### Out of scope +### Device selection -Local image generation is untouched. `modules/llm/router.py` carries four image -routes (`/image/local`, `/image/local/runtime`, `/image/local/{name}`, -`/image/local/{name}/install`) and `resolve_image_generation()` dispatches on -`sdcpp.PROVIDER` through `OpenAICompatibleImageProvider`. None of that changes, -and the sd-server sidecar keeps its own lifecycle. The only shared file is -`modules/llm/router.py`, which is edited around them. +A machine reports one entry per (backend, device) pair. Measured on the Windows +test machine: -Also unchanged: ingestion, the bge-small 384-dimensional index, retrieval, the -chat SSE protocol, Kokoro, and the OpenAI-compatible connection path. +```text +[0] Vulkan0 type=GPU 6002.0 MiB total, 5234.0 MiB free NVIDIA GeForce RTX 3050 +[1] Vulkan1 type=ACCEL 16198.3 MiB total AMD Radeon(TM) Graphics +[2] CPU type=CPU 31884.6 MiB total, 22750.2 MiB free AMD Ryzen 5 9600X +``` -### Constants: what v1 carries, and what it deliberately does not +`select_device()` takes the first device typed `GPU`, or `None`. -A constant here is a fixed number compiled into the fit path. Not user-facing, -not per-machine: decided once, written down, and wrong for everybody if wrong at -all. Earlier drafts of this phase needed five. **v1 carries one.** +- **First, not largest.** Sorting by memory picks the 16 GB integrated part over + the 6 GB discrete card and places every layer on the slower device. +- **Never a sum.** One device becomes the budget. The same card appearing twice + under two backends would otherwise turn a 6 GB card into 12 GB. +- **An integrated GPU is not a GPU for budgeting.** A part carving from system + RAM has no memory of its own to place layers in. It types as `IGPU`, and both + it and `ACCEL` are skipped by the `type is GPU` filter. + +`DeviceType` models all five ggml members plus an `UNKNOWN` sentinel, and +`parse()` maps an unrecognised integer to `UNKNOWN` rather than raising. That is +not defensiveness for its own sake: every Mac lists an Accelerate `BLAS` device +at raw type 3, so a three-member enum would raise `ValueError` at first paint on +a perfectly healthy machine. ggml has added a member twice. + +### Host memory does not come from ggml + +ggml's CPU device reports real available memory on native Windows and, so far, +nowhere else. Under WSL2 the figure is virtualised and `total == free`. On macOS +it is worse: measured on an M2 with roughly 2 GB genuinely free, the CPU device +reports 8192.0 MiB total against 8192.0 MiB free, physical RAM restated twice +with no live component at all. + +So `system_memory.available_bytes()` asks the OS. On Linux it parses +`MemAvailable` from `/proc/meminfo`; on macOS it runs `vm_stat` and sums free +plus inactive pages, which is what a large allocation can actually claim; and it +falls back to `sysconf` totals rather than to zero. `Device.reports_live_memory` +exists so a `free == total` reading is treated as "this device does not report +live memory" rather than as an idle machine. + +This matters because `ram_available_bytes` is what separates `PARTIAL` from +`TOO_BIG` on a discrete card. + +### Unified memory is one pool + +Apple Silicon reports two views of one memory: Metal states +`recommendedMaxWorkingSetSize` as both its total and its free, while the host +states the same physical memory again. Summed, an 8 GB Mac priced as though it +had 10.3 GB, and the refusal threshold came out at 10,581 MiB. -**Prediction constants are not runtime constants.** Almost everything below only -decides what the screen says. `--fit` does its own allocation at load and moves -layers if our sum was optimistic, so an inaccurate prediction produces a wrong -*label*, not a broken model. The single exception is the context floor, which is -passed to the runtime as `-c` and genuinely changes behaviour — which is why it -is derived from the codebase rather than chosen. +```python +UMA_HOST_FRACTION = 0.85 -#### In v1 +def unified_pool_bytes(working_set_bytes: int, host_bytes: int) -> int: + return min(working_set_bytes, int(UMA_HOST_FRACTION * host_bytes)) +``` -| Constant | Kind | Basis | -|---|---|---| -| **Context floor, 16K** | **runtime** | `HISTORY_BUDGET_TOKENS = 3000` plus ~8,000 tokens of grounding (~24k characters from `build_context`) plus a reply. Grounded, not chosen. | -| **Fit reserve** | **runtime** | **1024 MiB, and not ours.** It is llama.cpp's `fit_params_target` default, one GiB per device on every backend, read from source and observed on both Vulkan and Metal. Pass it as `-fitt` rather than predicting it. See **7.2**. | -| **Compute buffers** | prediction | **Measured: ~170 MiB** for a 4B at 16K. Part of `need`, scales with context and batch. | - -> **No longer a ponytail, and no longer a prediction.** The reserve began as an -> unmeasured placeholder at ~1,024 MiB. On an RTX 3050 `--fit` held back -> **1,028.34 MiB** of 5,234 free, which read as a lucky guess. It was not a guess -> and it was not lucky: `fit_params_target` defaults to `1024 * 1024*1024` per -> device in `common/common.h`, so the true figure is **exactly 1024 MiB** and the -> Windows excess was allocator rounding. Confirmed independently on Metal, where -> the fitter prints `cannot meet free memory target of 1024 MiB`. -> -> **So the generality worry was unfounded and the table it motivated is gone.** -> One point could not distinguish a constant from a proportion, which was a fair -> objection; the answer is that the source says constant. Hermes' equivalent is -> `max(2 GiB, 9% of total)` (`_MARGIN_FLOOR = 2 << 30`, `_MARGIN_FRACTION = 0.09` -> in `local_runtime/hardware.py`), roughly twice as conservative, and it is -> *their* number rather than the runtime's, which is the difference that matters: -> ours is the one the allocator will actually apply, and `-fitt` lets us set it. - -#### Not in v1, and why - -Each of these was in an earlier draft. Each was dropped because it would have -meant writing an invented number into a spec where it would be indistinguishable -from a measured one. That is the specific failure this phase exists to avoid: -llmfit called Qwen3 8B a "Perfect" fit for an 8 GB Mac because it was working -from a memory figure that was not true on that machine. +Two limits apply and the smaller governs. The working set is a ceiling Metal will +not allocate past, and it is exactly the figure llama.cpp's fitter subtracts its +own margin from, so discounting it again would refuse a model the runtime +demonstrably places. Host memory is what is actually there, and that is the leg +that lies: it is a snapshot, and loading weights takes seconds during which +another application can take memory. -| Constant | What it would enable | Why not now | What it needs first | -|---|---|---|---| -| `felt_cost` | a time estimate replacing v1's crude `MAX_OFFLOAD` ceiling | the roofline is specified under **Recommendation policy**; v1 ships the ceiling because it fixes the observed failure with one number | its constants, which come from completed turns rather than a benchmark — see **Where the constants come from** | -| **Speed floor** | a threshold in seconds rather than in spilled fraction | same | a `felt_cost` that means something, plus a threshold from real sessions rather than copied from an agent app | -| **Ladder rungs** | starting narrow so a **bigger model stays GPU-resident**, widening only when a conversation needs it. The benefit is memory, not long chats: Qwen3 4B on an 8 GB M2 is `FITS` at 16K and `PARTIAL` at 40K, purely from KV cache. | context is fixed at load | evidence that mid-conversation reloads are worth the complexity for document Q&A | -| **Growth threshold** | when to step up a rung | same | measured occupancy over real sessions | -| `decode_fraction` | correct speed prediction, and the only way an MoE is priced sanely | it is an **input to the speed model**, which v1 approximates with `MAX_OFFLOAD`. Compute and commit it now regardless: it is free from the header, and `1.0` for every dense entry shipped. | nothing — this one is ready | - -#### When they come back - -Each is additive and none requires a migration or a contract change. - -- **The speed gate** (`felt_cost` + floor) lands with the first real prefill - measurements. It slots into the recommendation policy as a filter over the - `FITS` set; the ordering and the badges are untouched. The reasoning about - *why* it must measure prefill and not decode is kept under the policy block so - it is not rediscovered. -- **The context ladder** (rungs + threshold) lands if fixed-at-load proves - limiting on large machines. It changes `POST /models/load` timing only. -- **`decode_fraction`** lands with the MoE rung, as a manifest field that - defaults to `1.0` for every dense entry already shipped. - -Shipping without them is not a gap to apologise for. Four of the five were -numbers nobody had measured, and a confident wrong badge is worse than a narrower -feature that tells the truth. - -## Phases - -Phases 7.0–7.5 keep Ollama registered. `REGISTRY` carries both adapters, so each -ships independently and is reversible. **7.6 is the only irreversible phase and -the only one that touches user data.** - -> **"Registered and working" is true of chat, not of catalog or install, and the -> difference is worth stating before someone plans around the stronger claim.** -> Two seams are llmfit-shaped and cannot serve both runtimes at once: -> `LocalRuntime.resolve(ScoredModel) -> InstallPlan` asks a runtime to map a -> *scored* model onto an install plan, and `LlamaCppProvider` has nothing to -> resolve because it installs a `GgufArtifact`; and `CatalogService.__init__` -> requires an advisor that 7.4 removes. So from 7.4 onward **Ollama's catalog and -> install path is dead even though its adapter is still registered**, and what -> survives to 7.6 is that an existing Ollama selection still answers, through -> `resolve_generation()` and the chat path. -> -> That is enough for the property the staging is for — no user loses chat -> mid-swap, and every phase before 7.6 reverts with one commit — but it is not -> "both runtimes fully working side by side", and trying to preserve that would -> mean keeping two catalog services alive for the sake of a runtime being -> deleted three phases later. - -### 7.0 — De-Ollama the domain - -No runtime change. `ScoredModel.ollama_name` → `artifacts: tuple[GgufArtifact, ...]` -where `GgufArtifact(repo, quantization, mmproj)`. `OllamaProvider` renders -`hf.co/{repo}:{quant}` itself, so the runtime's private vocabulary stops leaking -upward. - -Deleted: `_fallback_ollama_name()`, `clean_runtime_name()`, the `trusted_quant` -gate, the `FitLevel.MARGINAL` relabel, the `:latest` workaround. All five exist -only because Ollama's library was in the way. - -llmfit adapter switches to `recommend --force-runtime llamacpp --output-llamacpp --n --no-dashboard`. Bump `CACHE_VERSION`. - -**Tests:** `llmfit.py` fixtures. No UI change. - -> **Most of this phase cancels itself out, so do not ship it on its own.** The -> `CACHE_VERSION` bump is pointless because 7.4 deletes `CACHE_VERSION` and the -> scan cache; retargeting the llmfit adapter is pointless because 7.4 moves -> llmfit out of the app, leaving `recommendations/llmfit.py` with no consumer. -> What genuinely survives is the **domain change** at the top of this section — -> `ollama_name` becoming `artifacts`, and the five deletions — plus the llmfit -> *parsing*, which becomes the authoring script's. Fold the rest into 7.4 and -> land them together rather than writing code whose only reader is deleted two -> phases later. - -### 7.1 — GGUF header reader - -`modules/llm/gguf/header.py`, stdlib only. Local file or `Range: bytes=0-8388607` -against `huggingface.co`. Yields architecture, `block_count`, `context_length`, -`embedding_length`, `head_count_kv`, key/value lengths, `sliding_window`, -`expert_count`, `n_vocab`, chat template, exact tensor bytes. - -Verified: HF returns `206` with `accept-ranges: bytes`. A 135M model's metadata -ends at 1.77 MB and its tensor table at 1.79 MB, in 2.5 s. **Bigger models need -more:** Qwen3-Coder-30B-A3B's metadata alone runs to 5.94 MB, with 579 tensor -entries after it, so a 4 MB budget fails on it. Budget **8 MB** and retry wider -on a truncated parse. The vocabulary token list is what makes the metadata -large, so the figure scales with vocab size rather than with the model. - -**Tests:** a checked-in truncated GGUF; a fake HTTP server asserting the Range -header and a truncated-response retry. - -### 7.2 — Hardware budget and estimator - -`modules/llm/hardware/` — `ctypes` into `libggml`, then `--list-devices`, then -OS APIs. First hit wins, cached; warm the probe in the background at first -launch, not on the path of the first render: on macOS, 20 Metal shader libraries -are compiled on the first call, measured at **19.0 s once** and ~45 ms after. -That cold penalty is **Metal-only**: measured off Apple Silicon the equivalent -call is 77 ms with one backend and 205 ms with two. - -> **Warm the right call.** Earlier drafts attributed the cold cost to -> `ggml_backend_load_all()`. Measured on an M2, `load_all()` is **2.3 ms** and the -> compile happens inside the first **`ggml_backend_dev_count()`**, which is 19 s -> cold and 43 to 49 ms warm. A background warm that calls `load_all()` and stops -> therefore warms nothing, and the 19 s lands on whoever asks for the device list -> first, which is the first render. Enumerate the devices in the warm, not just -> load the backends. - -> **Folders, not two files.** Earlier drafts named `hardware.py` and `fit.py`. -> Between them they carry library loading, the working-directory dance, device -> enumeration, device selection, the `--list-devices` fallback parser, OS memory -> APIs, the OS-versus-ggml cross-check, budget assembly, calibration storage, the -> KV term, compute buffers, the reserve table, the need/usable comparison, KV -> precision selection, context sizing and badge copy. That is sixteen -> responsibilities in two files, against the repo's one-responsibility-per-file -> rule in `AGENTS.md`. Split them along those lines; the names above are the -> package, not the module. - -#### The probe must run from the library directory - -**This is the single most likely way this phase ships broken.** -`ggml_backend_load_all()` discovers backends by scanning **the directory of the -running executable**, not the directory the libraries were loaded from. The -backend probe runs inside the frozen `surfsense-api.exe`, which lives in -`resources/backend/`; the ggml libraries live in `resources/llamacpp/`. Different -directories, so the scan finds nothing. - -Verified on a Windows machine with a working RTX 3050, same minute, same process -otherwise: +The 0.85 earns that second job. Measured: spending the working set as though it +were live sized a 28,672 token window on an 8 GB Mac with 2.3 GB reclaimable, the +load paged, and it took 43 seconds, long enough for title generation to time out +at 30. -```text -python wprobe.py → device count 0 ← card invisible -cd && python wprobe.py → device count 2 ← CUDA0 + CPU -``` +A GPU sharing the CPU's description is the signal that the memory is shared. +Apple Silicon reports the same part name for both, which is cheap and reliable. -**Loading the libraries by absolute path is not sufficient.** The backend scan is -a separate step keyed on the host executable. Set the working directory around -`ggml_backend_load_all()`, or spawn the probe from the library directory. +### GPU status: four answers, not two -`GGML_BACKEND_PATH` is **not** the escape hatch — it expects a *file*, not a -directory. Passing one logs `load_backend: failed to load ` and leaves the -count at zero. +ggml answers an empty device list with exit 0 on two very different machines: a +laptop with no graphics card, and a workstation whose backend library did not +ship. Measured on Windows and Linux with a working card and `ggml-cuda.dll` +staged but no cudart beside it, `--list-devices` prints `(none)` and exits 0, and +`GGML_BACKEND_DEBUG=1` changes nothing. -The failure is silent and indistinguishable from a genuinely GPU-less machine: -every row badges *"Works here, runs on your processor"* and nothing reports an -error. See **Failure behavior**. +Badging the second as the first is the failure `gpu_status.py` exists to stop: +the user sees every model marked as running on the processor, with no suggestion +that the card they bought is idle because a file is missing. -#### Two library handles on Windows, one on Linux +| ggml sees a GPU | OS sees a GPU | Status | +|---|---|---| +| yes | either | `present` | +| no | yes | `broken_install` | +| no | no | `absent` | +| no | cannot say, a CPU device was listed | `absent` | +| no | cannot say, nothing was listed | `unknown` | -The exported symbols are split, and the intuitive single handle fails: +The OS half is `Win32_VideoController` through PowerShell on Windows, +`card*/device/vendor` under `/sys/class/drm` on Linux, and `True` on arm64 macOS. +Virtual adapters are filtered: the Windows test machine carried a `Parsec Virtual +Display Adapter` beside two real GPUs. -| Library | Exports | -|---|---| -| `ggml.dll` / `libggml.so` | `ggml_backend_load_all`, `dev_count`, `dev_get` | -| `ggml-base.dll` / `libggml-base.so` | `dev_name`, `dev_description`, `dev_type`, `dev_memory` | +`present` means ggml listed a GPU, which is not the same as the budget being +priced against one. An integrated part counts here and is still skipped by +`select_device`. The two questions are "is the runtime seeing the hardware" and +"what will hold the layers", and conflating them is how an AMD APU laptop would +end up badged as broken. -Linux resolves the second through the ELF dependency, so one -`CDLL("libggml.so")` works — but `CDLL("libggml-base.so")` raises `undefined -symbol: ggml_backend_load_all`. **Windows needs both handles**; PE exports do not -chain, and `ggml.dll` alone raises `AttributeError: function -'ggml_backend_dev_name' not found`. +The diagnosis travels beside the budget rather than inside it, on both +`GET /llm/system` and `GET /llm/catalog`. The budget is memory; this is a +diagnosis; and a broken install must not read as a machine that has no card. -#### Device selection: first `type == GPU`, never a sum +### Two budget modes -A machine reports one entry per *(backend, device)* pair, so the same physical -card can appear more than once and an integrated GPU can advertise more memory -than a discrete one. Measured: +The same subtraction answers two different questions. -```text -[0] Vulkan0 type=GPU 6002.0 MiB total, 5234.0 MiB free NVIDIA GeForce RTX 3050 -[1] Vulkan1 type=ACCEL 16198.3 MiB total AMD Radeon(TM) Graphics -[2] CPU type=CPU 31884.6 MiB total, 22750.2 MiB free AMD Ryzen 5 9600X -``` +**Capacity** prices a catalog, which is a shelf of models you might install +later and must not move with whatever the machine happens to be doing right now. +Host memory is `total - 2 GiB`, the reserve a host keeps for itself once a model +is resident. -```python -def select_device(devices): - """First GPU wins. ggml orders backends by preference, so this is also - backend selection.""" - return next((d for d in devices if d.type == DeviceType.GPU), None) -``` +**Live** decides a launch: this model, into this memory, in a moment. -- **First, not largest.** Sorting by memory picks the 16 GB integrated chip over - the 6 GB discrete card and places every layer on the slower device. -- **Never aggregate.** No `sum()` across devices. One device becomes - `HardwareBudget.usable_vram_bytes`. -- **An integrated GPU is not a GPU** for budgeting — a part carving from system - RAM has no memory of its own to place layers in. It types as `IGPU`, not - `ACCEL`; see below. - -> **The device type enum has five members, not three, and getting that wrong is -> a crash rather than a wrong answer.** Read from `ggml-backend.h` at `b11050`: -> -> ```c -> GGML_BACKEND_DEVICE_TYPE_CPU = 0 // system memory -> GGML_BACKEND_DEVICE_TYPE_GPU = 1 // dedicated memory -> GGML_BACKEND_DEVICE_TYPE_IGPU = 2 // integrated, host memory -> GGML_BACKEND_DEVICE_TYPE_ACCEL = 3 // BLAS, AMX -> GGML_BACKEND_DEVICE_TYPE_META = 4 // wraps several devices for tensor parallelism -> ``` -> -> Earlier drafts of this document, and the listing in -> [`08-cuda-backend.md`](08-cuda-backend.md), label the integrated Radeon on the -> Windows machine `ACCEL`. Its raw type is `2`, so it is **`IGPU`**. The selection -> *behaviour* is unaffected, since both are skipped by `type == GPU`, and the -> reasoning was right about the hardware. Only the name was wrong. -> -> **A three-member Python enum crashes on macOS.** Every Mac lists an Accelerate -> `BLAS` device, measured at raw type `3`, so `DeviceType(3)` raises `ValueError` -> on a machine that is otherwise perfectly healthy. This is the probe running at -> first paint, so it is a startup failure, not a degraded badge. Model all five, -> and treat an unknown integer as "not a GPU" rather than as an error, because -> ggml has added a member twice and will again. - -This rule is required here, under Vulkan alone. It becomes load-bearing if -[`08-cuda-backend.md`](08-cuda-backend.md) ever ships, because two loaded -backends list the same card twice, **both typed `GPU`**, so type filtering does -not deduplicate them. - -#### `ram_available_bytes` does not come from ggml reliably - -ggml's CPU device reports real available memory on native Windows (31884.6 MiB -total against 22750.2 MiB free, measured) and **nowhere else that has been -tried**. Under WSL2 the figure is virtualised and `total == free`. On macOS it is -worse: measured on an M2 with roughly 2 GB genuinely free, the CPU device reports -**8192.0 MiB total against 8192.0 MiB free** — physical RAM restated twice, with -no live component at all. The Metal device is barely better, at 5461.3 total -against 5461.0 free. - -So **native Windows is the exception, not the rule**. Keep the OS API in the -chain for the RAM half rather than trusting the CPU device, since -`ram_available_bytes` is what separates `PARTIAL` from `TOO_BIG`, and treat a -`free == total` reading as "this device does not report live memory" rather than -as an idle machine. - -Two budget modes. **Capacity** (total − margin) for catalog pricing; **live** -(free now) for launch decisions. Pricing against live-free while a model is -loaded makes every row read as too large. - -#### Two subtractions, on opposite sides of the comparison - -`modules/llm/fit/`. The single `overhead` term earlier drafts carried is **two -different quantities**, and collapsing them is what made the first prediction of -this wrong by 922 MiB: +Pricing the catalog against live free memory makes every large row read +`TOO_BIG` whenever a browser is open, which is a badge that lies in the direction +users notice. The catalog therefore defaults to capacity, and `plan_load` takes +both. + +One subtlety worth preserving. The 2 GiB reserve is only meaningful against +physical RAM. When nothing states the total, the live reading is a **floor** +rather than another figure to deduct from. Subtracting twice reported 0 GB on a +busy 8 GB machine and badged every model `Won't fit`. + +## The estimate + +### Three numbers, not two ```text -need = weights + KV(window) + compute_buffers what the model allocates -usable = device_free − fit_reserve what --fit will actually use -FITS iff need ≤ usable +need = weights + mmproj + KV(window) + compute_buffers +resident = what the model must fit inside to run at full speed +refusal = what physics allows at all ``` -Measured, Qwen3 4B Q4_K_M at 16K on an RTX 3050 with 5,234 MiB free: +The two subtractions sit on opposite sides of the comparison, and collapsing +them into one `overhead` term is what made the first prediction wrong by 922 MiB. +Measured, Qwen3 4B at 16K on an RTX 3050 with 5,234 MiB free: treating the +1,028 MiB the fitter holds back as part of `need` predicts `FITS` with 378 MiB +spare, while llama.cpp actually spilled 602 MiB of weights and 320 MiB of KV. +Same numbers, opposite verdict, purely from which side of the comparison the term +sits on. -| | MiB | -|---|---| -| `Vulkan0` model buffer | 2078.04 | -| `Vulkan0` KV buffer | 1984.00 | -| `Vulkan0` compute buffer | 143.62 | -| **placed on device** | **4205.66** | -| **left deliberately unused** | **1028.34** | +`resident` and `refusal` come from the budget rather than being assembled in the +estimator, because what they mean depends on the machine: -Treating that 1,028 MiB as part of `need` rather than as a subtraction from -available memory predicts `FITS` with 378 MiB spare. llama.cpp instead spilled -602 MiB of weights and 320 MiB of KV to the CPU — a `PARTIAL`. Same numbers, -opposite verdict, purely from which side of the comparison the term sits on. +```python +@property +def resident_bytes(self) -> int: + return self.usable_vram_bytes if self.has_gpu else self.ram_available_bytes + +@property +def refusal_bytes(self) -> int: + if not self.has_gpu: + return self.ram_available_bytes # nothing to spill from + if self.uma: + return self.ram_available_bytes # one memory, never a sum + return self.usable_vram_bytes + self.ram_available_bytes +``` -`compute_buffers` was absent from the formula entirely. It is ~170 MiB for a 4B -at 16K (143.62 device + 26.01 host) and scales with context and batch, so it is -not a rounding error on a 6 GB card. +where `usable_vram_bytes = device_free - fit_reserve`. -This replaces today's single `recommendation_reserve_gb: 2.0`, which was flagged -`ponytail` as uncalibrated. +Assembling these in the estimator is what badged every model on a CPU-only laptop +as spilling from a graphics card it does not have. With no GPU the two are the +same number, which makes `PARTIAL` unreachable there by construction rather than +by a special case. -#### The fit reserve is not a measurement. It is a flag we can set +On unified memory the refusal threshold is **host memory**, not the device's own +figure, because Metal's working set bounds what Metal will allocate rather than +what the machine can hold. Measured: a 1.7B at 40960 projected 6032 MiB against a +5460 MiB working set, and llama.cpp ran it with 23 of 29 layers on the CPU +backend, which reads the same chips without that ceiling. So `PARTIAL` stays +reachable on a Mac, in the band between the pool minus the fit reserve and host +memory. Refusing it would remove a real option from the users with the least +choice. -**Corrected by reading the source and the log, on an M2 / 8 GB at `b11050`.** -Earlier drafts of this section carried a `(platform, backend, memory model)` -table with one measured row and two blanks, on the reasoning that Metal, Vulkan -and CUDA reserve different amounts and none of it transfers. **That reasoning was -wrong.** The reserve is a single hardcoded default in llama.cpp's own fitter, -identical on every backend: +### The fit reserve is a flag, not a prediction ```cpp -// common/common.h — margin per device in bytes for fitting parameters to free memory +// common/common.h std::vector fit_params_target = std::vector(llama_max_devices(), 1024 * 1024*1024); ``` **1 GiB per device, platform independent.** The Windows figure of 1,028.34 MiB -was that 1024 MiB plus 4.34 MiB of allocator rounding, so the spec's original -"~1,024 MiB" guess was not lucky to within 4.3 MiB, it was **exactly the -constant**. Metal names it out loud under `-v`: +was that 1024 MiB plus allocator rounding. Metal names it out loud under `-v`: ```text common_params_fit_impl: projected to use 5752 MiB of device memory vs. 5460 MiB of free @@ -754,260 +697,336 @@ common_params_fit_impl: cannot meet free memory target of 1024 MiB, need to reduce device memory by 1315 MiB ``` -`5752 − (5460 − 1024) = 1316`. The comparison `need > free − 1024` is exact, on a -second backend, with no fitting. +`5752 - (5460 - 1024) = 1316`. The comparison is exact, on a second backend, with +no fitting. -**It is also a CLI flag: `-fitt` / `--fit-target`, in MiB, per device.** So the -reserve stops being a number to predict and becomes an input we control. Pass it -explicitly and `usable = device_free − fit_reserve` is true **by construction** -rather than by estimate, on every platform, including ones nobody has run. +It is also a CLI flag, `--fit-target`, in MiB per device. So the reserve stops +being a number to predict and becomes an input we control: we pass it, and +`usable = device_free - fit_reserve` is true by construction on every platform, +including ones nobody has run. `fit_target_mib()` is the one place that value is +computed, and it is the same function the preset writes and the badge subtracts. -| platform / backend | reserve | basis | -|---|---|---| -| any backend, any platform | **1024 MiB** | llama.cpp default, read from source at `b11050`; observed on Vulkan and Metal | -| whatever we pass to `-fitt` | that value | the flag exists; prefer pinning it to inheriting it | -| no GPU device | n/a | `PARTIAL` is unreachable; see **Fit states** | +### The four terms -**Three consequences.** The blank rows do not need measuring, and the plan to -measure them was work that could not have produced new information. The -self-calibration below is **no longer about the reserve**, only about the -compute-buffer half of the prediction. And pinning `-fitt` removes a class of -error rather than shrinking it, which is worth more than a better guess. +`itemise()` returns them separately rather than as a sum, for three reasons. A +sum is untestable, because it can be right for compensating wrong reasons. A test +can assert that the weights term does not move when the window does. And the +offload calculation needs to know which terms llama.cpp can actually move. -> **Errors here are still asymmetric, and the direction still matters** for the -> compute-buffer term that remains predicted. Too large predicts `PARTIAL` where -> reality is `FITS`: the user sees *Reduced speed*, installs anyway (`PARTIAL` -> installs exactly like `FITS`), and is not harmed. Too small predicts `FITS` -> where reality is `PARTIAL`, which is precisely the llmfit failure this phase -> exists to delete. *Unknown shapes round up; never underestimate memory.* +```python +@dataclass(frozen=True) +class NeedItems: + weights_bytes: int + mmproj_bytes: int + kv_bytes: int + compute_bytes: int +``` -#### What `--fit` actually does when it cannot fit +The projector is its own term because `--fit` does not count it, so a vision +model our sum calls resident can still fail to allocate, and because it is not a +thing the fitter can move. + +### The KV cache + +Architecture-derived and independent of how the weights were quantized, which is +what lets one header read price every build of a model. Three modules, one +question each. + +**How wide one layer's entry is.** Ordinarily +`head_count_kv x (key_length + value_length) x bytes_per_element`. For a latent +model (`kv_lora_rank > 0`) it is `kv_lora_rank + key_length_mla`, one entry per +token per layer rather than a key and a value per head. Priced by the ordinary +formula, a DeepSeek-class header reads enormously too large, because it reports a +single KV head while the model has a hundred and more. + +**Bytes per element** comes from the block layout, not from the bit width divided +by eight: `q8_0` stores 32 int8 weights plus one fp16 scale, so it is 34/32 bytes +per element and not 1.0. + +| | f32 | f16 / bf16 | q8_0 | q5_1 | q5_0 | q4_1 | q4_0 / iq4_nl | +|---|---|---|---|---|---|---|---| +| bytes/element | 4.0 | 2.0 | 34/32 | 24/32 | 22/32 | 20/32 | 18/32 | + +The load plan only ever chooses between `f16` and `q8_0`, and the preset writes +only those two. The rest are there because a searched model's own header can name +any of them and they have to be priceable. + +**How many cells a layer allocates.** Not the same as the token count. Every +cache is padded to 256 cells on every backend, and a sliding-window layer is +sized as `pad(min(n_ctx, n_swa + n_ubatch))`, because a batch being processed has +to sit in the cache beside the window it attends to. `n_ubatch` is +llama-server's default 512 and `n_seq_max` is 1, pinned by the `parallel = 1` +every preset writes. + +**Which layers hold the whole window.** `sliding_window.py` consults three +sources in the order llama.cpp itself does: a per-layer flag array in the header, +then a period in the header, then a per-architecture table of 14 entries read +from `load_swa_pattern` at `b11050`. An architecture nobody has verified is +deliberately **absent** from the table rather than guessed, and absence means +every layer is priced at full width, which over-states memory in the only +direction it is safe to be wrong in. + +Layers that allocate a cache are `block_count - shared_kv_layers`. Gemma 3n and +Gemma 4 reuse an earlier layer's cache on their last blocks, so charging those +layers a cache each over-states the window's cost on exactly the models chosen to +be cheap on a small machine. + +**The formula is exact, verified on both backends.** Qwen3 1.7B on Metal: +`2 x 28 layers x 8 kv_heads x 128 x 2 bytes x 16384` = 1792 MiB, reported as +`1792.00`, and 4480.00 at 40,960 cells. Qwen3 4B on Vulkan predicts 2304 MiB +against `1984 + 320` measured. + +### Compute buffers + +Absent from the formula entirely in the first draft, and then present as a +constant, which was nearly as wrong: scratch is sized by the widest thing one +layer computes, so it scales with the model rather than being a fee every model +pays alike. The constant was fitted to a 1.7B and under-stated a 4B by about +70 MiB, which is more than the margin that decides a 12 GB card's top row. -Measured, same session, Qwen3 1.7B at `-c 40960` against a 5,460 MiB working set. -It does **not** solve for a placement, it **searches** for one, reloading at each -step: +```python +activation_width = max(12 * n_embd, + 4 * n_ff, + n_used * (2 * n_embd + 3 * n_ff_exp) + 3 * n_ff_shared) -```text -29/29 layers → 0/29 → 29/29 → 22/29 → 23/29 (settles at 23/29, f ≈ 0.21) +flat = int((activation_width * 512 + n_vocab * min(512, 1)) * 4 * SAFETY) +total = flat + 5_120 * n_ctx ``` -Two behaviours that matter to us fall out of that log: +with `SAFETY = 1.20` and `SLOTS = 1`, matching the `parallel = 1` in every preset. + +The safety factor and the per-token slope are fitted, not derived, and the module +says so. Three measured points: Qwen3 1.7B on Metal reported 102.24 MiB at 16384 +and 222.24 MiB at 40960, and a 4B on Vulkan reported 143.62 device plus 26.01 +host at 16384. The slope is exact for the Metal pair, 120 MiB over 24,576 tokens. +The two backends disagree about the flat half by more than the widths explain, so +1.20 is fitted to cover both rather than to match either, and it over-states the +1.7B by about a third to do it. At 1.10 the 4B came out at 0.97 of what Vulkan +really allocated, and under-stating is the direction that ships a confident badge +about a model that spills. -- **It never touched the context.** KV stayed at 4,480 MiB / 40,960 cells through - every step. Because `-c` was set explicitly, the fitter treats it as fixed and - spills layers instead, which is the same "explicitly set means hands off" rule - that `-ngl` follows. Good: our context floor survives contact with the fitter. -- **If we do not set `-c`, it will reduce context on its own, as far as - `fit_params_min_ctx = 4096`.** That is well below our 16K floor and it happens - silently. So `-c` is not optional for us, and not only because we want a - particular window: leaving it unset hands the floor to llama.cpp. +A header that states no widths at all falls back to the old constant line, +23 MiB plus 5,120 bytes per token. Wrong in detail, but it is the measured floor +for the smallest model and certainly better than pricing scratch at nothing. The +manifest closes that door for curated entries by requiring `embedding_length` and +`feed_forward_length` to be present and positive. -Per-layer KV comes from the header; the formula is validated exactly in the -**Appendix**. +### Offload counted in layers, not bytes -> **Do not double-count.** ggml's `free` **already excludes** the desktop's -> allocation and the backend context. Measured on a 6144 MiB RTX 3050: 986 MiB -> was gone before a single weight loaded — ~445 MiB to the desktop, ~541 MiB to -> the CUDA context (`nvidia-smi` idle free 5699 MiB against ggml's 5158 MiB). -> Budget from `free` **and** add a ~1 GiB overhead term and the same memory is -> charged twice, demoting rows that fit. State explicitly which number `margin` -> and `overhead` are each subtracted from: against `free`, `margin` is ~0 and -> `overhead` covers per-model compute buffers only. +llama.cpp does not split a model by bytes. Its fitter fills devices with whole +layers, back to front, so the answer is a count of layers over the layer count: -> **ponytail:** the *per-model* half of the overhead constant is still a -> placeholder until measured against real RSS with a model loaded. The -> pre-model half is now measured at 986 MiB on a discrete card. On a 5.4 GB -> budget it moves rows across the fits/spills line. +```python +movable = items.weights_bytes + items.kv_bytes +per_layer = movable / shape.block_count +fraction = min(1.0, ceil(spilled / per_layer) / shape.block_count) +``` + +Only layers move. The projector is pinned by a flag rather than placed by the +fitter, and the compute buffer belongs to whichever device runs the graph, so +neither is part of what a layer costs. -#### The overhead constant is a bootstrap value, not a permanent guess +Measured on an RTX 3050, Qwen3 4B spilled 602 MiB of weights plus 320 MiB of +cache, which is 0.19 of the model. The byte ratio said 0.12 and this gives 5/36, +0.14. All three agree on the verdict and on the ordering, but the fraction grades +the reason line and gates the recommendation, so it is the fraction that has to +be right. -The shipped constant is a prediction for a machine nobody tested. **After one -successful load, that machine's real figure is known**, and the same rule the -rest of this phase runs on applies: anything that differs per machine is measured -on that machine by the runtime that will do the work. +### Fit states -The instrument is already in use — `ggml_backend_dev_memory()`, sampled either -side of `POST /models/load`: +There is no `unknown`. Every row has a file size, so every row has a badge. + +| State | Condition | At load | +|---|---|---| +| `FITS` | `need <= resident` | every layer on the device, full speed | +| `PARTIAL` | `resident < need <= refusal` | `--fit` places some layers on the CPU; runs, slower | +| `TOO_BIG` | floor `need > refusal` | physics refusal, the only state that blocks install | + +`TOO_BIG` is evaluated at `CONTEXT_FLOOR_TOKENS`, not at the requested window: a +model that will not fit at the floor cannot be rescued by a smaller context, and +the remedy to offer is a smaller build. + +There is one more `TOO_BIG` path, found by the property sweep rather than by a +fixture. When `need > refusal` at the *requested* window but the floor fits, the +window is refused rather than described as a partial offload. It is reachable +only where residency and refusal are the same number, which means a machine with +no GPU, where the model is already entirely on the processor and calling it +partly offloaded would name a device that is not there. + +`offload_fraction` is kept on the verdict and never collapsed into the state. +`PARTIAL` spans everything from barely noticeable to unusable, and the number is +already known from the same subtraction that produced the state. + +`CONTEXT_FLOOR_TOKENS` is **8192**, not llama.cpp's own 4096 reduction floor. +4096 is too short once a system prompt, retrieved excerpts and a reply share the +window, so the fit search would settle there rather than spill weights it could +spill instead. What makes 8192 safe rather than merely smaller is +`modules/chat/budget`, which sizes every part of a prompt from `n_ctx`; see +**The context budget**. + +### Speed tiers: one classification + +A badge and a recommendation both start from the same `FitVerdict`, and both used +to threshold its raw `offload_fraction` independently. The badge drew lines at a +quarter and a half to choose wording; the recommendation drew a separate line at +three quarters to choose whether to star a build. Nothing connected the two, so a +build could be starred while its own badge read *"Too big for the GPU, so part +runs on the CPU"* or worse. + +`fit/speed.py` is now the only module that thresholds that number: ```python -real_overhead = (free_before - free_after) - weights_bytes - kv_bytes(n_ctx) +TOO_BIG state is TOO_BIG +HEAVY_SPILL f >= 0.5 +MODERATE_SPILL 0.25 < f < 0.5 +LIGHT_SPILL f <= 0.25 +FULL state is FITS + +RECOMMENDABLE_TIERS = {FULL, LIGHT_SPILL} ``` -This residual **already contains the compute buffers**, which is why they need no -separate per-model table: the first load measures everything that is neither -weights nor KV, on the machine that will run it. +Collapsing both callers into one ordered tier removes the second line rather than +moving it. A recommended build can now only ever carry *"Full speed"* or *"Most +of it still fits"* wording, by construction rather than by two independently +tuned numbers staying in step. +`test_badge_matches_the_load.py` holds it with a property test sweeping every +curated model against every budget shape. + +### Badges + +A badge is a verdict plus one plain line of why. The verdict is what someone +choosing a model needs (fast, slower, or impossible); the mechanism is the +explanation, not the headline. Wording branches on two facts the budget already +carries, `uma` and `has_gpu`, and never on the runtime's name. + +**Discrete GPU** ```text -first run predict with the shipped constant -after first load record the machine's real overhead, keyed by device name -every run after prefer the recorded value +● Full speed Runs entirely on your graphics card +◐ Reduced speed A little too big for the graphics card. Most of it still fits. +◐ Reduced speed Too big for the graphics card, so part runs on the processor. +◐ Reduced speed Well over the graphics card's memory. Expect it to be slow. +○ Won't fit Needs about 21 GB. This PC has 13.6 GB ``` -Three reasons this is worth ~15 lines in the load path: - -- **The constant is wrong at most once per machine**, on a screen shown before - anything is installed, instead of wrong forever on every machine that was not - one of the three targets. -- **It self-corrects per backend and per driver.** A CUDA context alone costs - ~541 MiB (measured); Metal and Vulkan differ; a driver update changes it. A - committed table cannot track that and no longer has to. -- **It retires the last `ponytail` in the fit path.** Nobody repeats the - measurement session after the first release. - -Storage is a small JSON file in the app data dir keyed by device name — no -migration, and no settings table needs inventing for it. Invalidate on a -llama.cpp build bump or when the device name changes. - -**Tests:** a recorded value is preferred over the constant; a recorded value from -a different device name or build is ignored; a corrupt or missing file falls back -to the constant rather than raising, since this sits on the first-render path. - -**Tests:** decision-table over constructed profiles — no GGUF parsing in the fit -tests, so the estimator is testable independently of the reader. - -### 7.3 — `LlamaCppProvider` - -Registered alongside Ollama. `REGISTRY` already takes two; `CatalogService` -already loops `self._runtimes`. This is where 05a's acceptance criterion — -"adding a fake llama.cpp runtime requires no change to llmfit parsing, curated -partitioning, or the frontend catalog schema" — becomes a real test. - -Chat composes `OpenAICompatibleChatProvider` rather than reimplementing -streaming. Install fetches the GGUF into `--models-dir` with real filenames, not -llama.cpp's content-addressed HF cache layout (`LLAMA_CACHE` points at the app -data dir so nothing writes to `~/.cache`). - -> **Resolved: `POST /models/load` cannot set per-model args. `--models-preset` -> can.** Measured on macOS at `b11050`. Every payload shape is accepted and none -> of them does anything: `{"args": ["-c","4096"]}`, `{"args": -> ["--ctx-size","4096","-fa","on"]}` and `{"preset": "..."}` all return -> `200 {"success":true}` while the worker's argv stays **byte identical**. The -> router reports that argv under `GET /models` as `status.args`, which is the -> only reliable way to see what it actually did. -> -> The mechanism is a separate flag, `--models-preset PATH`, pointing at an INI -> file with one section per model: -> -> ```ini -> [Qwen3-8B-Q4_K_M] -> model = /path/to/Qwen3-8B-Q4_K_M.gguf -> ctx-size = 16384 -> flash-attn = on -> cache-type-k = q8_0 -> cache-type-v = q8_0 -> parallel = 1 -> ``` -> -> Verified: the worker then launches with `--ctx-size 16384 --cache-type-k q8_0 -> --cache-type-v q8_0 --flash-attn on --parallel 1`, and reports the requested -> `n_ctx_slot` rather than its own default. **So every per-model decision in this -> phase is implementable**, including the KV precision row in **Decisions**, -> which rides the same mechanism as the context window. -> -> **The INI is read once at router startup.** Appending a section while the -> router runs does not surface the model, so installing a model or changing its -> load plan means rewriting the file and restarting the sidecar. That is the -> lifecycle `watchImageModel()` already implements for sd-server in -> `electron/src/main/index.ts`: poll a small endpoint, compare the arg set, then -> `stopNamed` plus `startOne` on a change. Reuse it rather than inventing a -> second supervisor pattern. -> -> Also measured and worth passing: llama-server defaults to `--parallel 4`, so -> the KV cache is sized for four concurrent slots this app never uses. - -**Context is fixed at load, not grown.** `POST /models/load {"args": ["-c", N]}` -with `N` the largest window that keeps the verdict at **`FITS`** — not the -largest that fits at all. KV cache is allocated upfront and competes with the -weights for device memory, so maximising context silently demotes models out of -GPU residency: Qwen3 4B on an 8 GB M2 is `FITS` at 16K (4,632 MiB) and `PARTIAL` -at its native 40K (6,468 MiB). Floored at 16K (3,000 -history tokens plus ~8,000 of grounding plus a reply) and capped at the model's -own `context_length`. No ladder, no reload mid-conversation. Growing on -occupancy is additive later; it would mean reloading the model between turns and -two constants copied from Hermes with no evidence they suit RAG. - -**The router costs no device memory until a model loads** (measured: VRAM -unchanged at the idle baseline with the router up and three models discovered), -and it **auto-discovers** everything in `--models-dir` as `status: "unloaded"`, -spawning one worker per loaded model with `--port 0 --model `. Pass -`--sleep-idle-seconds` deliberately: at `b11050` a model self-evicted after -roughly 30 s idle without it, which would turn the second question of a -conversation into a reload. - -**Router mode runs with an empty models directory**, so the sidecar lifecycle is -testable before any model exists. Verified on Windows and Linux at `b11050`: -`GET /health` → `{"status":"ok"}`, `GET /models` → `{"data":[],"object":"list"}`, -and `GET /props` → `"role":"router"` — which is the reliable check that the -sidecar came up in router mode rather than single-model mode. SIGTERM shuts it -down cleanly (`cleaning up before exit`), and on Windows `taskkill /PID /T -/F` walks the tree and reports each child terminated. - -**Grandchild reaping verified**, with a model resident, on Windows at `b11050`: +**Apple Silicon** swaps "the graphics card" for "the GPU", "the processor" for +"the CPU", and "This PC" for "This Mac". -```text -router 132 llama-server.exe - child 10828 conhost.exe - child 8820 llama-server.exe ← the model worker - grandchild 18688 conhost.exe +**No GPU** collapses to two states, because `PARTIAL` is unreachable: -taskkill /PID 132 /T /F → all four terminated, depth first -SURVIVORS: none +```text +● Works here Runs on your processor +○ Won't fit Needs about 21 GB. This PC has 16 GB ``` -> **Windows only.** macOS and Linux reap through process groups and signals, not -> `taskkill /T`, and the macOS hardened runtime case (7.5) is still untested -> against a grandchild. +Three wording decisions that look small and are not. + +**"Full speed", not "Fast".** Fast is a promise the badge cannot keep: a 32B +running entirely on a 4090 is still slower than a 4B. *Full speed* is relative to +the model, which is exactly what the state means. + +**"Works here" on a machine with no GPU.** Technically that case is `FITS`, but +*Full speed* reads as a boast about a slow situation when there is no faster +alternative to contrast with. The useful information is simply: yes, you can run +this. -**Tests:** a fake llama-server (aiohttp) covering health, models, props, load, -chat stream, delete; install against a fake HF serving a real small GGUF. +**Sizes are decimal GB with one decimal, trailing zero dropped**, so a pair reads +"21 GB" and "13.6 GB" rather than "21.0 GB" and "14 GB". Rounding the second to a +whole number loses the half gigabyte that decided the answer. -### 7.4 — Catalog: curated + search +> **Copy rule for every user-facing string in this phase: no em dashes and no +> hyphens.** Commas, full stops or parentheses. Applies to badges, empty states, +> error copy and the recommendation line. -**`curated-models.json` stays.** It is the only tier that works with no network, -so it is the entire product airgapped, the default on first run, and the only -thing the recommendation can read. It ships frozen in the build; there is no -refresh path. +### The load plan -Manifest `schema_version` **3**. One entry, before and after: +`plan_load()` decides the window and the cache precision once, because context is +fixed at load and cannot be renegotiated mid-conversation. -```jsonc -// v2 — today -{ - "model_id": "Qwen/Qwen3-8B", - "family": "Qwen3", - "minimum_fit": "good", - "minimum_context": 8192, - "allowed_quantizations": ["Q4_K_M"], - "artifacts": { "ollama": { "name": "qwen3:8b", "quantization": "Q4_K_M" } }, - "label": "Qwen3 8B", - "parameter_count": "8B", - "size_bytes": 5225374496 -} +```python +ceiling = shape.context_length or CONTEXT_FLOOR_TOKENS +floor = min(CONTEXT_FLOOR_TOKENS, ceiling) +CONTEXT_RUNGS = (8192, 16384, 32768) +``` + +**Prefer residency over window.** KV is allocated upfront and competes with the +weights, so maximising context silently demotes a model out of GPU residency. +The search widens only while the verdict stays `FITS`. + +**Prefer `f16` over `q8_0`.** Quality is identical, but a quantized cache +requires a working flash-attention kernel, and where one is unavailable llama.cpp +falls back to CPU attention silently. `resident_precision()` returns the cheapest +cache that keeps the build resident, or `None` when none does, in which case the +plan holds the floor at `f16`: a cheaper cache buys nothing once layers are +spilling anyway, so it would take the flash-attention dependency for no gain. + +`planned_precision()` is the same rule with a fallback, and it is what +`rows.py`, `recommendation.py` and `service.repo()` all call. Before that shared +rule existed, the badge priced `f16` while the loader chose `q8_0`, so a row read +*Reduced speed* for a model the runtime then placed entirely on the device. + +**A fixed set of rungs rather than a continuous search**, so the whole surface of +windows a person can see is something a test can enumerate. The model's own +trained length is tried too when it falls between two rungs, so a model is not +held below what it was actually trained for, and the floor is always a candidate +so the search never returns nothing. + +**Two budgets, and widening stays inside the tighter one.** `budget` is capacity +and decides the verdict, so the plan agrees with the badge the catalog already +showed: a machine that is busy this second has not become one that cannot run the +model. `live` caps widening only, because context past the floor is +opportunistic. A cap must not raise a ceiling, so widening takes whichever budget +has less headroom. Widening against a more generous `live` and then reporting +against `budget` produced a plan that said `PARTIAL` for a row the catalog had +badged `FITS`. + +## The catalog + +### Two tiers + +| | Curated | Search | +|---|---|---| +| Source | `curated-models.json`, shipped frozen | `huggingface.co` | +| Network | none, ever | required, egress gated | +| Priced from | committed `shape`, on first paint | file size, then the header on open | +| Carries a `rank` | yes, never displayed | no | +| Can be recommended | yes | no | + +Two endpoints, deliberately. Curated plus installed renders offline and +instantly; search is paged and needs the network. One response covering both +would either block on the network or return partial results behind a flag, and +that flag is the `scanned` flag this phase deleted. + +### The manifest, schema 4 + +`curated-models.json` is **source, not build output.** A person runs the +authoring script, reads what it proposes, and commits the result. A rank moving +78 to 94 shows up in a pull request where someone notices; a tag rebuilds to the +same manifest forever; and the cadence is honest, since these change when someone +adds a model rather than when someone cuts a release. -// v3 +```jsonc { "model_id": "Qwen/Qwen3-8B", "family": "Qwen3", "label": "Qwen3 8B", "parameter_count": "8B", - // ── DERIVED, per model. The script writes all of this, no human input ever. - // Quantization-independent: measured, the architecture fields are - // identical across Q4_K_M, Q8_0 and f16 of the same model. + // DERIVED, per model. The script writes all of it, no human input ever. "shape": { - "architecture": "qwen3", - "block_count": 36, - "head_count_kv": 8, - "key_length": 128, - "value_length": 128, - "context_length": 40960, - "n_vocab": 151936 + "architecture": "qwen3", "block_count": 36, + "head_count_kv": 8, "key_length": 128, "value_length": 128, + "context_length": 40960, "n_vocab": 151936, + "embedding_length": 4096, "feed_forward_length": 12288, + "sliding_window": 0, "expert_count": 0, + "expert_feed_forward_length": 0, "expert_shared_feed_forward_length": 0, + "expert_used_count": 0, "sliding_window_pattern": 0, + "sliding_window_layers": [], "shared_kv_layers": 0, + "kv_lora_rank": 0, "key_length_mla": 0 }, - "capabilities": [], // user-facing only. Today: "vision" or nothing. - "decode_fraction": 1.0, // 1.0 dense; the active slice for MoE, computed - // from the header (see below) + "capabilities": [], // user facing only: "vision" or nothing + "decode_fraction": 1.0, // 1.0 dense; the active slice for a mixture of experts - // ── One entry per shipped build. v1 ships exactly one. "variants": [ { // DERIVED @@ -1018,10 +1037,9 @@ Manifest `schema_version` **3**. One entry, before and after: "mmproj": null, // JUDGEMENT. Three fields, typed by a person, beside the build they judge. - "rank": 78, // preference order for document Q&A. - // Not a benchmark. Never displayed. - "rank_basis": "llmfit-1.1.11", // what produced it - "validated": true // someone ran THIS file + "rank": 78, + "rank_basis": "llmfit-1.1.11", + "validated": false } ] } @@ -1030,1214 +1048,745 @@ Manifest `schema_version` **3**. One entry, before and after: **The split is the point.** Everything marked DERIVED comes free from the GGUF header and the Hugging Face listing; everything marked JUDGEMENT is three small decisions a person makes in a minute. That is what keeps the manifest -maintainable at twenty entries instead of six — the tedious half scales -automatically, the half that needs a brain stays tiny. - -**Why `rank` lives inside the variant.** Quality is a function of *(model, -quantization)*, not of the model alone: llmfit returns 75 · 78 · 81 · 82 · 83 for -Qwen3 8B at Q3_K_M · Q4_K_M · Q5_K_M · Q6_K · Q8_0 (see **Appendix**). A rank at -the top level therefore describes a build without naming it, and that is not -hypothetical — an earlier draft of this document carried five ranks taken at -Q8_0 while the manifest pinned Q4_K_M, and nothing in the schema could catch it. -Nested, the mismatch is **impossible to express**: the rank sits beside the -quantization it was measured at, and `test_curated_models.py` asserts the pair -mechanically instead of a person remembering a rule. - -`rank_basis` records what produced the number — `llmfit-` today, an -internal eval identifier later. It exists so a manifest part-way through that -migration is detectable, and so ranks from different bases are never compared. - -**`variants` is a list from the start, and v1 puts one thing in it.** Authoring -is unchanged: one pin, one rank, the same twenty minutes. The list costs nothing -now and is the only part of this schema that is expensive to add later, because -changing it means `schema_version: 4` and re-authoring every entry. It also -closes the gap named first under **The quantization ladder**: with two variants a -24 GB card and a 64 GB machine stop receiving the identical file. The ladder -sweep in **Authoring** already computes a rank for every rung and discards four -of five; this is where the rest would go. +maintainable at twenty entries instead of six. **`shape` is what makes the curated tier work offline.** Pricing a model means -reading its GGUF header, and a curated entry is not downloaded yet — so without -these fields an airgapped machine has no fit badge on the one tier it can use. -Committing what the header said at authoring time is what lets `weights + -KV(16K) + compute_buffers` run against the manifest alone, on first paint, with no -network. It is the same reason Hermes carries estimator inputs inline. - -`decode_fraction` is the fraction of the build's bytes read per decoded token: -`1.0` for a dense model, the active slice for MoE. **It is computed, not typed.** -The header carries `.expert_count` and `.expert_used_count` (verified: -128 and 8 on Qwen3-Coder-30B-A3B), and expert tensors are identifiable in the -tensor table the reader already parses — `blk.N.ffn_{down,gate,up}_exps.weight`, -144 of that model's 579 tensors. So: - -``` -decode_fraction = (non_expert_bytes + expert_bytes × used_count / expert_count) - ÷ total_bytes -``` - -Sanity check: Qwen3-Coder-**30B-A3B** is 30B total against 3B active, so the -formula should land near 0.1, which is the range Hermes hand-authored (0.08 to -0.15) for comparable models. Without it an MoE entry's speed -prediction is wrong by an order of magnitude, which matters the moment the -80B-A3B-class rung lands. - -`decode_fraction` stays **per model, not per variant**: it is an expert-to-total -byte ratio, so it barely moves with quantization. `shape` and `capabilities` are -per-model for the same reason — the architecture fields are identical across -quantizations, measured. - -`validated: true` means a person downloaded **that variant's** file and ran it. -False is allowed and honest; it is not a gate. Per-variant is the only place it -means anything precise once an entry ships more than one build. - -Dropped from v2: `artifacts` (replaced by `variants`, which can hold more than -one), `minimum_fit` (a gate — only physics gates now), `allowed_quantizations` -(redundant once a variant names its file), `minimum_context` (the model's own -`context_length` replaces it), and top-level `advisor_providers` (scoped an -llmfit scan that no longer runs). - -#### Authoring - -A script run by hand, **never CI**. The manifest is **source, not build output**. - -```bash -$ uv run scripts/refresh_curated_models.py \ - --add Qwen/Qwen3-Next-80B-A3B \ - --repo unsloth/Qwen3-Next-80B-A3B-GGUF --quant Q4_K_M +reading its GGUF header, and a curated entry is not downloaded yet, so without +these fields an airgapped machine would have no fit badge on the one tier it can +use. + +**`embedding_length` and `feed_forward_length` are required**, unlike their +optional counterparts on `ModelShape`. A committed entry is authored by a script +that read a real header, so a missing width here means a manifest written by hand +or by an older script, and the compute buffer would quietly price it as though +the model had no layers to compute. The app refuses to start on that rather than +shipping a confident wrong badge. + +**`rank` lives inside the variant** because quality is a function of (model, +quantization), not of the model alone: llmfit returns 75, 78, 81, 82, 83 for +Qwen3 8B at Q3_K_M through Q8_0. A rank at the top level describes a build +without naming it, and that is not hypothetical, since an earlier draft carried +five ranks taken at Q8_0 while the manifest pinned Q4_K_M. Nested, the mismatch +is impossible to express. + +`validate_manifest` rejects a wrong `schema_version`, a duplicate `model_id`, a +duplicate `(repo, file)` pin, and **more than one distinct `rank_basis`**, so +ranks from two scales are never silently sorted together. + +The shipped six, all Qwen3, all `Q4_K_M`, `rank_basis` `llmfit-1.1.11`: + +| model | file | rank | +|---|---|---| +| Qwen3 0.6B | 396,705,472 B | 33 | +| Qwen3 1.7B | 1,107,409,472 B | 48 | +| Qwen3 4B | 2,497,281,312 B | 63 | +| Qwen3 8B | 5,027,784,512 B | 78 | +| Qwen3 14B | 9,001,753,984 B | 85 | +| Qwen3 32B | 19,762,150,048 B | 92 | - resolving unsloth/Qwen3-Next-80B-A3B-GGUF … Q4_K_M found, 45.2 GB - reading GGUF header (8 MB range) … qwen3next, 48 layers, 2 kv-heads, ctx 262144 - llmfit … sweeping budgets for the quantization ladder - Q3_K_M 88 · Q4_K_M 94 · Q5_K_M 96 · Q6_K 97 · Q8_0 97 - pinned quant Q4_K_M found in ladder ✓ - llmfit proposes rank 94 (would be highest in the list — currently 92) - writing variants[0]: Q4_K_M, rank 94, basis llmfit-1.1.11 +### Authoring - MoE detected: 128 experts, 8 used → decode_fraction 0.09 - ⚠ validated=false until you run this file. +`scripts/refresh_curated_models.py`, run by hand, **never in CI**. The three +judgement fields live in an `ENTRIES` table in the script where a person edits +them; everything else is derived from the live listing and a real header read. - wrote curated-models.json (+1 entry) -``` +`scripts/curated/` holds the derivation: `huggingface.py` resolves a repo and +file and reads the header, `projector.py` finds an `mmproj-*.gguf` sibling and +prefers F16 over BF16 over Q8_0, and `tensor_bytes.py` computes +`decode_fraction`. -The person then downloads it, chats with it, confirms citations resolve, sets -`variants[0].validated: true`, and commits. Twenty minutes, most of it -waiting for the download. +`decode_fraction` is the fraction of a build's bytes read per decoded token, 1.0 +for a dense model and the active slice for a mixture of experts. It is computed, +not typed: ```text -every build - read the committed JSON. No llmfit, no network, no subprocess. +decode_fraction = (non_expert_bytes + expert_bytes x used / experts) / total_bytes ``` -**What triggers a refresh** — events, not a schedule; realistically four or five -a year: +Both halves of that come from llama.cpp rather than from conventions restated in +our tree: sizes from `GGML_QUANT_SIZES`, and the expert tensor set from +`TENSOR_NAMES`, matching the same seven members llama.cpp's own fitter matches +with `blk.\d+.ffn_(up|down|gate_up|gate)_(ch|)exps`. Deliberately not +`ffn_norm_exps`: a per-expert norm is a vector beside these matrices, and the set +that decides what llama.cpp moves to the host is the set that decides what it +reads from the host. + +Every shipped entry is dense and carries `1.0`. It is computed now anyway because +it is free from the header, and because without it a mixture-of-experts entry's +speed prediction would be wrong by roughly an order of magnitude the moment such +a rung lands. + +**What triggers a refresh**, realistically four or five times a year: a model +worth recommending ships; a quantizer deletes or re-uploads a pinned file so +users hit a 404; the pinned llama.cpp build is bumped and pins want reverifying; +a hardware class turns out to be under-served; or someone reports a bad +recommendation. -| Trigger | Cadence | -|---|---| -| A model worth recommending ships (a new family, a good MoE, a vision entry) | the main one | -| A quantizer deletes or re-uploads a pinned file, so users hit a 404 | reactive | -| The pinned llama.cpp build is bumped and pins want re-verifying | on bump | -| A hardware class is under-served (see **The quantization ladder**) | occasional | -| Someone reports a bad recommendation | reactive | - -#### Scoring at the pinned quantization - -**llmfit has no flag for this, so the script has to work for it.** There is no -way to ask for a score at a named quantization: `recommend` and `info` grade -whatever `best_quant` the detected hardware selects, and `plan --quant` accepts a -quantization but returns only `weight_gb`, `kv_cache_gb`, `total_vram_gb`, -`estimated_tps` and `recommended_gpu` — **no quality field**. Verified on -`1.1.11`. - -That matters because quality tracks `best_quant` exactly (see **Appendix**). -Score at the wrong budget and you have graded a different file than the one in -`artifacts`. Qwen3 8B is 78 at the pinned Q4_K_M and 83 at Q8_0; run the script -on a workstation and you commit 83 for a file that behaves like 78. - -**The procedure: build the model's ladder, then index into it.** Do not search -for a single budget. Sweep the device budget across the model's whole -quantization ladder, record the `(best_quant → quality)` pair at each step, and -look up the pinned quantization in the resulting map. +### Ranking -``` -for budget in multiplicative_steps(lower, upper, ratio=1.01): - row = llmfit recommend --force-runtime llamacpp --no-dashboard \ - --memory {budget} --ram 128G -n 20000 - ladder[row.best_quant] = row.score_components.quality +`rank` does two things and nothing else: it orders the curated rows, and it +selects the star. It is never displayed, never compared outside the app, and +never applied to a searched model. Because it is only ever a sort key it does not +need to be a measurement, it needs to be an **order**. Hence `rank`, not +`quality`: calling it quality would imply we measured something we did not. -variant.rank = ladder[pinned_quant] # KeyError is a hard failure -variant.rank_basis = f"llmfit-{llmfit_version}" -``` +**It means "good at this app's job"**, answering from the user's documents with +citations that resolve, not general capability. That distinction does real work: +llmfit rates Qwen2.5-Coder 7B at 89 against Qwen3 8B's 83, because it is +measuring coding ability, and for document Q&A the coder is the weaker model. -Because the rank is written **into the variant**, it is stored beside the -quantization it was read at and cannot drift from it. The `best_quant` assertion -below is what catches a bad *read*; the schema is what prevents a bad *write*. -Adding a second variant later means indexing the same ladder at a second rung — -the sweep already computed it. - -Three properties this needs, each learned from a measured failure: - -- **Multiplicative steps at 1 % or finer.** The window in which a given - quantization is `best_quant` scales with the model, so fixed steps are wrong at - both ends. Qwen3 0.6B reads `Q3_K_M` at 1400M, `Q4_K_M` at 1420M and `Q5_K_M` - at 1500M — a Q4_K_M window roughly 6 % wide, which a 1G-step sweep skips - entirely and silently. Qwen3 32B's is 22G to 24G, with 21G reading Q3_K_M and 25G reading Q5_K_M. -- **`--ram` pinned high and constant.** It is swept as one variable, not two. -- **A missing rung is a hard failure, never a fallback.** If the pinned - quantization never appears in the ladder, the script **exits non-zero and says - so**. It must not fall back to the nearest rung or to the bare-hardware score: - that is precisely the silent mis-grade this section exists to prevent. The - operator then types the rank by hand, which is always allowed. - -The `best_quant` assertion in the transcript above is the visible half of this. -The ladder sweep is what makes it something the script can actually assert -rather than something the operator has to remember. - -> **ponytail:** this whole subsection is a workaround for a missing llmfit flag. -> If `plan --quant` ever grows a quality field, or `recommend` grows -> `--quant`, the sweep collapses to one call and this text should be deleted -> rather than kept for history. - -Numbers are source rather than build output for three reasons beyond speed: a -rank moving 78 → 94 appears in a pull request where someone notices, a tag -rebuilds to the same manifest forever, and the cadence is honest — these change -when someone adds a model, not when someone cuts a release. - -#### Ranking - -`rank` orders the curated rows and selects the ★. It does **two** things and -nothing else. It is never displayed, never compared outside the app, -and never applied to a searched model. - -It is a property of a **variant**, not of a model: the same model at two -quantizations is two builds that answer differently, and the schema stores the -number beside the build. Where this section says "a model's rank", read "the rank -of the variant under consideration". With one variant per entry, as v1 ships, the -two readings coincide. - -Because it is only ever a sort key, it does not need to be a measurement — it -needs to be an **order**. Hence `rank`, not `quality`: calling it quality would -imply we measured something we did not. - -**`rank` means "good at this app's job"** — answering from the user's documents -with citations that resolve — not general capability. That distinction does real -work. llmfit rates Qwen2.5-Coder 7B at **89** against Qwen3 8B's **83**, because -it is measuring coding ability; for document Q&A the coder is the weaker model -and should rank *below* it. A proposal that grades the wrong task is exactly what -the human review step exists to correct. - -**llmfit proposes; a person decides.** It carries real model-specific signal — -grouping ~4,000 scored models by *(parameter count, quantization)*, 224 of 230 +**llmfit proposes; a person decides.** It carries real model-specific signal: +grouping about 4,000 scored models by (parameter count, quantization), 224 of 230 groups show quality varying between models of identical size and quant, so it is not parameter count in disguise. But it is unreliable on models it does not know -well: - -``` -22.1B @ Q6_K - 100.0 mconcat/Qwen3.5-27B-Claude-4.6-Opus-Reasoning-Distilled-NV - 100.0 crushleorey/Qwopus3.6-27B-v2-NVFP4 - 88.0 upstage/solar-pro-preview-instruct -``` - -Community fine-tunes scoring a perfect 100, above published models. That number -is closer to the name than to measured behaviour. It behaves on models it knows -properly — the shipped six come back `33 · 48 · 63 · 78 · 85 · 92`, monotonic -and sanely spread — which is exactly the population the curated tier draws from. - -> **Those figures are at the pinned Q4_K_M**, recovered by the ladder sweep -> above. An earlier draft of this document carried `38 · 53 · 68 · 83 · 90 · 92`, -> which is the same six graded at whatever `best_quant` a large budget selected — -> Q8_0 for five of them. The order was unaffected, so no pick or test changed, -> but the numbers described files the manifest does not ship. **Any rank quoted -> anywhere must name the quantization it was taken at.** - -Four checks before committing a rank: - -1. within a family, rank rises with parameter count — the invariant in **Tests** -2. anything at or near 100 is suspect; nothing worth curating maxes a scale -3. crossing families, cross-check a public leaderboard — that is the comparison - llmfit is least reliable at -4. a specialist (a coder, a reasoning-tuned variant) is ranked for *this* task, - not for the task it was tuned on - -If llmfit has no entry, or proposes something absurd, **type the integer by -hand**. The field is an int; nothing requires llmfit to have produced it. A -number written by someone who ran the model beats a scraped 100. - -**Parameter count is not a sufficient substitute.** Within the shipped six it -happens to work, since they are one family of dense models. It breaks the moment -the list grows: MoE (80B total, ~3B active reads nothing like a dense 80B), -a second dense family where sizes stop being comparable, and any specialist whose -size says nothing about how it handles documents. - -**One list, not tracks.** SurfSense has one job — the Studio formats are summary, -flashcards, mindmap, quiz, podcast, office, web and visuals, and chat is document -Q&A. All twenty-three prompt cases are the same shape: read documents, write -prose or JSON. A single ranking is therefore meaningful, and splitting it would -import a general-catalog framing this app does not need. +well, returning a perfect 100 for community fine-tunes with names like +`Qwen3.5-27B-Claude-4.6-Opus-Reasoning-Distilled-NV`, above published models. +That number is closer to the name than to measured behaviour. The shipped six +come back 33, 48, 63, 78, 85, 92: monotonic and sanely spread, which is the +population the curated tier draws from. + +Four checks before committing a rank: within a family it should rise with +parameter count; anything at or near 100 is suspect; across families cross-check +a public leaderboard, which is the comparison llmfit is least reliable at; and a +specialist is ranked for *this* task, not the one it was tuned on. If llmfit has +no entry or proposes something absurd, type the integer by hand. The field is an +`int` and nothing requires llmfit to have produced it. + +**If llmfit disappears tomorrow**, committed ranks are unaffected, the schema is +unaffected, the refresh script loses its proposer, and the app never knew about +it. That is the test a long-term dependency should pass, and the reason to keep +it outside the product. **Vision is a capability, not a rank position.** Nobody chooses a vision model -*instead of* a general one; they need one when the documents are images. It is -already in the manifest, derived from the header and `input_modalities` (and -gated on `typed_content` — see **7.7**): +instead of a general one; they need one when the documents are images. So the +star stays "the best-ranked build that runs well", and a vision entry surfaces +when it is relevant, never as a competitor in the ordering. -```jsonc -"capabilities": ["vision"] -``` - -So the ★ stays "the best-ranked model that fits", and a vision entry surfaces -when it is relevant (*your PDFs contain scanned pages, and this model can read -them*), never as a competitor in the ordering. `vision` is the only capability -that reaches the UI; everything else `chat_template_caps` reports is an internal -constraint. - -Tracks would only be warranted if SurfSense gained jobs wanting genuinely -different models — a conversational model for podcasts against an analytical one -for chat, say. Nothing today does. - -**If llmfit disappears tomorrow:** committed ranks are unaffected, the schema is -unaffected (`rank` is an int), the refresh script loses its proposer and a person -types the number, and the app never knew about it. That is the test a long-term -dependency should pass, and the reason to keep it outside the product. - -> **Long-term destination, not phase 7.** llmfit grades general capability. -> SurfSense needs something narrower: *does this model answer from the user's -> documents, and cite the right chunks?* No external leaderboard measures that. -> The honest end state is a small internal eval — 20-40 fixed questions over a -> fixed document set, run through the real chat path, scored on whether the -> answer came from the sources and whether `[n]` citations resolved — with `rank` -> set from that. It measures the actual pipeline, catches the RAG failure that -> matters most (answering confidently *without* the sources), reruns in minutes, -> and would also tell you whether the compact/capable/frontier tier split still -> earns its keep after 7.8 lands. Nothing measures that today. - -**Staleness is acceptable, by design.** Airgapped means no refresh path, so a +**Staleness is acceptable by design.** Airgapped means no refresh path, so a shipped list ages. That would be fatal if curated were the only way to get a -model. It is not: 204,797 models are one search away, badged and installable, -with no rank attached. So curated ages into "a starting point we tested a while -ago" rather than a boundary — which is what makes a few-times-a-year cadence a -choice instead of a liability. - -**Search:** `?filter=gguf&search=&sort=downloads&direction=-1&limit≤50`, 300 s -TTL cache (HF allows 500 requests per 5 minutes), `tree/main?recursive=true` for -quants. Group split parts (`-00001-of-00003`); exclude `mmproj*` and `*draft*` -from the quant list. - -Gates: architecture in the 152-name list · chat template present · repo not -`gated` · disk space · physics. Nothing else. - -**Gate on the header, not on Hugging Face's tags.** Counted on `library=gguf`: - -| `pipeline_tag` | GGUF repos | -|---|---| -| `text-generation` | 37,849 | -| `image-text-to-text` | 5,327 | -| `any-to-any` | 315 | -| `audio-text-to-text` | 27 | -| `automatic-speech-recognition` | 547 — not chat | -| `feature-extraction` | 681 — embeddings, exclude | - -That totals ~43,500 of 204,797, so **roughly 80% of GGUF repos carry no useful -pipeline tag at all** — filtering the search by tag would hide four-fifths of the -catalog, including working chat models whose uploader left the field blank. The -two header gates do the job structurally instead: a supported architecture -excludes Whisper-type models, and a present chat template excludes embedding -models. Pipeline tags are worth offering as an optional facet ("vision models -only"); they are not the gate. - -**Search ordering.** All five HF sort fields work on `filter=gguf`; only one is -usable as a default: - -| `sort=` | Top result | Verdict | -|---|---|---| -| `downloads` | `unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF` | **default** | -| `likes` | `unsloth/Qwen3.8-27B-GGUF` | surfaces uncensored derivatives in the top few | -| `trendingScore` | `prism-ml/Ternary-Bonsai-2-27B-gguf` | same | -| `lastModified` | `tapiocaTakeshi/Qubit` | surfaces half-finished uploads | -| `createdAt` | `mradermacher/HuatuoGPT-3-9B-i1-GGUF` | same | - -Every order surfaces abliterated and uncensored fine-tunes somewhere near the -top. That is what an open catalog means, and it is not a reason to reintroduce -grading — the curated tier is what serves someone who does not want to evaluate -models, and it is the only tier the recommendation reads. Copy must present the -count as popularity, not endorsement. - -**What a search row shows** — facts about the file, never a judgement of it: -fit verdict · size per quantization · architecture · context length · chat -template present · vision capability · license · gated · downloads, likes, -last updated · **provenance**. - -Provenance is free and worth surfacing: 32 of the top 40 GGUF repos carry a -`base_model:quantized:` tag, so a row can read *"quantized from -`Qwen/Qwen3.8-27B`"*. That is a statement about where the file came from, not a -grade — and it is most of what a quality score was doing for a user anyway, -letting them recognise an unfamiliar repo as a repackaging of a model they know. -The 8 of 40 with no such tag are informative by their absence. - -**Fit badges appear on both tiers. Quality appears on neither.** These look -alike and are not: - -| | What it is | Needs | Where | -|---|---|---|---| -| fit badge | subtraction — does this file fit in this memory | device memory + file size | **every row, both tiers** | -| `rank` | a preference order over models we tested | a person to have decided | curated only, and never displayed | +model, and it is not: 204,797 models are one search away, badged and installable. +Curated ages into "a starting point we tested a while ago" rather than a +boundary. -"This 40 GB file will not fit in your 8 GB of memory" is no more a judgement than -"this file is 40 GB", and it is the single most useful thing to tell someone -browsing 204,797 models they have no size intuition for. Withholding it does not -protect them; it makes them download 40 GB to find out. +> **Long-term destination, not this phase.** llmfit grades general capability. +> SurfSense needs something narrower: does this model answer from the user's +> documents, and cite the right chunks? No external leaderboard measures that. +> The honest end state is a small internal eval, 20 to 40 fixed questions over a +> fixed document set, run through the real chat path and scored on whether the +> answer came from the sources and whether `[n]` citations resolved, with `rank` +> set from that. -Two stages, so the list stays cheap: +### Search ```text -search list file size is already in the HF response - → coarse badge from size alone, no extra request - -open a model read 2-4 MB of its GGUF header (2.5 s) - → exact badge, conversation memory included - -curated shape is in the manifest - → exact badge offline, on first paint +GET /api/models?filter=gguf&search=&sort=downloads&direction=-1&limit=<=50&full=true +GET /api/models/{repo}/tree/main?recursive=true +GET /api/models/{repo}?expand[]=gguf ``` -#### Fit states - -Badge vocabulary changes meaning with the swap. It stops being llmfit's -five-level score (`perfect`/`good`/`marginal`/`too_tight`/`unknown`) and becomes -a three-state statement about **where the weights will live**. There is no -`unknown`: every row has a file size, so every row has a badge. - -Let `need = weights + KV(window) + compute_buffers`, `vram` = -`HardwareBudget.usable_vram_bytes` (device free **minus** the fit reserve), and -`ram` = host memory available to the budget. The two subtractions sit on opposite -sides of the comparison and must not be collapsed into one term — see 7.2. - -| State | Condition | What happens at load | -|---|---|---| -| `FITS` | `need ≤ vram` | every layer on the device, full speed | -| `PARTIAL` | `vram < need ≤ vram + ram` | `--fit` places some layers on the CPU; runs, slower | -| `TOO_BIG` | `need > vram + ram` at the **16K floor** | physics refusal — the only state that blocks install | - -`TOO_BIG` is evaluated at the floor, not at the requested window: a model that -will not fit at 16K cannot be rescued by a smaller context, and the remedy to -offer is a smaller quantization or a smaller model. - -**Keep `offload_fraction`, do not collapse it into the state.** `PARTIAL` spans -everything from barely-noticeable to unusable, and the number is already known -from the same subtraction that produced the state. Two consumers need it: the -recommendation gate, and the graded reason line below. Discarding it is what -made an earlier draft star a 1.7B on a machine happily running an 8B. - -**Wording is per platform. The state set is not.** One label set is wrong on two -of the three targets, so the copy branches on two facts the budget already -carries: `uma`, and whether any non-CPU device exists. - -A badge is a **verdict plus one plain line of why**. The verdict is what someone -choosing a model needs (fast, slower, or impossible); the mechanism is the -explanation, not the headline. - -**Discrete GPU** (Windows or Linux with an NVIDIA or AMD card) - -```text -● Full speed Runs entirely on your graphics card -◐ Reduced speed Too big for your graphics card, so part runs on the processor -○ Won't fit Needs about 21 GB. This PC has 13.6 GB -``` - -**Apple Silicon** (`uma`) - -```text -● Full speed Runs entirely on the GPU -◐ Reduced speed Too big for the GPU, so part runs on the CPU -○ Won't fit Needs about 21 GB. This Mac has 13.6 GB -``` +300 s cache, because Hugging Face allows 500 requests per 5 minutes and a search +list is re-fetched on every keystroke a debounce lets through. + +**Sorted by downloads**, which is the only sort usable as a default. All five HF +sort fields work on `filter=gguf`; `likes` and `trendingScore` surface +uncensored derivatives in the top few, and `lastModified` and `createdAt` surface +half-finished uploads. Every order surfaces abliterated fine-tunes somewhere near +the top, which is what an open catalog means and is not a reason to reintroduce +grading. The count is presented as popularity, never as endorsement. + +**The gate is the header, not Hugging Face's tags.** Counted on `library=gguf`, +the pipeline tags cover roughly 43,500 of 204,797 repos, so about 80% of GGUF +repos carry no useful tag at all, including working chat models whose uploader +left the field blank. Two header facts do the job structurally instead: a +supported architecture excludes Whisper-shaped models, and a present chat +template excludes embedding models. `SUPPORTED` holds 113 `LLM_ARCH_*` names as +data rather than fetching them, because the app is airgapped and the list changes +only when the pinned build does. + +**Eligibility is read before the header.** Hugging Face has already parsed the +first GGUF's header and serves the result under `expand[]=gguf`, giving the +architecture, the context length and whether a chat template exists. Asking that +first means an architecture llama.cpp cannot run costs no range request against a +file we were never going to install. It does not carry the KV shape, so it +replaces nothing for a model we *can* run: the header read still happens, it just +no longer happens for models we cannot. + +A repo llama.cpp cannot run still lists its builds at their real sizes, badged +`Cannot run` and not installable, because the screen is answering "what is in +here" as well as "can I run it", and an empty repo reads as a broken page rather +than an unsupported model. + +A repo that ships **no chat template** is installable and warns. It will answer +badly in a chat UI, which is a real warning and not a fit state. + +**A header that cannot be read does not lose the repo.** The builds fall back to +a size-only shape, every architecture-derived term goes to zero so `need` is the +file size and nothing else, and the rows are marked `approximate` so the screen +can say so. Refusing the whole repo over one unreadable header would hide builds +the user can run. + +**What a search row shows**: fit verdict, size per quantization, architecture, +context length, chat template present, license, gated, downloads, likes, last +updated, and provenance. Provenance is free and worth surfacing: 32 of the top 40 +GGUF repos carry a `base_model:quantized:` tag, so a row can say where the +file came from. That is a fact about provenance, not a grade, and it is most of +what a quality score was doing for a reader. The 8 of 40 without one are +informative by their absence. + +`list_builds()` excludes split sets (`-of-`), projectors (`mmproj`) and draft +models (`draft`): none of them is a thing a user installs on its own. The +quantization comes out of the filename with a regex tight enough to require a +real quant token, because matching loosely reads `Qwen3` out of +`Qwen3-Coder-30B-A3B-Instruct-UD-TQ1_0.gguf`, seen live. The last match wins, +because the quant is conventionally the final token. + +### Install ids + +`POST /llm/install` keys on a `catalog_id`, and +[`../frontend/05-install-ux.md`](../frontend/05-install-ux.md) requires the +renderer to send only that: no repo, file, artifact URL, local path or +quantization. A curated row's id comes from the manifest; a search hit has no +manifest entry, so there is nothing to key on. Closing that by letting the +renderer post a repo and file would hand the frontend the ability to name an +arbitrary download, which is exactly the capability the install contract exists +to withhold. -**No GPU device** (`--list-devices` prints `(none)`) +So the server mints ids for both tiers and hands back only the id. -```text -● Works here Runs on your processor -○ Won't fit Needs about 21 GB. This PC has 16 GB -``` +- **Curated**: `secrets.token_urlsafe(18)`, minted once per process, keyed on + `(repo, file)`. Opaque so the renderer cannot assemble one. +- **Searched**: an `InstallTicket` with a 300 s TTL, the same window the search + cache uses. A ticket outliving its row would let a stale screen install + something the user is no longer looking at. -Terse variant, if rows are tight: verdict as the badge, reason as dimmed -trailing text, full explanation in the row's detail view. +`resolve_install()` checks curated ids first and then tickets, and cannot tell +them apart from the route's point of view. Both failure modes collapse into the +existing `422 catalog id is stale or unknown; refresh the catalog`, because it is +the same failure and already has copy. -```text -● Full speed -◐ Reduced speed (part runs on the processor) -○ Won't fit (needs 21 GB, you have 13.6 GB) -``` +### List order and the recommendation policy -**"Full speed", not "Fast".** Fast is a promise the badge cannot keep: a 32B -running entirely on a 4090 is still slower than a 4B. *Full speed* is relative -to the model, which is exactly what the state means — as fast as this machine -can run this particular model. Same reason *Reduced speed* beats *Slower*: it -says reduced from what it could be, not slow in absolute terms. +Curated sorts by `(fit state, -rank, model_id)`: fit coarsely, rank finely. +Sorting by rank alone would put a refused 32B at the top of a small machine's +screen, which is the one thing a model chooser must not do. -**"Works here" on a machine with no GPU.** Technically that case is `FITS`, but -"Full speed" reads as a boast about a slow situation when there is no faster -alternative to contrast with. The useful information is simply: yes, you can run -this. That is the one place the verdict word differs and not just the -explanation. +**One row per model, not per variant.** A model with two builds is one row, which +takes the best state and rank among them and installs the build that produced +them. Listing builds separately would show the same model twice on a screen whose +job is choosing a model. -On Apple Silicon there is no separate system RAM to spill into. A model over -Metal's working set is not copied anywhere; llama.cpp runs some layers on the CPU -backend against the same physical memory, so "uses system RAM" would describe a -transfer that does not happen. With no GPU at all, `FITS` means the processor and -`PARTIAL` is unreachable because there is nothing to spill from. +**Neither list displays a rank.** `CatalogRow` does not carry one, `schemas.py` +does not declare one, and `router._row()` does not serialise one. The surest way +to keep "never displayed" true is for the renderer never to receive it. -> **Copy rule for every user-facing string in this phase: no em dashes and no -> hyphens.** Use commas, full stops or parentheses. Applies to badges, empty -> states, error copy and the recommendation line. - -**Grade the `PARTIAL` reason line by `offload_fraction`.** Not optional: the -range is wide enough that one sentence is wrong at both ends, and the number is -already on the verdict. - -```text -f ≲ 0.25 ◐ Reduced speed A little too big for the graphics card. Most of it still fits. -f ≳ 0.5 ◐ Reduced speed Well over your graphics card's memory. Expect it to be slow. +```python +def recommend(curated, budget) -> Recommendation | None: + candidates = [ + (entry, variant, verdict) + for entry in curated + for variant in entry.variants + # at the cache the loader would actually choose + for verdict in [estimate(..., precision=planned_precision(...))] + if speed_tier(verdict) in RECOMMENDABLE_TIERS + ] + return max(candidates, key=lambda c: (c.variant.rank, -c.variant.size_bytes), + default=None) ``` -The verdict word does not change, because the state has not changed. Only the -explanation sharpens. An earlier draft filed this as a copy refinement; it is -really the same information the recommendation gate needs, surfaced. - -> **Decision: `PARTIAL` stays available on unified memory.** Hermes' -> `_uma_budget()` sets `ram_available_bytes = 0`, so on Apple Silicon their -> physics check is `need ≤ vram` alone and the middle state cannot occur — on an -> 8 GB M2 that refuses Qwen3 8B (7,043 MiB against ~5,461) outright. We do not -> copy that. It genuinely runs with CPU layers, and refusing it removes a real -> option from the users with the least choice. On UMA the refusal threshold is -> total RAM, not the device working set. +`TOO_BIG`'s own tier is never in `RECOMMENDABLE_TIERS`, so that is the one check: +physics and speed are judged by the same classification rather than a state check +plus a separate threshold. + +**`None` is an honest answer rather than a failure.** Every build physics does +not refuse stays installable; it simply goes unstarred. The one case where that +was wrong has been fixed: a CPU-only machine used to have every row come back as +a full spill, which the gate then refused, so the one screen whose job is +choosing a model chose nothing on the hardware that most needs the help. Splitting +residency from refusal made `PARTIAL` unreachable there, and the star came back. + +**The policy ranges over builds, not models**, because a build is what the user +installs and what `rank` describes. With one variant per entry the cross-product +is the entry list and the behaviour is identical, but writing it this way is the +difference between adding a second build later as a manifest edit and rewriting +the module, its tests and every fixture. + +**Why the gate is on speed rather than on residency.** Residency is a mechanism +and speed is the goal, and they coincide on a large card and diverge on a small +one. Measured on an RTX 3050, Qwen3 4B spills about a fifth and Qwen3 8B about a +quarter, and the machine's owner runs the 8B without noticeable lag. A +`FITS`-only rule stars a 1.7B on that machine, which is not conservative, it is +wrong: it bans a configuration that demonstrably works. -**One modifier, not a fourth state.** A search row badged from file size alone, -and a row whose header read was truncated, carry the same three states at lower -precision — rendered as approximate (a `~`, or a lighter treatment) and resolving -to the firm badge once the header lands. +The reason a fifth spilled is barely noticeable is that decode is +bandwidth-bound and the split is not proportional. Measured, using `-ngl` to set +the fraction directly so model size is held constant: -**Eligibility is not fit.** These block or warn independently and must not be -rendered as fit states — a model can be `FITS` and still gated: +| layers on GPU | `f` | prefill ratio | decode ratio | +|---|---|---|---| +| 36/36 | 0.00 | 1.00 | 1.00 | +| 28/36 | 0.22 | 0.81 | 0.63 | +| 18/36 | 0.50 | 0.67 | 0.38 | +| 0/36 | 1.00 | 0.50 | 0.25 | -| Condition | Effect | -|---|---| -| architecture not in llama.cpp's 152-name list | not installable; no amount of memory helps | -| no chat template in the header | installable, warns — will produce garbage in a chat UI | -| repo is `gated` | needs a Hugging Face account before the download resolves | +Fully on the CPU, prefill still runs at half device speed while decode drops to a +quarter. This app is prefill-dominated, roughly 8,000 prompt tokens against 300 +decoded, so a turn loses substantially less to spilling than any decode-focused +benchmark implies. That is also why no threshold borrowed from an agent-shaped +application belongs here. -#### List order +> **Open tension, stated rather than smoothed over.** The measured claim above +> is that the 3050's owner runs the 8B happily at a byte ratio of about 0.28. The +> layer-granular fraction puts that same configuration at 0.33, which is +> `MODERATE_SPILL`, which is not recommendable, so the current policy stars the +> 4B on that machine. Either the tier boundary is a little tight or the measured +> experience was generous, and one prefill measurement settles it. The boundary +> is not a physical constant, and it is deliberately the same number the badge +> uses, so moving it moves both together. -Curated sorts by `(fit state, -rank, model_id)` — fit coarsely, rank finely — -which is what today's `_sort_key` already does. Sorting by rank alone would put a -`TOO_BIG` 32B at the top of an 8 GB machine's screen. +## The wire -**One row per model, not per variant.** A model with two builds is one row; the -row takes the best state and rank among its variants, and the row's install -action uses that variant. Listing builds separately would show the same model -twice on a screen whose job is choosing a model. +### HTTP routes -The recommendation is the top row of the `FITS` bucket, subject to the speed -floor. Search sorts by downloads. -**Neither list displays a rank**; rank only orders curated rows and selects the -recommendation. +| Route | Returns | Network | +|---|---|---| +| `GET /llm/system` | budget, devices, `gpu_status` | none | +| `GET /llm/catalog` | budget, `gpu_status`, curated rows, installed rows, `recommended_model_id` | **none** | +| `GET /llm/search?q=&limit=` | HF hits, described not judged | `huggingface.co` | +| `GET /llm/search/{repo:path}` | builds with exact sizes and badges | `huggingface.co` | +| `POST /llm/install` | NDJSON progress stream | `huggingface.co` | +| `DELETE /llm/models/{model_name:path}` | `ModelDeleteRead`, `selection_cleared` | none | + +`GET /llm/search/{repo}` is where the header read happens, so the trigger is +**opening a search result**, not hovering or typing. The list-level badge stays +approximate until then. + +Unchanged: `GET /llm/providers`, `GET /llm/providers/{provider}/models`, +`GET|PUT /llm/selection/{role}`, `GET|POST /llm/onboarding`, everything under +`/llm/connections`, and the four `/llm/image/local*` routes, which are a separate +runtime with its own lifecycle and are out of scope here. -**A searched model can be installed, selected, and used. It can never be -recommended**, because recommending requires ranking and ranking requires a -`rank` that only the manifest carries. The two tiers differ in kind, not degree. +Deleted with the Ollama adapter: `GET /llm/providers/{provider}/catalog` and +`POST /llm/providers/{provider}/pull`. -Deleted: the collision-resolution block, `_placeholder_row()`, -`_synthetic_curated_model()`, `_runtime_target_model()`, the disk scan cache, -`CACHE_VERSION`, and the `scanned` flag through `AdvisorCatalog` → -`CatalogResult` → `RecommendationCatalogRead` → the frontend. +Search and repo routes guard on `egress.MODEL_SEARCH` and install on +`egress.MODEL_DOWNLOAD`, and an unreachable host is a `503` naming the +destination rather than a generic error. With egress off, search is **absent +rather than degraded**, which is the airgapped product: curated, installed and a +local `.gguf` import all still work. -**Recommendation policy**, its own module and its own test file: +### Response shapes -```text -eligible = (entry, variant) pairs that are not TOO_BIG - and whose offload_fraction ≤ MAX_OFFLOAD -eligible → max(variant.rank, -variant.size_bytes) reason: best-rank-eligible -else → no recommendation; every non-TOO_BIG pair stays installable, - just never starred -``` +`SystemRead` and `CatalogRead` both carry `budget` and `gpu_status`. +`CatalogRowRead` carries `fit` (state, need, budget, offload fraction, +approximate), `badge` (verdict, reason), `capabilities`, `installed`, `selected`, +`can_install`, `recommended`, and an opaque `catalog_id`. `RepoRead` carries +`architecture`, `context_length`, `supported`, `chat_template`, an optional +`ineligible_reason`, and a list of `BuildRead`. **No row of any kind carries a +`rank`, and no response carries a `scanned` flag**, because there is no scan: the +budget comes from the allocator in milliseconds. -```python -candidates = [(e, v) for e in curated for v in e.variants - for verdict in [fit(e.shape, v.size_bytes)] - if verdict.state is not FitState.TOO_BIG - and verdict.offload_fraction <= MAX_OFFLOAD] -pick = max(candidates, key=lambda ev: (ev[1].rank, -ev[1].size_bytes), - default=None) -``` +### The install stream -> **`MAX_OFFLOAD` fires, and it is load bearing.** An earlier draft suspected it -> might never trigger and should perhaps not ship. Run against the six profiles -> with the implemented estimator, the opposite holds: **without a ceiling, five -> of six machines star Qwen3 32B**, including a 6 GB RTX 3050 at `f = 0.82`. -> -> | machine | no ceiling | ceiling at 0.75 | `f` of the unceilinged pick | -> |---|---|---|---| -> | M2 8 GB | Qwen3 8B | Qwen3 8B | 0.38 | -> | RTX 3050 6 GB | **Qwen3 32B** | Qwen3 8B | **0.82** | -> | M4 16 GB | Qwen3 32B | Qwen3 32B | 0.57 | -> | RTX 4070 12 GB | Qwen3 32B | Qwen3 32B | 0.57 | -> | RTX 4090 24 GB | Qwen3 32B | Qwen3 32B | 0.05 | -> | M4 Max 64 GB | Qwen3 32B | Qwen3 32B | 0.00 | -> -> The earlier suspicion came from sweeping **one** model and finding every -> offload fraction usable. That is true, and it is the wrong population: the gate -> exists to stop the *largest* build being starred on a small card, and only the -> ladder exposes that. -> -> **0.75 is still too loose and should not ship as it stands.** It leaves the -> 4070 and the 16 GB Mac on a 32B at `f = 0.57`, which the roofline puts at about -> 37% of resident decode speed. Set the number from the roofline against a stated -> speed floor rather than picking a round one. The point here is only that the -> ceiling must exist. - -`MAX_OFFLOAD` is a v1 placeholder for the speed gate below, not a latency -judgement. Set it generously: the failure it exists to prevent is starring a -model that crawls, and the failure it replaced was refusing models that work. - -**The policy ranges over builds, not models**, because that is what the user -installs and what `rank` describes. With v1's single variant per entry the -cross-product is the entry list and the behavior is identical — but writing it -this way now is the difference between adding a `Q6_K` build later as a manifest -edit and rewriting the policy, its test file and every fixture. A model whose -larger variant is `TOO_BIG` and whose smaller one `FITS` is still recommendable -on its smaller build; that falls out of the cross-product rather than needing a -special case. - -#### Why the gate is on speed, not on residency - -An earlier draft gated on `FITS` alone, on the reasoning that a latency gate -needed constants nobody had measured. That is true and it still produced the -wrong answer, because **residency is a mechanism and speed is the goal.** They -coincide on a large card and diverge on a small one: +NDJSON, `{"type": …}` frames, one per line: ```text -RTX 3050, 6 GB, measured - Qwen3 4B f ≈ 0.20 spilled → PARTIAL - Qwen3 8B f ≈ 0.28 spilled → PARTIAL, and runs without noticeable lag +starting "Preparing download" +downloading completed / total, repeated +verifying "Checking the model" +preparing "Preparing the model runtime" <- reprice(), then wait +selecting "Selecting model" <- only when select: true +complete "Model is ready" + or "Downloaded. It becomes available once the runtime restarts." +error "The model could not be installed. Retry the download." ``` -A `FITS`-only rule stars **Qwen3 1.7B** on that machine while its owner is -running the 8B perfectly happily. The rule is not conservative, it is wrong: it -bans a configuration that works. +The download writes to a sibling `.part` file and renames only once the whole +file is present and verified, so the router never discovers a partial model, and +a cancelled download keeps the `.part` for a later resume. A server that ignores +our `Range` header is detected by the absence of a `206`, and the bytes already +held are discarded rather than appended to. -The reason a fifth spilled is barely noticeable is that decode is -bandwidth-bound and the split is not proportional. Which is also why a single -threshold on `f` is a crude instrument, and why the real gate is a time estimate. +One install at a time, enforced by an `asyncio.Lock` and a `409` when it is held: +two concurrent downloads compete for the same disk and the screen has one +progress bar. -#### The speed model +## Chat -Keep the offload fraction the fit calculation already computes. Decode reads the -weights once per token, so: +### The context budget -```text -f = bytes_on_cpu / total_bytes 0.0 resident … 1.0 all CPU -t_token = W × decode_fraction × ( f/BW_cpu + (1−f)/BW_gpu ) -decode_tps = efficiency / t_token -``` +`HISTORY_BUDGET_TOKENS` used to be a constant with no relationship to `n_ctx`, +which was safe only while the window was always the same number. Lowering the +floor to 8192 made it unsafe, so `modules/chat/budget.py` prices the four named +parts of a turn against the real window: -**Measured, and it holds.** Qwen3 4B on an RTX 3050, `-ngl` used to set `f` -directly so model size is held constant: - -| layers on GPU | `f` | decode t/s | vs resident | implied `r` | -|---|---|---|---|---| -| 36/36 | 0.00 | **52.45** | 1.00 | — | -| 28/36 | 0.22 | 33.17 | 0.63 | 3.7 | -| 18/36 | 0.50 | 19.85 | 0.38 | 4.3 | -| 9/36 | 0.75 | 12.93 | 0.25 | 5.1 | -| 0/36 | 1.00 | 13.13 | 0.25 | 4.0 | - -The slowdown reduces to `1/(f·r + 1 − f)` where `r = BW_gpu / BW_cpu`, and every -point fits **r ≈ 4** within ±20%. Cross-checked against absolutes: 131 GB/s -effective on the device (78% of the card's rating) against 33 GB/s on the host, -ratio 4.0. So `r` comes from published bandwidths and needs no per-card -measurement. - -> **`r` is what makes spill tolerable here, and it is hardware-specific.** On a -> high-bandwidth card `r` is far larger — roughly 12 on a 4090 — so the same -> `f = 0.3` would cost ~78% rather than ~31%. Small cards have modest bandwidth -> *and* are the only ones that spill, which limits the damage, but a threshold -> calibrated on a 3050 is **not** conservative for other hardware. On unified -> memory `r` approaches 1 and spilling barely means anything. - -**Prefill degrades roughly half as much as decode**, which matters more here than -anywhere because this app is prefill-dominated: - -| `f` | prefill ratio | decode ratio | +| Part | Tokens | Basis | |---|---|---| -| 0.22 | 0.81 | 0.63 | -| 0.50 | 0.67 | 0.38 | -| 1.00 | **0.50** | **0.25** | - -Fully on the CPU, prefill still runs at half device speed while decode drops to a -quarter. A turn of ~8,000 prefill against ~300 decoded therefore loses -substantially less to spilling than any decode-focused benchmark implies — and -than Hermes' `PLEASANT_FLOOR_TOK_S = 20.0`, which was tuned for agentic bursts, -would suggest. **Do not import that threshold.** - -`decode_fraction` is the manifest field already defined and computed from the -header — **this is the thing it is for.** The spec previously filed it as -"nothing to apply it to"; the speed model is the application. - -Decode alone still mis-ranks for this app, because a turn is ~8,000 prefill -tokens against ~300 decoded. Prefill is compute-bound rather than -bandwidth-bound and degrades differently under offload, so it needs its own -term: - -```text -felt_time ≈ prefill_tokens / prefill_tps + decode_tokens / decode_tps -eligible = pairs whose felt_time ≤ budget -pick = max(eligible, key=rank) -``` - -#### Where the constants come from: every turn is a measurement - -`BW_gpu`, `BW_cpu`, `efficiency` and the prefill rate are per-machine. Both -reference implementations ship them as hardcoded tables — Hermes has a -`GPU_BANDWIDTH` dict and `_estimate_speed(..., offload_frac=0.0)`; Odysseus adds -`FALLBACK_K = {"cuda": 220, "rocm": 180, "metal": 150, …}`; llmfit's own output -admits `"method": "gpu_bandwidth_roofline"`, `"efficiency": 0.55`. +| Answer reserve | 1024 | Reserved first. llama.cpp stops a reply wherever the window runs out, so spending this on history is how a good answer gets cut off mid-sentence with no error at all. | +| System prompt | 400 | A rendered prompt plus the grounding header at the largest tier. Approximate; the exact figure is one `/apply-template` call away. | +| Excerpts | 2400 | The retrieval cap already limits chunks to 480 tokens at up to 5 hits, so this holds even before retrieval runs. | +| Question | 1024 | `MessageText` enforces this at the wire. | -**We do not copy that.** A table of bandwidths for cards nobody tested is the -llmfit failure in a new costume. Instead: +History gets whatever is left: `max(0, n_ctx - 4848)`, which is 3,344 tokens at +the 8192 floor and grows with the window. A narrower window therefore means a +shorter history rather than a turn that silently exceeds the window. -> **A completed chat turn already carries everything the model needs** — prompt -> tokens, generated tokens, time to first token, total time — and the offload -> fraction of the loaded model is known. That is enough to solve for this -> machine's constants, with no benchmark, no extra binary and no user-visible -> step. +**An unknown window is not a narrow one.** A remote endpoint that does not report +`n_ctx` is an absent fact, not a small number, so the fallback is the previous +constant, 3000, unchanged. For the same reason `answer_max_tokens()` returns +`None` there: capping a reply to this app's reserve on an endpoint whose real +window might be far larger would truncate answers for a limit that was never +theirs. -```text -first run predict with a conservative shipped default -after turn 1 record (prompt_tokens, decode_tokens, ttft, total, f) -thereafter predict with this machine's own numbers -``` - -Identical in shape to the memory self-calibration in **7.2**, and it reuses the -same per-device store. It is also why no `llama-bench` run ever has to ship. +### Counting tokens -#### Staging - -| | | -|---|---| -| **v1** | Retain `f` on the verdict. Gate on an offload-fraction ceiling instead of `FITS`. One number, conservative, and it fixes the 3050 case. | -| **next** | The roofline above with a shipped default, plus badge copy graded by `f`. | -| **then** | Turn-based self-calibration, which retires the shipped constants. | -| **never** | A hardcoded per-GPU bandwidth table. | - -The v1 step is deliberately crude: an `f` ceiling is not a latency judgement and -should not be described as one. It is a placeholder that is wrong in the safe -direction — it stars models that run, rather than refusing models that work. - -> **RAG is prefill-dominated, which is why the gate must measure prefill.** -> `HISTORY_BUDGET_TOKENS = 3000` plus ~24k characters of grounding is roughly -> 8,000 prefill tokens against ~300 decoded — the inverse of an agent's ratio. -> Decode is memory-bound; prefill is compute-bound. A decode-only prediction -> mis-ranks for this app. (An earlier draft cited a 36–40% CUDA lead here; -> measured, it is **9.1%** — see [`08-cuda-backend.md`](08-cuda-backend.md).) -> -> **There is no manifest field to fall back on.** An earlier draft ended this -> block with *"use llmfit's `prefill_tps`/`ttft_ms` from the manifest until -> measured class constants exist"*. Those were `ScoredModel` fields, produced by -> a scan that no longer runs, and schema 3 carries `shape`, `capabilities`, -> `decode_fraction` and `variants` and nothing else. Until the turn-based -> calibration below lands there is a shipped default and no second source. +`Generator.token_count()` prices a turn by the model's own tokenizer through +llama.cpp's `/tokenize`, proxied to the loaded worker the same way `/props` is. +The fallback heuristic was about 4 characters per token, which under-counts dense +text by 15 to 20%. -**Tests:** policy decision-table across the six hardware profiles above. +`None` is the answer for anything short of a clean count: no such endpoint, a +transient failure, or a model this generator does not run. A caller must fall +back to an estimate rather than read `None` as zero tokens, and `build_messages` +never mixes the two within one turn. -### 7.5 — Packaging, three targets +A remote endpoint has no equivalent, so `OpenAICompatibleChatProvider` keeps the +heuristic exactly as before. -`fetch-llamacpp.mjs`, pinned build number **and SHA-256** (`fetch-llmfit.mjs` -verifies a checksum; `fetch-ollama.mjs` does not — close that gap here, since -llama.cpp has no stable channel). +### Capabilities and the system role -| Target | Asset | Size | -|---|---|---| -| darwin-arm64 | `bin-macos-arm64.tar.gz` | 11 MB | -| win32-x64 | `bin-win-vulkan-x64.zip` | 31 MB | -| linux-x64 | `bin-ubuntu-vulkan-x64.tar.gz` | 30 MB | - -Verified: the Vulkan archive is the CPU archive plus exactly one file -(`libggml-vulkan.so` / `ggml-vulkan.dll`), and all CPU micro-architecture -variants ship inside — 15 on Windows, 10 on Linux (the CUDA archive carries 14 -on Linux, so the figure is per archive, not per platform). - -> **No CUDA payload ships, and the CUDA staging notes have moved.** Earlier -> drafts described the flat-folder probe order and the `cudart64_*` / -> `cublas64_*` / `cublasLt64_*` DLLs here, before the backend decision was made. -> This phase stages Vulkan on every platform off Apple Silicon and nothing else, -> so those notes belong to [`08-cuda-backend.md`](08-cuda-backend.md), which -> already carries them. Leaving them in a Vulkan-only packaging section reads as -> work to do. - -Prune to `llama-server` plus its libraries: 24 executables → 1, which also -shrinks the macOS notarization surface. - -- **Linux:** `libggml-vulkan.so` has `DT_NEEDED: libvulkan.so.1`. deb adds - `libvulkan1` to `Depends`; AppImage bundles the loader or accepts CPU (the ICD - must come from the host driver). glibc floor is **2.34**. -- **Pin `ubuntu-22.04`.** `ubuntu-latest` migrates to 26.04 between 19 Oct and - 19 Nov 2026 and will silently raise the AppImage's glibc floor. The - PyInstaller binaries already require 2.39 from the current runner, so this is - a pre-existing portability bug that llama.cpp does not cause and this phase - should fix. -- **macOS:** router mode spawns grandchildren under the hardened runtime. -- **Windows:** verify `taskkill /t /f` on the router PID reaps its model workers. - -**Tests:** staged-checksum verification; per-OS smoke running `--list-devices` -from the final packaged resource path. - -### 7.6 — Remove Ollama completely - -421 references across 66 files. The only irreversible phase. - -**Delete:** +Two sources, two kinds of fact, and only one of them reaches a person: ```text -backend/modules/llm/providers/ollama/ provider.py, catalog.py, __init__.py -backend/tests/unit/llm/recommendations/test_ollama_runtime.py -backend/tests/unit/llm/test_ollama_num_ctx.py -electron/src/main/sidecars/ollama.ts -electron/scripts/fetch-ollama.mjs -electron/ollama/ 501 MB staged +GET /models -> architecture.input_modalities what the model can ACCEPT +GET /props -> chat_template_caps what the TEMPLATE supports ``` -**llmfit's packaging goes in the same phase**, and is easy to miss because none -of it has "ollama" in the name, so the definition-of-done grep below will not -catch a single one: - -```text -electron/scripts/fetch-llmfit.mjs -electron/scripts/check-llmfit.mjs -backend/tests/fixtures/llmfit/ fit.json, system.json -``` +Template-derived rather than guessed from a name. A regex over model names is how +you end up telling someone a model reads images when nothing can hand it one. -plus `SidecarContext.llmfitPath`, `python.ts`'s `SURFSENSE_LOCAL_LLMFIT_PATH`, -the four `llmfit_*` settings in `shared/config.py`, `recommendation_reserve_gb` -(replaced by the reserve table in 7.2), and the llmfit staging and smoke steps -in `release-local.yml`. **Decisions** says llmfit is not shipped and not in the -build; this is what that costs in files. `modules/llm/recommendations/llmfit.py` -itself is deleted earlier, in 7.4, when its parsing moves into the authoring -script. - -**Rewrite:** `shared/config.py` (`ollama_base_url`/`ollama_models_dir` → -`llamacpp_*`), `sidecars/types.ts` (`SidecarContext.ollama*`), -`sidecars/python.ts` (stops passing `OLLAMA_*`), `electron-builder.yml`, -`electron/package.json` (`build:ollama` and the `dist` chain), -`.github/workflows/release-local.yml`, `modules/llm/router.py`, -`modules/llm/resolution.py`, `modules/llm/selection.py`, `modules/chat/errors.py`. - -**Two files in `modules/llm/profile/` are Ollama-shaped and neither appears in -any phase's file list.** `classify.py` is covered by the **Decisions** table -(*Prompt tier fallback* — key on loopback, not on a provider name) but is not -listed anywhere as work. `fingerprint.py` is not covered at all: -`from_ollama(name, tag, show)` is built from `/api/tags` and `/api/show`, and it -needs a `from_llamacpp()` reading `GET /models` and `GET /props` instead. Both -land here. - -**Frontend:** `selected-roles.tsx` (`llamacpp` → "Local"; a loopback connection -is arguably also "Local"), `chat-error-notice.tsx`. **Keep** -`connection-form.tsx`'s Ollama preset. - -**Egress:** `ollama_pull → registry.ollama.ai` becomes two destinations against -`huggingface.co` — `model_download` (a repo you named) and `model_search` (text -you typed). Same host, different consent. `ollama_pull_host()`'s `hf.co/` -special-case disappears, and with both destinations now static the dynamic -host-resolution path goes with it. - -> **The migration maps an existing grant to `model_download` only.** -> `model_search` starts unset. This is the one place "same host, different -> consent" has to be honoured rather than restated: a user who allowed model -> downloads allowed *fetching a file they named*, and search sends text they are -> typing to the same host for a different reason. Splitting one destination into -> two and carrying the grant across both would manufacture consent nobody gave, -> which is precisely the distinction the split exists to draw. The asymmetry is -> safe in the other direction: an unset `model_search` degrades to the airgapped -> product, which **Failure behavior** already specifies. - -**Three things that will bite:** - -1. **Do not touch revisions `0004` and `0009`.** They mention Ollama and are - applied history on every installed machine. A new revision does the data work. - `tests/integration/test_migrations.py` asserts historical schema — read it - first. -2. **Existing users have a broken selection.** `SelectedModel(provider="ollama")` - makes `resolve_generation()` raise `unknown provider` and chat dies. The - migration clears it and the UI states why; the weights are in a blob format - we no longer manage, so there is no silent remap. -3. **`/ollama/` is orphaned and gigabytes.** Offer to delete it. Leaving - it silently is not acceptable in a local-first app. - -**Definition of done:** - -```bash -grep -rn -i ollama surfsense_local \ - --exclude-dir=node_modules --exclude-dir=release --exclude-dir=.venv -``` +`supports_system_role` is read generously, defaulting to `True` when absent, +because a runtime that does not report it is more likely to be old than to be +incapable, and dropping the system message takes the grounding and the citation +instructions with it while the model answers exactly as confidently as it would +have otherwise. Where a template genuinely has no system role, `for_template()` +folds the system content into the first user turn rather than losing it. The +adapter downgrades at that seam so `modules/chat` never learns that templates +differ. -returns only the `connection-form.tsx` preset and its test, revisions `0004` -and `0009`, and the new migration's `WHERE destination = 'ollama_pull'`. +`vision` requires **both** halves: `IMAGE in inputs` and `typed_content`. A model +can accept images architecturally while its template takes only string content, +which leaves no way to send it one. It is the only capability that reaches the +UI; `system_role`, `typed_content` and `tools` are constraints that change how a +request is built and mean nothing to a person. -**Tests:** a migration test over a database seeded with an Ollama selection and -an `ollama_pull` egress row, asserting the grant lands on `model_download` and -that `model_search` is **not** created. +### Constrained decoding -> **`test_ollama_num_ctx.py` is deleted here and its subject is not.** Those two -> tests assert that a full Studio grounding gets a window large enough to hold -> it, which is a real regression guard: Ollama's 4096 default silently dropped a -> 24k-character grounding to a third of its length, measured. llama.cpp fixes -> context at load, so `num_ctx()` has no successor, but the behaviour does. The -> replacement belongs in 7.2, against the context floor: assert the 16K floor -> covers `HISTORY_BUDGET_TOKENS` plus `build_context`'s grounding budget plus a -> reply, so the floor is checked against its stated derivation rather than -> asserted as a constant. Write it before deleting the old one. +48 of the 69 prompt files demand a JSON object. `response_format` with a +`json_schema` masks every token that would produce invalid JSON, so a malformed +answer stops being something to repair and becomes something that cannot be +emitted. Chat prose is deliberately unconstrained. -### 7.7 — Capabilities and multimodal +The schema is not injected into the prompt. Fields are still described, once, +because compliance no longer depends on persuasion. -`Capabilities(inputs, tools, system_role, typed_content, context_tokens)` at the -`Generator` seam, from `GET /models.architecture.input_modalities` and -`GET /props.chat_template_caps` — template-derived truth, not a name regex. +## Packaging -**Two kinds, two sources, and only one of them reaches the UI.** +`electron/scripts/fetch-llamacpp.mjs`, pinned to build `b11050` **and to a +SHA-256 per target**, which closed the gap left by the previous fetcher. -```text -GET /models → architecture.input_modalities what the model can ACCEPT -GET /props → chat_template_caps what the TEMPLATE supports -``` - -| | Field | Surfaced? | +| Target | Asset | Size | |---|---|---| -| **Capability** | `vision` | **yes** — a row badge and a search facet | -| **Constraint** | `supports_system_role` | no — changes how the request is built | -| **Constraint** | `supports_typed_content` | no — precondition for vision | -| Recorded, unused | `tools`, `tool_calls` | no | -| Recorded, unused | reasoning support | no | +| darwin-arm64 | `bin-macos-arm64.tar.gz` | 11.2 MB | +| win32-x64 | `bin-win-vulkan-x64.zip` | 31.8 MB | +| linux-x64 | `bin-ubuntu-vulkan-x64.tar.gz` | 30.4 MB | + +The Vulkan archive is the CPU archive plus exactly one file, and all CPU +micro-architecture variants ship inside: 15 on Windows, 10 on Linux. The staging +step prunes to `llama-server` plus its libraries, 24 executables down to 1, which +also shrinks the macOS notarization surface. + +- **Linux**: `libggml-vulkan.so` has `DT_NEEDED: libvulkan.so.1`. The deb adds + `libvulkan1` to `Depends`; the AppImage bundles the loader or accepts CPU, + since the ICD must come from the host driver. +- **`ubuntu-22.04` is pinned** in the workflow. `ubuntu-latest` migrates and + would silently raise the AppImage's glibc floor. +- **Windows**: `vulkan-1.dll` is present in `System32` on a stock install, so + nothing needs bundling. `taskkill /PID /T /F` walks the router's tree; + verified with a model resident at `b11050`, four processes terminated depth + first with no survivors. +- **macOS and Linux** reap through process groups and signals rather than + `taskkill /T`. + +`bundling/api.spec` ships `modules/llm/catalog/curated-models.json` at +`modules/llm/catalog`. It is read by path, so the analyser cannot see it, and the +path is asserted by a `packaging`-marked test because the previous value pointed +at a directory this branch deleted. -One struct is fine; one *list that reaches the renderer* is not. -`supports_system_role` is meaningless to a person and `typed_content` is an -implementation detail. The manifest's `capabilities` array carries only the -user-facing set, which today means `["vision"]` and nothing else. +## Failure behavior -**`supports_system_role` matters immediately.** `build_messages()` always emits a -system message and silently degrades on templates that do not support one. That -is a live bug the swap fixes. +- **Runtime missing or crashed**: the catalog stays visible, installs are + disabled, and an already-selected remote connection still answers. +- **No Vulkan loader**: `dlopen` fails, the backend is skipped silently, CPU + runs. +- **A GPU exists and ggml cannot see it**: reported as `broken_install` rather + than badged as a CPU-only machine. See **GPU status**. +- **Quantized KV without working flash attention**: `q8_0` requires it, and when + the fused kernel is unavailable llama.cpp falls back to CPU attention silently, + with the device near 0% utilisation. Both cache types are set together or + neither, which removes the one trigger we control. +- **HF unreachable**: curated and installed render from the manifest and disk; + search reports the destination is unavailable, which is a `503` naming the + host, not an error dialog. +- **Header read truncated**: retry wider once, then fall back to a file-size + estimate with the badge marked approximate. +- **A broken manifest**: the service substitutes an empty manifest rather than + failing to start, so installed models and a local `.gguf` import still work. +- **An unreadable file in the models directory**: skipped with a warning when the + preset is written. It costs that one model; failing would leave the runtime + dead over a file nobody asked it to load. +- **`TOO_BIG`**: state the required and available bytes and name a smaller build. + Never a bare disabled control. +- **Cancelled download**: the partial file stays as `.part`, the model is never + discovered, and the next attempt resumes. +- **Dead curated pin**, where the repo or file 404s at download: say the model is + no longer available from this source and fall through to the next-best entry. + This has to be handled at runtime rather than checked at build time, because a + repo can disappear between the release and a user's first install. + +> **Three of these are the same failure wearing different clothes**, and the +> pattern is worth naming: llama.cpp degrades quietly and exits 0. The probe's +> backend search path, a missing backend dependency, and `-ngl` disabling the +> fitter all produce a working-looking system that is running on the CPU. Assume +> nothing worked until something says it did. -**Vision needs both halves.** "The model accepts images" is the obvious check and -it is not sufficient: +## Tests -```python -can_see = Modality.IMAGE in caps.inputs and caps.typed_content -``` +2,661 unit tests pass across `tests/unit/llm` and `tests/unit/chat`. The ones +worth knowing about, because each encodes a failure that actually happened: -A model can accept images architecturally while its chat template takes only -string content, leaving no way to hand it one. Checking `input_modalities` alone -produces a vision badge on a model that cannot be sent a picture. +**Estimator** -**What is deliberately absent.** `audio` and `video` are not modelled, because -nothing can feed them: Docling's `allowed_formats` in `worker/ingestion/parsing.py` -is PDF, DOCX, PPTX, XLSX, HTML, CSV, MD and **IMAGE**. Podcasts are audio *out* -and Kokoro owns that; audio *in* would need ingestion to accept and transcribe -media first, at which point `audio` becomes relevant and not before. `tts` and -`embedding` are fixed (Kokoro, and bge-small against a 384-dimensional index). +- The two subtractions stay on their own sides: a profile where + `need <= device_free` but `need > device_free - fit_reserve` asserts `PARTIAL`, + which is the real RTX 3050 case and the one a collapsed `overhead` term gets + backwards. +- `compute_buffers` is part of `need`: removing the term flips at least one + fixture from `PARTIAL` to `FITS`. +- The Vulkan 4B measurement is not under-predicted, and a header stating no + widths never prices scratch at zero. +- KV cells round up to the 256 boundary (16385 becomes 16640); the measured Qwen3 + numbers stay exact; a latent model prices one compressed head. +- A header's sliding-window period beats the built-in table; a per-layer array is + exact; `shared_kv_layers` allocate nothing. +- `PARTIAL` is reachable on a unified budget and unreachable with no GPU; + `can_install` is true for `PARTIAL` and false only for `TOO_BIG`. +- A property sweep over four shapes, five windows, two precisions and five budget + shapes asserts that the items sum to the verdict's `need`, that weights do not + move with the window, that KV and compute are non-decreasing in it, and that a + `PARTIAL` fraction times the block count is an integer. + +**Badge and recommendation** + +- `test_badge_matches_the_load.py` sweeps every curated model against every + budget shape and asserts a starred row's badge is never spill wording. +- A `PARTIAL` build below the light-spill ceiling is starrable, and a + higher-ranked spilling build beats a fully resident lower-ranked one. +- `TOO_BIG` is refused regardless of rank: the tier relaxes residency, never + physics. +- A CPU-only machine still gets a star. +- A quantized cache is allowed to rescue a larger model, because the star has to + name the model the loader will actually run. + +**Hardware** + +- Device lines round-trip through a description containing a tab. +- The three GPU-status rows, plus integrated-only and OS-unknown. +- A probe raising `OSError` while the OS reports a card is `broken_install`. +- Two `ctypes` handles on Windows, one elsewhere. +- Two threads asking for the inventory produce exactly one probe call. + +**Catalog and runtime** + +- Curated renders with no network at all; search is `403` with egress off. +- A search row carries no `rank` field, asserted on the serialized response so it + cannot be reintroduced silently. +- A repo whose architecture is unsupported short-circuits with no `Range` request + reaching the mock transport. +- A 500 on the resolve URL yields `approximate` builds rather than a failed repo. +- A projector file never gets its own preset section, and the model's section + names it. +- The preset writes `fit-target = 1024`, or 1624 with a 600 MiB projector. +- A fake router covers health, models, props, load, tokenize, chat stream and + delete; router mode against an empty models directory asserts + `/props` reports `role: router`. +- Manifest: schema 3 is refused, a shape without `embedding_length` is refused, + two `rank_basis` values are refused, and every shipped entry has positive + widths. + +**Migration**: a database seeded with an Ollama selection and an `ollama_pull` +egress row upgrades so the grant lands on `model_download` and `model_search` is +**not** created. + +No new CI job. The build stays install, freeze, package, sign; neither llmfit nor +`huggingface.co` appears in it. -`IMAGE` being in that list already is what makes vision worth doing at all: -SurfSense accepts standalone images today, on top of PDF pages that are -effectively pictures. A vision model has something to look at on day one. +## Acceptance -`Message.content` widens from `str` to `str | tuple[ContentPart, ...]`. The -adapter **downgrades at the seam**: a text-only model gets the image replaced by -its extracted text, never an error, so `modules/chat` never learns about -modality. +- A clean machine on all three OSes opens the catalog with a hardware line and + badged rows, with no scan button and no network. +- The recommended model installs and answers with citations. +- A model found through search, one Ollama's library never carried, installs and + answers. +- A Windows machine with an RTX card runs on Vulkan from the installer alone, and + the catalog prices against that card rather than the integrated GPU. +- Renaming `ggml-vulkan.dll` makes `/llm/system` report `broken_install`, the + hardware line says the card was not detected, and no row reads *Reduced speed*. + Restoring it returns `present`. +- A machine with no usable GPU runs on CPU with no error and still gets a star. +- Airgapped: curated install from a local `.gguf`, chat, and Studio all work. +- `/health` answers within a second of boot while `/llm/system` on a cold shader + cache takes up to about 25 s. +- After an install, `models.ini` carries the planned `ctx-size` and + `fit-target`, and the server log shows the fitter honouring the target. +- Quit leaves no orphan `surfsense-*` or `llama-server` process. -Retrieval stays text. Ingestion already writes page files; `image_url.url` -accepts a **local file path**, so the path Docling wrote is handed over directly -with no base64 round-trip. `--mmproj-auto` is on by default with `-hf`, so a -vision model brings its own projector. +## What each machine gets -> **ponytail:** [#19980](https://github.com/ggml-org/llama.cpp/issues/19980) — -> `--fit` does not account for mmproj VRAM, so vision models can OOM on tight -> machines. Add the projector bytes to our own sum. +Computed from the shipped manifest through `build_budget`, `plan_load` and +`recommend`, in capacity mode. Cells are `state/offload fraction`. -Touches `Hit`, the chunk model, and citations. Ships after the swap, not with it. +| machine | resident | refusal | 0.6B | 1.7B | 4B | 8B | 14B | 32B | star | +|---|---|---|---|---|---|---|---|---|---| +| M2 8 GB | 4198 MiB | 6144 MiB | fits | fits | fits | 0.33 | too big | too big | 4B | +| RTX 3050 6 GB | 4210 MiB | 34046 MiB | fits | fits | fits | 0.33 | 0.60 | 0.81 | 4B | +| M4 16 GB | 9898 MiB | 14336 MiB | fits | fits | fits | fits | fits | too big | 14B | +| RTX 4070 12 GB | 9976 MiB | 40696 MiB | fits | fits | fits | fits | fits | 0.55 | 14B | +| RTX 4090 24 GB | 21976 MiB | 85464 MiB | fits | fits | fits | fits | fits | fits | 32B | +| M4 Max 64 GB | 48128 MiB | 63488 MiB | fits | fits | fits | fits | fits | fits | 32B | +| No GPU 16 GB | 14336 MiB | 14336 MiB | fits | fits | fits | fits | fits | too big | 14B | -### 7.8 — Constrained decoding and prompt consolidation +Every machine gets a star, including the CPU-only one. Three gaps remain, and all +three are manifest work rather than estimator work: -69 prompt files exist (23 cases × 3 tiers); **48 demand JSON**. -`response_format: {"type": "json_schema", "schema": {…}}` masks every token that -would produce invalid JSON, so malformed output becomes mechanically impossible. +1. **The top collapses.** 24 GB and 64 GB, 2.7x apart, get the same file. A + higher quant does not fix it; 32B at Q8 is about 35 GB and still leaves room. + The top needs a bigger model, and a mixture of experts is the right shape for + those machines: large in memory but reading only a few billion parameters per + token. `decode_fraction` is already computed and shipped for exactly that. +2. **A second variant on 8B and 32B**, say `Q6_K`, would fill the 16 GB band and + give a 4090 somewhere to go. Two extra pins, not fifty, and `variants` is + already a list. +3. **A vision entry.** All six are text-only, in an app with a PDF pipeline and + an ingestion path that already accepts standalone images. -Two caveats to design around now: +## Appendix: measurements -- The schema is **not** injected into the prompt. Fields still get described — - once, not once per tier, because compliance no longer depends on persuasion. -- [#29006](https://github.com/ggml-org/llama.cpp/issues/29006): json_schema on - the chat endpoint returns 400 for some templates while equivalent GBNF on - `/completion` succeeds. Studio needs that fallback path. +Taken against llama.cpp `b11050` (some earlier figures at `b11043`) and llmfit +`1.1.11`, on an M2 8 GB and on an RTX 3050 / Ryzen 5 9600X. -`params_b` becomes exact from the GGUF header, retiring `_estimated_b()`'s -blob-size division and its `gemma3:4b` reads-as-5.5B `ponytail`. +### Catalog scale -Tiers do not disappear, but what they carry changes: after constrained decoding -and template caps, a tier carries **reasoning depth only**, not format -compliance. That plausibly collapses three tiers to two. +| Measurement | Value | +|---|---| +| GGUF repos on Hugging Face | 204,797 | +| Ollama library | 240 models | +| llmfit rows with an `ollama_name` | 138 of 9,590, 1.4% | +| llama.cpp `LLM_ARCH_*` names | 152 upstream; 113 in our shipped gate | +| GGUF repos with a useful pipeline tag | about 43,500, so roughly 80% have none | +| Top 40 GGUF repos carrying `base_model:quantized:` | 32 | +| GGUF header over HTTP Range | 1.79 MB, 2.5 s, for a 135M model | +| macOS DMG before the swap | 1,507 MB | +| `electron/ollama/` staged on macOS | 501 MB, of which 380 MB MLX | -## Failure behavior +### Device probing -- Runtime missing or crashed: catalog stays visible, installs disabled, an - already-selected remote connection still answers. -- No Vulkan loader: `dlopen` fails, backend skipped silently, CPU runs. (Not - observed on Windows — `vulkan-1.dll` is present in `C:\Windows\System32` on a - stock install, so nothing needs bundling there. Linux supplies it through - `libvulkan1`; see 7.5.) -- **Quantized KV without working flash attention: assert, do not assume.** - `q8_0` cache requires flash attention, and when the fused kernel is - unavailable llama.cpp falls back to CPU attention **silently** — no warning, - the device sits near 0% utilisation and throughput collapses. Two known - triggers: asymmetric `-ctk`/`-ctv`, and Vulkan on non-NVIDIA hardware, whose - vendor-neutral `GL_KHR_cooperative_matrix` path was only optimised in mid-2026 - and has an open report of extreme degradation on AMD. So: set both cache types - identically, verify flash attention engaged after load, and fall back to `f16` - — accepting the memory cost — rather than running at CPU-attention speed. - **This is the third silent-degradation case in this phase**, after the probe's - backend search path (7.2) and a missing backend dependency below. The pattern - is constant: llama.cpp degrades quietly and exits 0. Assume nothing worked - until something says it did. -- **A GPU exists and ggml cannot see it: say so.** The silent skip above is - correct when there is no GPU and wrong when there is one, and the two are - indistinguishable from ggml alone — both print `(none)` and exit **0**, with no - warning even under `GGML_BACKEND_DEBUG=1`. Two real causes: the probe ran - outside the library directory (7.2), or a backend library's dependencies are - missing from the package. Cross-check against the OS: - - | OS reports a GPU | ggml reports a GPU | Meaning | - |---|---|---| - | yes | yes | normal | - | no | no | genuine CPU-only machine, badge accordingly | - | **yes** | **no** | **broken install.** Say so; never badge CPU-only. | - - Sources: `Win32_VideoController` on Windows, `/sys/class/drm` on Linux. Filter - virtual adapters — the test machine carried a `Parsec Virtual Display Adapter` - beside two real GPUs. Odysseus codes around the same failure explicitly - (*"nvcc found but CUDA runtime is not visible"*), which is independent evidence - it is common enough to handle rather than assume away. -- HF unreachable: curated and installed render from the manifest and disk; - search reports the destination is unavailable, not an error. -- Header read truncated: retry wider once, then fall back to a file-size estimate - with the badge marked approximate. -- `TOO_BIG` (the physics refusal): state required and available bytes and name a - smaller quantization or a smaller model. Never a bare disabled control. -- Cancelled download: `POST /models/unload`, partial file removed, never selected. -- **Dead curated pin** (the repo or file 404s at download): say the model is no - longer available from this source, drop it from the recommendation, and fall - through to the next-best entry. This must be handled here rather than checked - at build time — a repo can disappear between the release and a user's first - install, so no pre-flight check can prevent it. +| Measurement | Value | +|---|---| +| `ggml_backend_load_all()`, M2 | 2.3 ms | +| first `ggml_backend_dev_count()`, M2 (compiles 20 Metal libraries) | **19.0 s** cold, 43 to 49 ms warm | +| `ggml_backend_load_all()`, one backend | 77 ms Windows, 194 ms Linux | +| `ggml_backend_load_all()`, two backends | 205 ms | +| `ggml_backend_dev_memory()` first call, CUDA / Vulkan / CPU | 47.89 / 3.50 / 0.01 ms | +| `llama-server --list-devices` | 174 ms cold, 60 ms warm | +| CPU device, native Windows | 31884.6 MiB total, 22750.2 MiB free | +| CPU device, WSL2 | 26048.6 MiB total, 26048.6 MiB free, virtualised | +| CPU device, macOS | 8192.0 / 8192.0 MiB on a machine with about 2 GB free | +| M2 `recommendedMaxWorkingSetSize` | 5726.63 MB = 5461.3 MiB, which is MTL0's reported total | +| M2 devices listed | `MTL0` (GPU), `BLAS` (ACCEL, raw type 3), `CPU` | +| RTX 3050, `nvidia-smi` idle | 6144 MiB total, 5699 MiB free | +| RTX 3050, ggml after context init | 6143.5 MiB total, 5158 MiB free | +| Consumed before any weight loads | **986 MiB**, about 445 desktop and 541 context | -## Tests +That last row is why the budget starts from ggml's `free` and not from a +nameplate total. `free` already excludes the desktop's allocation and the backend +context, so adding an overhead term on top charges the same memory twice. -- GGUF reader: truncated file, oversized vocab, split parts, Range retry. -- Estimator: decision-table over constructed profiles; f16 vs q8_0 KV (a 1 GB - swing on an 8 GB machine, measured). -- **KV precision is chosen, not defaulted**: a profile where `f16` fits selects - `f16`; one where only `q8_0` fits selects `q8_0` and records why; one where - neither fits is `PARTIAL` or `TOO_BIG` as the sizes dictate. -- **`-ctk` and `-ctv` are always equal** in any launch argument set the provider - builds — asserted on the spawned command line, since a mismatch silently - disables the fused kernel. -- **The two subtractions stay on their own sides.** A profile where - `weights + KV + compute ≤ device_free` but `> device_free − fit_reserve` - asserts **`PARTIAL`**, not `FITS` — the real RTX 3050 case, and the one a - single collapsed `overhead` term gets backwards. -- **An unmeasured `(platform, backend)` row rounds up.** A budget for a platform - with no measured reserve never yields a *more* optimistic verdict than the - measured Windows row on equivalent memory. -- `compute_buffers` is part of `need`: removing the term flips at least one - fixture from `PARTIAL` to `FITS`, which is the regression to catch. -- Fit states: each of `FITS` / `PARTIAL` / `TOO_BIG` at its boundary, ±1 byte; - `TOO_BIG` evaluated at the 16K floor, not the requested window; `PARTIAL` - reachable on a `uma` budget (the Hermes divergence — a regression here silently - refuses Qwen3 8B on an 8 GB Mac); `PARTIAL` unreachable with no GPU device; - `can_install` true for `PARTIAL` and false only for `TOO_BIG`; copy resolves to - the right platform wording for all three budget shapes. -- Recommendation policy: the six hardware profiles above, asserting both pick - and reason key; and that a searched row is never eligible to be the pick. -- **A `PARTIAL` pair below `MAX_OFFLOAD` is starrable.** The regression fixture - is the measured RTX 3050: a higher-ranked model at `f ≈ 0.28` beats a fully - resident lower-ranked one. A `FITS`-only policy fails this test, which is the - point of having it. -- **`offload_fraction` survives to the verdict** and is not recomputed by the - renderer; the graded reason line is selected from it, and the same fraction - drives both the gate and the copy. -- `TOO_BIG` is still refused regardless of rank: the ceiling relaxes residency, - never physics. -- Catalog: curated renders with no network; search gated on egress; arch, - template, and gated-repo rejections; a search row carries no `rank` field at - all — asserted on the serialized response, so it cannot be reintroduced - silently. -- Manifest (extends `test_curated_models.py`, which already covers unique ids, - display metadata, and each rejection path): every entry has a `shape` and at - least one variant; every variant has a `quantization`, a `rank` and a - `rank_basis`; every `shape.architecture` is in the 152-name list; a curated - row prices and badges with no network at all. Note - `test_packaged_manifest_has_eight_unique_curated_models` asserts `len == 8` - and `advisor_providers == ["Alibaba"]` — both become wrong, and both are - updated in the same commit as the manifest, never before. -- **A model with two variants is one row**, taking the best state and rank among - them, and recommending the variant that produced them. Asserted with a fixture - carrying a `TOO_BIG` large build and a `FITS` small one: the row is - recommendable on the smaller build. -- **Ranks are never compared across `rank_basis` values.** A fixture mixing an - `llmfit-*` rank with an eval-derived one is rejected rather than silently - sorted. -- **`rank` rises with parameter count within a family.** Today this passes - trivially — the shipped six are one family and score `33 · 48 · 63 · 78 · 85 · - 92` at the pinned Q4_K_M. The first failure is the signal to look, and it is expected: a - mixture-of-experts, a second dense family, or a specialist ranked for document - Q&A rather than for what it was tuned on will all break size-ordering - legitimately. The assertion exists to make that a decision, not a drift. -- **Authoring: every committed `rank` was taken at that entry's pinned - quantization.** The ladder sweep records which rung it read; the script fails - non-zero when the pinned rung is absent rather than grading a neighbour. This - is a script-level test, not an app test — nothing in the product can check it, - which is exactly why it is written down. -- A curated entry with a `vision` capability is never preferred over a - higher-ranked text model for the ★; capabilities do not enter the ordering. -- Capabilities: `vision` requires **both** `IMAGE in inputs` and - `typed_content`; a model with image input and a string-only template reports - no vision badge. `supports_system_role` false routes the system message into - the first user turn rather than dropping it. Only the user-facing set reaches - the serialized response — `system_role` and `typed_content` never appear in - an API payload. -- **Device selection**: `select_device()` over fixtures — a discrete 6 GB card - beside a 16 GB integrated one selects the **discrete** card; `ACCEL` alone - yields no GPU; `usable_vram_bytes` never exceeds any single device's memory - (asserted against a listing where the same card appears twice, which a naive - sum turns into 12 GB on a 6 GB card); a CPU-only listing returns `None`. -- **Library loading**: two `ctypes` handles on Windows, one on Linux. A - regression is an `AttributeError` at startup, not a wrong answer, so it is - worth an explicit test rather than leaving it to the first packaged run. -- **Probe working directory**: devices are found when the probe runs from the - library directory and **not** found when it does not. The second assertion is - what stops the search-path bug from returning; without it the test passes in a - dev shell and the bug ships. -- **Broken-install detection**: the three OS-versus-ggml rows in **Failure - behavior**; the mismatch row produces a distinct error state, never a - `has_gpu: false` budget. -- Runtime: fake llama-server for health/models/props/load/chat/delete; router - mode against an empty models directory, asserting `/props` reports - `role: router`. -- Migration: an Ollama selection and an `ollama_pull` row, upgraded. -- Packaging: staged checksum; `--list-devices` from the packaged path per OS. - -No new CI job. The build stays install → freeze → package → sign; neither llmfit -nor `huggingface.co` appears in it. +### Fit terms -## Acceptance +Qwen3 4B Q4_K_M at `-c 16384`, `--fit` on, RTX 3050 with 5,234 MiB free, from +llama.cpp's own allocation log: -- A clean machine on all three OSes opens the catalog with a hardware line and - badged rows, with no scan button and no network. -- The recommended model installs and answers with citations. -- A model found through search — one Ollama's library never carried — installs - and answers. -- A Windows machine with an RTX card runs on Vulkan, from the installer alone, - and the catalog prices against that card rather than an integrated GPU. -- A machine with no usable GPU runs on CPU with no error. -- Airgapped: curated install from a local `.gguf`, chat, and Studio all work. -- `grep -i ollama` returns only the intended survivors. -- Quit leaves no orphan `surfsense-*` or `llama-server` process. - -## Appendix — measurements +| buffer | MiB | +|---|---| +| `Vulkan0` model | 2078.04 | +| `CPU_Mapped` model, spilled | 602.16 | +| `Vulkan0` KV | 1984.00 | +| `CPU` KV, spilled | 320.00 | +| `Vulkan0` compute | 143.62 | +| `Vulkan_Host` compute | 26.01 | +| `Vulkan_Host` output | 2.32 | +| **placed on device** | **4205.66** | +| **fit reserve, left unused** | **1028.34** | -Taken on an M2 / 8 GB against llama.cpp `b11043` and llmfit `1.1.11`, except -where noted. +Qwen3 1.7B Q4_K_M at `-c 16384` on the M2: -| Measurement | Value | +| buffer | MiB | |---|---| -| GGUF repos on Hugging Face | 204,797 | -| llmfit embedded database | 11,271 models, 3,454 publishers | -| Ollama library | 240 models | -| llmfit rows with an `ollama_name` | 138 of 9,590 (1.4%) | -| llmfit rows resolvable under `--force-runtime llamacpp` | 1,109 of 6,804 fitting | -| llama.cpp supported architectures | 152 (`LLM_ARCH_*`) | -| `llmfit --json system` | 0.68 s | -| `llmfit --json fit` (full scan) | **0.84 s**, 9,590 rows, 17 MB JSON | -| `llmfit recommend -n 20000 --force-runtime llamacpp` | 0.72 s, 6,804 rows | -| `llama-server --list-devices` | 174 ms cold, 60 ms warm | -| `ggml_backend_dev_memory()` via ctypes | 180 ms warm, ~20 s first call on Metal | -| GGUF header over HTTP Range | 1.79 MB, 2.5 s (135M model) | -| macOS DMG today | 1,507 MB | -| `electron/ollama/` staged (macOS) | 501 MB — of which 380 MB is MLX, 66 MB the Go binary | +| `MTL0_Mapped` model | 1050.43 | +| `CPU_Mapped` model | 243.43 | +| `MTL0` KV | 1792.00 | +| `MTL0` compute | 102.24 | +| `CPU` compute | 24.01 | +| **the fitter's own projection** | **2944** | -Hardware-dependence of llmfit fields, from two simulated machines -(`--memory 8G --ram 8G` versus `--memory 80G --ram 256G`, 3,047 models in common): +`1050 + 1792 + 102 = 2944`, against `projected to use 2944 MiB`. The three-term +`need` is exact on Metal, as it was on Vulkan. -| Field | Changes with hardware | -|---|---| -| `parameter_count`, `params_b`, `context_length`, `license`, `use_case`, `capability_ids`, `runtime` | **no** — identical | -| `best_quant`, `memory_required_gb`, `estimated_tps`, `fit_level`, `score_components.quality` | yes | +**243.43 MiB stayed on the CPU at "offloaded 29/29 layers to GPU".** The +non-repeating tensors are host-side even at full offload, so 29/29 does not mean +all weights on the device, and an offload fraction derived from the layer count +alone reads 0.00 while 19% of the file is elsewhere. + +**Metal reports `0.00 MiB` buffer sizes during the fitter's probe pass**, before +the real load, so anything scraping buffer sizes must ignore the first pass. +`use shared buffers = true` also means Metal allocations are not the separate +arenas Vulkan and CUDA report. -Holding the quantization fixed (2,424 models): **quality differs 0/2424, memory -differs 0/2424.** Quality and memory are functions of *(model, quantization)*, -not of hardware — hardware only enters by choosing the quantization. That is -what makes a pinned manifest's numbers deterministic and build-time computable. +### The estimator against those measurements -**The same result on a single model, swept.** The population statistic above says -hardware does not move quality. This says *why*, and it is the observation the -whole authoring design rests on. Qwen3 8B, `--ram 64G` held fixed, sweeping the -device budget: +Current code, same shapes and windows: -| `--memory` | `best_quant` | `quality` | `estimated_tps` | `score` | +| | KV predicted | KV measured | compute predicted | compute measured | |---|---|---|---|---| -| 6G | Q3_K_M | 75 | 17.9 | 65.1 | -| 7G | Q4_K_M | 78 | 13.4 | 63.8 | -| 8G | Q5_K_M | 81 | 10.7 | 63.6 | -| 10G | Q6_K | 82 | 9.0 | 66.2 | -| 12G | Q8_0 | **83** | 6.7 | 63.6 | -| 16G | Q8_0 | **83** | 6.7 | 67.4 | -| 24G | Q8_0 | **83** | 6.7 | 67.4 | -| 48G | Q8_0 | **83** | 6.7 | 67.4 | +| 4B at 16384, Vulkan | 2304 MiB | 2304 MiB | 172 MiB | 170 MiB | +| 1.7B at 16384, Metal | 1792 MiB | 1792 MiB | 138 MiB | 126 MiB | +| 1.7B at 40960, Metal | 4480 MiB | 4480 MiB | 258 MiB | 246 MiB | -Read the last four rows: 12 GB to 48 GB, four times the memory, and `quality` -does not move. **It stops moving the instant the quantization stops moving.** -Hardware reaches quality through exactly one channel, `best_quant`, and that -channel is one we close by pinning the file. - -Three decisions follow from that, and this table is the reason for all three: - -1. **`rank` is authored once and shipped frozen.** A number that does not vary - with the machine is data, not a per-machine computation. There is nothing for - a scan on the user's device to discover. -2. **Only `score_components.quality` is read; `score` is discarded.** In the same - sweep `score` runs 65.1, 63.8, 63.6, 66.2, 63.6, 67.4 — non-monotonic, and - moving with hardware because it blends fit and speed into the quality term. - The composite is a statement about a laptop; the component is a statement - about a file. -3. **The declared budget must land llmfit on the pinned quantization.** Since - quality tracks `best_quant` exactly, scoring at the wrong budget scores the - wrong file. See **Authoring**, where this is a hard assertion rather than a - convention. - -**Windows and Linux, measured on an RTX 3050 / Ryzen 5 9600X at `b11050`.** -Both platforms were previously reasoned from documentation only. +KV is exact. Compute is over in every case, which is the safe direction and the +reason the safety factor is 1.20. -| Measurement | Value | -|---|---| -| `ggml_backend_load_all()`, one backend | 77 ms (Win) · 194 ms (Linux) | -| `ggml_backend_load_all()`, two backends | 205 ms | -| `ggml_backend_dev_memory()` first call, CUDA | 47.89 ms | -| `ggml_backend_dev_memory()` first call, Vulkan | 3.50 ms | -| `ggml_backend_dev_memory()`, CPU | 0.01 ms | -| RTX 3050, `nvidia-smi` idle | 6144 MiB total, 5699 MiB free | -| RTX 3050, ggml after context init | 6143.5 MiB total, 5158 MiB free | -| Consumed before any weight loads | **986 MiB** (~445 desktop, ~541 context) | -| CPU device, native Windows | 31884.6 MiB total, 22750.2 MiB free | -| CPU device, WSL2 | 26048.6 MiB total, **26048.6 MiB free** (virtualised) | -| `vulkan-1.dll` on stock Windows | present in `System32`, nothing to bundle | +> **One unexplained gap, recorded rather than smoothed.** The model buffers in +> both logs exceed the pinned file sizes: 2680 MiB against 2382 for the 4B, and +> 1294 MiB against 1056 for the 1.7B. The likeliest explanation is that the +> measured runs used a different publisher's build of the same quantization, +> since the logs name a quant but not a repo. It is worth re-measuring against +> the exact pinned file before concluding the weights term needs a correction, +> because if it does, it is a 12 to 22% under-estimate on the largest term. + +### Offload and speed -**Offload sweep**, Qwen3 4B Q4_K_M, `-ngl` controlling `f` directly, f16 KV, -`-p 512 -n 300`, Vulkan, RTX 3050: +Qwen3 4B Q4_K_M, `-ngl` controlling the fraction directly, f16 KV, `-p 512 +-n 300`, Vulkan, RTX 3050: | ngl | `f` | pp512 | tg300 | |---|---|---|---| @@ -2250,20 +1799,17 @@ Both platforms were previously reasoned from documentation only. `ngl 0` edging out `ngl 9` is real: splitting across devices costs transfers that outweigh nine layers of device work. -**KV precision.** Resident, depth 0: `f16` pp8192 1437.66 / tg300 52.47 against -`q8_0` 1300.44 / 51.34. +The slowdown reduces to `1 / (f x r + 1 - f)` with `r = BW_gpu / BW_cpu`, and +every point fits `r` near 4 within ±20%. Cross-checked against absolutes: 131 +GB/s effective on the device, 78% of the card's rating, against 33 GB/s on the +host, ratio 4.0. On a high-bandwidth card `r` is far larger, roughly 12 on a +4090, so the same fraction costs much more; and on unified memory `r` approaches +1 and spilling barely means anything. A threshold calibrated on a 3050 is +therefore not automatically conservative elsewhere. -> **That comparison is at the wrong depth and should not be quoted as q8_0's -> cost.** At depth 0 the cache is nearly empty, so its precision cannot matter; -> published CUDA figures showing ~18% at 8K depth were measured under different -> conditions. The valid comparison — `f16` at 16K depth — **could not be taken**, -> because it fails to allocate on this card. What is known: `q8_0` at depth 16384 -> decodes at 27.46 t/s against 51.34 at depth 0, and that halving is the cost of -> attending over a long context, not of quantizing it. **q8_0's real cost at -> depth on Vulkan is unmeasured**; it needs a card with enough VRAM to run `f16` -> at 16K. +### KV precision -**The decisive result.** `f16` KV at a 16K window on a 6 GB card: +The decisive result, `f16` KV at a 16K window on a 6 GB card: ```text ggml_vulkan: Device memory allocation of size 316407808 failed. @@ -2272,134 +1818,103 @@ llama_bench: error: failed to create context ``` `q8_0` at the same depth runs. On this class of card the planned configuration -does not fit, which is what makes KV precision a fit-calculation output rather -than a default. +does not fit, which is what makes cache precision an output of the fit +calculation rather than a global default. -**Fit terms, measured.** Qwen3 4B Q4_K_M at `-c 16384`, `--fit` on, RTX 3050 -with 5,234 MiB free, from llama.cpp's own allocation log: +Resident at depth 0: `f16` pp8192 1437.66 / tg300 52.47 against `q8_0` 1300.44 / +51.34. -| buffer | MiB | -|---|---| -| `Vulkan0` model | 2078.04 | -| `CPU_Mapped` model (spilled) | 602.16 | -| `Vulkan0` KV | 1984.00 | -| `CPU` KV (spilled) | 320.00 | -| `Vulkan0` compute | 143.62 | -| `Vulkan_Host` compute | 26.01 | -| `Vulkan_Host` output | 2.32 | -| **placed on device** | **4205.66** | -| **fit reserve (unused)** | **1028.34** | - -**The KV formula is exact, not approximate.** -`2 × 36 layers × 8 kv_heads × 128 × 2 bytes × 16384` = **2304 MiB**, against -`1984 + 320` measured. Qwen3 0.6B independently: predicted `378 + 1792 + ~60` = -2,234 MiB, measured VRAM delta **2,234 MiB**. That is the evidence that a curated -row can be priced offline from committed `shape` fields alone. +> **That comparison is at the wrong depth and must not be quoted as q8_0's +> cost.** At depth 0 the cache is nearly empty, so its precision cannot matter. +> The valid comparison, `f16` at 16K depth, could not be taken because it fails +> to allocate on this card. What is known: `q8_0` at depth 16384 decodes at 27.46 +> t/s against 51.34 at depth 0, and that halving is the cost of attending over a +> long context, not of quantizing it. -**Apple Silicon, measured on an M2 / 8 GB at `b11050`.** The platform the -reserve table listed as unmeasured. Qwen3 1.7B Q4_K_M, `llama-server -v`. - -| Measurement | Value | -|---|---| -| `ggml_backend_load_all()` | **2.3 ms** | -| first `ggml_backend_dev_count()` (compiles 20 Metal libraries) | **19.0 s** cold, 43 to 49 ms warm | -| `ggml_backend_dev_memory()` | 0.02 ms | -| devices listed | `MTL0` (GPU), `BLAS` (ACCEL, raw type 3), `CPU` | -| `recommendedMaxWorkingSetSize` | 5726.63 MB = **5461.3 MiB**, which is what ggml reports as MTL0 total | -| MTL0 total / free | 5461.3 / 5461.0 MiB — **not a live figure** | -| CPU device total / free | 8192.0 / 8192.0 MiB — physical RAM, twice, on a machine with ~2 GB free | -| fit margin the fitter names | **1024 MiB** | - -At `-c 16384`, `--parallel 1` would have been the right flag and was not passed; -llama-server defaults to **4 slots**, so pass `--parallel 1` or the KV cache is -sized for concurrency this app does not use. - -| buffer, `-c 16384` | MiB | -|---|---| -| `MTL0_Mapped` model | 1050.43 | -| `CPU_Mapped` model | 243.43 | -| `MTL0` KV | 1792.00 | -| `MTL0` compute | 102.24 | -| `CPU` compute | 24.01 | -| `CPU` output | 2.32 | -| **fitter's own projection** | **2944** | +### Backend comparison -`1050 + 1792 + 102 = 2944`, against the fitter's `projected to use 2944 MiB`. -**The three-term `need` is exact on Metal**, as it was on Vulkan. +Qwen3 4B Q4_K_M, `-ngl 99`, same card: -**The KV formula is exact on Metal too.** `2 × 28 layers × 8 kv_heads × 128 × -2 bytes × 16384` = **1792 MiB**, reported as `1792.00`. At 40,960 cells it is -`4480.00`, again exact. Compute buffers scale with context as stated: 102.24 MiB -at 16K, 222.24 MiB at 40K. - -**243.43 MiB stayed on the CPU at "offloaded 29/29 layers to GPU".** The -non-repeating tensors are host-side even at full offload, so *29/29 offloaded* -does not mean *all weights on the device*, and an `offload_fraction` derived from -the layer count alone will read 0.00 when 19% of the file is elsewhere. - -**Metal reports `0.00 MiB` buffer sizes during the fitter's probe pass**, before -the real load. Anything scraping buffer sizes must ignore the first pass, and -`use shared buffers = true` means Metal allocations are not the separate arenas -Vulkan and CUDA report. +| test | CUDA 13.4 | Vulkan | CUDA advantage | +|---|---|---|---| +| `pp512` | 1929.18 ± 8.84 | 1767.62 ± 1.43 | +9.1% | +| `pp8192` | 1537.12 ± 5.84 | 1394.77 ± 5.04 | +10.2% | +| `tg300` | 52.19 ± 0.22 | 51.13 ± 0.11 | +2.1% | -**`--fit` searches rather than solves.** At `-c 40960`, needing 5752 MiB against -5460 free with a 1024 margin: `29/29 → 0/29 → 29/29 → 22/29 → 23/29`, reloading -at each step, settling at 23/29 (`f ≈ 0.21`). It never reduced the context, -because `-c` was explicit. +0.66 s on an 11 s turn, for 685 MB of payload. Full working in +[`08-cuda-backend.md`](08-cuda-backend.md). -**Router lifecycle.** Router up with three models discovered: VRAM unchanged from -idle. Worker spawn: `--port 0 --model `. Idle eviction ~30 s without -`--sleep-idle-seconds`. Grandchild reaping via `taskkill /T /F`: four processes, -depth first, no survivors. +### Why `rank` can be authored at all -**`-ngl` disables `--fit`.** With `-ngl 99`: `common_fit_params: failed to fit -params to free device memory: n_gpu_layers already set by user to 99, abort`, -after which the model loaded entirely on CPU with 478 MiB of VRAM touched, no -error, exit 0. +This is the load-bearing fact under the whole authoring design. Qwen3 8B, +`--ram 64G` held fixed, sweeping the device budget: -**`POST /models/load` ignored `{"args": ["-c","16384"]}`** — workers came up at -`n_ctx_slot` 34304 / 28160 / 12288 (model defaults). The same flag on the -`llama-server` command line **is** honoured (`n_ctx_slot = 16384`). See 7.3. +| `--memory` | `best_quant` | `quality` | `estimated_tps` | `score` | +|---|---|---|---|---| +| 6G | Q3_K_M | 75 | 17.9 | 65.1 | +| 7G | Q4_K_M | 78 | 13.4 | 63.8 | +| 8G | Q5_K_M | 81 | 10.7 | 63.6 | +| 10G | Q6_K | 82 | 9.0 | 66.2 | +| 12G | Q8_0 | **83** | 6.7 | 63.6 | +| 16G | Q8_0 | **83** | 6.7 | 67.4 | +| 24G | Q8_0 | **83** | 6.7 | 67.4 | +| 48G | Q8_0 | **83** | 6.7 | 67.4 | -GPU backend comparison, same card and model (Qwen3 4B Q4_K_M, `-ngl 99`): +Read the last four rows: four times the memory, and quality does not move. **It +stops moving the instant the quantization stops moving.** Hardware reaches +quality through exactly one channel, `best_quant`, and pinning the file closes +it. Confirmed across the population too: holding the quantization fixed over +2,424 models, quality differs in 0 of 2,424 and memory in 0 of 2,424. -| test | CUDA 13.4 | Vulkan | CUDA advantage | -|---|---|---|---| -| `pp512` | 1929.18 ± 8.84 | 1767.62 ± 1.43 | +9.1% | -| `pp8192` | 1537.12 ± 5.84 | 1394.77 ± 5.04 | +10.2% | -| `tg300` | 52.19 ± 0.22 | 51.13 ± 0.11 | +2.1% | +Three consequences. `rank` is authored once and shipped frozen, because a number +that does not vary with the machine is data and a scan on the user's device has +nothing to discover. Only `score_components.quality` is read and `score` is +discarded, because `score` runs 65.1, 63.8, 63.6, 66.2, 63.6, 67.4 in the same +sweep, non-monotonic and moving with hardware since it blends fit and speed into +the quality term: the composite is a statement about a laptop, the component is a +statement about a file. And the declared budget must land llmfit on the pinned +quantization, which is why the authoring script sweeps the ladder and indexes it +rather than taking one reading. -0.66 s on an 11 s turn, for 685 MB. Full working in -[`08-cuda-backend.md`](08-cuda-backend.md) — **Appendix**, including why the -earlier 36–40% figure was retired. +### Router lifecycle -Asset sizes at `b11050`, verified against the live release: macos-arm64 11.2 MB, -win-vulkan 31.8 MB, ubuntu-vulkan 30.4 MB — the figures in **7.5** are accurate. +| Measurement | Value | +|---|---| +| Router up, three models discovered | VRAM unchanged from idle | +| Worker spawn argv | `--port 0 --model ` | +| Idle eviction without `--sleep-idle-seconds` | about 30 s | +| Model dropped into a running router's directory | still invisible after 26 s | +| Sidecar restart to pick up a preset change | 0.15 s | +| Grandchild reaping, `taskkill /T /F` | four processes, depth first, no survivors | +| `POST /models/load` with `{"args": ["-c","16384"]}` | ignored; worker used the model default | +| The same flag on the command line | honoured, `n_ctx_slot = 16384` | -Device query versus advisor, same machine: +### Device query against the advisor it replaced | | llmfit | `ggml_backend_dev_memory()` | |---|---|---| -| Memory budgeted | 8.0 GB | **5,461 MiB** (Metal working set) | -| Context priced | 8,192 | 16,384 (our real turn) | -| Qwen3 8B verdict | **Perfect** | **uses system RAM** | +| Memory budgeted | 8.0 GB | 5,461 MiB, the Metal working set | +| Context priced | 8,192 | the window we actually load | +| Qwen3 8B verdict | **Perfect** | **spills to system RAM** | + +This is the failure the whole phase exists to delete: a confident label produced +from a memory figure that was not true on that machine. -## Follow-on work, with the measurement each one waits on +## Open measurements and deferred work -The estimator's terms are implemented and its two verdict bugs are fixed. Three -things are deliberately left, each because it needs a number nobody has taken -rather than code nobody has written. +Three things are deliberately left. Each needs a number nobody has taken rather +than code nobody has written. -### Measurement: a quantized cache against a small spill +### A quantized cache against a small spill `plan_load` drops to `q8_0` whenever that turns a spill into residency. The assumption underneath is that a resident model at `q8_0` beats a partly spilled -one at `f16`, and it has never been measured. Unsloth measured `q8_0` generation -at **35 % slower** and ships it off by default, which is evidence the assumption -can be wrong. +one at `f16`, and it has never been measured here. Published figures for +quantized caches report materially slower generation, which is evidence the +assumption can be wrong. -One model, one window, three configurations, on the two machines already used -for the rest of this phase (RTX 3050 under Vulkan, M2 8 GB under Metal): +One model, one window, three configurations, on the two machines already used for +the rest of this phase: | | cache | context | expected placement | |---|---|---|---| @@ -2407,46 +1922,68 @@ for the rest of this phase (RTX 3050 under Vulkan, M2 8 GB under Metal): | B | `q8_0` with `flash-attn on` | 16384 | fully resident, 36/36 | | C | `f16` | 8192 | fully resident, the control | -Qwen3 4B Q4_K_M, `parallel 1`, `fit-target 1024`. Three prompts of roughly 2K, -8K and 14K grounding tokens, three runs each, median of -`prompt_tokens_seconds` and `predicted_tokens_seconds` from `/metrics`. +Qwen3 4B Q4_K_M, `parallel 1`, `fit-target 1024`. Three prompts of roughly 2K, 8K +and 14K grounding tokens, three runs each, median of `prompt_tokens_seconds` and +`predicted_tokens_seconds` from `/metrics`. **The rule, fixed before the run so the result cannot be argued with.** If B -decodes at 0.9x of A or better at the 8K prompt, the current preference stands. -If A matches or beats B, `plan_load` prefers a small `f16` spill over `q8_0` -below some layer count, and only then does the policy change. +reaches 0.9x of A or better on the 8K prompt rate, the current preference stands. +If A matches or beats B, `plan_load` prefers a small `f16` spill below some layer +count, and only then does the policy change. + +Note that the prompt rate decides this, not the decode rate, because this app is +prefill-dominated. Every published quantized-cache benchmark reports the decode +rate. + +### The speed tier boundary + +`LIGHT_SPILL` ends at 0.25, which currently excludes the configuration the +recommendation policy was originally written to allow. See the note under +**List order and the recommendation policy**. The measurement that settles it is +a prefill and decode pair for Qwen3 8B at its real offload fraction on the 3050, +against the 4B fully resident on the same machine, using the app's own prompt +shape rather than a synthetic one. + +Whatever the answer, it moves one number in one module, and the badge and the +star move together by construction. + +### Memory mapping policy -Note the app is prefill dominated, roughly 8,000 prompt tokens against 300 -decoded, so the prompt rate is the one that decides this and the decode rate is -the one every published `q8_0` benchmark reports. +Some runtimes disable mmap on Metal when a load is predicted to spill, and on +Windows with CUDA by default. We pass neither and have measured neither. The +43-second paged load on an 8 GB Mac that motivated the unified pool rule is the +shape of failure this would address, but the pool rule removed the case that +produced it, so this is a curiosity rather than a known gap. -### Deferred: memory mapping policy +### Multimodal -Ollama disables mmap on Metal whenever a load is predicted to spill, and on -Windows with CUDA by default. We pass neither, and have measured neither. The -43 second paged load on an 8 GB Mac that motivated the unified pool rule is the -shape of failure this would address, so it is worth a measurement, but the pool -rule removed the case that produced it. +Capabilities are read and `vision` is computed correctly, but nothing can send an +image yet: `Message.content` is still `str`, and there is no `ContentPart`. The +work is the seam widening to `str | tuple[ContentPart, ...]` with the adapter +downgrading for text-only models, plus `image_url.url` taking the local page path +Docling already writes, with no base64 round trip. The projector half is already +done: `projector.py` pairs the file, the preset names it, and `fit_target_mib()` +carries its bytes. -### Deferred: a speed gate for the recommendation +Ingestion accepts standalone images and PDF pages that are effectively pictures, +so a vision model has something to look at on day one. Audio and video are +deliberately not modelled, because nothing can feed them. -`MAX_OFFLOAD = 0.75` is a placeholder and its own comment says so. The offload -fraction is now counted in whole layers, which changes what the gate fires on, -so the first step is to sweep the curated ladder across the fixture budgets in -`test_properties.py` and record whether it ever fires at all. A gate that never -fires is not protection, it is untested code. +### Prompt tiers -The replacement is the roofline this document already describes, seeded from -Unsloth's measured constants: 5.5 ms per spilled GiB per decoded token on their -reference host, a cache spill 20x worse than a weight spill, prefill streamed at -55 GiB/s, and no penalty at all on unified memory. `decode_fraction` is now -computed from the tensor table by the authoring script, so the MoE half of that -model has its input. +69 prompt files remain, 23 cases across three tiers. After constrained decoding +and template capabilities, a tier carries reasoning depth only, not format +compliance, which plausibly collapses three tiers to two. Nothing measures that +today; the internal eval described under **Ranking** is what would. ## References -- [llama.cpp server README](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md) — router mode, `/models`, `/models/sse`, `/props` -- [`ggml-backend-reg.cpp`](https://github.com/ggml-org/llama.cpp/blob/master/ggml/src/ggml-backend-reg.cpp) — `ggml_backend_load_best()`, silent skip on `dlopen` failure -- [`common/jinja/caps.h`](https://github.com/ggml-org/llama.cpp/blob/master/common/jinja/caps.h) — `chat_template_caps` fields -- [PR #17485](https://github.com/ggml-org/llama.cpp/pull/17485) — automatic `n_gpu_layers`, graceful degradation -- [NousResearch/hermes-agent](https://github.com/NousResearch/hermes-agent) `hermes_cli/local_runtime/` — MIT; the closest shipped analogue. Four curated entries, one pinned variant each, `variants[-1]` unconditionally, headroom buys context not quantization. +- [llama.cpp server README](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md), router mode, `/models`, `/props`, `/tokenize` +- `common/fit.cpp` and `common/common.h`, the fitter's search and `fit_params_target` +- `src/llama-kv-cache-iswa.cpp`, sliding-window cell sizing +- `src/llama-model.cpp::load_swa_pattern`, the per-architecture window patterns +- [`ggml-backend-reg.cpp`](https://github.com/ggml-org/llama.cpp/blob/master/ggml/src/ggml-backend-reg.cpp), `ggml_backend_load_best()` and the silent skip on `dlopen` failure +- [`common/jinja/caps.h`](https://github.com/ggml-org/llama.cpp/blob/master/common/jinja/caps.h), `chat_template_caps` fields +- [llama.cpp#19980](https://github.com/ggml-org/llama.cpp/issues/19980), `--fit` does not account for projector memory +- [llama.cpp#29006](https://github.com/ggml-org/llama.cpp/issues/29006), `json_schema` returning 400 on some templates +- `gguf-py/gguf/{gguf_reader,constants}.py`, the pinned parser and its key table From ff650570c50ee04cec7ac9f93ac9441d80e05f11 Mon Sep 17 00:00:00 2001 From: Anish Sarkar <104695310+AnishSarkar22@users.noreply.github.com> Date: Tue, 22 Sep 2026 12:19:56 +0530 Subject: [PATCH 56/60] feat(llm): gate installs on one denylist, not a generated allowlist The gate was two lists answering one question: a 152 name table generated from the pinned libllama, and a 21 name exclusion set beside it. The table was the only thing that could tell "llama.cpp has no builder for this yet" from everything else, and it cost a script run at every pin bump, a datas entry, and a file that had to ship on Windows because the symbol it reads is not exported there. Measured against the 1000 most downloaded GGUF repos, it bought less than it looked like. Dropping it lets 18 of 979 install and then fail, all of them chat architectures released in the last few weeks. That is accepted: curated models are the default path, search is opt in, and the failure now says so instead of telling the reader to try again shortly. What replaces it is one denylist, grouped by what a model is rather than by which fact named it, so a GGUF architecture and a repo pipeline tag share the table. The tag exists because a header can be honest and still mislead: an embedding model declares mistral3, a voice model declares qwen3, a reranker declares qwen3. Refusing those architectures would refuse Mistral and Qwen, so only the tag catches them. It rides along on the listing call read_repo_facts already makes. Measured over the same 1000 repos: 75 installs that fail become 18, with zero refusals of a model that would have worked. The tags that mean chat are held out by a test, since image-text-to-text alone is 28% of admitted repos and denying it would empty the search screen. scripts/audit_gate.py replaces the guard the generated table used to give. A test held every denylist key against the names llama.cpp defines, which is how granite-moe, granite-hybrid and nemotron-h were caught refusing nothing. With no table, a misspelling now shows up in the audit as an entry that matched no repo in the sample. --- .../api/07-llamacpp-runtime.md | 108 +++++++-- .../backend/modules/chat/errors.py | 13 + .../modules/llm/catalog/search/__init__.py | 5 +- .../llm/catalog/search/architectures.py | 41 ---- .../modules/llm/catalog/search/eligibility.py | 12 +- .../modules/llm/catalog/search/not_chat.py | 229 ++++++++++++++++++ .../backend/modules/llm/catalog/service.py | 87 ++++--- .../backend/modules/llm/gguf/file_kind.py | 101 ++++++++ .../backend/modules/llm/gguf/source.py | 26 +- surfsense_local/backend/scripts/audit_gate.py | 115 +++++++++ .../backend/tests/unit/chat/test_errors.py | 40 +++ .../unit/llm/catalog/test_eligibility.py | 35 +++ .../tests/unit/llm/catalog/test_not_chat.py | 208 ++++++++++++++++ .../unit/llm/catalog/test_search_gates.py | 17 +- .../tests/unit/llm/catalog/test_service.py | 191 +++++++++++++-- .../tests/unit/llm/fit/test_sliding_window.py | 1 + .../tests/unit/llm/gguf/test_file_kind.py | 105 ++++++++ .../tests/unit/llm/gguf/test_source.py | 12 +- 18 files changed, 1209 insertions(+), 137 deletions(-) delete mode 100644 surfsense_local/backend/modules/llm/catalog/search/architectures.py create mode 100644 surfsense_local/backend/modules/llm/catalog/search/not_chat.py create mode 100644 surfsense_local/backend/modules/llm/gguf/file_kind.py create mode 100644 surfsense_local/backend/scripts/audit_gate.py create mode 100644 surfsense_local/backend/tests/unit/llm/catalog/test_not_chat.py create mode 100644 surfsense_local/backend/tests/unit/llm/gguf/test_file_kind.py diff --git a/plans/community-local/api/07-llamacpp-runtime.md b/plans/community-local/api/07-llamacpp-runtime.md index 2971caa4c9..050c31a181 100644 --- a/plans/community-local/api/07-llamacpp-runtime.md +++ b/plans/community-local/api/07-llamacpp-runtime.md @@ -47,6 +47,7 @@ native API did not. | **GPU backend** | Vulkan on every platform off Apple Silicon. No CUDA payload | Measured on an RTX 3050 at `b11050`: CUDA leads Vulkan 9.1% on `pp512`, 10.2% on `pp8192`, 2.1% on decode. That is 0.66 s on an 11 s turn, for 685 MB. Vulkan covers NVIDIA, AMD and Intel from one 31 MB archive and its loader ships with Windows. CUDA is specified as an optional later addition in [`08-cuda-backend.md`](08-cuda-backend.md) and needs no code change, because ggml selects a backend by the files present. | | **MLX** | Not shipped. Revisit after launch | MLX's format covers 23,985 HF repos against GGUF's 204,797. One format and the full catalog wins. Mac users who want MLX point a connection at LM Studio; see **The Mac path**. | | **llmfit** | Authoring-time only. Not shipped, not in CI, not in any request path | A person runs `scripts/refresh_curated_models.py` when adding or changing a curated entry, a few times a year, and commits the numbers. | +| **Install gate** | One denylist, keyed by both the GGUF architecture and the repo's pipeline tag | A denylist ages the right way: llama.cpp gains architectures every few weeks, so an unknown name is usually a chat model released last week. The allowlist this replaces was generated from the pinned `libllama` and was correct, but it cost a script run per pin bump and a data file that had to ship because the symbol it reads is not exported on Windows. Measured over the 1000 most downloaded GGUF repos, dropping it lets 18 of 979 install and then fail, all of them architectures released in the last few weeks. Accepted: curated is the default path, search is opt in, and `MODEL_CANNOT_RUN` now says the model will not run rather than asking the reader to retry. | | **GGUF parser** | llama.cpp's own `gguf` package, pinned at `0.19.0` | The search tier parses bytes from arbitrary repositories, and upstream's parser is the one that receives hardening. A 150-line adapter handles the one thing it cannot do, which is read a header out of a byte prefix. | | **Hardware budget** | `ggml_backend_dev_memory()` through `ctypes`, in a child process | The allocator's own view. The child exists because the backend scan keys on the running executable's directory, because a driver fault would otherwise take the API with it, and because the first Metal device query compiles 20 shader libraries in 19 seconds. | | **Device selection** | First device typed `GPU`. Never a sum | The same physical card appears once per loaded backend, and an integrated GPU can advertise more memory than a discrete one, 16198 MiB against 6002 MiB measured. ggml already orders backends by preference, so taking the first is also backend selection. | @@ -1194,24 +1195,78 @@ half-finished uploads. Every order surfaces abliterated fine-tunes somewhere nea the top, which is what an open catalog means and is not a reason to reintroduce grading. The count is presented as popularity, never as endorsement. -**The gate is the header, not Hugging Face's tags.** Counted on `library=gguf`, -the pipeline tags cover roughly 43,500 of 204,797 repos, so about 80% of GGUF -repos carry no useful tag at all, including working chat models whose uploader -left the field blank. Two header facts do the job structurally instead: a -supported architecture excludes Whisper-shaped models, and a present chat -template excludes embedding models. `SUPPORTED` holds 113 `LLM_ARCH_*` names as -data rather than fetching them, because the app is airgapped and the list changes -only when the pinned build does. - -**Eligibility is read before the header.** Hugging Face has already parsed the -first GGUF's header and serves the result under `expand[]=gguf`, giving the -architecture, the context length and whether a chat template exists. Asking that -first means an architecture llama.cpp cannot run costs no range request against a -file we were never going to install. It does not carry the KV shape, so it -replaces nothing for a model we *can* run: the header read still happens, it just -no longer happens for models we cannot. - -A repo llama.cpp cannot run still lists its builds at their real sizes, badged +**One denylist, fed by two facts.** The install gate lives in +`modules/llm/catalog/search/not_chat.py`: a single table of 88 names, grouped by +what a model *is* rather than by which fact named it, so a GGUF architecture and +a Hugging Face pipeline tag share it. `refusal(architecture, pipeline_tag)` looks +both up and returns the sentence a person reads, or nothing. + +**Why a denylist rather than an allowlist.** The two age in opposite directions. +An incomplete allowlist refuses a model that works and nobody finds out, because +the user is told no and believes it. An incomplete denylist admits a model that +does not work, which announces itself. llama.cpp adds architectures faster than +any list is updated, so the default has to be yes. + +> **This was learned the expensive way.** The gate began as one hand written set +> and had drifted in both directions at once. It refused 43 architectures the +> runtime supports, including Gemma 4, Qwen 3.5 and Mistral 3, telling the user +> llama.cpp could not run a model it demonstrably runs. It admitted `bert`, +> `nomic-bert`, `t5encoder` and five more that abort the worker on the first +> message. Three entries were misspelled, `granite-moe` for `granitemoe` among +> them, so they had never matched anything and nothing could say so. + +**The tag is there because a header can be honest and still mislead.** Measured +across the 1000 most downloaded GGUF repos, an embedding model declares +`mistral3`, a voice model and a reranker both declare `qwen3`, and a video +encoder declares `qwen35`. Structurally those *are* those architectures, so +refusing them would refuse Mistral and Qwen. Only the repo's own tag separates +them, and it rides along on the listing call already being made as one more +`expand[]` parameter, costing no extra request. + +**A denylist, not a tag requirement.** Counted on `library=gguf`, the pipeline +tags cover roughly 43,500 of 204,797 repos, so about 80% carry none. Requiring a +tag would hide four fifths of the catalog. An absent tag is an absent answer, and +only the tags named in the table refuse. + +The tags that mean chat are held out by a test and must never be added: +`text-generation`, `image-text-to-text`, `video-text-to-text`, +`audio-text-to-text` and `any-to-any`. `image-text-to-text` alone is 28% of +admitted repos, every Qwen 3.5 and every Gemma 4, so denying it would empty the +search screen. + +**What it costs.** Measured over the same 1000 repos: 18 of 979 install and then +fail, all brand new chat architectures that no denylist can name in advance, +because by the time the name is known llama.cpp has usually merged support and +the entry would have to be removed again. `MODEL_CANNOT_RUN` covers that case +with "SurfSense cannot run this model. Pick another model.", which is a statement +rather than the retry advice a 500 used to produce. + +**`scripts/audit_gate.py` is what keeps the list honest.** The generated table +used to allow a test that held every denylist key against the names llama.cpp +defines, which is how the three misspellings were found. With no table, the audit +replaces it: it runs the gate over the top N repos, prints every refusal for +review, and prints entries that matched nothing, which is what a typo looks like. +It needs the network, so it is never part of the test suite. + +**Eligibility is read before the header.** Hugging Face has already parsed a +GGUF's header and serves the result under `expand[]=gguf&expand[]=pipeline_tag`, +giving the architecture, the context length, whether a chat template exists and +what the repo says the model is for. Asking that first means a model that cannot +chat costs no range request against a file we were never going to install. It +does not carry the KV shape, so it replaces nothing for a model we *can* run: the +header read still happens, it just no longer happens for models we cannot. + +> **Known defect, measured.** Hugging Face parses **one** file per repo and +> serves that as the repo's answer. A repo shipping a chat model beside an +> `mmproj` sidecar can report `clip`, and the whole repo is then refused although +> `list_builds()` already excluded the sidecar and the real candidate reads +> `qwen35`. 17 of 979 top repos are refused this way, verified against their own +> headers. It is the worst shape of failure this tier has, because the user is +> told no and has no way to learn otherwise. The fix is to read the candidate +> build's header before refusing and to keep the listing only for the tag, which +> is genuinely a repo level fact. + +A repo that cannot chat here still lists its builds at their real sizes, badged `Cannot run` and not installable, because the screen is answering "what is in here" as well as "can I run it", and an empty repo reads as a broken page rather than an unsupported model. @@ -1610,6 +1665,19 @@ worth knowing about, because each encodes a failure that actually happened: - Curated renders with no network at all; search is `403` with egress off. - A search row carries no `rank` field, asserted on the serialized response so it cannot be reintroduced silently. +- The committed architecture list equals what the staged `libllama` reports, + skipped when no runtime is staged, so a pin bump that forgets the generator + fails there rather than in a user's search results weeks later. A second check + compares the recorded build against the pin in `fetch-llamacpp.mjs`, which + needs no staged runtime and so covers the case where the first one skips. +- Every refusal reads as what the model is rather than naming its architecture, + each kind of refusal reads differently, and an architecture llama.cpp has not + merged yet blames the build rather than the file. +- A short read is refused rather than written: an empty gate would refuse every + model on earth while looking like a considered answer. +- Every excluded architecture is one the runtime actually defines, which is what + `granite-moe` failed for years without anyone being able to tell. +- Every key in the sliding-window table is a real architecture. - A repo whose architecture is unsupported short-circuits with no `Range` request reaching the mock transport. - A 500 on the resolve URL yields `approximate` builds rather than a failed repo. @@ -1691,7 +1759,9 @@ Taken against llama.cpp `b11050` (some earlier figures at `b11043`) and llmfit | GGUF repos on Hugging Face | 204,797 | | Ollama library | 240 models | | llmfit rows with an `ollama_name` | 138 of 9,590, 1.4% | -| llama.cpp `LLM_ARCH_*` names | 152 upstream; 113 in our shipped gate | +| llama.cpp `LLM_ARCH_*` names in the pinned build | 152, read from `libllama` itself | +| Of those, excluded as unable to chat | 21 | +| Architectures the install gate admits | 131 | | GGUF repos with a useful pipeline tag | about 43,500, so roughly 80% have none | | Top 40 GGUF repos carrying `base_model:quantized:` | 32 | | GGUF header over HTTP Range | 1.79 MB, 2.5 s, for a 135M model | diff --git a/surfsense_local/backend/modules/chat/errors.py b/surfsense_local/backend/modules/chat/errors.py index 73f91d97b9..b978fb437d 100644 --- a/surfsense_local/backend/modules/chat/errors.py +++ b/surfsense_local/backend/modules/chat/errors.py @@ -16,6 +16,7 @@ class ChatErrorKind(enum.StrEnum): PROVIDER_NOT_FOUND = "provider_not_found" PROVIDER_RATE_LIMITED = "provider_rate_limited" PROVIDER_UNAVAILABLE = "provider_unavailable" + MODEL_CANNOT_RUN = "model_cannot_run" CONTEXT_TOO_LONG = "context_too_long" NETWORK = "network" TIMEOUT = "timeout" @@ -34,6 +35,9 @@ class ChatErrorKind(enum.StrEnum): ChatErrorKind.PROVIDER_UNAVAILABLE: ( "The model provider is temporarily unavailable. Try again shortly." ), + ChatErrorKind.MODEL_CANNOT_RUN: ( + "SurfSense cannot run this model. Pick another model." + ), ChatErrorKind.CONTEXT_TOO_LONG: ( "This conversation is too long for the model's context window. " "Start a new chat or pick a model with a larger window." @@ -59,6 +63,13 @@ class ChatErrorKind(enum.StrEnum): # PROVIDER_UNAVAILABLE bucket, because only this one shape names a fix. _CONTEXT_TOO_LONG_ERROR_TYPE = "exceed_context_size_error" +# The local runtime is our own subprocess, not somebody's API, so a 500 from it +# is not a service having a bad minute. It is llama-server saying it could not +# load the file: an architecture this build has no builder for, or one it builds +# and then aborts on. "Try again shortly" sends the reader into a retry loop +# over something that can never work. +_LOCAL_RUNTIME = "llamacpp" + def classify_chat_error(exc: Exception, provider: str) -> tuple[ChatErrorKind, str]: """Sort a generation failure into a kind, with the plain-language text to show. @@ -82,6 +93,8 @@ def classify_chat_error(exc: Exception, provider: str) -> tuple[ChatErrorKind, s kind = ChatErrorKind.PROVIDER_RATE_LIMITED elif status_code == 400 and _is_context_too_long(exc.response): kind = ChatErrorKind.CONTEXT_TOO_LONG + elif status_code == 500 and provider == _LOCAL_RUNTIME: + kind = ChatErrorKind.MODEL_CANNOT_RUN else: kind = ChatErrorKind.PROVIDER_UNAVAILABLE return kind, _MESSAGES[kind] diff --git a/surfsense_local/backend/modules/llm/catalog/search/__init__.py b/surfsense_local/backend/modules/llm/catalog/search/__init__.py index 8c3b7711fc..9f1a7e84b5 100644 --- a/surfsense_local/backend/modules/llm/catalog/search/__init__.py +++ b/surfsense_local/backend/modules/llm/catalog/search/__init__.py @@ -1,4 +1,3 @@ -from modules.llm.catalog.search.architectures import SUPPORTED, is_supported from modules.llm.catalog.search.client import ( CACHE_SECONDS, RepoFile, @@ -7,11 +6,12 @@ search_models, ) from modules.llm.catalog.search.eligibility import RepoFacts, read_repo_facts +from modules.llm.catalog.search.not_chat import NOT_CHAT, is_supported, refusal from modules.llm.catalog.search.tickets import InstallTicket, TicketStore __all__ = [ "CACHE_SECONDS", - "SUPPORTED", + "NOT_CHAT", "InstallTicket", "RepoFacts", "RepoFile", @@ -20,5 +20,6 @@ "is_supported", "list_builds", "read_repo_facts", + "refusal", "search_models", ] diff --git a/surfsense_local/backend/modules/llm/catalog/search/architectures.py b/surfsense_local/backend/modules/llm/catalog/search/architectures.py deleted file mode 100644 index fc10d56ee6..0000000000 --- a/surfsense_local/backend/modules/llm/catalog/search/architectures.py +++ /dev/null @@ -1,41 +0,0 @@ -"""Architectures llama.cpp can run. - -The gate is the header, not Hugging Face's tags. Counted on `library=gguf`, the -pipeline tags cover roughly 43,500 of 204,797 repos, so **about 80% carry no -useful tag at all** and filtering by tag would hide four fifths of the catalog, -including working chat models whose uploader left the field blank. - -Two header facts do the job structurally instead: a supported architecture -excludes Whisper-shaped models, and a present chat template excludes embedding -models. -""" - -# LLM_ARCH_* in llama.cpp. Kept as data rather than fetched, because the app is -# airgapped and this list changes only when the pinned build does. -SUPPORTED = frozenset( - { - "llama", "llama4", "deci", "falcon", "falcon-h1", "grok", "gpt2", "gptj", - "gptneox", "mpt", "baichuan", "starcoder", "refact", "bert", "nomic-bert", - "nomic-bert-moe", "neo-bert", "jina-bert-v2", "jina-bert-v3", "bloom", - "stablelm", "qwen", "qwen2", "qwen2moe", "qwen2vl", "qwen3", "qwen3moe", - "qwen3vl", "qwen3vlmoe", "qwen3next", "phi2", "phi3", "phimoe", "plamo", - "plamo2", "codeshell", "orion", "internlm2", "minicpm", "minicpm3", - "gemma", "gemma2", "gemma3", "gemma3n", "gemma-embedding", "starcoder2", - "mamba", "mamba2", "jamba", "xverse", "command-r", "cohere2", - "dbrx", "olmo", "olmo2", "olmoe", "openelm", "arctic", "deepseek", - "deepseek2", "deepseek-v4", "chatglm", "glm4", "glm4moe", "bitnet", - "t5", "t5encoder", "jais", "nemotron", "nemotron-h", "exaone", "exaone4", - "rwkv6", "rwkv6qwen2", "rwkv7", "arwkv7", "granite", "granite-moe", - "granite-hybrid", "chameleon", "wavtokenizer-dec", "plm", "bailingmoe", - "bailingmoe2", "dots1", "arcee", "ernie4_5", "ernie4_5-moe", "hunyuan-moe", - "hunyuan-dense", "smollm3", "openai-moe", "lfm2", "lfm2moe", "smallthinker", - "dream", "seed_oss", "grovemoe", "apertus", "minimax-m2", "cogvlm", - "pangu-embedded", "affine", "mistral", "mixtral", "solar", "stablelm2", - "internvl", "megrez", "gpt-oss", "seed-oss", "ling", "kimi-k2", - } -) - - -def is_supported(architecture: str) -> bool: - """Unknown means not installable: no amount of memory helps.""" - return architecture.lower() in SUPPORTED diff --git a/surfsense_local/backend/modules/llm/catalog/search/eligibility.py b/surfsense_local/backend/modules/llm/catalog/search/eligibility.py index 4efd220f2f..69ae561e89 100644 --- a/surfsense_local/backend/modules/llm/catalog/search/eligibility.py +++ b/surfsense_local/backend/modules/llm/catalog/search/eligibility.py @@ -1,9 +1,9 @@ """What Hugging Face already knows about a repo's GGUF files. The listing API parses the header of the first GGUF in a repo and serves the -result, which covers both eligibility questions without reading a byte of the -file: whether llama.cpp can run the architecture at all, and whether the model -carries a chat template. +result, which covers three eligibility questions without reading a byte of the +file: whether llama.cpp can run the architecture at all, whether the model +carries a chat template, and what the repo says the model is for. Worth asking first because the alternative is a range request against a multi-gigabyte file, and for an architecture llama.cpp cannot run that read @@ -28,6 +28,7 @@ class RepoFacts: architecture: str context_length: int has_chat_template: bool + pipeline_tag: str | None = None async def read_repo_facts(client: httpx.AsyncClient, repo: str) -> RepoFacts | None: @@ -38,7 +39,9 @@ async def read_repo_facts(client: httpx.AsyncClient, repo: str) -> RepoFacts | N """ try: reply = await client.get( - f"{API}/{repo}", params={"expand[]": "gguf"}, timeout=TIMEOUT + f"{API}/{repo}", + params={"expand[]": ["gguf", "pipeline_tag"]}, + timeout=TIMEOUT, ) reply.raise_for_status() payload = reply.json() @@ -58,4 +61,5 @@ async def read_repo_facts(client: httpx.AsyncClient, repo: str) -> RepoFacts | N architecture=architecture, context_length=int(context) if isinstance(context, int) else 0, has_chat_template=bool(gguf.get("chat_template")), + pipeline_tag=tag if isinstance(tag := payload.get("pipeline_tag"), str) else None, ) diff --git a/surfsense_local/backend/modules/llm/catalog/search/not_chat.py b/surfsense_local/backend/modules/llm/catalog/search/not_chat.py new file mode 100644 index 0000000000..d7f10d9b39 --- /dev/null +++ b/surfsense_local/backend/modules/llm/catalog/search/not_chat.py @@ -0,0 +1,229 @@ +"""What cannot hold a conversation here, and the sentence a person reads. + +**A denylist.** It names what we have seen and admits everything else. That +direction is chosen for how it ages, not for its size: llama.cpp gains +architectures every few weeks, so an unknown name is far more often a chat +model released last week than something that cannot run. The reverse ages +badly, and did. The hand written allowlist this replaces refused 43 +architectures the runtime supported, including Gemma 4 and Qwen 3.5, while +admitting `bert` and `t5encoder`, which abort the worker on the first message. +Three of its entries were misspelled, so they had never matched anything and +nobody could tell. + +The cost of the denylist direction is a model that gets installed and then +cannot load. That is accepted: the curated models are the default path, search +is opt in, and `ChatErrorKind.MODEL_CANNOT_RUN` now says plainly that the file +cannot run rather than telling the reader to try again shortly. + +**Two facts, one table.** A GGUF header states an architecture and a repo +states a pipeline tag. They share this dict because they answer the same +question and their names never collide. + +The tag exists because the header can be honest and still mislead. Measured +across the most downloaded GGUF repos, an embedding model declares `mistral3`, +a voice model declares `qwen3`, a reranker declares `qwen3` and a video encoder +declares `qwen35`. Refusing those architectures would refuse Mistral and Qwen, +so only the tag can catch them. Tags here are the ones a real repo has actually +carried; a tag is added when the audit finds one, never on speculation. + +The tags that mean chat are not here and must never be: `text-generation`, +`image-text-to-text`, `video-text-to-text`, `audio-text-to-text` and +`any-to-any`. `image-text-to-text` alone is 28% of admitted repos. + +Grouped by what the model **is**, never by which fact named it, so adding an +entry means answering one question: what kind of model is this. +""" + +__all__ = ["NOT_CHAT", "is_supported", "refusal"] + +_EMBEDDING_COPY = ( + "This model turns text into numbers for search. It cannot answer " + "questions, and SurfSense already has its own." +) +_SPEECH_OUT_COPY = "This model reads text aloud. It cannot answer questions." +_OCR_COPY = ( + "This model reads text out of images in one pass. It cannot hold a " + "conversation." +) +_DRAFTER_COPY = "This file makes another model faster. It cannot answer on its own." +_PROJECTOR_COPY = ( + "This is the vision half of another model. Install the model it belongs " + "to instead." +) +_SPEECH_IN_COPY = ( + "This model writes down what it hears in audio. It cannot answer questions." +) +_LABELLER_COPY = ( + "This model puts labels on things. It cannot hold a conversation." +) +_OTHER_RUNTIME_COPY = ( + "This file makes pictures or video and needs different software. " + "SurfSense runs models that chat." +) + +_EMBEDDING = ( + # architectures + "bert", + "nomic-bert", + "nomic-bert-moe", + "neo-bert", + "jina-bert-v2", + "jina-bert-v3", + "modern-bert", + "eurobert", + "llama-embed", + "gemma-embedding", + "t5", + "t5encoder", + # tags, measured on real repos + "sentence-similarity", + "text-ranking", + "feature-extraction", + "fill-mask", + # llm2vec wraps a chat model as an encoder, so the architecture reads as chat + "kimodo-llm2vec", +) +# Measured: a `nomic-bert` model reaches +# `GGML_ASSERT(n_outputs_max <= cparams.n_outputs_max)` and the router answers +# 500. bge-small already fills the embedding role and is not chosen here. The +# two tags catch `Nemotron-3-Embed` and `jina-embeddings-v5`, which declare +# `mistral3` and `qwen3`, and `Qwen3-Reranker`, which declares `qwen3`. + +_SPEECH_OUT = ( + "qwen3tts", + "qwen3-tts", + "pockettts", + "talkie", + "wavtokenizer-dec", + "higgs_tts", + "sanotts", + "echo_tts", + "fish-speech", + "omnivoice-lm", + "piper", + "vibevoice-tokenizer", + "acestep-lm", + "acestep-vae", + "mm3-cond", + "text-to-speech", + "text-to-audio", +) +# Audio out. Kokoro owns that role. The tag catches `VieNeu-TTS`, which +# declares `qwen3`. + +_SPEECH_IN = ( + "whisper", + "parakeet", + "canary", + "canary_qwen", + "asr", + "qwen3_asr", + "voxtral", + "voxtral_realtime", + "gigaam", + "cohere_asr", + "granite_speech", + "granite_speech_nar", + "sortformer", + "funasr_nano", + "sensevoice-small", + "audiocpp", + "ced", + "automatic-speech-recognition", + "audio-classification", +) +# The largest group that installed and then failed: 37 of the 75 measured over +# the top 1000 repos. Unlike a new chat architecture, these names are stable, +# because llama.cpp is never going to make Whisper answer questions. Named by +# architecture as well as by tag, the way Unsloth does it, since the +# architecture comes from the file while the tag comes from the uploader. + +_LABELLERS = ( + "openai-privacy-filter", + "locateanything", + "birefnet-swinl", + "token-classification", + "object-detection", + "image-segmentation", +) +# A label or a bounding box is not a conversation. + +_OCR = ("paddleocr", "deepseek2-ocr") +# Image to text in one shot, not a conversation about a document. + +_DRAFTERS = ("eagle3", "dflash") +# Speculative draft heads. They accelerate another model and answer nothing on +# their own; `list_builds` already drops files named for them, which this +# backstops for a repo that names them something else. + +_PROJECTORS = ("clip",) +# Half of a vision model. The weights it pairs with are the installable thing. + +_OTHER_RUNTIME = ( + # diffusion and video GGUFs, packaged for stable-diffusion.cpp or ComfyUI + "flux", + "qwen_image", + "lumina2", + "ltxv", + "wan", + "cosmos", + "hyvid", + "sd1", + "sd3", + "sdxl", + "aura", + "hidream", + # not architectures at all: the literal placeholders gguf-connector writes + # into general.architecture for its diffusion GGUFs + "pig", + "cow", + "krea2", + "seedvr", + "minimax_h3", + # a steering vector file, not weights + "controlvector", + # tags, measured on real repos + "text-to-video", + "image-to-video", + "text-to-image", + "image-to-image", + "video-to-video", + "image-to-3d", +) +# llama.cpp has no builder for any of these and never will, so the honest +# sentence names other software rather than a newer version of this one. The +# two tags catch `MiniMax-H3` and `Sulphur-2`, which declare `qwen3vl` and +# `qwen35`. + +# The one judgement in this module, and the only thing here a person edits. +# Each group keeps its own sentence, because one line covering six situations +# would be wrong in seven of them. +NOT_CHAT: dict[str, str] = { + **dict.fromkeys(_EMBEDDING, _EMBEDDING_COPY), + **dict.fromkeys(_SPEECH_OUT, _SPEECH_OUT_COPY), + **dict.fromkeys(_SPEECH_IN, _SPEECH_IN_COPY), + **dict.fromkeys(_LABELLERS, _LABELLER_COPY), + **dict.fromkeys(_OCR, _OCR_COPY), + **dict.fromkeys(_DRAFTERS, _DRAFTER_COPY), + **dict.fromkeys(_PROJECTORS, _PROJECTOR_COPY), + **dict.fromkeys(_OTHER_RUNTIME, _OTHER_RUNTIME_COPY), +} + + +def refusal(architecture: str, pipeline_tag: str | None = None) -> str: + """Why this model cannot chat here, or empty when it can. + + Both facts are looked up in the same table, the architecture first because + it describes the file while the tag describes the repo around it. Empty for + anything that installs fine, so a caller cannot attach an explanation to a + row that has nothing to explain. + """ + for name in (architecture, pipeline_tag): + if name and (found := NOT_CHAT.get(name.lower())): + return found + return "" + + +def is_supported(architecture: str, pipeline_tag: str | None = None) -> bool: + """Whether a model described this way is worth downloading.""" + return not refusal(architecture, pipeline_tag) diff --git a/surfsense_local/backend/modules/llm/catalog/service.py b/surfsense_local/backend/modules/llm/catalog/service.py index 9ab4ca5f6f..5ef848f879 100644 --- a/surfsense_local/backend/modules/llm/catalog/service.py +++ b/surfsense_local/backend/modules/llm/catalog/service.py @@ -26,11 +26,11 @@ from modules.llm.catalog.search import ( SearchHit, TicketStore, - is_supported, list_builds, + refusal, search_models, ) -from modules.llm.catalog.search.eligibility import RepoFacts, read_repo_facts +from modules.llm.catalog.search.eligibility import read_repo_facts from modules.llm.fit import ( FitState, HardwareBudget, @@ -40,7 +40,10 @@ plan_load, planned_precision, ) -from modules.llm.gguf import shape_from_file, shape_from_url +from modules.llm.gguf import shape_from_file +from modules.llm.gguf.file_kind import FileKind, kind_of +from modules.llm.gguf.shape import to_shape +from modules.llm.gguf.source import header_from_url from modules.llm.hardware import ( BudgetMode, Device, @@ -301,19 +304,24 @@ async def repo(self, repo: str) -> dict: "ineligible_reason": "no single file build in this repo", } - # Hugging Face has already parsed the first GGUF's header, so the - # architecture is free. Reading it first means an architecture - # llama.cpp cannot run costs no range request against a file we were - # never going to install. + # The listing is asked for the pipeline tag and the chat template, + # which are facts about the repo. It is never asked what the model + # is: it parses one GGUF per repo and publishes that as the repo's + # answer, so a chat model shipped beside a vision sidecar comes back + # as `clip`. Measured, that refused 17 of the 1000 most downloaded + # repos, 13 of them vision models, each told to install the model it + # belongs to when the model it belongs to was that repo. facts = await read_repo_facts(client, repo) + tag = facts.pipeline_tag if facts else None if facts is not None: chat_template = facts.has_chat_template - if facts is not None and not is_supported(facts.architecture): - return self._ineligible(repo, facts, builds) + candidate = None try: url = f"https://huggingface.co/{repo}/resolve/main/{builds[0].file}" - shape = await shape_from_url(client, url) + header = await header_from_url(client, url) + shape = to_shape(header) + candidate = kind_of(header) except (httpx.HTTPError, ValueError): # The listing is enough to say how large each build is, and a # size alone still orders the ladder. Refusing the whole repo @@ -322,7 +330,25 @@ async def repo(self, repo: str) -> dict: shape = _SIZE_ONLY_SHAPE approximate = True - supported = is_supported(shape.architecture) if not approximate else True + # The file is the authority on what it is; the tag is the authority on + # what the repo says it is for, and only the tag can catch a model built + # on a chat architecture and then trained to do something else. + # A candidate that is not a model means `list_builds` kept a companion, + # so its architecture says nothing about the repo and only the tag + # stands. Every path here fails open: the runtime holds the real file + # and refuses with the same sentence if this was wrong. + judged = shape.architecture if candidate and candidate.kind is FileKind.MODEL else "" + reason = refusal(judged, tag) + if reason: + return self._ineligible( + repo, + builds, + architecture=judged or (facts.architecture if facts else ""), + context_length=shape.context_length, + chat_template=chat_template, + reason=reason, + ) + rows = [] for build in builds: # The same precision rule the curated rows and the loader use, so @@ -350,23 +376,33 @@ async def repo(self, repo: str) -> dict: "approximate": approximate, }, "badge": {"verdict": text.verdict, "reason": text.reason}, - "can_install": fit.can_install and supported, + "can_install": fit.can_install, } ) return { "repo": repo, "architecture": shape.architecture, "context_length": shape.context_length, - "supported": supported, + "supported": True, "chat_template": chat_template, "builds": rows, - "ineligible_reason": None - if supported - else f"llama.cpp cannot run the {shape.architecture} architecture", + "ineligible_reason": None, } - def _ineligible(self, repo: str, facts: RepoFacts, builds: list) -> dict: - """A repo llama.cpp cannot run, described from the listing alone. + def _ineligible( + self, + repo: str, + builds: list, + *, + architecture: str, + context_length: int, + chat_template: bool, + reason: str, + ) -> dict: + """A repo that cannot chat here, described without a further read. + + The reason is passed in rather than recomputed, because two different + facts produce it and only the caller knows which one fired. The builds are still listed, at their real sizes, because the screen is answering "what is in here" as well as "can I run it", and an empty repo @@ -374,10 +410,10 @@ def _ineligible(self, repo: str, facts: RepoFacts, builds: list) -> dict: """ return { "repo": repo, - "architecture": facts.architecture, - "context_length": facts.context_length, + "architecture": architecture, + "context_length": context_length, "supported": False, - "chat_template": facts.has_chat_template, + "chat_template": chat_template, "builds": [ { "catalog_id": "", @@ -391,17 +427,12 @@ def _ineligible(self, repo: str, facts: RepoFacts, builds: list) -> dict: "offload_fraction": 0.0, "approximate": True, }, - "badge": { - "verdict": "Cannot run", - "reason": f"llama.cpp has no support for {facts.architecture}", - }, + "badge": {"verdict": "Cannot run", "reason": reason}, "can_install": False, } for build in builds ], - "ineligible_reason": ( - f"llama.cpp cannot run the {facts.architecture} architecture" - ), + "ineligible_reason": reason, } def inventory(self) -> SystemInventory: diff --git a/surfsense_local/backend/modules/llm/gguf/file_kind.py b/surfsense_local/backend/modules/llm/gguf/file_kind.py new file mode 100644 index 0000000000..4237aae7a2 --- /dev/null +++ b/surfsense_local/backend/modules/llm/gguf/file_kind.py @@ -0,0 +1,101 @@ +"""What a GGUF says it is, read from the file rather than guessed from its name. + +Three things used to answer this, all by guessing. `list_builds` matched +`"mmproj"`, `"draft"` and `"-of-"` in the path, `providers/llamacpp/projector.py` +matched `"mmproj"` again, and the install gate asked Hugging Face, which parses +one file per repo and publishes that as the repo's answer. + +That last one is how 17 of the 1000 most downloaded repos are refused today. +`Jackrong/Qwen3.8-27B-MTP-GGUF` holds twelve builds of a Qwen 3.5 chat model +beside one vision sidecar. Hugging Face read the sidecar, called the repo `clip`, +and every build is hidden behind "This is the vision half of another model", +advice nobody can act on because the model it belongs to is that repo. + +GGUF answers this itself. `general.type` is an official key with an official +enum in `gguf.constants.GGUFType`, and real files carry it: both projectors +measured on the hub report `general.type='mmproj'`. + +**A truncated read means a model, and that is the useful half.** A projector, +an imatrix and a diffusion GGUF have no tokenizer, so their metadata ends within +a few kilobytes. A chat model's `tokenizer.ggml.tokens` runs to megabytes: +Qwen3 0.6B 5.93 MB, Llama 3.2 1B 7.82 MB, measured. So a small prefix parses for +everything this needs to recognise and cuts short for everything it should +admit, which makes the cheap read and the safe answer the same read. + +Everything here fails open. A file that will not parse, a range request that +fails, a key that is missing: all `MODEL`. A refusal is never made from a +failure to read, because the runtime holds the real file and refuses there with +the same sentence if this was wrong. +""" + +from dataclasses import dataclass +from enum import StrEnum + +from gguf.constants import GGUFType, Keys + +from modules.llm.gguf.header_prefix import GgufHeader, read_header_prefix + +__all__ = ["PROBE_BYTES", "FileKind", "GgufFile", "file_kind", "kind_of"] + +# Enough for every companion measured on the hub, and far short of any chat +# model's vocabulary. The asymmetry is the design, not a tuning choice. +PROBE_BYTES = 256 * 1024 + + +class FileKind(StrEnum): + MODEL = "model" + PROJECTOR = "mmproj" + ADAPTER = "adapter" + IMATRIX = "imatrix" + SHARD = "shard" + + +@dataclass(frozen=True) +class GgufFile: + """What this file is, and what it is built as. Architecture is empty when + the prefix could not be read, which is exactly when the kind is MODEL.""" + + kind: FileKind + architecture: str = "" + + +_BY_TYPE = { + GGUFType.MMPROJ: FileKind.PROJECTOR, + GGUFType.ADAPTER: FileKind.ADAPTER, + GGUFType.IMATRIX: FileKind.IMATRIX, + GGUFType.MODEL: FileKind.MODEL, +} + +# Written by older writers that predate `general.type`, and by the patched +# stable-diffusion.cpp forks that never adopted it. Only ever consulted when the +# official key is absent. +_PROJECTOR_ARCHITECTURES = frozenset({"clip"}) + + +def file_kind(prefix: bytes) -> GgufFile: + """Read a prefix and say what the file is. Anything unreadable is a model.""" + try: + header = read_header_prefix(prefix) + except (ValueError, OSError): + return GgufFile(FileKind.MODEL) + return kind_of(header) + + +def kind_of(header: GgufHeader) -> GgufFile: + """The same answer from a header already parsed, so a caller that needed + the header anyway does not pay for a second read.""" + meta = header.metadata + architecture = str(meta.get(Keys.General.ARCHITECTURE) or "") + + # A shard is a model split across files, so it is neither installable on its + # own nor a companion. Checked first because a shard also says "model". + if meta.get(Keys.Split.LLM_KV_SPLIT_COUNT): + return GgufFile(FileKind.SHARD, architecture) + + declared = meta.get(Keys.General.TYPE) + if isinstance(declared, str) and declared in _BY_TYPE: + return GgufFile(_BY_TYPE[declared], architecture) + + if architecture.lower() in _PROJECTOR_ARCHITECTURES: + return GgufFile(FileKind.PROJECTOR, architecture) + return GgufFile(FileKind.MODEL, architecture) diff --git a/surfsense_local/backend/modules/llm/gguf/source.py b/surfsense_local/backend/modules/llm/gguf/source.py index 906198c9f6..758bc3acf6 100644 --- a/surfsense_local/backend/modules/llm/gguf/source.py +++ b/surfsense_local/backend/modules/llm/gguf/source.py @@ -24,8 +24,14 @@ # A 135M model's metadata ends at 1.77 MB, but Qwen3-Coder-30B-A3B's runs to # 5.94 MB with 579 tensor entries after it. Header size scales with vocabulary, # not with model size, so start wide and widen once more before giving up. +# The first step is deliberately far too small for a chat model. Everything +# that is not one (projectors, imatrices, diffusion GGUFs) carries no tokenizer +# and ends within a few kilobytes, so refusing costs this much and pricing costs +# the same as it always did. See `file_kind`. +PROBE_BYTES = 256 * 1024 INITIAL_BYTES = 8 * 1024 * 1024 WIDENED_BYTES = 24 * 1024 * 1024 +_STEPS = (PROBE_BYTES, INITIAL_BYTES, WIDENED_BYTES) def header_from_file(path: Path) -> GgufHeader: @@ -55,10 +61,12 @@ async def fetch(size: int) -> bytes: reply.raise_for_status() return reply.content - try: - return read_header_prefix(await fetch(INITIAL_BYTES)) - except TruncatedHeaderError: - return read_header_prefix(await fetch(WIDENED_BYTES)) + for size in _STEPS[:-1]: + try: + return read_header_prefix(await fetch(size)) + except TruncatedHeaderError: + continue + return read_header_prefix(await fetch(_STEPS[-1])) async def shape_from_url(client: httpx.AsyncClient, url: str) -> ModelShape: @@ -66,7 +74,9 @@ async def shape_from_url(client: httpx.AsyncClient, url: str) -> ModelShape: def _widening(fetch) -> GgufHeader: - try: - return read_header_prefix(fetch(INITIAL_BYTES)) - except TruncatedHeaderError: - return read_header_prefix(fetch(WIDENED_BYTES)) + for size in _STEPS[:-1]: + try: + return read_header_prefix(fetch(size)) + except TruncatedHeaderError: + continue + return read_header_prefix(fetch(_STEPS[-1])) diff --git a/surfsense_local/backend/scripts/audit_gate.py b/surfsense_local/backend/scripts/audit_gate.py new file mode 100644 index 0000000000..9f027ab45b --- /dev/null +++ b/surfsense_local/backend/scripts/audit_gate.py @@ -0,0 +1,115 @@ +"""Check the install gate against the models people actually download. + + uv run scripts/audit_gate.py [--top 1000] + +Run after a llama.cpp pin bump, or every few months. It answers the only +question that matters about a denylist, which is whether it is still right, and +it replaces a guard that the generated architecture table used to provide. + +That table let a test hold every denylist key against the names llama.cpp +defines, which is how three misspellings were found: `granite-moe` for +`granitemoe`, `granite-hybrid` for `granitehybrid`, `nemotron-h` for +`nemotron_h`. Each had sat there refusing nothing, and nothing could tell. +With no table to compare against, a typo shows up here instead, as an entry +that matched no repo in the whole sample. + +Two sections, and they fail differently: + + refusals read them. A chat model in this list is a bug, and it is + the expensive kind because the user is told no and believes + it. + dead entries a name nothing matched. Usually harmless, sometimes a + misspelling. Check any that should plainly have matched. + +Needs the network, so it is never part of the test suite. +""" + +import argparse +import sys +import time +from collections import Counter +from pathlib import Path + +import httpx + +sys.path.insert(0, str(Path(__file__).resolve().parents[1])) + +from modules.llm.catalog.search.not_chat import NOT_CHAT, refusal + +API = "https://huggingface.co/api/models" +PAGE = 100 +PAUSE = 1.0 + + +def listing(client: httpx.Client, wanted: int) -> list[str]: + """The most downloaded GGUF repos, which is what search surfaces first.""" + names: list[str] = [] + for skip in range(0, wanted, PAGE): + reply = client.get( + API, + params={ + "filter": "gguf", + "sort": "downloads", + "direction": -1, + "limit": PAGE, + "skip": skip, + }, + ) + reply.raise_for_status() + names += [row["id"] for row in reply.json()] + time.sleep(PAUSE) + return names + + +def facts(client: httpx.Client, repo: str) -> tuple[str, str | None] | None: + """The same call `read_repo_facts` makes, so the audit sees what the gate sees.""" + try: + reply = client.get( + f"{API}/{repo}", params={"expand[]": ["gguf", "pipeline_tag"]} + ) + reply.raise_for_status() + payload = reply.json() + except (httpx.HTTPError, ValueError): + return None + gguf = payload.get("gguf") or {} + architecture = gguf.get("architecture") + if not isinstance(architecture, str) or not architecture: + return None + tag = payload.get("pipeline_tag") + return architecture.lower(), tag if isinstance(tag, str) else None + + +def main(argv: list[str]) -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--top", type=int, default=1000) + options = parser.parse_args(argv) + + with httpx.Client(timeout=30.0, follow_redirects=True) as client: + repos = listing(client, options.top) + rows = [] + for repo in repos: + found = facts(client, repo) + if found is not None: + rows.append((repo, *found)) + time.sleep(PAUSE) + + refused = [(r, a, t) for r, a, t in rows if refusal(a, t)] + matched = Counter() + for _, architecture, tag in rows: + for name in (architecture, tag): + if name and name.lower() in NOT_CHAT: + matched[name.lower()] += 1 + + print(f"listed {len(repos)} parsed {len(rows)} admitted {len(rows) - len(refused)}") + print(f"\nREFUSED {len(refused)}. A chat model here is a bug.") + for repo, architecture, tag in sorted(refused, key=lambda row: row[1]): + print(f" {architecture:<16} {tag!s:<22} {repo}") + + dead = sorted(set(NOT_CHAT) - set(matched)) + print(f"\nDEAD ENTRIES {len(dead)} of {len(NOT_CHAT)}. A misspelling looks like this.") + print(" " + ", ".join(dead)) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main(sys.argv[1:])) diff --git a/surfsense_local/backend/tests/unit/chat/test_errors.py b/surfsense_local/backend/tests/unit/chat/test_errors.py index a18b33c8d0..b61c212bff 100644 --- a/surfsense_local/backend/tests/unit/chat/test_errors.py +++ b/surfsense_local/backend/tests/unit/chat/test_errors.py @@ -78,3 +78,43 @@ def test_an_ordinary_400_is_still_provider_unavailable() -> None: exc = httpx.HTTPStatusError("400", request=request, response=response) assert classify_chat_error(exc, "llamacpp")[0] is ChatErrorKind.PROVIDER_UNAVAILABLE + + +def test_a_model_the_local_runtime_cannot_load_says_so() -> None: + """The failure the search gate used to prevent before the download. + + llama-server answers 500 when a model fails to load: an architecture this + build has no builder for, or one it builds and then aborts on. Bucketed + under PROVIDER_UNAVAILABLE that reads "try again shortly", which sends + someone into a retry loop over something that can never work. There is no + remote provider here either. It is our own subprocess. + """ + request = httpx.Request("POST", "http://x/v1/chat/completions") + response = _response_with_body( + 500, + { + "error": { + "code": 500, + "type": "server_error", + "message": "model name=GLM-5.3-Flash-Q4_K_M failed to load", + } + }, + ) + exc = httpx.HTTPStatusError("500", request=request, response=response) + + kind, message = classify_chat_error(exc, "llamacpp") + + assert kind is ChatErrorKind.MODEL_CANNOT_RUN + assert "try again" not in message.casefold() + assert "—" not in message + assert "-" not in message + + +def test_a_remote_provider_500_is_still_temporary() -> None: + """A hosted endpoint answering 500 really can recover on a retry. Only the + local runtime is making a statement about the file on disk.""" + request = httpx.Request("POST", "http://x/v1/chat/completions") + response = httpx.Response(500, request=request, content=b"") + exc = httpx.HTTPStatusError("500", request=request, response=response) + + assert classify_chat_error(exc, "openai")[0] is ChatErrorKind.PROVIDER_UNAVAILABLE diff --git a/surfsense_local/backend/tests/unit/llm/catalog/test_eligibility.py b/surfsense_local/backend/tests/unit/llm/catalog/test_eligibility.py index c29f15674e..8939a257c6 100644 --- a/surfsense_local/backend/tests/unit/llm/catalog/test_eligibility.py +++ b/surfsense_local/backend/tests/unit/llm/catalog/test_eligibility.py @@ -57,3 +57,38 @@ async def test_a_listing_that_fails_is_not_a_refusal_either() -> None: """The header read is still available, and it is the authoritative one.""" async with httpx.AsyncClient(transport=serving({}, status=500)) as client: assert await read_repo_facts(client, "some/repo") is None + + +async def test_the_pipeline_tag_is_read_in_the_same_call() -> None: + """A model built on a chat architecture and then trained to do something + else is invisible to the header gate. Measured: `Nemotron-3-Embed-1B` + declares `mistral3` and tags itself `sentence-similarity`. + + It rides along on the call already being made, so the check costs no extra + round trip. + """ + seen: list[httpx.URL] = [] + + def handle(request: httpx.Request) -> httpx.Response: + seen.append(request.url) + return httpx.Response( + 200, json={"gguf": GGUF, "pipeline_tag": "sentence-similarity"} + ) + + async with httpx.AsyncClient(transport=httpx.MockTransport(handle)) as client: + facts = await read_repo_facts(client, "some/embed-gguf") + + assert facts is not None + assert facts.pipeline_tag == "sentence-similarity" + assert len(seen) == 1 + assert "pipeline_tag" in str(seen[0]) + + +async def test_an_untagged_repo_reports_no_tag_rather_than_failing() -> None: + """Roughly four fifths of GGUF repos carry no tag, so the common case has + to be a quiet None and not a refusal.""" + async with httpx.AsyncClient(transport=serving({"gguf": GGUF})) as client: + facts = await read_repo_facts(client, "some/repo") + + assert facts is not None + assert facts.pipeline_tag is None diff --git a/surfsense_local/backend/tests/unit/llm/catalog/test_not_chat.py b/surfsense_local/backend/tests/unit/llm/catalog/test_not_chat.py new file mode 100644 index 0000000000..8c9502da99 --- /dev/null +++ b/surfsense_local/backend/tests/unit/llm/catalog/test_not_chat.py @@ -0,0 +1,208 @@ +"""The one list: what cannot hold a conversation, and what to say about it. + +A denylist, not an allowlist. It names what we have seen and admits everything +else, which is the only way a list stays right as llama.cpp adds architectures. +The reverse ages badly: the hand written allowlist this replaces refused 43 +architectures the runtime ran, including Gemma 4, while admitting `bert`, which +aborts the worker on the first message. + +Two facts feed it and they share the table, because they answer the same +question and their names never collide: + + the GGUF header's architecture what the file is built as + the repo's pipeline tag what the uploader says it is for + +The second exists because the first can be honest and still mislead. Measured +across the 1000 most downloaded GGUF repos, an embedding model declares +`mistral3`, a voice model declares `qwen3` and a video encoder declares +`qwen35`. Refusing those architectures would refuse Mistral and Qwen. +""" + +import pytest + +from modules.llm.catalog.search import is_supported, refusal + +pytestmark = pytest.mark.unit + + +def test_an_architecture_this_list_has_never_heard_of_is_admitted() -> None: + """The denylist's whole contract, and the change from what came before. + + llama.cpp gains architectures faster than any list is updated. An unknown + name is far more often a chat model released last week than something that + cannot run, so the default is yes. The cost is a wasted download when we + are wrong, and the chat error now says so plainly rather than telling the + reader to try again shortly. + """ + assert is_supported("glm5next") + assert is_supported("something-invented-yesterday") + + +def test_an_embedding_architecture_is_refused() -> None: + """It builds, loads, then aborts the worker on the first message. + + Measured against the pinned binary: a `nomic-bert` model reaches + `GGML_ASSERT(n_outputs_max <= cparams.n_outputs_max)` and the router + answers 500. Nothing downstream recovers, so the refusal belongs before + the download. + """ + assert not is_supported("nomic-bert") + assert not is_supported("bert") + assert not is_supported("gemma-embedding") + + +def test_a_chat_architecture_is_admitted() -> None: + """The ordinary case, and the one the old allowlist got wrong: it refused + Gemma 4 because nobody had retyped the list since llama.cpp added it.""" + assert is_supported("qwen3") + assert is_supported("gemma4") + + +def test_the_gate_is_case_insensitive() -> None: + """A header states its own architecture and publishers disagree on case.""" + assert is_supported("QWEN3") + assert not is_supported("Nomic-Bert") + + +def test_a_model_wearing_a_chat_architecture_is_refused_by_its_tag() -> None: + """The hole the header cannot see. `Nemotron-3-Embed-8B` declares + `mistral3`, which is Mistral's architecture and cannot be refused.""" + assert is_supported("mistral3") + assert not is_supported("mistral3", "sentence-similarity") + + +def test_a_vision_chat_model_is_not_refused_by_its_tag() -> None: + """`image-text-to-text` is 28% of the catalog: every Qwen 3.5, every + Gemma 4, every Qwen3 VL. Denying it would empty the search screen.""" + for tag in ("text-generation", "image-text-to-text", "any-to-any"): + assert is_supported("qwen35", tag), tag + + +def test_an_untagged_repo_is_admitted() -> None: + """Roughly four fifths of GGUF repos carry no tag. A missing tag is a + missing answer, not a negative one.""" + assert is_supported("qwen3", None) + assert is_supported("qwen3", "") + + +def test_a_file_for_different_software_says_that_rather_than_blaming_the_build( +) -> None: + """Stable Diffusion and video GGUFs are packaged for other runtimes. + + Telling someone a newer SurfSense would run their FLUX checkpoint is a + promise nothing will keep, and `pig` and `cow` are not architectures at + all: they are literal placeholders gguf-connector writes into the header. + """ + for name in ("flux", "wan", "sdxl", "pig", "cow"): + reason = refusal(name) + assert reason, name + assert "newer" not in reason.lower(), name + + +def test_a_refusal_says_what_the_model_is_rather_than_naming_its_architecture( +) -> None: + """The reader chose a model, not an architecture.""" + reason = refusal("nomic-bert") + + assert "nomic-bert" not in reason + assert "search" in reason.lower() + + +def test_an_embedding_reads_the_same_whichever_fact_caught_it() -> None: + """One table, one sentence. A user refused an embedding model should not + get different words depending on which fact happened to catch it.""" + assert refusal("nomic-bert") == refusal("mistral3", "sentence-similarity") + + +def test_each_kind_of_refusal_reads_differently() -> None: + """One sentence covering six situations is wrong in five of them.""" + reasons = { + refusal(name) + for name in ("nomic-bert", "qwen3tts", "paddleocr", "eagle3", "clip", "flux") + } + + assert len(reasons) == 6 + + +def test_a_supported_model_has_nothing_to_explain() -> None: + """Asking why a runnable model was refused is a question with no answer.""" + assert refusal("qwen3") == "" + assert refusal("qwen3", "text-generation") == "" + + +def test_refusal_copy_keeps_the_house_style() -> None: + """No em dashes and no hyphens in anything a person reads.""" + from modules.llm.catalog.search.not_chat import NOT_CHAT + + for name in NOT_CHAT: + reason = refusal(name) + assert "—" not in reason + assert "-" not in reason, reason + + +def test_the_tags_that_mean_chat_are_never_keys() -> None: + """The most expensive mistake available in this file. + + `image-text-to-text` alone is 28% of architecture admitted repos. Adding + any of these would turn away a whole family, silently and permanently. + """ + from modules.llm.catalog.search.not_chat import NOT_CHAT + + for tag in ( + "text-generation", + "image-text-to-text", + "video-text-to-text", + "audio-text-to-text", + "any-to-any", + ): + assert tag not in NOT_CHAT, tag + + +def test_speech_to_text_models_are_refused() -> None: + """The largest group that was installing and then failing. + + Measured over the 1000 most downloaded GGUF repos: 37 of the 75 that got + through and could not load are transcription models. Their names are stable + in a way a new chat architecture's is not, because llama.cpp is never going + to turn Whisper into something that answers questions. + """ + for name in ("whisper", "parakeet", "canary", "voxtral", "gigaam", "qwen3_asr"): + assert not is_supported(name), name + assert not is_supported("qwen2", "automatic-speech-recognition") + + +def test_transcription_reads_differently_from_reading_aloud() -> None: + """Opposite directions. One sentence for both would be wrong for one.""" + assert refusal("whisper") != refusal("qwen3tts") + + +def test_models_that_label_things_are_refused() -> None: + """Classifiers and detectors produce a label, not a conversation.""" + assert not is_supported("openai-privacy-filter") + assert not is_supported("qwen3", "token-classification") + assert not is_supported("qwen3", "object-detection") + + +def test_more_media_tools_are_refused() -> None: + """Image and video tools that reached the top 1000 and installed.""" + for name in ("krea2", "seedvr", "birefnet-swinl", "minimax_h3"): + assert not is_supported(name), name + assert not is_supported("qwen3", "text-to-image") + + +def test_a_control_vector_is_not_a_model() -> None: + """`creative-writing-control-vectors` ships steering vectors, not weights.""" + assert not is_supported("controlvector") + + +def test_a_new_chat_architecture_is_still_admitted() -> None: + """The line this list must not cross. + + `glm5next` and `inkling` are chat models llama.cpp has not merged yet, and + they carry `text-generation` like any other. Naming them here would refuse + them for the few weeks they are unsupported and then keep refusing them + forever, which is the drift this whole file is shaped to avoid. + """ + assert is_supported("glm5next", "text-generation") + assert is_supported("inkling", "image-text-to-text") + assert is_supported("deepseek41", None) diff --git a/surfsense_local/backend/tests/unit/llm/catalog/test_search_gates.py b/surfsense_local/backend/tests/unit/llm/catalog/test_search_gates.py index 71e79e6a28..93bc3d70f9 100644 --- a/surfsense_local/backend/tests/unit/llm/catalog/test_search_gates.py +++ b/surfsense_local/backend/tests/unit/llm/catalog/test_search_gates.py @@ -11,16 +11,21 @@ pytestmark = pytest.mark.unit -def test_a_supported_architecture_passes() -> None: - """The 152 name list, matched without caring about case.""" +def test_an_ordinary_chat_model_passes() -> None: + """Matched without caring about case: publishers disagree on it.""" assert is_supported("qwen3") assert is_supported("GEMMA3") -def test_an_architecture_llama_cpp_cannot_run_is_refused() -> None: - """No amount of memory helps, so this blocks rather than badges.""" - assert not is_supported("whisper") - assert not is_supported("something-invented") +def test_a_model_that_cannot_chat_is_refused() -> None: + """Eligibility blocks rather than badges: no amount of memory helps. + + A denylist, so an unknown name passes. `modules/llm/catalog/search/ + not_chat.py` explains why that direction is the one that ages well. + """ + assert not is_supported("nomic-bert") + assert not is_supported("flux") + assert is_supported("something-invented") def test_a_ticket_resolves_to_the_build_the_server_priced() -> None: diff --git a/surfsense_local/backend/tests/unit/llm/catalog/test_service.py b/surfsense_local/backend/tests/unit/llm/catalog/test_service.py index 5e5a5392e0..eab4423911 100644 --- a/surfsense_local/backend/tests/unit/llm/catalog/test_service.py +++ b/surfsense_local/backend/tests/unit/llm/catalog/test_service.py @@ -8,6 +8,7 @@ import pytest from modules.llm.catalog import CatalogService, load_curated_models +from modules.llm.gguf.source import PROBE_BYTES from modules.llm.hardware import GpuStatus from tests.unit.llm.gguf.build import STRING, UINT32, array, gguf, kv @@ -117,6 +118,23 @@ def no_runtime(_): assert catalog.devices == () +def gguf_bytes(*, architecture: str = "qwen3") -> bytes: + """A header a repo read can parse, for whatever architecture a test needs.""" + return gguf( + [ + kv("general.architecture", STRING, architecture), + kv(f"{architecture}.block_count", UINT32, 36), + kv(f"{architecture}.attention.head_count_kv", UINT32, 8), + kv(f"{architecture}.attention.key_length", UINT32, 128), + kv(f"{architecture}.attention.value_length", UINT32, 128), + kv(f"{architecture}.context_length", UINT32, 40960), + kv(f"{architecture}.embedding_length", UINT32, 4096), + kv(f"{architecture}.feed_forward_length", UINT32, 12288), + array("tokenizer.ggml.tokens", STRING, ["a", "b"]), + ] + ) + + _REAL_ASYNC_CLIENT = httpx.AsyncClient @@ -133,7 +151,13 @@ def _serve_repo(monkeypatch, transport: httpx.MockTransport) -> None: ) -def _repo_transport(recorder: list[str], *, gguf: dict | None, header: bytes | None): +def _repo_transport( + recorder: list[str], + *, + gguf: dict | None, + header: bytes | None, + tag: str | None = None, +): """Answers the three calls `repo()` makes, and records which were made.""" def handle(request: httpx.Request) -> httpx.Response: @@ -150,26 +174,46 @@ def handle(request: httpx.Request) -> httpx.Response: if header is None: return httpx.Response(500) return httpx.Response(200, content=header) - return httpx.Response(200, json={"gguf": gguf} if gguf else {}) + listing: dict = {"gguf": gguf} if gguf else {} + if tag is not None: + listing["pipeline_tag"] = tag + return httpx.Response(200, json=listing) return httpx.MockTransport(handle) -async def test_an_architecture_llama_cpp_cannot_run_costs_no_header_read( +async def test_a_refusal_is_made_from_the_file_and_costs_only_the_probe( tmp_path, monkeypatch ) -> None: - """The listing already said so. Reading a range out of a multi gigabyte file - to re-learn it is bytes spent on a question that was already answered.""" - calls: list[str] = [] - transport = _repo_transport(calls, gguf={"architecture": "whisper"}, header=None) - _serve_repo(monkeypatch, transport) + """A refusal used to be free and was wrong 17 times per 1000 repos. + + It now reads the candidate build, and reads it cheaply: a file that is not a + chat model carries no tokenizer, so its header ends within the probe and the + widening steps are never asked for. Being right costs a quarter of a + megabyte, once, on a repo nobody was going to install. + """ + ranges: list[str] = [] + + def handle(request): + if "/tree/main" in request.url.path: + return httpx.Response( + 200, json=[{"path": "model-Q4_K_M.gguf", "size": 5_000_000_000}] + ) + if "/resolve/main/" in request.url.path: + ranges.append(request.headers["Range"]) + return httpx.Response(206, content=gguf_bytes(architecture="nomic-bert")) + return httpx.Response(200, json={"gguf": {"architecture": "nomic-bert"}}) + + _serve_repo(monkeypatch, httpx.MockTransport(handle)) service = CatalogService(load_curated_models(), tmp_path, tmp_path, probe=lambda _: []) - body = await service.repo("some/whisper-gguf") + body = await service.repo("some/embed-gguf") assert body["supported"] is False - assert "whisper" in body["ineligible_reason"] - assert not any("/resolve/main/" in url for url in calls) + # The wording itself is `not_chat.refusal`'s contract and is asserted there; + # a repo refused without one would leave a blank row on the screen. + assert body["ineligible_reason"] + assert ranges == [f"bytes=0-{PROBE_BYTES - 1}"] async def test_an_unsupported_repo_still_lists_what_is_in_it( @@ -177,11 +221,13 @@ async def test_an_unsupported_repo_still_lists_what_is_in_it( ) -> None: """The screen answers "what is in here" as well as "can I run it", and an empty repo reads as a broken page rather than an unsupported model.""" - transport = _repo_transport([], gguf={"architecture": "whisper"}, header=None) + transport = _repo_transport( + [], gguf=None, header=gguf_bytes(architecture="nomic-bert") + ) _serve_repo(monkeypatch, transport) service = CatalogService(load_curated_models(), tmp_path, tmp_path, probe=lambda _: []) - body = await service.repo("some/whisper-gguf") + body = await service.repo("some/embed-gguf") assert len(body["builds"]) == 2 assert all(not build["can_install"] for build in body["builds"]) @@ -208,20 +254,9 @@ async def test_a_readable_header_is_never_called_approximate( ) -> None: """The flag means "priced from the file size alone", so a real read must clear it or the screen hedges a number it did not need to hedge.""" - header = gguf( - [ - kv("general.architecture", STRING, "qwen3"), - kv("qwen3.block_count", UINT32, 36), - kv("qwen3.attention.head_count_kv", UINT32, 8), - kv("qwen3.attention.key_length", UINT32, 128), - kv("qwen3.attention.value_length", UINT32, 128), - kv("qwen3.context_length", UINT32, 40960), - kv("qwen3.embedding_length", UINT32, 4096), - kv("qwen3.feed_forward_length", UINT32, 12288), - array("tokenizer.ggml.tokens", STRING, ["a", "b"]), - ] + transport = _repo_transport( + [], gguf={"architecture": "qwen3"}, header=gguf_bytes() ) - transport = _repo_transport([], gguf={"architecture": "qwen3"}, header=header) _serve_repo(monkeypatch, transport) service = CatalogService(load_curated_models(), tmp_path, tmp_path, probe=lambda _: []) @@ -229,3 +264,107 @@ async def test_a_readable_header_is_never_called_approximate( assert body["supported"] is True assert not any(build["fit"]["approximate"] for build in body["builds"]) + + +async def test_a_model_wearing_a_chat_architecture_is_refused_by_its_tag( + tmp_path, monkeypatch +) -> None: + """The hole the header gate cannot see. + + Measured across the 500 most downloaded GGUF repos: `Nemotron-3-Embed-8B` + declares `mistral3`, which llama.cpp builds and `NOT_CHAT` has no reason to + refuse, so it installs as though it were a chat model. Its repo tag says + `sentence-similarity`, and that is the only thing that knows. + """ + transport = _repo_transport( + [], + gguf={"architecture": "mistral3"}, + header=gguf_bytes(architecture="mistral3"), + tag="sentence-similarity", + ) + _serve_repo(monkeypatch, transport) + service = CatalogService(load_curated_models(), tmp_path, tmp_path, probe=lambda _: []) + + body = await service.repo("nvidia/Nemotron-3-Embed-8B-GGUF") + + assert body["supported"] is False + assert "search" in body["ineligible_reason"].lower() + + +async def test_an_untagged_repo_is_priced_normally(tmp_path, monkeypatch) -> None: + """Four fifths of GGUF repos carry no tag. Treating a missing tag as a + refusal would empty the search screen, which is why this is a denylist.""" + transport = _repo_transport([], gguf={"architecture": "qwen3"}, header=None, tag=None) + _serve_repo(monkeypatch, transport) + service = CatalogService(load_curated_models(), tmp_path, tmp_path, probe=lambda _: []) + + body = await service.repo("some/qwen3-gguf") + + assert body["supported"] is True + assert body["builds"] + + +async def test_a_vision_chat_model_is_not_refused_by_its_tag( + tmp_path, monkeypatch +) -> None: + """`image-text-to-text` is how a multimodal chat model is tagged. Reading it + as a picture model would turn away the whole vision family.""" + transport = _repo_transport( + [], gguf={"architecture": "qwen3vl"}, header=None, tag="image-text-to-text" + ) + _serve_repo(monkeypatch, transport) + service = CatalogService(load_curated_models(), tmp_path, tmp_path, probe=lambda _: []) + + body = await service.repo("some/qwen3-vl-gguf") + + assert body["supported"] is True + + +async def test_a_vision_repo_is_judged_by_its_model_not_its_sidecar( + tmp_path, monkeypatch +) -> None: + """The measured defect this replaces. + + Hugging Face parses one GGUF per repo and serves that as the repo's answer. + `Jackrong/Qwen3.8-27B-MTP-GGUF` holds twelve builds of a Qwen 3.5 chat model + beside one vision sidecar; Hugging Face read the sidecar and reported the + repo as `clip`. 17 of the 1000 most downloaded repos are refused this way, + 13 of them vision models, and the refusal tells the reader to install the + model it belongs to, which is that repo. + + `list_builds` already excludes the sidecar, so the candidate build's own + header is the answer and we already fetch it. + """ + header = gguf_bytes(architecture="qwen35") + transport = _repo_transport( + [], + gguf={"architecture": "clip"}, + header=header, + tag="image-text-to-text", + ) + _serve_repo(monkeypatch, transport) + service = CatalogService(load_curated_models(), tmp_path, tmp_path, probe=lambda _: []) + + body = await service.repo("Jackrong/Qwen3.8-27B-MTP-GGUF") + + assert body["supported"] is True + assert body["architecture"] == "qwen35" + assert len(body["builds"]) == 2 + + +async def test_a_repo_whose_model_really_cannot_chat_is_still_refused( + tmp_path, monkeypatch +) -> None: + """The other half: the header is now the authority in both directions, so a + real embedding repo is refused on its own bytes rather than on a summary.""" + header = gguf_bytes(architecture="nomic-bert") + transport = _repo_transport( + [], gguf={"architecture": "nomic-bert"}, header=header, tag=None + ) + _serve_repo(monkeypatch, transport) + service = CatalogService(load_curated_models(), tmp_path, tmp_path, probe=lambda _: []) + + body = await service.repo("nomic-ai/some-embed-gguf") + + assert body["supported"] is False + assert "search" in body["ineligible_reason"].lower() diff --git a/surfsense_local/backend/tests/unit/llm/fit/test_sliding_window.py b/surfsense_local/backend/tests/unit/llm/fit/test_sliding_window.py index 1262720ca4..a88dc1dd04 100644 --- a/surfsense_local/backend/tests/unit/llm/fit/test_sliding_window.py +++ b/surfsense_local/backend/tests/unit/llm/fit/test_sliding_window.py @@ -134,3 +134,4 @@ def test_layers_that_share_a_cache_allocate_none_of_their_own() -> None: assert kv_cache_bytes(shared, 16384, KvPrecision.F16) == kv_cache_bytes( alone, 16384, KvPrecision.F16 ) + diff --git a/surfsense_local/backend/tests/unit/llm/gguf/test_file_kind.py b/surfsense_local/backend/tests/unit/llm/gguf/test_file_kind.py new file mode 100644 index 0000000000..5ab0d12802 --- /dev/null +++ b/surfsense_local/backend/tests/unit/llm/gguf/test_file_kind.py @@ -0,0 +1,105 @@ +"""What a GGUF says it is, read from the file rather than guessed from its name. + +Three different things used to answer this question, all by guessing: +`list_builds` matched `"mmproj"` in the path, `projector.py` matched the same +string again, and the gate asked Hugging Face, which parses one file per repo +and serves that as the repo's answer. + +The last one is how 17 of the 1000 most downloaded repos come to be refused +today. `Jackrong/Qwen3.8-27B-MTP-GGUF` ships twelve builds of a Qwen 3.5 chat +model beside one vision sidecar. Hugging Face read the sidecar, reported the +repo as `clip`, and the whole thing is turned away with "This is the vision half +of another model", advice that cannot be followed because the model it belongs +to is that repo. + +GGUF answers this itself. `general.type` is an official key with an official +enum in `gguf.constants.GGUFType`, and real files populate it: both projectors +measured on the hub carry `general.type='mmproj'`. + +The truncation rule is the other half. A projector has no tokenizer, so its +header fits in a small prefix and parses. A chat model's `tokenizer.ggml.tokens` +runs to megabytes, so the same prefix cuts mid metadata. Measured: Qwen3 0.6B +5.93 MB, Llama 3.2 1B 7.82 MB. A header that does not fit is therefore a real +model, which makes the cheap read and the safe answer the same read. +""" + +import pytest + +from modules.llm.gguf.file_kind import PROBE_BYTES, FileKind, file_kind +from tests.unit.llm.gguf.build import STRING, UINT32, gguf, kv + +pytestmark = pytest.mark.unit + + +def test_a_projector_says_so_whatever_it_is_called() -> None: + """The bug this replaces, stated as a contract: the name is not consulted.""" + raw = gguf([kv("general.type", STRING, "mmproj"), + kv("general.architecture", STRING, "clip")]) + + assert file_kind(raw).kind is FileKind.PROJECTOR + + +def test_a_model_says_so() -> None: + """The ordinary case for a small model whose header does fit.""" + raw = gguf([kv("general.type", STRING, "model"), + kv("general.architecture", STRING, "qwen3")]) + + kind = file_kind(raw) + + assert kind.kind is FileKind.MODEL + assert kind.architecture == "qwen3" + + +def test_a_header_too_big_for_the_prefix_is_a_model() -> None: + """The measured rule, and the reason the refusal path is the cheap one. + + A chat model's vocabulary pushes its metadata past any prefix worth + fetching. Nothing else in a repo does that, so a truncated read is not a + failure to answer, it is the answer. + """ + raw = gguf([kv("general.type", STRING, "model"), + kv("general.architecture", STRING, "qwen3")]) + + assert file_kind(raw[:20]).kind is FileKind.MODEL + + +def test_a_file_that_is_not_a_gguf_at_all_is_a_model() -> None: + """Fails open, like every other step. A refusal is never made from a + failure to read, because the runtime is the authority and it will refuse + with the same sentence if this is wrong.""" + assert file_kind(b"this is not a gguf").kind is FileKind.MODEL + + +def test_an_older_file_with_no_type_key_falls_back_to_the_architecture() -> None: + """`general.type` postdates GGUF v1. A file without it is judged the way + everything was judged before, by what it declares itself to be.""" + raw = gguf([kv("general.architecture", STRING, "clip")]) + + kind = file_kind(raw) + + assert kind.kind is FileKind.PROJECTOR + assert kind.architecture == "clip" + + +def test_an_imatrix_and_an_adapter_are_named_too() -> None: + """Both are valid GGUF containers holding something that is not a model, so + only the header separates them. `list_builds` matches neither by name.""" + assert file_kind(gguf([kv("general.type", STRING, "imatrix")])).kind is FileKind.IMATRIX + assert file_kind(gguf([kv("general.type", STRING, "adapter")])).kind is FileKind.ADAPTER + + +def test_a_shard_is_recognised_from_its_split_keys() -> None: + """`-of-` in a filename is the guess this replaces. `split.count` is the + file saying it.""" + raw = gguf([kv("general.type", STRING, "model"), + kv("general.architecture", STRING, "qwen3"), + kv("split.count", UINT32, 4)]) + + assert file_kind(raw).kind is FileKind.SHARD + + +def test_the_probe_is_small_enough_that_a_refusal_is_cheaper_than_a_pricing_read() -> None: + """The economics that make this worth doing at all. Today a refusal costs + nothing and is wrong 1.7% of the time; this makes it cost a quarter of a + megabyte and be right.""" + assert PROBE_BYTES <= 512 * 1024 diff --git a/surfsense_local/backend/tests/unit/llm/gguf/test_source.py b/surfsense_local/backend/tests/unit/llm/gguf/test_source.py index 446aa3fbca..dd78a09a07 100644 --- a/surfsense_local/backend/tests/unit/llm/gguf/test_source.py +++ b/surfsense_local/backend/tests/unit/llm/gguf/test_source.py @@ -4,7 +4,7 @@ import pytest from modules.llm.gguf import shape_from_file, shape_from_url -from modules.llm.gguf.source import INITIAL_BYTES +from modules.llm.gguf.source import INITIAL_BYTES, PROBE_BYTES from tests.unit.llm.gguf.build import STRING, UINT32, array, gguf, kv pytestmark = pytest.mark.unit @@ -27,7 +27,13 @@ def a_gguf(vocab: int = 3) -> bytes: @pytest.mark.asyncio async def test_only_the_front_of_the_file_is_requested() -> None: - """The point of the whole exercise: price a 40 GB model without fetching it.""" + """The point of the whole exercise: price a 40 GB model without fetching it. + + The first ask is the probe rather than the pricing read. A file that is not + a chat model answers there and costs a quarter of a megabyte, which is what + makes reading the real file affordable on the refusal path. A chat model + truncates and widens, so it pays exactly what it paid before. + """ seen: list[str] = [] def handler(request: httpx.Request) -> httpx.Response: @@ -38,7 +44,7 @@ def handler(request: httpx.Request) -> httpx.Response: async with httpx.AsyncClient(transport=transport) as client: shape = await shape_from_url(client, "https://example.invalid/m.gguf") - assert seen == [f"bytes=0-{INITIAL_BYTES - 1}"] + assert seen == [f"bytes=0-{PROBE_BYTES - 1}"] assert shape.block_count == 28 From b713ad56c28f849dbb3f31885cb3224360f38e2d Mon Sep 17 00:00:00 2001 From: Anish Sarkar <104695310+AnishSarkar22@users.noreply.github.com> Date: Tue, 22 Sep 2026 12:30:26 +0530 Subject: [PATCH 57/60] feat(llm): gate installs on the file's own header, not names or repo summaries One denylist replaces the generated architecture table and its script. The gate reads the candidate build's header instead of Hugging Face's repo summary, which parses one file per repo and reported a chat model shipped beside a vision sidecar as `clip`. Measured over the top 1000 GGUF repos: wrong refusals 17 to 0, installs that then fail 75 to 18, and those now say so instead of asking the reader to retry. --- .../backend/modules/llm/catalog/service.py | 75 ++++++++++++---- .../backend/modules/llm/gguf/file_kind.py | 20 ++++- .../llm/providers/llamacpp/__init__.py | 3 +- .../llm/providers/llamacpp/projector.py | 18 ++-- .../tests/unit/llm/catalog/test_reprice.py | 21 ++++- .../tests/unit/llm/catalog/test_service.py | 89 +++++++++++++++++++ .../tests/unit/llm/gguf/test_file_kind.py | 34 +++++++ .../llm/providers/llamacpp/test_projector.py | 19 ++-- 8 files changed, 238 insertions(+), 41 deletions(-) diff --git a/surfsense_local/backend/modules/llm/catalog/service.py b/surfsense_local/backend/modules/llm/catalog/service.py index 5ef848f879..a162de09ee 100644 --- a/surfsense_local/backend/modules/llm/catalog/service.py +++ b/surfsense_local/backend/modules/llm/catalog/service.py @@ -40,10 +40,9 @@ plan_load, planned_precision, ) -from modules.llm.gguf import shape_from_file -from modules.llm.gguf.file_kind import FileKind, kind_of +from modules.llm.gguf.file_kind import FileKind, GgufFile, kind_of from modules.llm.gguf.shape import to_shape -from modules.llm.gguf.source import header_from_url +from modules.llm.gguf.source import header_from_file, header_from_url from modules.llm.hardware import ( BudgetMode, Device, @@ -63,7 +62,6 @@ ModelPreset, RouterClient, download_gguf, - is_projector, projector_for, write_presets, ) @@ -73,6 +71,12 @@ # Priced from the listing alone when a header cannot be read. Every # architecture-derived term is zero, so `need` is the file size and nothing # else, and the row is marked approximate so the screen can say so. +# How many builds the candidate walk will read before giving up. A repo +# needs more than a handful of companions sorting ahead of its model for +# this to matter, and each read is a probe, so the cap is about bounding a +# pathological repo rather than a cost anyone pays. +_CANDIDATES = 4 + _SIZE_ONLY_SHAPE = ModelShape( architecture="", block_count=0, @@ -183,13 +187,16 @@ def reprice(self) -> None: live = self.budget(BudgetMode.LIVE) presets = [] for path in sorted(self._models_dir.glob("*.gguf")): - # A projector is half of a vision model, not a model. It has a - # header and a size like any other file here, so without this it - # would be offered as something to chat with. - if is_projector(path): - continue try: - shape = shape_from_file(path) + header = header_from_file(path) + # A projector is half of a vision model, not a model. It has a + # header and a size like any other file here, so without this it + # would be offered as something to chat with. Asked of the file + # rather than of its name: this loop reads the header anyway, so + # the name only ever saved a read it was about to do. + if kind_of(header).kind is not FileKind.MODEL: + continue + shape = to_shape(header) except (OSError, ValueError): # A truncated or foreign file in the directory. Skipping it # costs that one model; failing would leave the runtime dead @@ -316,13 +323,8 @@ async def repo(self, repo: str) -> dict: if facts is not None: chat_template = facts.has_chat_template - candidate = None - try: - url = f"https://huggingface.co/{repo}/resolve/main/{builds[0].file}" - header = await header_from_url(client, url) - shape = to_shape(header) - candidate = kind_of(header) - except (httpx.HTTPError, ValueError): + candidate, shape = await self._candidate(client, repo, builds) + if shape is None: # The listing is enough to say how large each build is, and a # size alone still orders the ladder. Refusing the whole repo # over one unreadable header would hide builds the user can run. @@ -389,6 +391,45 @@ async def repo(self, repo: str) -> dict: "ineligible_reason": None, } + async def _candidate( + self, client: httpx.AsyncClient, repo: str, builds: list + ) -> tuple[GgufFile | None, ModelShape | None]: + """The first build whose own header says it is a model. + + `list_builds` orders by size and a draft head is always smaller than the + model it accelerates, so in a repo shipping both, the smallest file is a + companion. Judging it would refuse every build of a working model, which + is the failure Hugging Face's repo summary used to cause, reproduced + from our own ordering. + + So the order is a suggestion and the header is the answer. A companion + costs one probe to rule out, because it carries no tokenizer and its + metadata ends inside it, while a chat model truncates there and widens + exactly as it did before. + + The question is "does this repo hold anything runnable", so a build that + is a model but a denied one keeps the walk going. A draft head declares + `general.type = model` and is one, structurally; only the denylist knows + it answers nothing. Refusing on the first denied build would turn away + a working model shipped beside its own accelerator. + + Stops after `_CANDIDATES` reads, and then reports the last thing it saw + so the refusal names something real. That is the right answer for a repo + of nothing but sidecars, which is what a drafter-only repo is. + """ + last: tuple[GgufFile | None, ModelShape | None] = (None, None) + for build in builds[:_CANDIDATES]: + url = f"https://huggingface.co/{repo}/resolve/main/{build.file}" + try: + header = await header_from_url(client, url) + except (httpx.HTTPError, ValueError): + return last + found = kind_of(header) + last = (found, to_shape(header)) + if found.kind is FileKind.MODEL and not refusal(found.architecture): + return last + return last + def _ineligible( self, repo: str, diff --git a/surfsense_local/backend/modules/llm/gguf/file_kind.py b/surfsense_local/backend/modules/llm/gguf/file_kind.py index 4237aae7a2..af3a9b5059 100644 --- a/surfsense_local/backend/modules/llm/gguf/file_kind.py +++ b/surfsense_local/backend/modules/llm/gguf/file_kind.py @@ -48,6 +48,7 @@ class FileKind(StrEnum): ADAPTER = "adapter" IMATRIX = "imatrix" SHARD = "shard" + NOT_LOADABLE = "not_loadable" @dataclass(frozen=True) @@ -59,11 +60,13 @@ class GgufFile: architecture: str = "" -_BY_TYPE = { +# Only the companion types. `GGUFType.MODEL` is deliberately absent: a file +# claiming to be a model still has to name an architecture, and the check below +# is what catches one that does not. +_COMPANION_TYPES = { GGUFType.MMPROJ: FileKind.PROJECTOR, GGUFType.ADAPTER: FileKind.ADAPTER, GGUFType.IMATRIX: FileKind.IMATRIX, - GGUFType.MODEL: FileKind.MODEL, } # Written by older writers that predate `general.type`, and by the patched @@ -93,9 +96,18 @@ def kind_of(header: GgufHeader) -> GgufFile: return GgufFile(FileKind.SHARD, architecture) declared = meta.get(Keys.General.TYPE) - if isinstance(declared, str) and declared in _BY_TYPE: - return GgufFile(_BY_TYPE[declared], architecture) + if isinstance(declared, str) and declared in _COMPANION_TYPES: + return GgufFile(_COMPANION_TYPES[declared], architecture) if architecture.lower() in _PROJECTOR_ARCHITECTURES: return GgufFile(FileKind.PROJECTOR, architecture) + + # Silence as an answer, rather than a missing answer. `general.architecture` + # is what llama.cpp's loader dispatches on, so a file that declares none + # fails every time. Measured: MiniMax-H3's video GGUFs carry a bare tensor + # header with zero metadata and are among the repos that install and then + # fail. Reachable only after a successful parse, which is what keeps it off + # the truncated path where silence means nothing at all. + if not architecture: + return GgufFile(FileKind.NOT_LOADABLE) return GgufFile(FileKind.MODEL, architecture) diff --git a/surfsense_local/backend/modules/llm/providers/llamacpp/__init__.py b/surfsense_local/backend/modules/llm/providers/llamacpp/__init__.py index 40c507944f..9edcb20edf 100644 --- a/surfsense_local/backend/modules/llm/providers/llamacpp/__init__.py +++ b/surfsense_local/backend/modules/llm/providers/llamacpp/__init__.py @@ -11,7 +11,7 @@ render_presets, write_presets, ) -from modules.llm.providers.llamacpp.projector import is_projector, projector_for +from modules.llm.providers.llamacpp.projector import projector_for from modules.llm.providers.llamacpp.provider import PROVIDER, LlamaCppProvider from modules.llm.providers.llamacpp.router_client import RouterClient, RouterModel @@ -26,7 +26,6 @@ "RouterModel", "download_gguf", "for_template", - "is_projector", "projector_for", "read_capabilities", "render_presets", diff --git a/surfsense_local/backend/modules/llm/providers/llamacpp/projector.py b/surfsense_local/backend/modules/llm/providers/llamacpp/projector.py index f278c1044f..70bd079d51 100644 --- a/surfsense_local/backend/modules/llm/providers/llamacpp/projector.py +++ b/surfsense_local/backend/modules/llm/providers/llamacpp/projector.py @@ -5,9 +5,10 @@ second only when told to. The fitter does not count its memory either, so the margin has to carry it. -Both files sit in the models directory, so this is also what stops a projector -being offered as a model in its own right: it has a GGUF header and a size, and -nothing else about it says it cannot answer a question. +Pairing only. What a file *is* comes from its header, in `gguf/file_kind.py`, +and `reprice` asks there before offering anything as a model. The name is still +how the pair is found, because that search globs a whole directory and a header +read per file is the cost this was always avoiding. """ from collections.abc import Iterable @@ -16,12 +17,11 @@ _MARKER = "mmproj" -def is_projector(path: Path) -> bool: - """Whether this file is a projector rather than a model. +def _named_as_projector(path: Path) -> bool: + """Whether the name says projector, which is how the pair is found. - Named by convention rather than read from the header: every publisher marks - them this way, and the alternative is a header read per file on a path that - runs at startup. + Not a verdict on what the file is. `file_kind` answers that from the header, + and `reprice` asks it before anything is offered as a model. """ return _MARKER in path.stem.lower() @@ -41,5 +41,5 @@ def projector_for(model_path: Path, named: Iterable[str] = ()) -> Path | None: if candidate.exists(): return candidate - beside = [path for path in sorted(directory.glob("*.gguf")) if is_projector(path)] + beside = [path for path in sorted(directory.glob("*.gguf")) if _named_as_projector(path)] return beside[0] if len(beside) == 1 else None diff --git a/surfsense_local/backend/tests/unit/llm/catalog/test_reprice.py b/surfsense_local/backend/tests/unit/llm/catalog/test_reprice.py index 8e60564915..831b7c9970 100644 --- a/surfsense_local/backend/tests/unit/llm/catalog/test_reprice.py +++ b/surfsense_local/backend/tests/unit/llm/catalog/test_reprice.py @@ -38,6 +38,23 @@ def a_model(path: Path, *, blocks: int = 28, ctx: int = 40960) -> None: ) +def a_projector(path: Path) -> None: + """A real projector on disk, which is what the name used to stand in for. + + `reprice` now asks the header rather than the filename, so a file has to + carry `general.type = mmproj` to be one. Every projector measured on the hub + does. + """ + path.write_bytes( + gguf( + [ + kv("general.type", STRING, "mmproj"), + kv("general.architecture", STRING, "clip"), + ] + ) + ) + + @pytest.fixture def service(tmp_path: Path) -> CatalogService: """A service over an empty models directory.""" @@ -156,7 +173,7 @@ def test_a_projector_is_never_offered_as_a_model(tmp_path: Path) -> None: other file here, so without a rule it gets its own section and the user is offered something that cannot answer a question.""" a_model(tmp_path / "Qwen3-VL-Q4_K_M.gguf") - a_model(tmp_path / "mmproj-F16.gguf") + a_projector(tmp_path / "mmproj-F16.gguf") service = CatalogService(load_curated_models(), tmp_path, tmp_path) service.reprice() @@ -173,7 +190,7 @@ def test_a_vision_model_names_its_projector_and_reserves_room_for_it( does not count it while deciding, so the margin has to carry its bytes.""" a_model(tmp_path / "Qwen3-VL-Q4_K_M.gguf") projector = tmp_path / "mmproj-F16.gguf" - a_model(projector) + a_projector(projector) padding = 600 * 1024**2 - projector.stat().st_size projector.write_bytes(projector.read_bytes() + b"\0" * padding) service = CatalogService(load_curated_models(), tmp_path, tmp_path) diff --git a/surfsense_local/backend/tests/unit/llm/catalog/test_service.py b/surfsense_local/backend/tests/unit/llm/catalog/test_service.py index eab4423911..048ffbe8b5 100644 --- a/surfsense_local/backend/tests/unit/llm/catalog/test_service.py +++ b/surfsense_local/backend/tests/unit/llm/catalog/test_service.py @@ -368,3 +368,92 @@ async def test_a_repo_whose_model_really_cannot_chat_is_still_refused( assert body["supported"] is False assert "search" in body["ineligible_reason"].lower() + + +async def test_a_sidecar_sorting_first_does_not_refuse_the_whole_repo( + tmp_path, monkeypatch +) -> None: + """The last place a filename still decided something. + + `list_builds` orders by size and a draft head is always smaller than the + model it accelerates, so in a repo shipping both, the head is `builds[0]`. + Judging it would refuse every build of a working model, which is the same + failure Hugging Face's repo summary used to cause, from our own ordering + instead. + + The file says what it is, so the walk moves on rather than believing the + order it was handed. + """ + served = { + "Qwen3-Q8_0-MTP.gguf": gguf_bytes(architecture="eagle3"), + "model-Q4_K_M.gguf": gguf_bytes(architecture="qwen3"), + } + + def handle(request: httpx.Request) -> httpx.Response: + if "/tree/main" in request.url.path: + return httpx.Response( + 200, + json=[ + {"path": "MTP/Qwen3-Q8_0-MTP.gguf", "size": 1_000_000_000}, + {"path": "model-Q4_K_M.gguf", "size": 5_000_000_000}, + ], + ) + if "/resolve/main/" in request.url.path: + name = request.url.path.rsplit("/", 1)[-1] + return httpx.Response(206, content=served[name]) + return httpx.Response(200, json={"gguf": {"architecture": "qwen3"}}) + + _serve_repo(monkeypatch, httpx.MockTransport(handle)) + service = CatalogService(load_curated_models(), tmp_path, tmp_path, probe=lambda _: []) + + body = await service.repo("some/qwen3-with-a-drafter") + + assert body["supported"] is True + assert body["architecture"] == "qwen3" + + +async def test_a_repo_of_nothing_but_sidecars_is_still_refused( + tmp_path, monkeypatch +) -> None: + """The walk admits nothing on its own. A repo that never yields a model is + described by the last thing it did yield, not quietly let through.""" + def handle(request: httpx.Request) -> httpx.Response: + if "/tree/main" in request.url.path: + return httpx.Response( + 200, json=[{"path": "MTP/Qwen3-Q8_0-MTP.gguf", "size": 1_000_000_000}] + ) + if "/resolve/main/" in request.url.path: + return httpx.Response(206, content=gguf_bytes(architecture="eagle3")) + return httpx.Response(200, json={}) + + _serve_repo(monkeypatch, httpx.MockTransport(handle)) + service = CatalogService(load_curated_models(), tmp_path, tmp_path, probe=lambda _: []) + + body = await service.repo("some/drafters-only") + + assert body["supported"] is False + + +def test_a_projector_named_anything_is_not_offered_as_a_model(tmp_path) -> None: + """The last filename guess on the local side. + + `reprice` reads every header anyway, to price the model, so checking the + name first only saved a read it was about to do. Reading first and asking + the file makes it right for a publisher who did not follow the convention, + and `general.type = mmproj` is what every projector measured on the hub + actually carries. + """ + models = tmp_path / "models" + models.mkdir() + (models / "vision-half-BF16.gguf").write_bytes( + gguf([kv("general.type", STRING, "mmproj"), + kv("general.architecture", STRING, "clip")]) + ) + (models / "Qwen3-8B-Q4_K_M.gguf").write_bytes(gguf_bytes()) + service = CatalogService(load_curated_models(), models, tmp_path, probe=lambda _: []) + + service.reprice() + + written = (models / "models.ini").read_text() + assert "vision-half-BF16" not in written + assert "Qwen3-8B-Q4_K_M" in written diff --git a/surfsense_local/backend/tests/unit/llm/gguf/test_file_kind.py b/surfsense_local/backend/tests/unit/llm/gguf/test_file_kind.py index 5ab0d12802..14f6791fe8 100644 --- a/surfsense_local/backend/tests/unit/llm/gguf/test_file_kind.py +++ b/surfsense_local/backend/tests/unit/llm/gguf/test_file_kind.py @@ -103,3 +103,37 @@ def test_the_probe_is_small_enough_that_a_refusal_is_cheaper_than_a_pricing_read nothing and is wrong 1.7% of the time; this makes it cost a quarter of a megabyte and be right.""" assert PROBE_BYTES <= 512 * 1024 + + +def test_a_header_that_parsed_and_named_no_architecture_cannot_be_loaded() -> None: + """The one case where silence is an answer rather than a missing answer. + + `general.architecture` is mandatory for anything llama.cpp can load: it is + what the loader dispatches on. A file that declares none will fail every + time. Measured on the hub: MiniMax-H3's video GGUFs carry a bare tensor + header with zero metadata, and they are among the repos that install and + then fail today. + + Safe only because the parse succeeded. A short read raises before reaching + this, which is what separates "I read it and there was nothing" from "I + could not finish reading", and the second must always admit. + """ + raw = gguf([kv("general.type", STRING, "model")]) + + assert file_kind(raw).kind is FileKind.NOT_LOADABLE + + +def test_a_bare_tensor_header_with_no_metadata_cannot_be_loaded() -> None: + """The measured shape: a valid GGUF container holding nothing that says + what it is.""" + assert file_kind(gguf([])).kind is FileKind.NOT_LOADABLE + + +def test_a_truncated_read_still_admits() -> None: + """The line this must never cross. A chat model's word list pushes its + header past any prefix worth fetching, so truncation is the common case and + refusing on it would refuse almost everything.""" + raw = gguf([kv("general.type", STRING, "model"), + kv("general.architecture", STRING, "qwen3")]) + + assert file_kind(raw[:20]).kind is FileKind.MODEL diff --git a/surfsense_local/backend/tests/unit/llm/providers/llamacpp/test_projector.py b/surfsense_local/backend/tests/unit/llm/providers/llamacpp/test_projector.py index 2fb3b31145..9e95db2cf4 100644 --- a/surfsense_local/backend/tests/unit/llm/providers/llamacpp/test_projector.py +++ b/surfsense_local/backend/tests/unit/llm/providers/llamacpp/test_projector.py @@ -4,7 +4,7 @@ import pytest -from modules.llm.providers.llamacpp import is_projector, projector_for +from modules.llm.providers.llamacpp import projector_for pytestmark = pytest.mark.unit @@ -15,12 +15,17 @@ def touch(path: Path) -> Path: return path -def test_a_projector_is_recognised_however_it_is_named() -> None: - """Every publisher marks them the same way, which is why the name is enough - and a header read per file at startup is not needed.""" - assert is_projector(Path("mmproj-F16.gguf")) - assert is_projector(Path("Qwen3-VL-mmproj-BF16.gguf")) - assert not is_projector(Path("Qwen3-8B-Q4_K_M.gguf")) +def test_the_pair_is_found_by_name_whichever_way_it_is_written() -> None: + """Finding the pair is a directory search, so it stays a name match: a + header read per file is the cost this avoids. What a file *is* is decided + from its header in `gguf/file_kind.py`, and `reprice` asks there before + offering anything as a model, so a badly named projector is caught even + though this never sees it.""" + from modules.llm.providers.llamacpp.projector import _named_as_projector + + assert _named_as_projector(Path("mmproj-F16.gguf")) + assert _named_as_projector(Path("Qwen3-VL-mmproj-BF16.gguf")) + assert not _named_as_projector(Path("Qwen3-8B-Q4_K_M.gguf")) def test_the_manifest_name_wins_when_there_is_one(tmp_path: Path) -> None: From e8e01d7615e9aa1bc596d0d4209ca11dc9224d36 Mon Sep 17 00:00:00 2001 From: Anish Sarkar <104695310+AnishSarkar22@users.noreply.github.com> Date: Tue, 22 Sep 2026 12:32:58 +0530 Subject: [PATCH 58/60] docs(plans): record the header gate and what vision still needs --- .../api/07-llamacpp-runtime.md | 103 ++++++++++++++++-- 1 file changed, 93 insertions(+), 10 deletions(-) diff --git a/plans/community-local/api/07-llamacpp-runtime.md b/plans/community-local/api/07-llamacpp-runtime.md index 050c31a181..aef86ff350 100644 --- a/plans/community-local/api/07-llamacpp-runtime.md +++ b/plans/community-local/api/07-llamacpp-runtime.md @@ -1256,15 +1256,34 @@ chat costs no range request against a file we were never going to install. It does not carry the KV shape, so it replaces nothing for a model we *can* run: the header read still happens, it just no longer happens for models we cannot. -> **Known defect, measured.** Hugging Face parses **one** file per repo and -> serves that as the repo's answer. A repo shipping a chat model beside an -> `mmproj` sidecar can report `clip`, and the whole repo is then refused although -> `list_builds()` already excluded the sidecar and the real candidate reads -> `qwen35`. 17 of 979 top repos are refused this way, verified against their own -> headers. It is the worst shape of failure this tier has, because the user is -> told no and has no way to learn otherwise. The fix is to read the candidate -> build's header before refusing and to keep the listing only for the tag, which -> is genuinely a repo level fact. +**The file is asked, not the repo.** Hugging Face parses **one** file per repo +and serves that as the repo's answer, so a chat model shipped beside an `mmproj` +sidecar came back as `clip` and the whole repo was refused, 17 of 979 top repos +among them and 13 of those vision models. Each was told to install the model it +belonged to, which was that repo. Nobody could learn otherwise, because being +told no is indistinguishable from a considered answer. + +So `service.repo()` reads the candidate build's own header and gates on that. +`_candidate()` walks the builds until one is both a model and one the denylist +allows, because `list_builds` orders by size and a draft head is always smaller +than the model it accelerates: judging the first file would reproduce the same +failure from our own ordering. Normally one read; two or three in a repo whose +sidecars sort first. + +**Refusing became the cheap path.** `source.py` widens from `PROBE_BYTES` +(256 KiB) to 8 MiB to 24 MiB. A projector, an imatrix or a diffusion GGUF has no +tokenizer, so its metadata ends inside the probe and `general.type` names it +outright. A chat model's `tokenizer.ggml.tokens` runs to megabytes, measured at +5.93 MB for Qwen3 0.6B and 7.82 MB for Llama 3.2 1B, so it truncates there and +widens to exactly what it read before. The truncation is the verdict: a header +that does not fit belongs to a real model. + +`modules/llm/gguf/file_kind.py` reads only official keys, `general.type` against +`gguf.constants.GGUFType`, plus `split.count`. A header that parses and names no +architecture at all is `NOT_LOADABLE`, because that key is what llama.cpp +dispatches on. Everything else fails open: a short read, a failed request, an +unparseable file all admit, since a refusal made from a failure to read is the +one mistake this tier cannot afford. A repo that cannot chat here still lists its builds at their real sizes, badged `Cannot run` and not installable, because the screen is answering "what is in @@ -1289,7 +1308,10 @@ what a quality score was doing for a reader. The 8 of 40 without one are informative by their absence. `list_builds()` excludes split sets (`-of-`), projectors (`mmproj`) and draft -models (`draft`): none of them is a thing a user installs on its own. The +models (`draft`): none of them is a thing a user installs on its own. It is a +**display filter and decides nothing**. Eligibility comes from the candidate's +header, and the walk moves on when this filter kept something it should not +have, so a name being wrong costs a row on a page rather than a refusal. The quantization comes out of the filename with a regex tight enough to require a real quant token, because matching loosely reads `Qwen3` out of `Qwen3-Coder-30B-A3B-Instruct-UD-TQ1_0.gguf`, seen live. The last match wins, @@ -1534,6 +1556,67 @@ which leaves no way to send it one. It is the only capability that reaches the UI; `system_role`, `typed_content` and `tools` are constraints that change how a request is built and mean nothing to a person. +### Vision: what is built, and what is missing + +**A vision model is two files.** The weights answer questions and the projector +turns a picture into something the weights can read. Every vision repo ships +both, and `list_builds` offers only the first, because a projector is not a +thing anyone installs on its own. + +**Today a vision model installs and gives a text chat.** `install()` fetches one +file, the build the user chose, so the projector stays on Hugging Face. +`projector_for` finds nothing beside the model, `mmproj_bytes` is 0, the preset +carries no `mmproj` line, and `llama-server` loads the weights alone. Search +prices it the same way, so the badge, the preset and the load all agree: this is +a text model. + +That is deliberate rather than unfinished. Nothing can send an image, so +fetching the projector would cost roughly a gigabyte of disk and its bytes of +device memory for a file that would be loaded and never touched. Consistency is +worth more than readiness here: a projector on disk with no way to use it would +make every memory estimate wrong by its size. + +**What is already built and dormant**, each tested, waiting only for a second +file to exist on disk: + +| | | +|---|---| +| `providers/llamacpp/projector.py` | finds the projector paired with a model | +| `providers/llamacpp/preset.py` | emits `mmproj = ` | +| `fit_target_mib()` | raises the fitter's margin by the projector's bytes, because `--fit` allocates it after placing layers and does not count it while deciding. Measured: a 600 MiB projector gives `fit-target = 1624` | +| `fit/itemisation.py` | prices `mmproj_bytes` as its own term | +| `capabilities.py` | `can_see`, gated on both halves | + +**What has to be added**, in order: + +1. **`InstallPlan` gains the projector.** It is `model_id, repo, file, size_bytes` + today. The repo listing already names the file, and + `scripts/curated/projector.py` already encodes the preference order, F16 over + BF16 over Q8_0. +2. **`install()` fetches both files.** One extra `download_gguf` into the same + directory. From that point everything above starts firing on its own. +3. **Search pricing passes `mmproj_bytes`.** `service.repo()` calls `estimate()` + without it, so a vision build is currently priced as though the projector + were free. `reprice()` already passes it. Both must agree or the badge + promises a fit the load does not deliver. +4. **The chat request carries an image.** This is the real work and the only + part with nothing behind it: no `image_url` anywhere in the request path and + no attach control in the UI. `typed_content` from `chat_template_caps` is + what says whether a given model's template can accept one. + +**Check before starting.** A vision model should not show a `vision` badge +today, because `read_capabilities` derives it from what the loaded server reports +it accepts, and with no projector loaded that should be text only. This has not +been verified against a running server. If the badge does appear, that is the +worst intermediate state available: the catalog promising picture support while +the chat screen offers no way to send one. Fix that before anything else here. + +**Audio is one step further back.** llama.cpp supports audio input through the +same mechanism, `clip.has_audio_encoder` and the whole `Keys.ClipAudio` group, +but `Modality` carries only `TEXT` and `IMAGE`, so an audio capable model is not +even detected as one. Out of scope, recorded so it is not mistaken for an +oversight. + ### Constrained decoding 48 of the 69 prompt files demand a JSON object. `response_format` with a From 10398e6658a66003dae3ecea81e2a14cb289ec74 Mon Sep 17 00:00:00 2001 From: Anish Sarkar <104695310+AnishSarkar22@users.noreply.github.com> Date: Tue, 22 Sep 2026 13:17:08 +0530 Subject: [PATCH 59/60] fix(llm): let the router load models, and stop racing it _ensure_loaded read /models then posted /models/load, losing to the request already loading the model: 400 `model is already running`, which failed title generation. The proxy autoloads, so the call was a duplicate. --- .../llm/providers/llamacpp/provider.py | 29 ++++----- .../llm/providers/llamacpp/router_client.py | 28 ++++++++- .../llm/providers/llamacpp/fake_router.py | 20 +++++- .../llm/providers/llamacpp/test_provider.py | 63 +++++++++++++++---- .../src/main/sidecars/llamacpp.test.ts | 16 +++++ .../electron/src/main/sidecars/llamacpp.ts | 6 ++ 6 files changed, 132 insertions(+), 30 deletions(-) diff --git a/surfsense_local/backend/modules/llm/providers/llamacpp/provider.py b/surfsense_local/backend/modules/llm/providers/llamacpp/provider.py index 3fa9e7bf5c..c772ed7bdf 100644 --- a/surfsense_local/backend/modules/llm/providers/llamacpp/provider.py +++ b/surfsense_local/backend/modules/llm/providers/llamacpp/provider.py @@ -7,6 +7,14 @@ `/v1/chat/completions`, so the streaming, error handling and message shaping in `OpenAICompatibleChatProvider` already work against it. +Deliberately **no** load either. `--models-autoload` is on by default and the +router's proxy calls `ensure_model_ready` before forwarding, so a cold model +loads on the request that needs it. Asking as well was a check-then-act across a +socket, and it lost the race to the request already loading the model: the +router answers 400 `model is already running`, which took out title generation +on every new thread. The router owns model lifecycle; we hold no opinion about +it. + Deliberately **no** `pull()`. Fetching its own weights from a name made sense when the runtime owned the download. Here SurfSense fetches the GGUF itself, because that is the only place `egress.require()` can hold, and because it buys @@ -67,10 +75,11 @@ async def capabilities(self, model: str) -> Capabilities: Cached per model rather than fetched per message: two files in the same directory can carry entirely different templates, so the cache is keyed by model id, never assumed to hold across them. What it does - hold across is repeated calls for the *same* resident model within one - turn, or between turns, since nothing about a loaded model's template - or window changes on its own. `_ensure_loaded` clears an entry the one - time that stops being true: a fresh load. + hold across is repeated calls for the *same* resident model, since + nothing about a loaded model's template or window changes on its own. + Nothing invalidates it, and nothing needs to: `get_provider()` builds a + fresh adapter per call, so the cache cannot outlive the resolution that + created it, let alone the sidecar restart a preset rewrite triggers. """ cached = self._capabilities_cache.get(model) if cached is not None: @@ -140,7 +149,6 @@ async def chat( reasoning: bool | None = None, json_schema: dict | None = None, ) -> AsyncIterator[str]: - await self._ensure_loaded(model) # Downgrade at the seam: `modules/chat` assembles one conversation and # never learns that templates differ. shaped = for_template(messages, await self.capabilities(model)) @@ -172,14 +180,3 @@ async def chat( reasoning=reasoning, ): yield chunk - - async def _ensure_loaded(self, model: str) -> None: - """Models are discovered `unloaded`, so the first turn has to ask. - - Reloading a resident model would evict and re-read gigabytes between two - turns, so this checks before it acts. - """ - resident = {m.id for m in await self._router.models() if m.loaded} - if model not in resident: - self._capabilities_cache.pop(model, None) - await self._router.load(model) diff --git a/surfsense_local/backend/modules/llm/providers/llamacpp/router_client.py b/surfsense_local/backend/modules/llm/providers/llamacpp/router_client.py index 6716e1c245..6e7c4ca378 100644 --- a/surfsense_local/backend/modules/llm/providers/llamacpp/router_client.py +++ b/surfsense_local/backend/modules/llm/providers/llamacpp/router_client.py @@ -109,13 +109,23 @@ async def tokenize(self, model_id: str, text: str) -> int: return len(reply.json().get("tokens", [])) async def load(self, model_id: str) -> None: - """Bring a model into memory. + """Bring a model into memory, for a caller that means "load now". + + The chat path does not use this: the router loads on demand. This is for + a deliberate warm up, such as after an install. Deliberately sends no arguments. The router accepts an `args` field and ignores it, measured; per-model flags come from the preset INI instead. + + A model that is already running is the state the caller wanted, so the + 400 llama.cpp answers in that case is not an error. Anything crossing + this socket can be beaten to it by a request that loaded the model + first, and that race is not worth reporting as a failure. """ async with self._client() as client: reply = await client.post("/models/load", json={"model": model_id}) + if reply.status_code == 400 and _already_running(reply): + return reply.raise_for_status() async def unload(self, model_id: str) -> None: @@ -129,3 +139,19 @@ async def delete(self, model_id: str) -> None: "DELETE", "/models", params={"model": model_id} ) reply.raise_for_status() + + +def _already_running(reply: httpx.Response) -> bool: + """Whether a 400 is llama.cpp's `model is already running`. + + Read from the body it sent rather than from the status alone, because every + other 400 on this route (`model is not found`, a missing name) really is a + caller mistake. + """ + try: + payload = reply.json() + except (httpx.HTTPError, ValueError): + return False + error = payload.get("error") if isinstance(payload, dict) else None + message = error.get("message") if isinstance(error, dict) else None + return isinstance(message, str) and "already running" in message diff --git a/surfsense_local/backend/tests/unit/llm/providers/llamacpp/fake_router.py b/surfsense_local/backend/tests/unit/llm/providers/llamacpp/fake_router.py index ea93edee3a..bbf003df2d 100644 --- a/surfsense_local/backend/tests/unit/llm/providers/llamacpp/fake_router.py +++ b/surfsense_local/backend/tests/unit/llm/providers/llamacpp/fake_router.py @@ -36,6 +36,11 @@ def __init__(self, models: list[str] | None = None) -> None: # How many times `/props` was actually asked, so a test can tell a # cached read apart from a fresh round trip. self.props_calls = 0 + # Whether the router already holds this model. The real one answers 400 + # `model is already running` to a load in that state, which is what a + # check-then-act across this socket races into. + self.already_running: set[str] = set() + self.load_calls: list[str] = [] def transport(self) -> httpx.MockTransport: return httpx.MockTransport(self._handle) @@ -83,7 +88,20 @@ def _handle(self, request: httpx.Request) -> httpx.Response: }, ) if path == "/models/load": - self.loaded.add(json.loads(request.content)["model"]) + name = json.loads(request.content)["model"] + self.load_calls.append(name) + if name in self.already_running: + return httpx.Response( + 400, + json={ + "error": { + "code": 400, + "type": "invalid_request_error", + "message": "model is already running", + } + }, + ) + self.loaded.add(name) return httpx.Response(200, json={"success": True}) if path == "/models/unload": self.loaded.discard(json.loads(request.content)["model"]) diff --git a/surfsense_local/backend/tests/unit/llm/providers/llamacpp/test_provider.py b/surfsense_local/backend/tests/unit/llm/providers/llamacpp/test_provider.py index c212dea2af..9948df2d8f 100644 --- a/surfsense_local/backend/tests/unit/llm/providers/llamacpp/test_provider.py +++ b/surfsense_local/backend/tests/unit/llm/providers/llamacpp/test_provider.py @@ -51,15 +51,17 @@ async def test_installed_models_come_from_the_router() -> None: @pytest.mark.asyncio -async def test_a_model_is_brought_into_memory_before_it_is_asked_anything() -> None: - """The router discovers models as `unloaded`, so the first turn would - otherwise go to a model that is not resident.""" +async def test_a_cold_model_answers_without_us_loading_it() -> None: + """The router discovers models as `unloaded`, and loads one on the request + that needs it: `--models-autoload` is on by default and the proxy calls + `ensure_model_ready` before forwarding. So the first turn works against a + model nobody has asked for yet, and asking as well only created a race.""" fake = FakeRouter(["m"]) chunks = [c async for c in provider_for(fake).chat("m", [Message("user", "hi")])] - assert "m" in fake.loaded assert "".join(chunks) == "Hello" + assert fake.load_calls == [] @pytest.mark.asyncio @@ -177,19 +179,56 @@ async def test_capabilities_are_cached_for_a_resident_model() -> None: @pytest.mark.asyncio -async def test_a_fresh_load_invalidates_the_cached_capabilities() -> None: - """The one event that can actually change what `/props` would say: the - idle timeout evicts a model, and the next turn reloads it, possibly with - a rewritten preset. The cache must not go on quoting the old answer.""" +async def test_the_capability_cache_cannot_outlive_its_adapter() -> None: + """Why nothing invalidates it any more. + + The answer `/props` gives is a property of the file and the preset, and a + preset rewrite restarts the sidecar. But `get_provider()` builds a fresh + adapter per call, so a cache entry cannot survive even one resolution, let + alone a restart. Within one adapter the answer cannot change, and across + adapters there is no cache to be stale. + """ fake = FakeRouter(["qwen3"]) provider = provider_for(fake) async for _ in provider.chat("qwen3", [Message("user", "hi")]): pass - assert fake.props_calls == 1 + async for _ in provider.chat("qwen3", [Message("user", "hi")]): + pass + assert fake.props_calls == 1, "a resident model's template does not change" - fake.loaded.discard("qwen3") # the router's own idle timer evicted it + async for _ in provider_for(fake).chat("qwen3", [Message("user", "hi")]): + pass + assert fake.props_calls == 2, "a new adapter carries nothing over" - async for _ in provider.chat("qwen3", [Message("user", "hi")]): + +async def test_a_turn_never_asks_the_router_to_load(anyio_backend) -> None: + """The router loads on demand and we stop having an opinion about it. + + `--models-autoload` is enabled by default and the proxy calls + `ensure_model_ready` before forwarding, so a chat request loads a cold model + on its own. Asking as well was a check-then-act across a socket: read + `/models`, see `loaded: false`, post `/models/load`, and lose the race to + the request that was already loading it. The router answers 400 `model is + already running`, which took out title generation on every new thread. + """ + fake = FakeRouter(["Qwen3-1.7B-Q4_K_M"]) + provider = provider_for(fake) + + async for _ in provider.chat("Qwen3-1.7B-Q4_K_M", [Message("user", "hi")]): pass - assert fake.props_calls == 2 + + assert fake.load_calls == [] + + +async def test_a_load_that_lost_the_race_is_not_an_error(anyio_backend) -> None: + """For the callers that do mean "load now", such as a warm up after an + install. `model is already running` is the state they wanted, and anything + crossing this socket can be beaten to it.""" + fake = FakeRouter(["Qwen3-1.7B-Q4_K_M"]) + fake.already_running.add("Qwen3-1.7B-Q4_K_M") + provider = provider_for(fake) + + await provider._router.load("Qwen3-1.7B-Q4_K_M") + + assert fake.load_calls == ["Qwen3-1.7B-Q4_K_M"] diff --git a/surfsense_local/electron/src/main/sidecars/llamacpp.test.ts b/surfsense_local/electron/src/main/sidecars/llamacpp.test.ts index a057b71d8a..67de527810 100644 --- a/surfsense_local/electron/src/main/sidecars/llamacpp.test.ts +++ b/surfsense_local/electron/src/main/sidecars/llamacpp.test.ts @@ -113,3 +113,19 @@ test("never sets a router wide reasoning budget", () => { assert.ok(spec) assert.ok(!spec.args.some((arg) => arg.startsWith("--reasoning-budget"))) }) + +test("the router is told to load models on demand, rather than relying on its default", () => { + // The chat path no longer asks the router to load anything: the proxy calls + // ensure_model_ready before forwarding, so a cold model loads on the request + // that needs it. Asking as well was a check-then-act across a socket and lost + // the race to the request already loading the model, which took out title + // generation with a 400 `model is already running`. + // + // That correctness now rests on an upstream default. Stating it means the day + // it flips we get a clear failure here rather than a silent one in a chat. + const { ctx } = staged() + const spec = llamacppSpec(ctx) + + assert.ok(spec) + assert.ok(spec.args.includes("--models-autoload")) +}) diff --git a/surfsense_local/electron/src/main/sidecars/llamacpp.ts b/surfsense_local/electron/src/main/sidecars/llamacpp.ts index 8e807a874a..718d077d97 100644 --- a/surfsense_local/electron/src/main/sidecars/llamacpp.ts +++ b/surfsense_local/electron/src/main/sidecars/llamacpp.ts @@ -67,6 +67,12 @@ export function llamacppSpec(ctx: SidecarContext): SidecarSpec | null { // turns the second question of a conversation into a reload. "--sleep-idle-seconds", "300", + // The chat path never asks the router to load anything: the proxy calls + // ensure_model_ready before forwarding, so a cold model loads on the request + // that needs it. Asking as well was a check-then-act across a socket, and it + // lost the race to the request already loading the model. This is the + // upstream default; stated because chat is now correct only while it holds. + "--models-autoload", // llama-server ships its own web UI, which we neither need nor want exposed. "--no-ui", "--jinja", From 9d7ee9e2444108e256b4b9e043c143dbdd25cbef Mon Sep 17 00:00:00 2001 From: Anish Sarkar <104695310+AnishSarkar22@users.noreply.github.com> Date: Tue, 22 Sep 2026 14:15:03 +0530 Subject: [PATCH 60/60] docs(plans): match phase 7 to the router owning model loads --- .../api/07-llamacpp-runtime.md | 26 +++++++++++++++---- 1 file changed, 21 insertions(+), 5 deletions(-) diff --git a/plans/community-local/api/07-llamacpp-runtime.md b/plans/community-local/api/07-llamacpp-runtime.md index aef86ff350..33050c0914 100644 --- a/plans/community-local/api/07-llamacpp-runtime.md +++ b/plans/community-local/api/07-llamacpp-runtime.md @@ -156,6 +156,7 @@ From `electron/src/main/sidecars/llamacpp.ts`: --host 127.0.0.1 --port --models-max 1 --sleep-idle-seconds 300 +--models-autoload --no-ui --jinja --reasoning-format deepseek @@ -170,6 +171,17 @@ directory. - `--sleep-idle-seconds 300` is not optional. Measured at `b11050`, a model without it self-evicted after roughly 30 s idle, which turns the second question of a conversation into a reload. +- `--models-autoload` is the upstream default, stated because the chat path + depends on it. The router's proxy calls `ensure_model_ready` before + forwarding, so a cold model loads on the request that needs it, and **nothing + in SurfSense ever calls `POST /models/load` on the chat path**. Asking as well + was a check-then-act across a socket: read `/models`, see `loaded: false`, + post a load, and lose the race to the request already loading the model. The + router answers 400 `model is already running`, which took out title generation + on every new thread. `RouterClient.load()` survives for a caller that means + "load now", such as a warm up after an install, and treats that 400 as + success. An Electron test holds this flag so an upstream change to the default + fails there rather than in a chat. - `--no-ui` because llama-server ships its own web UI, which we neither need nor want exposed. - `--reasoning-format deepseek` routes `` blocks to @@ -340,11 +352,15 @@ measured, `model name=… is not removable (not from cache)`, a 500, with the fi left on disk. Everything SurfSense installs lands in `--models-dir`, so that call can never succeed for us. -`/props` is read once per load rather than once per message, cached by model id -and cleared only by `_ensure_loaded()` issuing a fresh load, which is the one -event that can make the old answer wrong. Before that cache, `context_tokens()` -and `chat()`'s own template shaping each issued an independent round trip on -every turn. +`/props` is read once rather than once per message, cached by model id. Before +that cache, `context_tokens()` and `chat()`'s own template shaping each issued +an independent round trip on every turn. + +Nothing invalidates the cache, and nothing needs to. The answer is a property of +the file and the preset, and a preset rewrite restarts the sidecar, but +`get_provider()` builds a fresh adapter per call, so an entry cannot outlive the +resolution that created it. Within one adapter the answer cannot change; across +adapters there is no cache to be stale. ## Reading a model