feat(local): replace Ollama with a bundled llama.cpp runtime, a curated GGUF catalog, and a hardware-aware fit estimator

.github/workflows/release-local.yml

@@ -36,19 +36,18 @@ jobs:
       fail-fast: false
       matrix:
         include:
-          - os: ubuntu-latest
+          # Pinned: `ubuntu-latest` migrates to 26.04 and would silently raise
+          # the AppImage's glibc floor. llama.cpp's Vulkan build needs 2.34.
+          - os: ubuntu-22.04
             platform: --linux
             electron_cache: ~/.cache/electron
-            llmfit_resource: release/linux-unpacked/resources/llmfit/llmfit
           # Apple Silicon only: torch (2.3+) and onnxruntime (1.24+) no longer ship Intel Mac wheels.
           - os: macos-15
             platform: --mac
             electron_cache: ~/Library/Caches/electron
-            llmfit_resource: release/mac-arm64/SurfSense.app/Contents/Resources/llmfit/llmfit
           - os: windows-latest
             platform: --win
             electron_cache: ~\AppData\Local\electron\Cache
-            llmfit_resource: release/win-unpacked/resources/llmfit/llmfit.exe
 
     defaults:
       run:
@@ -193,15 +192,9 @@ jobs:
           pnpm install --frozen-lockfile
           pnpm build
 
-      - name: Stage Ollama
+      - name: Stage llama.cpp
         working-directory: surfsense_local/electron
-        run: node scripts/fetch-ollama.mjs
-
-      - name: Stage and smoke llmfit
-        working-directory: surfsense_local/electron
-        run: |
-          node scripts/fetch-llmfit.mjs
-          node scripts/check-llmfit.mjs
+        run: node scripts/fetch-llamacpp.mjs
 
       - name: Package installer
         working-directory: surfsense_local/electron
@@ -238,12 +231,8 @@ jobs:
 
           "${CMD[@]}"
 
-      - name: Smoke llmfit from final resources
-        working-directory: surfsense_local/electron
-        run: node scripts/check-llmfit.mjs "${{ matrix.llmfit_resource }}"
-
       - name: Smoke the frozen API (Linux)
-        if: matrix.os == 'ubuntu-latest'
+        if: matrix.os == 'ubuntu-22.04'
         run: |
           set -e
           export SURFSENSE_LOCAL_DATA_DIR="$RUNNER_TEMP/ss-smoke"
@@ -258,20 +247,19 @@ jobs:
           done
           echo "::error::packaged API never became healthy"; kill $API; exit 1
 
-      - name: Smoke the bundled Ollama (Linux)
-        if: matrix.os == 'ubuntu-latest'
+      - name: Smoke the bundled llama.cpp (Linux)
+        if: matrix.os == 'ubuntu-22.04'
         run: |
           set -e
-          export OLLAMA_HOST=127.0.0.1:11565
-          ./surfsense_local/electron/release/linux-unpacked/resources/ollama/bin/ollama serve &
-          OLL=$!
-          for i in $(seq 1 30); do
-            if curl -fsS "http://127.0.0.1:11565/" >/dev/null; then
-              echo "ollama OK"; kill $OLL; exit 0
-            fi
-            sleep 1
-          done
-          echo "::error::bundled ollama never responded"; kill $OLL; exit 1
+          DIR=./surfsense_local/electron/release/linux-unpacked/resources/llamacpp
+          # Run from the library directory: ggml scans the running executable's
+          # own directory for backends and silently reports none anywhere else,
+          # which is indistinguishable from a machine with no GPU.
+          OUT=$(cd "$DIR" && ./llama-server --list-devices 2>&1)
+          echo "$OUT"
+          if ! grep -qiE "Available devices|no devices found" <<<"$OUT"; then
+            echo "::error::packaged llama-server could not list devices"; exit 1
+          fi
 
       - name: Upload installers
         uses: actions/upload-artifact@v4

plans/community-local/00-umbrella-plan.md

@@ -29,6 +29,8 @@ Same phase number = integrate together.
 | **4** | [`04-studio.md`](frontend/04-studio.md) ◐ | [`04-studio.md`](api/04-studio.md) ◐ | [`04-studio.md`](worker/04-studio.md) ◐ |
 | **5** | [`05-install-ux.md`](frontend/05-install-ux.md) ◐ | [`05a-model-recommendations.md`](api/05a-model-recommendations.md) ✓ + [`05b-openai-compatible-connections.md`](api/05b-openai-compatible-connections.md) ✓ + [`05c-packaging.md`](api/05c-packaging.md) ◐ | [`05-packaging.md`](worker/05-packaging.md) ✓ |
 | **6** | import, license, egress settings ([`00d-pivot-plan.md`](00d-pivot-plan.md)) ✓ | `modules/migration/`, `modules/license/`, keychain, auto-update ([`00d-pivot-plan.md`](00d-pivot-plan.md)) ◐ | — (import reuses `ingest_document`) |
+| **7** | catalog rework, fit badges, HF search (in [`api/07-llamacpp-runtime.md`](api/07-llamacpp-runtime.md)) | [`07-llamacpp-runtime.md`](api/07-llamacpp-runtime.md) — replace Ollama with llama.cpp | — |
+| **8** | — | [`08-cuda-backend.md`](api/08-cuda-backend.md) — CUDA as an optional Windows backend. **Not scheduled**: measured at 0.66 s per turn for 685 MB | — |
 
 **Demo:** phase 3 all streams. **Ship:** phase 6 = SurfSense 2.0.0 (1.0.x tags are taken by the old project versioning; see [`00d-pivot-plan.md`](00d-pivot-plan.md)).
 
@@ -102,9 +104,9 @@ Docker Compose, Postgres, Zero, Redis, Celery, LangGraph, git KB, scrapers, MCP,
 | **HTTP stack** | **FastAPI** + uvicorn | Same stack as cloud backend; native OpenAPI for frontend; SSE streaming for chat. PyInstaller risk is handled in [`api/00-spike.md`](api/00-spike.md) — not a reason to downgrade. |
 | **Retrieval** | **FTS5 + sqlite-vec hybrid, cosine rescore** | Semantic + keyword from day one: both legs widen recall, cosine orders. Embeddings on ingest must be queried properly — not keyword-only, not in-memory scan over BLOBs. |
 | **Embed provider** | Bundled bge-small-en-v1.5 int8, in-process on onnxruntime | 384-dim, ~66MB, runs offline on CPU with no model server. Docling parses, Chonkie chunks. Remote embedding is a later opt-in, not a launch dependency. |
-| **Generation architecture** | llmfit catalog/advisor + curated-model policy + runtime adapters; Ollama local, multiple OpenAI-compatible connections remote | [`api/05a-model-recommendations.md`](api/05a-model-recommendations.md) and [`api/05b-openai-compatible-connections.md`](api/05b-openai-compatible-connections.md). llmfit supplies broad model metadata, hardware detection, and fit estimates; it is not a `Generator`, installer, or inference runtime. Local adapters resolve trusted artifacts and Ollama is first. Remote endpoints are named connection instances with their own URL and optional key; models are discovered live, not synchronized into SQLite. `SelectedModel(role)` stores the provider, connection identity when remote, and exact model id. SurfSense selects endpoints but does not load-balance their replicas. |
-| **llmfit integration** | **Pinned official binary, short-lived JSON CLI, normalized behind `ModelAdvisor`** | The API runs `llmfit --json system` and `llmfit --max-context 8192 --json fit`, caches one scan, and exposes only SurfSense DTOs. No fork, patch, Python import, or permanent llmfit server. CLI label/machine-code differences are normalized at one seam and fixture-tested before a version bump. Failure removes ranking, not installed-model selection or chat. |
-| **Ollama runtime** | **Bundled as a supervised sidecar (packaged); dev uses the developer's own `ollama serve`** | Chat can't depend on a daemon the user may not have installed. The packaged app ships the standalone Ollama archive (`electron/scripts/fetch-ollama.mjs` stages it, electron-builder carries it in `resources/ollama`) and Electron runs it as a third sidecar on a chosen port, passing `SURFSENSE_LOCAL_OLLAMA_BASE_URL` to the API. Models aren't shipped — the user pulls into the writable data dir after install. The Python sidecars and Ollama share one supervisor (`electron/src/main/sidecars/`): the supervisor spawns/reaps, one spec file per sidecar carries its identity. Boot now starts four — `api`, `worker-ingest`, `worker-studio`, and Ollama when packaged — and `watchImageModel` adds a fifth, sd-server, later, because only the API knows which image model was chosen. Ollama is best-effort at boot (only the API gates the window; its state surfaces via `/llm/providers`). |
+| **Generation architecture** *(Phase 7 supersedes the local half — [`api/07-llamacpp-runtime.md`](api/07-llamacpp-runtime.md): llama.cpp is the one local runtime, llmfit moves to build time, the catalog opens to every GGUF on Hugging Face)* | llmfit catalog/advisor + curated-model policy + runtime adapters; Ollama local, multiple OpenAI-compatible connections remote | [`api/05a-model-recommendations.md`](api/05a-model-recommendations.md) and [`api/05b-openai-compatible-connections.md`](api/05b-openai-compatible-connections.md). llmfit supplies broad model metadata, hardware detection, and fit estimates; it is not a `Generator`, installer, or inference runtime. Local adapters resolve trusted artifacts and Ollama is first. Remote endpoints are named connection instances with their own URL and optional key; models are discovered live, not synchronized into SQLite. `SelectedModel(role)` stores the provider, connection identity when remote, and exact model id. SurfSense selects endpoints but does not load-balance their replicas. |
+| **llmfit integration** *(Phase 7: the binary stops shipping and **never runs in CI**. A person runs `scripts/refresh_curated_models.py` by hand a few times a year, llmfit proposes a rank, and the number is committed as source. It is never a gate, a hardware source, or on a request path)* | **Pinned official binary, short-lived JSON CLI, normalized behind `ModelAdvisor`** | The API runs `llmfit --json system` and `llmfit --max-context 8192 --json fit`, caches one scan, and exposes only SurfSense DTOs. No fork, patch, Python import, or permanent llmfit server. CLI label/machine-code differences are normalized at one seam and fixture-tested before a version bump. Failure removes ranking, not installed-model selection or chat. |
+| **Ollama runtime** *(**superseded by Phase 7** — [`api/07-llamacpp-runtime.md`](api/07-llamacpp-runtime.md) replaces it with `llama-server` in router mode: 11–31 MB instead of 501 MB, Vulkan on every platform off Apple Silicon, and no MLX until after launch)* | **Bundled as a supervised sidecar (packaged); dev uses the developer's own `ollama serve`** | Chat can't depend on a daemon the user may not have installed. The packaged app ships the standalone Ollama archive (`electron/scripts/fetch-ollama.mjs` stages it, electron-builder carries it in `resources/ollama`) and Electron runs it as a third sidecar on a chosen port, passing `SURFSENSE_LOCAL_OLLAMA_BASE_URL` to the API. Models aren't shipped — the user pulls into the writable data dir after install. The Python sidecars and Ollama share one supervisor (`electron/src/main/sidecars/`): the supervisor spawns/reaps, one spec file per sidecar carries its identity. Boot now starts four — `api`, `worker-ingest`, `worker-studio`, and Ollama when packaged — and `watchImageModel` adds a fifth, sd-server, later, because only the API knows which image model was chosen. Ollama is best-effort at boot (only the API gates the window; its state surfaces via `/llm/providers`). |
 | **Persistence** | SQLAlchemy 2.0 + Alembic, same as cloud | Models are the source of truth. `versions/` ships as PyInstaller data, resolved from the package's own `__file__` — de-risked in [`api/00-spike.md`](api/00-spike.md). |
 | **Migrations** | **Hand-written; autogenerate is off** | Autogenerate cannot see a rename — it emits drop + add, which deletes a column's data silently. The target database is one user's laptop, unbacked and uninspectable, so every revision is written and read by a person. `env.py` carries no `target_metadata`, so `--autogenerate` cannot be used by accident. Mature SQLite-backed apps make the same call — hand-written revisions throughout. |
 | **Schema owner** | Alembic only; **never** `create_all` | Cloud's `create_all`-on-startup races its own migrations and breaks releases. Local has one path to a schema, and a test fails if models and migrations drift. |

plans/community-local/00b-diagrams.md

@@ -2,6 +2,10 @@
 
 > Workstreams: [`frontend/`](frontend/), [`api/`](api/), [`worker/`](worker/).
 
+> Updated for [`api/07-llamacpp-runtime.md`](api/07-llamacpp-runtime.md):
+> **Phase 7** replaces the Ollama sidecar with `llama-server` and removes the
+> bundled llmfit binary.
+
 ## Process layout at runtime
 
 ```text
@@ -10,7 +14,7 @@
 │  • BrowserWindow → frontend/dist (workspace UI)             │
 │  • spawn surfsense-api                                      │
 │  • spawn surfsense-worker                                   │
-│  • packaged Ollama sidecar; bundled llmfit executable       │
+│  • packaged llama-server sidecar (router mode)              │
 │  • on quit: SIGTERM workers, wait, exit                     │
 └─────────────────────────────────────────────────────────────┘
          │                              │
@@ -20,7 +24,8 @@
 │ FastAPI :8xxx   │          │ huey_consumer -w1│
 │ LLM catalog     │          │ Docling + embed  │
 │ runtime adapters│          │                 │
-│ llmfit JSON CLI │          │                 │
+│ GGUF header +   │          │                 │
+│ ggml budget     │          │                 │
 │ SQLite R/W      │          │                 │
 └────────┬────────┘          └────────┬─────────┘
          │                            │
@@ -42,20 +47,18 @@
          ▼                   ▼                   ▼
    ┌───────────┐      ┌─────────────┐     ┌───────────┐
    │ A Airgap  │      │ B Slim +    │     │ C BYO     │
-   │ full pack │      │ on-demand   │     │ Ollama/   │
+   │ full pack │      │ on-demand   │     │ endpoint  │
    └───────────┘      └─────────────┘     └───────────┘
                              ▼
-                    llmfit hardware scan
+              ggml device query (~180 ms, no scan)
                              ▼
-                  Curated + Explore
-           (Curated and Installed render with no
-            scan; only Explore needs one)
+              Curated (offline)  +  HF search (network)
+            every row badged FITS / PARTIAL / TOO_BIG
                              ▼
                  one-click Download & Use
                              ▼
                     runtime install plan
-                    ├─ Ollama tag (v1)
-                    └─ verified GGUF (later)
+                    └─ pinned GGUF (repo + file + quant)
                              ▼
               Create workspace → upload document
 ```
@@ -64,15 +67,15 @@
 
 ```text
 LOCAL                                      REMOTE
-llmfit catalogue + fit                    provider_connections
-        │                                           │
-        ▼                                           ▼
-SurfSense adapter + policy                live GET /models per endpoint
+curated manifest + HF search              provider_connections
         │                                           │
         ▼                                           ▼
-runtime artifact resolver                 SelectedModel(connection_id, name)
-  ├── Ollama ── pull tag ── chat             ├── generation
-  └── llama.cpp (future) ── chat             │   └── core vLLM/gateway
+fit estimator (ggml budget +              live GET /models per endpoint
+ GGUF header), rank from manifest                   │
+        │                                           ▼
+        ▼                                 SelectedModel(connection_id, name)
+GgufArtifact resolver                        ├── generation
+  └── llama.cpp ── fetch file ── chat        │   └── core vLLM/gateway
                                               │       └── /chat/completions
                                               └── image_generation
                                                   ├── vLLM-Omni/gateway
@@ -81,8 +84,10 @@ runtime artifact resolver                 SelectedModel(connection_id, name)
                                                       └── /images on 404/405
 ```
 
-llmfit never handles a SurfSense chat request. It can disappear and an already
-installed model still answers through its runtime adapter. Remote model lists
+llmfit is not in the app at all: it runs on a maintainer's machine a few times a
+year and its numbers ship as data in the manifest. The fit badge comes from the
+runtime's own allocator plus the model's GGUF header, so it works offline and
+covers every GGUF, not a curated subset. Remote model lists
 are fetched live and never copied into SQLite. Two endpoints with the same model
 id remain distinct because selection identity includes the connection id.
 

plans/community-local/00c-data-model.md

@@ -265,7 +265,7 @@ and the Fernet key comes from the OS keychain through Electron.
 
 `selected_models` stores one active model per `role`: `generation` or
 `image_generation`. `provider` chooses the runtime adapter, `name` is the exact
-model id, and nullable `connection_id` identifies the remote endpoint. Ollama
+model id, and nullable `connection_id` identifies the remote endpoint. Local
 **and `sdcpp`**, the bundled local image runtime admitted by revision `0009`,
 use no connection; an OpenAI-compatible selection requires one. The FK uses
 `ON DELETE CASCADE`, so disconnecting an endpoint clears only roles that use it.
@@ -275,7 +275,7 @@ which is computed on read rather than stored, so retuning a threshold needs no
 migration ([`api/05a-model-recommendations.md`](api/05a-model-recommendations.md)).
 
 The offerable local catalog, remote `/models` responses, hardware profile,
-llmfit scores, install plans, capabilities, and curated models are **not**
+Fit estimates, install plans, capabilities, and curated models are **not**
 stored. Local recommendations are recomputed from the packaged inputs; remote
 inventory is fetched live. Persisting either would create synchronization work
 without improving inference. See
@@ -291,7 +291,7 @@ without improving inference. See
 | `selected_models` | chosen model per role (above) |
 | `onboarding_completion` | the durable "model onboarding is done" marker (revision `0003`); selecting or clearing a model never writes it |
 | `license_state` | singleton (`CHECK id = 1`): the imported certificate, when it was imported, and `clock_watermark`, the highest instant ever seen. Plan and expiry are re-derived from the certificate on every read rather than stored (revision `0006`) |
-| `egress_destinations` | one row per destination — `ollama_pull`, `image_model_pull`, or `host:<hostname>` for a BYO provider — with `enabled` defaulting to **false** and `last_call_at` (revision `0008`) |
+| `egress_destinations` | one row per destination — `model_download` and `model_search` (both `huggingface.co`, renamed from `ollama_pull` by revision `0012`, see [`api/07-llamacpp-runtime.md`](api/07-llamacpp-runtime.md)), `image_model_pull`, or `host:<hostname>` for a BYO provider — with `enabled` defaulting to **false** and `last_call_at` (revision `0008`) |
 
 `workspaces` also gained a nullable `cloud_id` in revision `0005`: cloud-to-local
 import looks a workspace up by it and reuses the existing row rather than