feat(local): local speech recognition, for recordings as sources and voice input

.agents/skills/translate/glossary.md

@@ -41,6 +41,10 @@ SurfSense, Studio, Markdown, llama.cpp, GGUF, Hugging Face, API, MCP, and every
 | study guide | 学習ガイド | 학습 가이드 | 学习指南 | स्टडी गाइड | Lernzettel † | guía de estudio | Guide de révision | Guia de estudos | конспект |
 | podcast | ポッドキャスト | 팟캐스트 | 播客 | पॉडकास्ट | Podcast | podcast | Podcast | Podcast | подкаст |
 | speech (a model reading text aloud) | 音声合成 | 음성 | 语音 | वॉइस | Sprache | voz | voix | voz | речь |
+| speech recognition (turning recordings and the voice into text, on this computer) | 音声認識 | 음성 인식 | 语音识别 | स्पीच रिकग्निशन | Spracherkennung | reconocimiento de voz | reconnaissance vocale | reconhecimento de voz | распознавание речи |
+| voice input (dictating into the chat composer) | 音声入力 | 음성 입력 | 语音输入 | वॉइस इनपुट | Spracheingabe | entrada de voz | saisie vocale | entrada de voz | голосовой ввод |
+| transcript (a recording’s text) | 文字起こし | 스크립트 | 文字稿 | ट्रांसक्रिप्ट | Transkript | transcripción | transcription | transcrição | расшифровка |
+| recording (an audio file added as a source) | 録音 | 녹음 | 录音 | रिकॉर्डिंग | Aufnahme | grabación | enregistrement | gravação | запись |
 | sidebar (the workspace’s left column) | サイドバー | 사이드바 | 侧边栏 | साइडबार | Seitenleiste | barra lateral | barre latérale | barra lateral | боковая панель |
 | workspace | ワークスペース | 워크스페이스 | 工作区 | वर्कस्पेस | Workspace | espacio de trabajo | espace de travail | Workspace | рабочая область |
 | artifact (a Studio output) | 生成物 | 생성물 | 产物 | आर्टिफ़ैक्ट | Artefakt | artefacto | artefact | artefato | артефакт |

.github/workflows/build-audiocpp.yml

@@ -1,8 +1,9 @@
 # Compile audio.cpp's server for Windows and Linux with the recipe in
 # surfsense_local/electron/scripts/audiocpp/, check it against the app's
 # floors, and hand the staged folder to the desktop release as an artifact.
-# A pull request that changes the recipe, or the app code that drives the
-# server, runs it too.
+# Check that upstream's macOS archive, which the release ships as it is, has
+# every curated family too. A pull request that changes the recipe, or the
+# app code that drives the server, runs it too.
 #
 # TODO: ask upstream (0xShug0/audio.cpp) to publish Linux archives built on
 # Ubuntu 22.04 and Windows ones with per-CPU ggml libraries, as llama.cpp does.
@@ -20,7 +21,14 @@ on:
       - surfsense_local/backend/modules/llm/providers/audiocpp/**
       - surfsense_local/backend/modules/llm/catalog/local/engines/audiocpp/**
       - surfsense_local/backend/modules/llm/catalog/local/manifest/models.json
+      - surfsense_local/backend/modules/llm/transcription/**
+      - surfsense_local/backend/worker/ingestion/audio/**
+      - surfsense_local/backend/worker/audio_decode/**
+      - surfsense_local/backend/modules/documents/recordings/**
       - surfsense_local/backend/tests/packaging/test_audiocpp_voicing.py
+      - surfsense_local/backend/tests/packaging/test_audiocpp_transcription.py
+      - surfsense_local/backend/tests/packaging/test_audiocpp_transcription_config.py
+      - surfsense_local/backend/tests/packaging/test_recording_transcription.py
       - .github/workflows/build-audiocpp.yml
 
 permissions:
@@ -71,6 +79,8 @@ jobs:
           echo "CC=gcc-13" >> "$GITHUB_ENV"
           echo "CXX=g++-13" >> "$GITHUB_ENV"
 
+      # Staging starts the server once per curated family and fails unless
+      # that family's loader runs, so a family the recipe drops never ships.
       - name: Compile and stage
         if: steps.cache.outputs.cache-hit != 'true'
         run: node scripts/audiocpp/stage.mjs --strict
@@ -139,6 +149,46 @@ jobs:
           # addopts pins -m 'not packaging'; the later -m on the command line wins.
           uv run pytest tests/packaging/test_audiocpp_voicing.py -m packaging
 
+      - id: recogniser
+        name: Read the curated recogniser's pin
+        working-directory: surfsense_local/backend
+        run: |
+          uv run python - >> "$GITHUB_OUTPUT" <<'EOF'
+          from modules.llm.catalog.local.manifest import load_local_manifest
+
+          (model,) = [m for m in load_local_manifest().models if m.transcription]
+          pinned = model.builds[0].files[0]
+          print(f"sha256={pinned.sha256}")
+          print(f"url=https://huggingface.co/{pinned.repo}/resolve/{pinned.revision}/{pinned.path}")
+          print(f"name={pinned.path.rsplit('/', 1)[-1]}")
+          EOF
+
+      # 930 MB, so kept by its pinned hash rather than downloaded every run.
+      - uses: actions/cache@v5
+        with:
+          path: ${{ runner.temp }}/recogniser
+          key: recogniser-${{ steps.recogniser.outputs.sha256 }}
+
+      # Kokoro voices sentences in six languages and the transcription server,
+      # started as Electron's second sidecar from the transcription.json the app
+      # writes, writes them down through the app's client; then a recording goes
+      # through ingest, the API and that server. The test checks the file
+      # against the manifest's hash; Silero's check runs once it is staged.
+      - name: Gate on speech recognition through the app
+        working-directory: surfsense_local/backend
+        env:
+          SURFSENSE_TEST_AUDIOCPP_DIR: ${{ github.workspace }}/surfsense_local/electron/audiocpp
+          SURFSENSE_TEST_AUDIO_MODELS: ${{ runner.temp }}/audio-models
+          RECOGNISER: ${{ runner.temp }}/recogniser/${{ steps.recogniser.outputs.name }}
+        run: |
+          if [ ! -f "$RECOGNISER" ]; then
+            mkdir -p "$(dirname "$RECOGNISER")"
+            curl -fsSL --retry 3 -o "$RECOGNISER" "${{ steps.recogniser.outputs.url }}"
+          fi
+          cp "$RECOGNISER" "$SURFSENSE_TEST_AUDIO_MODELS/"
+          uv run scripts/fetch_silero_vad.py models
+          uv run pytest tests/packaging/test_audiocpp_transcription_config.py tests/packaging/test_audiocpp_transcription.py tests/packaging/test_recording_transcription.py -m packaging
+
       # An artifact drops symlinks and file modes; a tar keeps both.
       - name: Pack
         run: tar -cf audiocpp.tar audiocpp
@@ -148,3 +198,25 @@ jobs:
           name: audiocpp-${{ runner.os }}
           path: surfsense_local/electron/audiocpp.tar
           retention-days: 7
+
+  # macOS ships upstream's archive, which the recipe does not build, so a
+  # family added to the pin is checked against the archive itself. The release
+  # job stages it again; this only fails earlier, on the pull request.
+  macos-archive:
+    name: macOS arm64, upstream's archive
+    runs-on: macos-15
+    defaults:
+      run:
+        working-directory: surfsense_local/electron
+    steps:
+      - uses: actions/checkout@v6
+
+      - uses: actions/setup-node@v5
+        with:
+          node-version: 24
+          package-manager-cache: false
+
+      # Downloads the pinned archive, copies each curated family's spec, which
+      # fails on one it lacks, and starts the server once per family.
+      - name: Stage and check every curated family
+        run: node scripts/audiocpp/stage.mjs --strict

.github/workflows/release-local.yml

@@ -173,8 +173,9 @@ jobs:
           "$WORKER" --check-vision-runtime
 
       # A script is not analysed when the worker is frozen, so a library the
-      # spec leaves out shows only when a script runs in the shipped worker.
-      - name: Run document scripts in the frozen worker
+      # spec leaves out shows only when a script runs in the shipped worker;
+      # the same holds for the decode child's miniaudio.
+      - name: Run document scripts and decode a recording in the frozen worker
         working-directory: surfsense_local/backend
         # addopts pins -m 'not packaging'; the later -m on the command line wins.
         run: |
@@ -203,11 +204,12 @@ jobs:
           echo "::error::frozen API never became healthy"
           exit 1
 
-      - name: Stage embedding, voice, and Docling parser packs
+      - name: Stage embedding, voice, voice activity, and Docling parser packs
         working-directory: surfsense_local/backend
         run: |
           uv run scripts/fetch_embedding_model.py models
           uv run scripts/fetch_bundled_voice.py models
+          uv run scripts/fetch_silero_vad.py models
           uv run scripts/fetch_docling_models.py models
 
       - name: Build frontend SPA

docs/adr/0056-speech-recognition-runs-on-a-second-audiocpp-server.md

@@ -0,0 +1,33 @@
+# ADR 0056: Speech recognition runs on a second audio.cpp server that only the API calls
+
+- **Status:** Accepted
+- **Date:** 2026-10-14
+- **Source:** [Local speech recognition proposal, the runtime L119–147](https://github.com/MODSetter/SurfSense/blob/55b3ddc2e9d6423f194ff7e5ef0c62910cece95f/docs/proposals/local-speech-to-text.md#L119-L147)
+
+## Context
+
+Recordings as sources and a mic in the chat composer need a speech recogniser that runs on the user's CPU and takes as little of the machine as it can. The app already ships audio.cpp's `audiocpp_server` for podcast voices ([packaging](../architecture/packaging.md)), and its pinned v0.8.2 source has NVIDIA's Nemotron 3.5 ASR, which held 1.19 GB loaded and ran at 11.5× real time on that server's CPU path. Compiling its family adds 9 source files and no new library.
+
+The voices' server cannot also serve it. Under its flags a dictation would evict Kokoro between a podcast's turns; a request that finds the other model busy fails at once with 503; the podcast's closing `unload_all_models` would unload a warm speech model and wait without bound on a transcription in flight; and every voice install, delete and API start rewrites `server.json`, which restarts the server and kills whatever it is transcribing.
+
+Text models already reach their runtime only through the API, so one gate can order every request ([ADR 0048](0048-the-api-is-the-only-path-to-a-text-model.md)). A recording's windows from the ingest worker and a speaker's phrases from the composer compete for one model the same way.
+
+## Decision
+
+- Speech recognition runs on `audiocpp_server`, the binary staged for voices, compiled with `nemotron_asr` beside the voice families and started a second time as its own sidecar, `audiocpp-transcription`.
+- It reads its own config, `<data>/audio/transcription.json`, which the API writes, and listens on its own port. Electron passes the flags: the CPU backend, half the logical cores up to 8, one loaded model, an unload after 5 idle minutes, 1 GiB kept free, no web UI. It gets no eSpeak.
+- Electron starts it when that file exists, restarts it when the file changes and stops it when the file goes. After a crash it restarts it, at most once every 10 seconds, and leaves it stopped after 3 crashes in 5 minutes, until the file changes again.
+- Only the API is told where it is. Every other process reaches speech recognition through the API.
+- One gate in the API sends every request to it, one at a time, since it runs one model and one request at a time anyway. A dictation's phrase goes ahead of a recording's window and waits at most for the window in flight. While a dictation is open, from the mic press until its last phrase or 20 s without one, no window starts.
+- A hold of the text runtime, which a podcast takes while it voices and a local image while it draws, also starts no window until it ends, waits for the request in flight, and unloads the speech model by id. Dictation stays available during it, behind the memory check.
+- Before a request that would load the model, the API checks the entry's measured peak plus 1 GiB against the memory the operating system says is available, unless the server says the model is loaded. Speech never unloads or holds the chat model; a shortfall is refused in a sentence.
+- [transcribe.cpp](https://github.com/handy-computer/transcribe.cpp) is the fallback, replacing the API's client, the sidecar spec and the config writer, not the routes.
+
+## Consequences
+
+- Voices and speech recognition never unload, evict, restart or wait on each other. The second process costs 7 MiB idle and 10 MiB after its model is unloaded (measured).
+- A dictation does not wait for a recording behind the server's own queue, because the API decides the order. A recording pauses while someone dictates, and while a podcast voices or an image draws.
+- The server's own memory guard, which estimates from the file's size, is only a backstop: the app admits from a measured peak.
+- The languages are those of the families in the pin. audio.cpp will not run Whisper, so the long tail of languages needs a later model or the fallback.
+- One binary serves both, so a pin bump is measured for both: the voices' peaks and the speech model's.
+- Settings › Resources counts the second process under its audio.cpp row with no change, since it counts every `audiocpp_server`.

docs/adr/README.md

@@ -83,3 +83,4 @@ The Date is the day the decision was first written down. Source links are permal
 | 0053 | [First-party plugins are listed in the app, third-party plugins in a registry in this repository, and third-party plugins start off](0053-plugins-are-listed-in-a-registry-and-third-party-ones-start-off.md) | Accepted, in part superseded by 0055 |
 | 0054 | [SurfSense's own plugins, free and paid, run on one plugin host, with the registry served beside it](0054-surfsenses-plugins-run-on-one-plugin-host.md) | Accepted |
 | 0055 | [Verified is a maintainer's judgement, and CI contacts each server a pull request adds](0055-verified-is-a-maintainers-judgement-and-ci-contacts-each-server.md) | Accepted |
+| 0056 | [Speech recognition runs on a second audio.cpp server that only the API calls](0056-speech-recognition-runs-on-a-second-audiocpp-server.md) | Accepted |

docs/architecture/agent.md

@@ -201,7 +201,7 @@ The snapshot routes, mounted only beside an opencode ([`previews/router.py`](../
 
 Each but `file` refuses a caller with an `Origin` header, and one without `Authorization: Bearer <key>`, the key Electron made at boot and passed to the API as `SURFSENSE_LOCAL_DOCX_SNAPSHOT_KEY` ([`snapshot_key.py`](../../surfsense_local/backend/modules/agent/previews/snapshot_key.py)), because any process on the machine can reach loopback. The `file` route is fetched by the print window itself, which sends an Origin and holds no key, as it fetches an artifact's file: the request id, a random 128-bit value only the key's holder is handed, lets it in, and only while that request is being printed.
 
-Electron's side ([`docx-snapshot/`](../../surfsense_local/electron/src/main/docx-snapshot/)) takes the next request at once after taking one, prints up to three at a time, gives each print 25 seconds, under the API's 30, and each call to the API 10. A request names its format, `docx` or `pptx`, and its pages as `printToPDF` page ranges; one without them is a Word file's pages 1 to 4, and a format or range it cannot print is reported as the request's failure without printing. A PDF the API refuses is reported as the request's failure. The page lays a Word file out as above, and a deck with @aiden0z/pptx-renderer, the in-app viewer's library: every slide on a page of the deck's own slide size, read from the file, printed with no margin, once each slide's pictures, backgrounds and charts are drawn ([`features/docx-snapshot/`](../../surfsense_local/frontend/src/features/docx-snapshot/)). The document is untrusted, since a source can steer the script that wrote it, so the hidden window runs sandboxed with no preload, in its own in-memory session whose requests may reach only the page's own files and object URLs and the one file being printed, and it never navigates. The page's policy allows only its own script, inline styles, data-URL and object-URL images, data-URL fonts, and connections to loopback; docx-preview renders no altChunk, whose HTML it would put in an unsandboxed frame, and the deck renderer gets no pdf.js, which would start a worker. A page whose script did not start is reported as such, with the window's first console error.
+Electron's side ([`docx-snapshot/`](../../surfsense_local/electron/src/main/docx-snapshot/)) takes the next request at once after taking one, prints up to three at a time, gives each print 25 seconds, under the API's 30, and each call to the API 10. A request names its format, `docx` or `pptx`, and its pages as `printToPDF` page ranges; one without them is a Word file's pages 1 to 4, and a format or range it cannot print is reported as the request's failure without printing. A PDF the API refuses is reported as the request's failure. The page lays a Word file out as above, and a deck with @aiden0z/pptx-renderer, the in-app viewer's library: every slide on a page of the deck's own slide size, read from the file, printed with no margin, once each slide's pictures, backgrounds and charts are drawn ([`features/docx-snapshot/`](../../surfsense_local/frontend/src/features/docx-snapshot/)). The document is untrusted, since a source can steer the script that wrote it, so the hidden window runs sandboxed with no preload, in its own in-memory session whose requests may reach only the page's own files and object URLs and the one file being printed, and which refuses it every permission, the microphone and the clipboard among them; it never navigates. The page's policy allows only its own script, inline styles, data-URL and object-URL images, data-URL fonts, and connections to loopback; docx-preview renders no altChunk, whose HTML it would put in an unsandboxed frame, and the deck renderer gets no pdf.js, which would start a worker. A page whose script did not start is reported as such, with the window's first console error.
 
 ## A turn