Repository navigation
[LLM|Audio] Voice podcasts on audio.cpp - #1907
Merged
Merged
Conversation
Podcasts are voiced by an audio model the user downloads and chooses, run by audio.cpp's server as a sidecar, in place of the Python Kokoro in the worker. Kokoro-82M, Supertonic 3 and KittenTTS Mini are curated; every other TTS family audio.cpp runs is listed with the reason it is left out. The runtime settings come from measurements on a laptop CPU and from the apps that already drive audio.cpp: the podcast unloads the model when it ends, with a 5-minute idle backstop; half the logical cores; the CPU while the chat model holds the GPU; the app's own memory check, stepping Kokoro's chunk size down before it refuses. Windows and Linux builds are made in release CI, since upstream's need glibc 2.38 and its Windows build carries AVX-512 code. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The image work that introduced catalog/local/engines/ has merged, so the files the plan builds beside are links now, not code spans. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
audio.cpp's server refuses an empty model list, so Electron starts it only while the API's server.json names a model, and restarts or stops it when the file changes, as it does llama-server for its preset. Electron owns the machine-wide settings as flags, which win over the config: the CPU, half the logical cores up to 8, one resident model, a 5-minute idle backstop and a 1 GB memory floor. eSpeak comes from beside the server. Nothing writes server.json yet, so nothing starts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
scripts/audiocpp/ stages the server into electron/audiocpp in dev and in the release. macOS downloads upstream's pinned archive. Windows and Linux compile the pinned commit, because upstream's Linux archives need glibc 2.38 and its Windows one puts AVX-512 in the executable. The recipe builds the CPU backend only, one ggml library per CPU family, without OpenMP, and on Linux with libstdc++ static in every file. Without a toolchain, dev runs without local audio; --strict fails. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
build-audiocpp.yml compiles on ubuntu-22.04 and Windows, gates Linux on glibc 2.34, GCC 7's libgcc and no dynamic libstdc++, voices one Kokoro passage, and hands the staged folders to the packaging jobs. It caches by the recipe and runs on pull requests that change it. Nothing is published: a release on this repository would become the newest entry of the feed installed apps update from. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
audio.cpp writes its family before the files it embeds, which make Kokoro's header 38.6 MB and Supertonic's 57 MB, so the family is read by walking the keys from the front. kokoro_tts, supertonic and kitten_tts read text aloud; a bare audiocpp, which speech recognisers declare too, has its own group and is not offered from search. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An audio entry commits its voices, languages, sample rate and the memory measured while voicing, which the server reports none of. The refresh script pins each model's builds from its folder of audio-cpp/audio.cpp-gguf, and --only adds entries without re-pinning the rest. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The catalog offers the three audio models once Electron hands the API an audio folder, which it does where audio.cpp is staged. Installing, deleting and starting up rewrite server.json, which Electron starts the server from; the last delete removes it. Installs are matched by their recorded file, since every audio model shares one repo. The chat lists keep llama.cpp's rows only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Revision 0016 rebuilds selected_models' checks so audiocpp holds audio_gen, and only that, without a connection. Selection takes an installed audio build only. Audio rows carry voicing, the memory measured while voicing, voice count and languages, and the chosen audio model leads its row as in use. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Settings > Audio lists the models on this computer with Use and Delete, never Delete for the one in use, and Add model offers the curated audio models on one line each: build, size, memory while voicing, voices and languages. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The podcast requires text_gen and audio_gen. Its voice resolves from the audio selection to an adapter over audio.cpp's server: the roster comes from the model's entry, each voice with the languages it speaks; each turn is one request, joined with the 0.35 s pause; the model is unloaded when voicing ends, even on failure; and voicing refuses first when available memory is under the measured peak plus 1 GiB. A server's audio model is refused until a remote speech client exists. kokoro-onnx, its fetch script, the voice pack and its bundling go. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The brief form reads each voice's languages, so a Supertonic voice appears under all 31, and a speaker keeps a voice that still speaks the new language. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An opt-in packaging test starts the staged server with the sidecar's flags from the server.json the app writes, voices two turns of each curated model through the adapter, checks the manifest's sample rate, and asserts nothing stays loaded. It fails when the unload is removed. Two weak tests now compare against literals: the pause length, and the thread count for a machine of 12 and of 32 cores. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The memory check ran only at voicing, after minutes of drafting, and the queue's retry drafted the whole episode again to refuse again. The podcast now checks before drafting and again at voicing, and a memory refusal is not retried. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A refusal for memory now adds the first other curated audio model, in the manifest's order, that would fit, so a person short of memory knows which model to install instead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
audio.cpp sizes a request's working memory for its longest chunk, and the committed peaks came from passages whose chunks fell short of the limit. Through the server with every chunk full, Kokoro peaks at 2,347 MB, not 1,421, Kitten at 1,863 MB, not 1,023, and Supertonic at 486 MB. Kitten's sweep gives it chunk steps at 240, 120 and 60. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The audio.cpp workflow now runs the real-server test on both builds, cached or not, in place of the Kokoro smoke run: each curated model's pinned file, voiced through the app's adapter from the server.json the app writes, then unloaded. A pull request that changes the adapter, the engine, the manifest or the test runs it too. The test now fails without a staged server once it is given models, so CI cannot skip it unseen. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Resolves the models UI rework (MODSetter#1906) against local audio: Electron stages both sd.cpp and audio.cpp, predev builds both, and the catalog doc says which sections offer Delete. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
@CREDO23 is attempting to deploy a commit to the Rohan Verma's projects Team on Vercel. A member of the Team first needs to authorize it. |
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Advanced Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
docs/proposals/local-audio-models.md.build-audiocpp.yml), gated on the app's glibc 2.34 floor, and stages it with eSpeak-ng. macOS takes upstream's archive, pinned by sha256. Nothing is published as a release of this repo.audio_genis held byaudiocpp(revision0016).build-audiocpp.ymlvoices all three models on each build through the app's own adapter andserver.json(tests/packaging/test_audiocpp_voicing.py).devis merged in, with conflicts from feat(local-ui): add chat and image model steps to onboarding, with server dialogs and a logo loader #1906 resolved inindex.ts,package.jsonand two docs.Follows #1903.
🤖 Generated with Claude Code
High-level PR Summary
This PR migrates podcast voice generation from the Python worker's Kokoro ONNX implementation to audio.cpp's server, a sidecar process managed by Electron. It adds three curated audio models (Kokoro 82M, Supertonic 3, KittenTTS Mini) to the local catalog with download and selection UI, implements memory checks before voicing, and includes CI workflows to compile audio.cpp for Windows/Linux with CPU-only backends while macOS uses upstream archives.
⏱️ Estimated Review Time: 30-90 minutes
💡 Review Order Suggestion
.github/workflows/build-audiocpp.ymlsurfsense_local/electron/scripts/audiocpp/pins.mjssurfsense_local/electron/scripts/audiocpp/recipe.mjssurfsense_local/electron/scripts/audiocpp/compile.mjssurfsense_local/electron/scripts/audiocpp/stage.mjsdocs/proposals/local-audio-models.mdsurfsense_local/backend/modules/llm/catalog/local/engines/audiocpp/manifest_fields.pysurfsense_local/backend/modules/llm/catalog/local/engines/audiocpp/evidence.pysurfsense_local/backend/modules/llm/catalog/local/engines/audiocpp/engine.pysurfsense_local/backend/modules/llm/catalog/local/manifest/models.jsonsurfsense_local/backend/modules/llm/providers/audiocpp/speech.pysurfsense_local/backend/modules/llm/providers/audiocpp/memory.pysurfsense_local/backend/modules/llm/resolution.pysurfsense_local/backend/alembic/versions/0016_local_audio_provider.pysurfsense_local/backend/modules/llm/models.pysurfsense_local/electron/src/main/sidecars/audiocpp.tssurfsense_local/electron/src/main/index.tssurfsense_local/backend/worker/studio/job.pysurfsense_local/backend/worker/studio/media/audio/podcast/pipeline.pysurfsense_local/backend/modules/artifacts/formats.pysurfsense_local/backend/modules/artifacts/podcast/brief.pysurfsense_local/frontend/src/features/models/local/audio/api.tssurfsense_local/frontend/src/features/settings/models/audio-models-settings.tsxsurfsense_local/backend/bundling/worker.specsurfsense_local/backend/pyproject.tomlsurfsense_local/backend/scripts/fetch_kokoro_model.pysurfsense_local/backend/modules/llm/providers/kokoro/provider.pyREADME.mddocs/architecture/overview.mddocs/architecture/packaging.mddocs/architecture/studio.mddocs/architecture/local-models/catalog.md