fix: the whisper container bakes its model — the release moved, the pin cannot follow (#3906) by rbuergi · Pull Request #3956 · Systemorph/MeshWeaver

rbuergi · 2026-09-10T18:16:12Z

The premise, re-verified today

deploy/whisper/helm/values.yaml:22 pointed a model initContainer at a GitHub release on this repo. Measured 2026-09-10:

URL	Status
`github.com/Systemorph/MeshWeaver/releases/download/voice-model-swiss-german/ggml-swiss-german-turbo-q5_0.bin` (the removed pin)	404

#2593 moved that asset to the private Systemorph/MeshWeaver.Plugins on 2026-08-28, updated three READMEs and run-local.sh, and left values.yaml and docker-compose.yml behind. The initContainer runs curl -fSL … "$MODEL_URL" under set -e, so the 404 aborts the pod in Init:Error and speech-to-text could not be enabled on any deployment.

The decision

Repointing the URL is not a fix — a private asset needs an Authorization header and the API asset URL. Three real options; I picked the third.

❌ 1. Token-carrying fetch — rejected on blast radius

The only credential that can read a private release asset is one that can read the repository, and Systemorph/MeshWeaver.Plugins is the private home of the entire AI engine, the plugin sources and the fleet's module trees. That buys one 547 MB blob with read access to the whole private platform, held as a standing namespace secret that does not exist today and would need rotating forever. It also inverts the plugin-catalog credential model, whose whole point is that the registry holds the git credential so consumers never need one.

❌ 2. Republish it anonymously — rejected on LICENCE, and again on prior intent

I checked the licence rather than assuming it. The model derives from Flurin17/whisper-large-v3-turbo-swiss-german, published under CC BY-NC-4.0 — Attribution, NonCommercial. Verified two ways on 2026-09-10: the model card states "This model is distributed under the Creative Commons Attribution-NonCommercial 4.0 license (cc-by-nc-4.0)", and the HF API returns cardData.license: cc-by-nc-4.0 plus the tag license:cc-by-nc-4.0. The base model openai/whisper-large-v3-turbo is MIT, but that is not the binding half — a derivative cannot be more permissive than the work it derives from, so our GGML file inherits the NonCommercial restriction. Publishing it for anonymous world download from a commercial product's infrastructure is exactly what NC restricts.

Independently, #2593 already decided this: it moved the asset off such a URL deliberately, recording that "this is a Systemorph-tuned model, and anonymous world-download was an accident of hosting", that private hosting is "a feature, not a cost", and the guardrail "any future release of a large binary goes to artifact storage, never a release asset on this repo."

For completeness, both anonymous URLs that already exist were measured and both 404: the old core release asset, and the portal content route /api/content/MeshWeaver/static/Speech/… documented in OnDeviceVoice.md — which is expected to 404 (that page's own caveat says the static-assets share is not yet mounted) and is an access-controlled route in any case, so it was never anonymous.

✅ 3. Stage the model into the image — chosen

Bake it at /models/model.bin and let the image registry distribute it. The pod already authenticates to that registry to pull its own image, so this adds zero new credentials, zero secrets, zero standing access, and the model never leaves the authenticated boundary the licence requires. It is also the option #2593 named first — "ACR (an OCI artifact) …" — in the form kubelet can already fetch (a bare OCI artifact would need oras in an image that ships only curl).

Two bugs it fixes beyond the 404: the emptyDir was per-pod, so every restart re-downloaded 547 MB; and the failure now lands at build time, on the machine of the person who can fix it, naming the file and the exact gh release download command.

Cost, stated plainly: the image grows by 574,041,195 bytes (~547 MiB), and the build requires the model in the build context first (gh auth).

Also fixed, because a half-fix is what caused this

deploy/whisper/docker-compose.yml carried the same stale pin — repaired. Its ./models:/models:ro bind mount is removed, because an empty host dir would now shadow the baked model. NOTES.txt no longer tells operators the pod downloads the model, and the .gitignore note that "the Dockerfile bakes no model" is corrected.

What I did NOT do

None of the forbidden softenings: -f stays, no || true, the init is not made non-blocking, no retry count raised. Each turns a loud 404 into a pod that starts without a model and serves a silent 503.

image.tag is deliberately unchanged at 1.7.4: it names the whisper.cpp release the binary is built from, and moving it to a tag no image carries would red the nightly lock-pinned-digests axis-2 lane ("does not resolve in the registry, so there is nothing to protect"). I verified the extractor still finds the pin in the edited file.

👤 What a person must still do — two named, actionable items

Build and push a model-bearing whisper-swiss-german image (needs gh auth to the private repo + registry push; no new credential is created):
```
gh release download voice-model-swiss-german --repo Systemorph/MeshWeaver.Plugins \
  --pattern 'ggml-swiss-german-turbo-q5_0.bin' --output deploy/whisper/models/model.bin
az acr build -r meshweaver -t whisper-swiss-german:1.7.4 deploy/whisper
```
Until this happens, meshweaver.azurecr.io/whisper-swiss-german:1.7.4 has no model in it and the pod will start and fail to load one. I cannot do this: it needs registry push, and operations here go through the memex API — I did not touch a cluster.
Mirror this change into Systemorph/Memex at deployments/aks/memex-cloud/whisper/helm/ — filed as Systemorph/Memex#271 so it is tracked rather than living in this PR body. That is a verbatim vendored copy of this chart, not a dependency — I read it and it carries the identical broken URL. Nothing makes the two agree.

Separately, and not resolved by me because it is not mine to resolve: CC BY-NC-4.0 is a NonCommercial licence, and this model is used in a commercial product. That question predates this issue (it applies equally to the shipped on-device MAUI path) and my change does not alter the posture — but it is now written down rather than implicit, and it deserves a decision.

Conserved work product

src/MeshWeaver.Documentation/Data/Architecture/VoiceModelDistribution.md (/Doc/Architecture/VoiceModelDistribution) — the licence and how it was verified, the standing rule (this asset travels only through channels that already authenticate their reader), all three options with why two lost, every consumer including the Memex vendored copy, and a checklist for the next move. This is the second time the asset moved and left a consumer behind, and the reasoning existed nowhere in the repo. Linked from Architecture.md, OnDeviceVoice.md, CentralizedSpeech.md and the whisper README.

Verification — each of these can fail

Check	Result
removed pin (old core release)	404
private asset, unauthenticated	404 — this is why a repoint cannot work
private asset via `gh api` (the documented path)	200, `574,041,195` bytes, `sha256 2d56e77…5ea612`
portal content route from `OnDeviceVoice.md`	404 (and access-controlled)
`meshweaver.azurecr.io/v2/` anonymously	401 — the authenticated channel, which is the point
`helm lint` + `helm template`	clean
rendered Deployment	asserted: no initContainer, no `/models` volume/mount, no `MODEL_URL` — and the assertion was falsified against the pre-change chart (exit 1, naming all six)
`DocumentationLinkIntegrityTest`	Passed — after first injecting a broken link into the new page and confirming it goes RED naming that page
`lock-pinned-digests.py --self-test`	passed; axis-2 still extracts `whisper-swiss-german:1.7.4`
`check-workflow-timeouts.py` / `check-pr-secret-preflight.py`	0 violations
`dotnet build -c Release -warnaserror` (Documentation.Test)	0 Warning(s), 0 Error(s)

No What's New entry, and deliberately so: nothing here is user-visible yet. Speech was already 503 on every deployment and stays 503 until item 1 above is done and Speech:Enabled is set — this change makes that possible, it does not perform it. An entry now would announce a fix nobody can observe.

🤖 Generated with Claude Code

https://claude.ai/code/session_01S25PJZbVeCbkWhCfMub9P7

…in cannot follow (#3906) `deploy/whisper/helm/values.yaml:22` pointed a model initContainer at a GitHub release on THIS repo. #2593 moved that asset to the PRIVATE MeshWeaver.Plugins repo on 2026-08-28, updated three READMEs and a shell script, and left `values.yaml` and `docker-compose.yml` behind. Measured again 2026-09-10: the URL is HTTP 404. The initContainer runs `curl -fSL … "$MODEL_URL"` under `set -e`, so the 404 aborts the pod in Init:Error and speech-to-text could not be enabled anywhere. Repointing the URL is not available as a fix, and neither are the two obvious alternatives: 1. Token-carrying fetch — REJECTED on blast radius. The only credential that reads a private release asset reads the REPOSITORY, and Systemorph/MeshWeaver.Plugins is the private home of the whole AI engine and the fleet's module trees. That buys one 547 MB blob with read access to the entire private platform, as a standing namespace secret that does not exist today, and it inverts the plugin-catalog model whose point is that consumers never hold the git credential. 2. Republish it anonymously — REJECTED on LICENCE, and independently on #2593's stated intent. The model derives from Flurin17/whisper-large-v3-turbo-swiss-german, which is CC BY-NC-4.0 (checked on the model card and the HF API, 2026-09-10; the MIT base model is not the binding half — a derivative cannot be more permissive than its source). #2593 also recorded that "anonymous world-download was an accident of hosting" and guardrailed large binaries off this repo's releases. 3. Stage the model into the image — CHOSEN. The pod already authenticates to the registry to pull its own image, so this adds zero new credentials, zero secrets and zero standing access, and the model never leaves the authenticated boundary the licence requires. It is also what #2593 asked for ("ACR (an OCI artifact) …"), in the form kubelet can already fetch. Dockerfile COPY models/model.bin, plus a size floor that fails the BUILD on a truncated download or a saved error page, and prints size + sha256. The failure now lands on the machine of the person who can fix it, naming the exact gh command. deployment.yaml the initContainer and the /models emptyDir are gone; the container reads the baked path. Also fixes a latent waste: emptyDir is per-pod, so every restart re-downloaded 547 MB. values.yaml `model.url` deleted — there is nothing to repoint it at. `image.tag` deliberately UNCHANGED: it names the whisper.cpp release (v1.7.4), and moving it to a tag no image carries would red the nightly lock-pinned-digests axis-2 lane ("does not resolve in the registry"). Verified the extractor still finds the pin. docker-compose the same stale pin, and the ./models:/models bind mount removed — an empty host dir SHADOWS the baked model. Fixing only the chart would repeat #2593's shape. NOTES.txt no longer tells operators the pod downloads the model. .gitignore the "Dockerfile bakes no model" note is now false; models/ stays untracked. None of the forbidden softenings were used: `-f` stays, no `|| true`, the init is not made non-blocking, no retry count was raised. Each of those turns a loud 404 into a pod that starts without a model and serves a silent 503. Doc: Data/Architecture/VoiceModelDistribution.md records the licence, the standing rule (this asset travels only through channels that already authenticate their reader), all three options with why two lost, every consumer including the VERBATIM VENDORED COPY of this chart in Systemorph/Memex at deployments/aks/memex-cloud/whisper/helm, and a checklist for the next move. This is the second time the asset moved and left a consumer behind; the reasoning was nowhere in the repo. Measured, not assumed: old core release asset (removed pin) 404 private asset, unauthenticated 404 ← why a repoint cannot work private asset via gh api (documented path) 200, 574,041,195 bytes, sha256 2d56e773724a247360067b527417842b81d25ff891fed014341a6844f15ea612 portal content route in OnDeviceVoice.md 404 (also access-controlled — never anonymous) meshweaver.azurecr.io/v2/ anonymously 401 ← the authenticated channel, which is the point helm lint + template clean; rendered Deployment has no initContainer, no /models volume, no MODEL_URL (asserted, and the assertion falsifies against the pre-change chart) DocumentationLinkIntegrityTest Passed (and made to FAIL on this new page first) lock-pinned-digests --self-test passed; axis-2 still extracts whisper-swiss-german:1.7.4 check-workflow-timeouts / pr-secret-preflight 0 violations No What's New entry: nothing here is user-visible yet. Speech was already 503 on every deployment and stays 503 until a person builds and pushes a model-bearing image and enables Speech — this change makes that possible, it does not perform it. Fixes #3906 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S25PJZbVeCbkWhCfMub9P7

…gets the CC BY-NC section it was already being sent to clients/voice-gateway/README.md has carried a 'Speech model license' section all along, naming CC BY-NC 4.0 and telling a commercial reader to 'point the whisper container at a permissively licensed model (see deploy/whisper/README.md)'. That target said nothing about a licence — so the constraint existed only in the file least likely to be read by someone editing a Helm chart, which is exactly how the chart's model source came to be chosen without it. deploy/whisper/README.md now carries the licence and the permissive-model escape hatch, and VoiceModelDistribution.md cites the pre-existing note as independent corroboration of the CC BY-NC finding rather than resting on my HuggingFace read alone. It also states plainly that the NonCommercial-vs-commercial-product question is OPEN, predates this issue, and is not answered here. DocumentationLinkIntegrityTest: Passed. Release -warnaserror: 0 Warning(s), 0 Error(s). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S25PJZbVeCbkWhCfMub9P7

Copilot

🟢 Approval recommended

The deploy chart and container changes are internally consistent (no remaining model.url/initContainer references) and the only finding is a small documentation comment correction in .gitignore.

Pull request overview

This PR fixes a broken Whisper Swiss-German model download path by switching the deployment approach from “download model at pod startup” to “bake the model into the container image,” and updates documentation to capture the licensing and distribution constraints that drove that decision.

Changes:

Removes the Helm initContainer + model.url mechanism and assumes /models/model.bin is present in the image.
Updates the Whisper Dockerfile and docker-compose workflow so the model is a required build-context input and is verified during image build.
Adds/updates architecture documentation to document the rationale (private hosting + CC BY-NC-4.0 constraints) and link the new guidance from related pages.

File summaries

File	Description
src/MeshWeaver.Documentation/Data/Architecture/VoiceModelDistribution.md	New architecture page documenting constraints, options considered, consumers, and checklist for future moves.
src/MeshWeaver.Documentation/Data/Architecture/OnDeviceVoice.md	Links to the new distribution guidance before discussing model hosting/URLs.
src/MeshWeaver.Documentation/Data/Architecture/CentralizedSpeech.md	Updates container description to reflect baked-in model and references the new distribution doc.
src/MeshWeaver.Documentation/Data/Architecture.md	Adds the new “Voice model distribution” page to the Architecture index.
deploy/whisper/README.md	Updates operator workflow to fetch model into build context and warns against `/models` mounts shadowing the baked model.
deploy/whisper/helm/values.yaml	Removes `model.url` and documents that the image must be rebuilt with the model baked in.
deploy/whisper/helm/templates/NOTES.txt	Updates post-install notes to match “model baked into image” behavior and adds a troubleshooting hint for model-missing images.
deploy/whisper/helm/templates/deployment.yaml	Removes initContainer and `/models` emptyDir volume/mount; keeps runtime env pointed at baked model path.
deploy/whisper/Dockerfile	Bakes `models/model.bin` into the runtime image and adds build-time sanity checks (size threshold + sha256 print).
deploy/whisper/docker-compose.yml	Removes the runtime bind mount and updates setup instructions to fetch the model for build-time baking.
.gitignore	Keeps the model build-context directory untracked and updates explanatory comments about the new baking behavior.

Review details

Files reviewed: 10/11 changed files
Comments generated: 0
Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

github-actions · 2026-09-10T18:26:48Z