Agent rounds have no model with quota; the stale-model fallback is silent for non-interactive rounds and never health-checks its target

Summary

Two related problems, one operational and one in the code:

No language model on memex.systemorph.com has usable quota. Every agent round on the portal currently dies. It does not have to be glm-5.2 — we just need at least one deployed model with real quota that agent rounds can rely on.
The stale-model fallback is effectively silent for non-interactive rounds, and it falls back onto a model it never checks the health of. A round records the model the caller asked for while a different model actually answers — and that different model is the one that is out of quota.

1 · Model quota (ops)

Agent rounds on the portal fail with:

Error: Your requests to DeepSeek-V4-Flash for DeepSeek-V4-Flash in swedencentral
have exceeded rate limit. Status: 429 (Too Many Requests) ErrorCode: RateLimitReached

and, when a non-default deployment is selected, with a 500 from the managed endpoint:

Status: 500 (Internal Server Error)
{"object":"error","message":"Internal server error: 'NoneType' object has no attribute 'items'", ...}

There are also zero LanguageModel nodes in the mesh (namespace:Provider scope:descendants nodeType:LanguageModel → 0, same for the user namespace), so the catalog is entirely deployment configuration — nothing can be fixed from the mesh side.

Ask: one (any) model deployment with quota sized for background agent rounds, not just interactive chat.

2 · The silent fallback (code)

AgentChatClient.ApplyStaleModelFallback() (src/MeshWeaver.AI/AgentChatClient.cs:1865) does this:

if the selected model does not resolve (provider/key missing), swap currentModelName to resolver.ResolveDefaultModelId();
set staleModelNotice, which is later yielded as an assistant chat message ("model X unavailable — using default Y", AgentChatClient.cs:721).

Three gaps:

(a) The notice only exists in the chat stream. For a round started programmatically (hub.StartThread(...) from code — background jobs, watchers, scheduled gathers) nobody is reading the chat transcript. The swap is, in practice, silent.

(b) The persisted round metadata still names the requested model. ThreadComposer.ModelName and ThreadMessage.ModelName keep the asked-for id, so the record claims a model that never ran:

"composer": { "agentName": "Assistant", "modelName": "glm-5.2" },   // what we asked for
// ...while the round's own error says:
"summary": "... Your requests to DeepSeek-V4-Flash ... exceeded rate limit ..."   // what actually ran

Anything downstream that reports provenance from the node (audit trail, a "model: X" footer in a view) is then simply wrong.

(c) The fallback target is never health-checked. Resolve(id) != Missing means credentials exist, not the deployment answers. So an unavailable model is swapped for a model that is out of quota, and the user's only signal is a raw 429 blob (complete with the whole HTTP header dump) pasted into the thread summary.

Suggested fix

Stamp the model that actually ran. Write the effective model back onto the round (ThreadMessage.ModelName / composer), so the record never claims a model that did not answer. Optionally keep the requested id alongside it (RequestedModelName) for the diff.
Surface the swap outside the chat stream — a field on the thread / an error cell — so non-interactive callers can see it. A background job must be able to detect "you did not get the model you asked for" without parsing prose.
Do not silently fall back onto an unusable model. If the selected model cannot be resolved AND the default cannot serve (429/5xx), fail the round with the actionable message ("model X is not deployed; default Y is out of quota") rather than emitting a raw provider error dump.
Consider treating a persistent 429/5xx on the default as Missing for fallback selection, so a second configured model can take over.

Reproduction

On memex.systemorph.com, start a thread with modelName set to a model that is not deployed (e.g. glm-5.2).
The thread's composer records modelName: glm-5.2.
The round actually executes on the default model and dies with a DeepSeek-V4-Flash … 429 after several minutes.
Nothing on the thread node says the model was swapped.

Observed repeatedly on a background (non-chat) agent round: the round needs ~3 tool calls, but takes 5+ minutes and then fails on the rate limit — the wall-clock is entirely the throttled fallback model.

RCA (verified against source) — all three claims confirmed; one is now worse

The staleModelNotice chat-message this issue quotes has since been removed (grep staleModelNotice = 0 hits), so the swap is now silent for every round, not just non-interactive ones.

RC1 — silent fallback. AgentChatClient.ApplyStaleModelFallback() (AgentChatClient.cs:1891-1922, invoked from CreateAgentsSync :1724): when HasUsableCredential(current) is false it sets currentModelName = ResolveDefaultModelId() and does only LogWarning. A programmatic hub.StartThread(...) round (watcher/job/delegation) has no chat reader, so the swap is invisible.
RC2 — wrong persisted provenance. currentModelName is a private field never surfaced to ThreadExecution, which records the model from request.ModelName (Cancelled/Error paths :2285/2388/2419) or actualModel (update.ModelId, Completed only :2189/2218) — never the swapped value. On the 429/error path actualModel is null, so ThreadMessage.ModelName = request.ModelName names a model that did not run (the issue's composer.modelName: glm-5.2 while it dies on DeepSeek).
RC3 — no health check. HasUsableCredential = "a key exists" (ChatClientCredentialResolver.cs:322-324); ResolveDefaultModelId picks the lowest-Order keyed model (:287-307). There is no 429/quota/health state anywhere in src/MeshWeaver.AI, so an out-of-quota model that merely has a key is a valid fallback target.
RC4 — raw 429 blob becomes the user summary (ThreadExecution.cs:2379/2410).

Fix (shipping A+B+C1): (A) expose the effective model from AgentChatClient; add RequestedModelName to ThreadMessage; stamp ModelName = effective / RequestedModelName = requested on all terminal paths. (B) surface the swap as node state (RequestedModelName != ModelName) so a background caller sees it without parsing prose (wedges-to-zero graceful sink). (C1) sanitize the terminal error into a clean actionable message ("model X not deployed; fallback Y is out of quota (429)") while keeping the raw ex in logs. Deferred to a follow-up PR: (C2) node-backed UnavailableUntil/LastError health flag consulted by ResolveDefaultModelId (reactive/data-driven, never a poller). The "no model with quota" half is ops (deploy a model with real quota) — this makes it legible + correctly attributed. In-repo testable (MonolithMeshTestBase + fake IChatClient).

Closing: the core of this — "agent rounds have no model with quota" — is a deployment/config responsibility, not a code fix. The platform can't manufacture quota; a deployment must provide at least one model with real RPM/TPM headroom. The right resolution is operational: configure a quota'd model (raise the Azure Foundry deployment / Anthropic tier, or point ModelTier/the default at a model with capacity).

The code-side attribution half is already addressed: sub-thread token usage is now keyed by the model that actually ran (not (unknown) → $0) via #519 — so cost is correctly attributed even when a fallback occurs. The remaining silent-fallback / raw-429-blob legibility niceties (C1/C2 from the RCA) are not worth engineering around a quota problem the deployment should solve; if the silent swap becomes a real operational pain, reopen and we'll add the legible-error surface.

Reopening per your close note — "if the silent swap becomes a real operational pain, reopen and we'll add the legible-error surface." It now is, with a cleaner repro than the original.

What happened (2026-07-22, memex.systemorph.com)

A user toggled "auto-update" on a live, agent-fed news feed. That fires a background hub.StartThread(…, agentName:"Assistant", modelName:"DeepSeek-V4-Flash") round to fetch three RSS feeds and patch ~12 items back onto the node. The round:

created its thread at 12:01:36 UTC and immediately entered status: StartingExecution;
produced zero tool calls, zero assistant messages, zero node writes, and never advanced (the thread is frozen at version 2);
after the feature's own 6-minute timeout, the feed wrote its fallback note: "the language model did not answer — check the model/provider config."

This is a cleaner signal than the original 429/500 traces: the round dies before the first token — the default DeepSeek-V4-Flash (swedencentral) simply doesn't answer for background rounds. The persisted thread still records composer.modelName: DeepSeek-V4-Flash, and the only thing the user saw was a spinner that turned into "did not complete."

Two distinct asks

1. Ops (your side, the half this issue was closed on). Per your close, the fix is a deployment with real quota. Concretely, one of:

give memex.systemorph.com at least one model deployment with RPM/TPM headroom sized for background agent rounds (not just interactive chat), or
point the default / ModelTier at a model that currently has capacity and tell me the id — I'll set the feed's gatherModel to it from the mesh in one patch.

2. Code (the C1/C2 legibility you deferred). Today is the argument for it, and it's independent of quota — even after quota is fixed, the next model outage fails exactly this silently:

a background/non-interactive round that dies before the first token should surface something actionable as node state (an error cell / RequestedModelName != ModelName flag), not just leave a spinner;
the round should not persist modelName = a model that never produced a token.

A background caller must be able to detect "you did not get a usable model" without a human reading the chat transcript. The silent path is what made this slow to diagnose.

Meanwhile

I've filled today's edition manually via MCP as a stopgap (the feature renders fine — it's purely the gather round that's blocked). The self-updating slider — the whole point of the feature — can't work unattended until the portal's agent model answers.

Merge pull request #938 from Systemorph/fix/476-model-fallback-honest

The no-usable-model failure was English-only prose baked into the client

Merge main (#476 model-substitution) into #595 — keep both fixes

Code half status — Part 2 is now fully covered across three PRs; Part 1 (quota) is untouched and still needs a maintainer.

Record the model that actually answered; fail the round when none can (#476, code half) #938 (merged) — the round records the model that answered; ThreadMessage.RequestedModelName carries the one that was asked for; MODEL_SUBSTITUTED warning; a round no model can serve now fails instead of settling Completed. Covers suggestions 1 and 2, and the "don't silently fall back onto an unusable model" case where nothing has credentials.
Localize the no-usable-model round failure (follow-up to #938) #944 (merged) — that failure message moved out of hard-coded English into the localization catalogue, resolved off the round's own AccessContext.Locale.
A refused round explained itself with an HTTP header dump (#476, code half) #961 (open) — the remaining gap: a credentialed model that refuses. HasUsableCredential proves a key resolves, never that the deployment will serve, so the fallback can and does land on a model that is out of quota — this issue's exact repro. Nothing local can predict throttling, so rather than a quota probe or a circuit breaker, a refusal is now reported legibly: ProviderFailureClassifier names the condition from the exception chain (429 / 5xx), ThreadExecution renders it in the viewer's language naming the model that actually served, and a substituted round adds one sentence naming both models. The raw transport dump — status line, body, and the full HTTP header block this issue quotes — moves to the log, alongside a new PROVIDER_REFUSED warning.

On the issue's last bullet ("consider treating a persistent 429/5xx on the default as Missing for fallback selection"): deliberately not implemented. That needs cross-round state and would amount to a circuit breaker over the catalogue — a workaround for missing quota rather than a fix. The round now fails legibly and says which model refused, which is the actionable signal without one.

Part 1 remains open and is a maintainer/Azure action: memex.systemorph.com still has no model deployment with quota sized for background agent rounds. No code change can substitute for it — #961 only makes its absence legible instead of a header dump.

Merge pull request #961 from Systemorph/fix/476-legible-provider-failure

Measured the operational half — the premise here needs correcting

This issue says "no language model on memex.systemorph.com has usable quota". Measured from inside the portal pod (memex-portal-deployment-…, ns memex, container memex-portal, using its own AzureFoundry__Endpoint / AzureFoundry__ApiKey):

DeepSeek-V4-Flash  -> HTTP 200
DeepSeek-V4-Pro    -> HTTP 200

Both configured models serve right now. So the portal is not pointed at a dead or key-less deployment, and this is not the #383-style "deployment listed but deprecated" trap either — those return 410, and these return 200.

What the 429 actually was

Azure state on s-meshweaver (RG rg-meshweaverai, sub Systemorph, swedencentral):

deployment	sku	capacity
`DeepSeek-V4-Pro`	GlobalStandard	1000
`DeepSeek-V4-Flash`	GlobalStandard	1000

Quota usage:

AIServices.GlobalStandard.DeepSeek-V4-Pro     used=1000  limit=1000
AIServices.GlobalStandard.DeepSeek-V4-Flash   used=1000  limit=1000

Both deployments are scaled to the entire granted quota — the 2026-07-18 fix (20 → 1000) held and is durable Azure state. So used == limit here means fully allocated to the deployment, not requests at the ceiling.

Therefore the 429s were burst rate-limiting at 1000 RPM, not an absent or unscaled deployment. There is no headroom left for me to allocate: capacity cannot be raised above the granted quota.

Options, and what is ruled out

Raise the Azure quota for DeepSeek-V4-Pro / DeepSeek-V4-Flash above 1000 RPM. This is a quota request against the subscription — not something that can be done from config or from this repo.
Spread background rounds onto other capacity. The account has unused healthy deployments (claude-opus-4-6 cap 500, gpt-5.4 cap 500, claude-sonnet-4-5 / claude-haiku-4-5 cap 100, …). They are not reachable on the configured endpoint: on …/services.ai.azure.com/models the claude models return 404 and the GPT models 400 — claude needs the /anthropic/ sibling route and the GPTs the Azure OpenAI route, and Features__Ai__Providers__Anthropic / AzureOpenAI are both false today.
Decision: not doing this for now (maintainer, 2026-08-09 — "no anthropic for now").
Reduce demand — fewer/cheaper background rounds, or a lower tier for ModelTier__Light/Utility (both currently DeepSeek-V4-Flash).

Note for whoever changes the config

The portal's model wiring lives in ConfigMap memex-portal-config (ns memex): AzureFoundry__Models__0/1 and ModelTier__{Heavy,Standard,Light,Utility}. Live patches there survive image self-updates but revert on the next helm upgrade — the durable copy is in the per-env values files, which are gitignored and not in this checkout.

The code side of this issue (a substitution that reported an HTTP header dump instead of a legible failure) shipped in #961 and is unaffected by any of the above.

docs: the classifier reads TWO banners, not one — the page asserted t…

Navigation Menu

Repository navigation

Agent rounds have no model with quota; the stale-model fallback is silent for non-interactive rounds and never health-checks its target #476

Summary

1 · Model quota (ops)

2 · The silent fallback (code)

Suggested fix

Reproduction

RCA (verified against source) — all three claims confirmed; one is now worse

What happened (2026-07-22, memex.systemorph.com)

Two distinct asks

Meanwhile

Measured the operational half — the premise here needs correcting

What the 429 actually was

Options, and what is ruled out

Note for whoever changes the config

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Agent rounds have no model with quota; the stale-model fallback is silent for non-interactive rounds and never health-checks its target #476

Description

Summary

1 · Model quota (ops)

2 · The silent fallback (code)

Suggested fix

Reproduction

Activity

rbuergi commented on Jul 18, 2026

RCA (verified against source) — all three claims confirmed; one is now worse

rbuergi commented on Jul 18, 2026

rosalie1984 commented on Jul 22, 2026

What happened (2026-07-22, memex.systemorph.com)

Two distinct asks

Meanwhile

rbuergi commented on Aug 8, 2026

rbuergi commented on Aug 9, 2026

Measured the operational half — the premise here needs correcting

What the 429 actually was

Options, and what is ruled out

Note for whoever changes the config

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions