Repository navigation
Model catalog metadata is stale, and clients resolve it instead of Runtime Host #4398
Description
What happened
A user on a Z.AI plan selected GLM-5.3 and got no thinking-effort choices. The upstream catalog has carried the right facts since 2026-08-14; three separate mechanisms each failed to deliver them.
How to reproduce
- Add a Z.AI (
zai) connection with an API key, so protocol model discovery runs. - Pick
glm-5.3in the model picker. - No thinking-effort choices appear.
The committed snapshot has no glm-5.3 under zai at all, so lookupModelMetadata returns an empty record. The zai-coding-plan path does carry it with efforts: ["low","high","max"], so a plan connection on a current build is unaffected — an older installed build is not.
Environment
- Maka version or commit:
a8597ec460(currentmain) - OS and version: macOS 26.6.2; the defect is platform-independent
- Surface: Desktop, TUI, CLI, and Runtime Host
- Node.js version: v26.7.0 (running from source)
Root causes
1. The committed snapshot is stale, and nothing detects that. models.dev has listed glm-5.3 with reasoning_options: [{type: "effort", values: ["low","high","max"]}] since 2026-08-14; our snapshot was generated 2026-08-29 and still lacks it under zai. A full comparison against upstream shows 62 models present upstream and absent from the snapshot, plus 7 effort-set mismatches including tencent hy3, vercel openai/gpt-5-codex, and grok-4.3/4.5/4.6. No job refreshes the snapshot on a schedule. check:model-metadata only verifies that generated output matches the snapshot, never that the snapshot matches upstream, so it stays green while drift accumulates.
2. Clients resolve catalog metadata from their own compiled copy. The Desktop renderer (9 files) and TUI/CLI (4 files) import PROVIDER_DEFAULTS, model-catalog, and model-thinking and compute model capabilities locally. Per docs/runtime-host-remote-access.md, Desktop, TUI, and CLI all connect to a Runtime Host that is installed independently via npx maka-agent@latest and may run on another machine, so up to four independently versioned registries can disagree about the same connection. buildConnectionModelCatalogEntries already accommodates this by returning an empty list for a providerType the client does not recognize — a client deciding on its own that a connection has no models.
3. Nothing refreshes at runtime. Even with a current snapshot, an installed client cannot learn about a newly listed model until it is upgraded.
Proposed plan
Three changes, each independently mergeable and verifiable.
Runtime Host becomes the sole model-catalog authority. Host resolves catalog metadata and projects it; Desktop, TUI, and CLI consume that projection instead of their own tables. This covers writes as well as reads — the add-connection form validates against the registry too. It requires a RUNTIME_HOST_COMPATIBILITY_EPOCH bump. The pricing chain in packages/runtime is already Host-only and is the shape to follow.
Runtime refresh of the models.dev catalog. Host fetches once at startup and keeps the committed snapshot on timeout or failure. Refreshed data reaches clients over the existing HostChangeFeed as a fifth frame alongside configuration, project catalog, session catalog, and scheduled task; clients re-query on receipt. Deliberately excluded: TTL, on-disk cache, conditional GET, and an opt-out toggle. At one fetch per process none of them earns its maintenance cost, and the snapshot already covers the offline case.
Scheduled snapshot upkeep. A nightly or weekly job runs refresh:model-metadata and opens a pull request for human review. It must also add a snapshot-versus-upstream drift check, since the absence of one is why the first cause went unnoticed. Sanity floors on the refresh — a minimum model count and a required-provider list — following the approach in pi's publish-model-catalog workflow.
Architecture gap to close
ARCHITECTURE.md and AGENTS.md both scope Runtime Host's sole authority to execution: Session, Turn, continuation, terminal state, lifecycle. Model and provider catalog is unclaimed, which is how a second authority grew inside the clients without violating any written rule. The first change should extend that statement so clients are thin projections of Host-owned state, catalog included.
Omissions to cover
- Write paths, not only reads:
provider-add-submission.tsvalidates new connections against the local registry today. ModelCatalogEntrycarries nothinkingOptionsfield. Thinking levels have never been part of the projection, so adding them is part of the first change rather than a consequence of it.- A connection whose
providerTypethe running build does not know is not dropped silently, and the branch this bullet assumed is unreachable.decodeProviderTypeinpackages/core/src/runtime-policy/connection-catalog-codec.tsthrows during persisted decode, sodecodeCanonicalConnectionCatalogEntryfails the whole connection catalog read withinvalid_document: the Host loses every connection, not the one row.buildConnectionModelCatalogEntriesreturning[]for an unregistered provider never runs, because the strict codec rejects the row first. The seam that fixes this is raw-level survival at the storage read boundary, mirroring the existingisRetiredGeminiCliConnectionfilter inpackages/storage/src/runtime-policy/connection-catalog-document.ts. It has no coupling to the models.dev runtime refresh and is not part of that change. MODEL_METADATA_FALLBACKSentries that a refresh retires get deleted in the same change that introduces the refresh, not as a follow-up.
Simplifications found while auditing
makeEntry,makeMissingDefaultEntry, andmakeMissingUserChoiceEntryshare one 12-field projection table, and the latter two are identical in 46 of 48 lines. AddingthinkingOptionstoday means editing three copies, so this collapses first.PROVIDER_DEFAULTSandPROVIDER_REGISTRYare two exported names for one object across two modules, at 92 and 7 references respectively.ModelCatalogEntryandThinkingLevel[]are parallel projections of the samelookupModelMetadataresult. They are separate only becausemodel-catalog.tspredatesmodel-thinking.tsby six weeks and thinking level arrived as a session control rather than a catalog fact. Rendering one model picker requires calling both. Merging them before the protocol work avoids freezing the split into the wire format.pi-tui-runner.tshand-rolls the rule inthinkingVariantsForConnection— relay profile overrides catalog — in four places.
Provenance
Investigation and drafting were done with Claude Code. The upstream comparison, file references, and line counts were produced by reading current main and querying models.dev; a committer should verify them before work starts.
Activity
Astro-Han commentedon Aug 31, 2026
take
refactor(runtime-host): make the Host the authority for the model cat…
fix(core): accept the modalities models.dev now publishes
refactor(core): move the models.dev metadata projection into core
feat(core): let a refresh replace the models.dev metadata layer
feat(runtime-host): refresh the models.dev catalog at Host startup
refactor(core): drop static model facts models.dev now states itself
ci: schedule model metadata upkeep and check the snapshot against ups…
refactor(core): move the models.dev metadata projection into core
feat(core): let a refresh replace the models.dev metadata layer
feat(runtime-host): refresh the models.dev catalog at Host startup
refactor(core): drop static model facts models.dev now states itself
feat(runtime-host): refresh the models.dev catalog at Host startup (#…
Astro-Han commentedon Sep 1, 2026
Closing — all three proposed changes are on
main:connection.catalog.changed.Three of the four simplifications landed with them, verified against
main:makeMissingDefaultEntry/makeMissingUserChoiceEntryare gone; only comments still carry the names.PROVIDER_DEFAULTSis gone;provider-registry.ts:1626exportsPROVIDER_REGISTRYas the one name.ModelCatalogEntry.thinkingLevels(model-catalog.ts:81) merged the two projections, and its comment records why they had drifted.One item stays open, exactly as this issue predicted it would. The "Omissions to cover" section noted that raw-level survival at the storage read boundary "has no coupling to the models.dev runtime refresh and is not part of that change", and it isn't. On
maintoday,connection-catalog-document.ts:130maps every maintained row throughdecodeCanonicalConnectionCatalogEntry, which reachesdecodeProviderType(connection-catalog-codec.ts:155); that throws for a provider the running build does not know, so one unregistered row fails the whole catalog read withinvalid_documentand the Host loses every connection rather than that one. The seam is theisRetiredGeminiCliConnectionfilter directly above it at:123, which already survives a row before the strict decode runs.Rather than hold this issue open as a container for one unrelated item, I'm tracking that separately.
refactor(runtime-host): make the Host the authority for the model cat…
fix(model-metadata): follow the models.dev provider split and body-on…
fix(model-metadata): follow the models.dev provider split and body-on…