Repository navigation
feat(scripts): sync the full models.dev fact set (pricing, description, knowledge, input limits) #2329
Copy link
Copy link
Closed
Description
Problem
The models.dev pipeline (scripts/sync-model-metadata.mjs) downloads https://models.dev/api.json (~3.5 MB, 6149 models) on every sync, but only maps a fraction of the data. Several high-value facts are dropped:
- Pricing:
packages/runtime/src/telemetry/builtin-pricing.tsis a hand-maintained 17-entry snapshot ("as of 2026-05-20"); models.dev carriescoston 93% of models (5739/6149). Spot check: localclaude-sonnet-4-5(3 / 15 / 0.3 / 3.75) matches models.dev exactly — the snapshot was transcribed by hand. No provider exposes a machine-readable pricing API, so models.dev is the correct source, and it is already a dependency. - Description: present on 100% of models, never synced — the model picker shows only a display name, so users cannot tell same-family models apart.
- Knowledge cutoff: present on 53% (3271) of models, never synced — users see no "knowledge as of" anywhere.
- Input limit:
limit.inputexists on 19% (1195) of models (e.g. gpt-5.2: context 400K but input 272K). Onlycontextandoutputare synced today; the context budget (context-budget-policy.ts:410-424) budgets againstcontextWindow, which can over-admit input the model cannot actually take. - PDF attachment capability:
modalities.inputvalues includepdf(e.g. claude-sonnet-4-5), but the sync filter keeps onlytext/image/audio— the pdf fact is dropped (relevant to attachment preflight). - Structured output: present on 62% (3829) of models, never synced — distinct from
tool_call(function calling); a model can support tools without structured JSON output. - Freshness:
last_updatedpresent on 100% of models, never synced — generated facts carry no age marker, so stale entries rot silently (the exact failure mode of the 17-entry price snapshot). - Status nuance: models.dev
statushasdeprecated(170),beta(53),alpha(1); the sync maps onlydeprecated → deprecated, everything else →active— beta/alpha models are presented as stable.
Proposed shape (extend the existing seam, no new pipeline)
- Extend
sync-model-metadata.mjsto map the full fact set into generated modules:cost→model-pricing.generated.ts(base rates only; see non-goals)description,knowledge,limit.input,structured_output,last_updated→ extendModelMetadata/model-metadata.generated.tsmodalities.inputkeepspdf(alongside text/image/audio)status: mapbeta/alphadistinctly instead of collapsing intoactive
builtin-pricing.tsbecomes generated + a local supplement layer for entries models.dev cannot carry (e.g.zai-coding-plan:*,MiniMax-cn:*— special tiers models.dev does not price). KeepgetBuiltinPricing/buildPricingLookupsignatures stable (runtime, runtime-host, desktop all consume them).- Extend the sync-contract test pattern (
models-dev-sync-contract.test.tstruthfulness checks) to the new fields.
Non-goals
- Tiered pricing (OpenAI
tiers/context_over_200k): the flatPricingConfigtakes the base rate only. - Special tiers (Coding Plan, Education, Volume Pricing) stay overrides, matching the current comment's intent.
- Low-value fields (family, open_weights, release_date, temperature, interleaved, experimental, provider-level api/env) are intentionally not synced.
Impact
- Removes the silently-stale hand-maintained price snapshot.
- Model picker can show description + knowledge cutoff; budget decisions respect real input limits; attachment preflight gets the pdf fact; beta/alpha models are honestly marked.
- Complements feat: user-overridable model facts (context window etc.) — models.json-style override layer #2330 (the user override layer): generated facts = read-only built-in side, models.json = user side.
Validation
- Contract test asserting generated pricing/description/knowledge/limit entries match api.json.
- A diff test that the current 17 hand entries still resolve through the new lookup.
- Context budget test: a model with
limit.inputbudgets against the input limit, notcontextWindow.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels
Activity
[-]feat(scripts): sync builtin pricing from models.dev instead of a hand-maintained snapshot[/-][+]feat(scripts): sync the full models.dev fact set (pricing, description, knowledge, input limits)[/+]