Skip to content

feat: expose qualified AI model capabilities - #696

Open
alongubkin wants to merge 3 commits into
mainfrom
alon/alien-946-model-capability-metadata
Open

alongubkin wants to merge 3 commits into
mainfrom
alon/alien-946-model-capability-metadata

Conversation

@alongubkin

@alongubkin alongubkin commented Sep 22, 2026 •

Copy link
Copy Markdown
Member

Problem

GET /v1/models and getAvailableModels() say which models are routable through each public Gateway API, but not which optional behaviors work through it (function tools, image input, reasoning controls, server-managed continuation, stateless replay). Callers treat the routing list as a capability claim, and it is not one.

Fix

  • alien-core adds ModelApiCapabilities / ModelCapabilitySupport and CatalogModel::capabilities(): one record per public API, each fact supported, unsupported or unverified. Every API is always present, so a missing record can't be misread as "unsupported".
  • The gateway returns capabilities on every /v1/models entry. Provider-listed models that were never qualified get an all-unverified matrix.
  • Only one tuple is qualified so far, from a live run on 2026-09-22: AWS gpt-oss-20b on the Responses API. Function tools and server-managed continuation are supported, stateless replay is unsupported, and the rest is unverified. Chat Completions and Messages stay unverified for that model.
  • The TS client checks the matrix it receives. If the matrix is present but malformed or incomplete, the call fails. If it is missing, every API is reported as unverified, because a Worker image or BYO container can run a gateway binary older than the app's SDK.

Alternatives considered: inferring capabilities from the provider-native protocol or the model family was rejected, because that is exactly the false claim this PR removes. Rejecting a response with no matrix was rejected too, because it would break getAvailableModels() for every app running an older gateway.

Proof

Run in a Linux sandbox on this branch rebased on main (abdbe6acb):

  • cargo test -p alien-ai-gateway --lib supplied_availability_gates_listing_and_inference_without_probing passes. With the four "capabilities" lines removed from list_models, it fails with capability matrix.
  • vitest run src/__tests__/client.test.ts passes 29/29. With main's client.ts and these tests, 3 fail. With the first commit's strict client, the older-gateway test fails.
  • cargo test -p alien-core --lib ai_catalog passes 13/13, and cargo test -p alien-ai-gateway --lib passes 94/94.
  • pnpm --filter @alienplatform/core generate (schema exporter + kubb) leaves the generated files unchanged, so the committed artifacts match the generator.
  • Biome and tsc --noEmit are clean for packages/ai-gateway.

Rollout

The change is additive on the wire, and older SDKs ignore the new field. No migration is needed.

Linear: ALIEN-946

🤖 Generated with Claude Code

@greptile-apps

greptile-apps Bot commented Sep 22, 2026 •

Copy link
Copy Markdown

RetriggerConfidence Score: 5/5

[Medium impact] The PR appears safe to merge; the previous type-export issue is fixed and no new blocking issues were found.

Summary

The PR adds per-API model capabilities to the catalog, gateway responses, and TypeScript client.

  • Model listings show qualified capabilities for each public API.
  • The TypeScript client returns a capability matrix with every model.

Reviews (3) · Last reviewed commit: "fix: export the model capability types f..." · Reviewed by Greptile

alongubkin and others added 2 commits October 9, 2026 21:55
A Worker image or BYO container can run a gateway binary older than the
app's SDK. Rejecting its model list would break getAvailableModels for
every such app, so a missing matrix now means no qualification evidence.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@alongubkin
alongubkin force-pushed the alon/alien-946-model-capability-metadata branch from 6823a97 to d646d93 Compare October 9, 2026 19:02
@alongubkin

Copy link
Copy Markdown
Member Author

@greptileai review

Comment thread packages/ai-gateway/src/client.ts
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@alongubkin

Copy link
Copy Markdown
Member Author

@greptileai review

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant