feat(proxy): /v1/messages now serves any upstream — Anthropic protocol → OpenAI/Gemini/DeepSeek bridge - #100
Merged
Conversation
…l → OpenAI/Gemini/DeepSeek bridge
Closes the symmetry gap: previously /v1/chat/completions accepted any
upstream (the OpenAI bridge double-acts as an internal Hub layer that
dispatches to Anthropic/Gemini/DeepSeek bridges), but /v1/messages
422'd anything that wasn't an Anthropic upstream. Now both endpoints
support all four providers; clients pick the protocol that fits their
SDK.
Implementation pattern lifted from LiteLLM's `experimental_pass_through`
adapter (`litellm/llms/anthropic/experimental_pass_through/adapters/
{transformation.py, streaming_iterator.py}`), trimmed to the MVP fields
aisix supports today (text content blocks). Tool_use / image /
thinking blocks land in a follow-up.
New surface (`aisix-provider-anthropic`)
- `parse_inbound_request(body) → ChatFormat` — Anthropic body parser
(folds `system` field into a leading system message, concatenates
text content blocks, surfaces unrecognized keys via `extra`)
- `chat_response_into_anthropic_json(resp, alias) → Value` — render
internal ChatResponse as Anthropic non-streaming JSON
- `AnthropicSseEncoder` + `AnthropicSseEvent` — state machine that
re-encodes a `ChatChunk` stream as Anthropic SSE events:
`message_start` / `content_block_start` / `content_block_delta` /
`content_block_stop` / `message_delta` / `message_stop`
- `AnthropicInboundError` for the 400-class translation errors
Handler (`aisix-proxy::messages`)
- Forks on `model.provider`:
- Anthropic upstream: existing byte-for-byte passthrough (preserves
cache_control, thinking blocks, tool_use that the gateway-internal
ChatFormat can't lossily round-trip)
- else: cross_provider_dispatch → parse → Hub.get(provider) →
bridge.chat / bridge.chat_stream → render Anthropic JSON / SSE
- Streaming uses async-stream to pump bridge chunks through the SSE
encoder; upstream errors surface as `event: error` SSE frames so
Anthropic SDKs can raise rather than silently truncating
- The response `model` field echoes the operator alias (`my-claude-
alias`) rather than leaking the upstream id (`gpt-4o`) so callers
see a stable identifier across upstream swaps
Tests
- 17 new unit tests in `wire.rs` covering the parser (system shapes,
unknown role, missing model, content block array vs string,
unrecognized top-level keys), the response encoder (shape + every
finish_reason mapping), and the SSE state machine (first chunk
bootstrapping, mid-stream deltas, finish trio, force-close, finish-
without-content)
- 2 new integration tests in `messages.rs` covering both directions
through a wiremock OpenAI upstream:
- non-streaming: Anthropic body in → Anthropic JSON out, asserts
every wire field (id/type/role/model/content/stop_reason/usage)
- streaming: SSE response sequence asserts message_start →
content_block_* → message_delta → message_stop in order with
correct text fragments
- Replaces the obsolete `non_anthropic_model_returns_400` pin
Test counts
- aisix-provider-anthropic: 33 → 36 passing
- aisix-proxy: 105 → 107 passing
- workspace clippy + fmt clean
Dependency change
- `aisix-provider-anthropic` moved from [dev-dependencies] to
[dependencies] in `aisix-proxy/Cargo.toml`. The proxy is the only
consumer of the new public Anthropic translation surface; other
providers stay behind the Bridge trait.
ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Free Run ID: 📒 Files selected for processing (2)
📝 WalkthroughWalkthroughGateway POST /v1/messages now supports Anthropic passthrough and cross-provider translation: Anthropic-shaped requests parse into internal ChatFormat, are dispatched to the resolved Bridge, and responses are re-encoded to Anthropic JSON or Anthropic SSE when upstreams are non-Anthropic. ChangesCross-Provider Message Routing
Sequence Diagram(s)sequenceDiagram
actor Client
participant Gateway
participant AnthropicTranslator
participant Hub
participant Bridge
participant UpstreamAPI
Client->>Gateway: POST /v1/messages (Anthropic JSON)
Gateway->>AnthropicTranslator: parse_inbound_request()
AnthropicTranslator-->>Gateway: ChatFormat
Gateway->>Hub: resolve_bridge(model)
Hub-->>Gateway: Bridge
alt Non-Streaming
Gateway->>Bridge: chat(ChatFormat)
Bridge->>UpstreamAPI: upstream request
UpstreamAPI-->>Bridge: ChatResponse
Bridge-->>Gateway: ChatResponse
Gateway->>AnthropicTranslator: chat_response_into_anthropic_json()
AnthropicTranslator-->>Gateway: Anthropic JSON
Gateway-->>Client: Anthropic JSON response
else Streaming
Gateway->>Bridge: chat_stream(ChatFormat)
loop Each upstream chunk
Bridge-->>Gateway: ChatChunk
Gateway->>AnthropicTranslator: AnthropicSseEncoder::next_events()
AnthropicTranslator-->>Gateway: AnthropicSseEvent[]
Gateway-->>Client: SSE frames (Anthropic)
end
end
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Note 🎁 Summarized by CodeRabbit FreeYour organization is on the Free plan. CodeRabbit will generate a high-level summary and a walkthrough for each pull request. For a comprehensive line-by-line review, please upgrade your subscription to CodeRabbit Pro by visiting https://app.coderabbit.ai/login. Comment |
The earlier review noted that per-bridge wiremock tests prove each Bridge translates ChatFormat ↔ its wire shape, and the proxy lib tests prove /v1/chat/completions end-to-end against an OpenAi upstream — but the *integration* of an OpenAI-protocol inbound request hitting an Anthropic / Gemini / DeepSeek upstream had zero coverage. Same gap, mirrored, on the /v1/messages side. These tests fill the matrix. | Inbound | Upstream | Non-streaming | Streaming | |----------|-----------|---------------|-----------| | OpenAI | OpenAI | existing | existing | | OpenAI | Anthropic | NEW | NEW | | OpenAI | Gemini | NEW | (covered)*| | OpenAI | DeepSeek | NEW | (covered)*| | Anthropic| OpenAI | from f3140ab | from f3140ab | | Anthropic| Anthropic | existing | NEW | | Anthropic| Gemini | NEW | NEW | | Anthropic| DeepSeek | NEW | NEW | * Gemini and DeepSeek share the OpenAi-compat wire shape; their streaming behaviour is identical to OpenAi-on-OpenAi which is already covered. The non-streaming variants are added separately to pin that `Hub.get(Provider::Gemini|Deepseek)` resolves to the right Bridge instance (different metrics labels, default base URL defaults). Test counts - aisix-proxy/src/lib.rs : +4 tests (matrix_openai_in_*) - aisix-proxy/src/messages.rs : +5 tests (matrix_anthropic_in_*) - aisix-proxy lib total : 105 → 116 - workspace fmt + clippy + test : green The most valuable cell is `matrix_openai_in_anthropic_upstream_*` — that's the path where wire shapes genuinely differ in both directions. The streaming variant pins the Anthropic-typed-event → OpenAi-flat-delta translation inside `AnthropicBridge::chat_stream`, which until now was only smoke-tested at the bridge level (typed events in / typed chunks out) but never end-to-end as an SSE byte stream re-emitted in OpenAi shape.
There was a problem hiding this comment.
Pull request overview
Note
Copilot was unable to run its full agentic suite in this review.
Extends the proxy’s Anthropic /v1/messages endpoint to support forwarding to non-Anthropic upstreams (OpenAI/Gemini/DeepSeek) by translating Anthropic-shaped requests into internal ChatFormat and re-encoding responses back into Anthropic JSON/SSE, making /v1/messages symmetric with /v1/chat/completions on the inbound axis.
Changes:
- Add cross-provider dispatch path in
/v1/messages: parse Anthropic JSON →ChatFormat→Bridge→ render Anthropic JSON/SSE. - Introduce
aisix-provider-anthropic“wire” translation helpers (parser, response renderer, SSE encoder) as public surface. - Expand docs and add integration/unit tests covering cross-protocol and cross-upstream matrix (streaming + non-streaming).
Reviewed changes
Copilot reviewed 7 out of 8 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| docs/api-proxy.md | Documents the two /v1/messages paths (passthrough vs translation) and the current text-only limitation. |
| crates/aisix-proxy/src/messages.rs | Implements cross-provider dispatch for /v1/messages plus SSE re-encoding and new integration tests. |
| crates/aisix-proxy/src/lib.rs | Adds integration tests covering cross-protocol × upstream scenarios for /v1/chat/completions. |
| crates/aisix-proxy/Cargo.toml | Promotes aisix-provider-anthropic to a runtime dependency so the proxy can use wire helpers. |
| crates/aisix-provider-anthropic/src/wire.rs | Adds inbound Anthropic parser, outbound Anthropic JSON renderer, and SSE encoder + unit tests. |
| crates/aisix-provider-anthropic/src/lib.rs | Re-exports the new wire translation API for proxy consumption. |
| README.md | Updates the README to reflect /v1/messages working against any configured upstream. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+373
to
+377
| let frame = format!( | ||
| "event: error\ndata: {{\"type\":\"error\",\"error\":{{\"type\":\"{}\",\"message\":{}}}}}\n\n", | ||
| e.error_type(), | ||
| serde_json::to_string(&e.to_string()).unwrap_or_else(|_| "\"error\"".into()), | ||
| ); |
| chat.top_p = Some(t as f32); | ||
| } | ||
| if let Some(t) = obj.get("max_tokens").and_then(Value::as_u64) { | ||
| chat.max_tokens = Some(t as u32); |
Comment on lines
+435
to
+444
| Some(Value::Array(blocks)) => { | ||
| let mut parts = Vec::new(); | ||
| for block in blocks { | ||
| if let Some(text) = block.get("text").and_then(Value::as_str) { | ||
| parts.push(text); | ||
| } | ||
| } | ||
| parts.join("") | ||
| } | ||
| _ => return Err(AnthropicInboundError::UnsupportedContent { idx }), |
moonming
added a commit
that referenced
this pull request
May 7, 2026
PR #100 (cross-provider /v1/messages — Anthropic protocol over non-Anthropic upstreams) landed on main with the pre-Phase-B Model API: model.provider() (method call), gemini_model(name, api_base) helpers, etc. After rebasing Phase B on top of #100, the Anthropic matrix tests + cross_provider_dispatch all stop compiling. This commit ports the survivors: - cross_provider_dispatch: switched to model.provider field access, picks up provider_key via dispatch::resolve_provider_key, threads it through BridgeContext::new(req_id, model, pk). - gemini_model / deepseek_model / anthropic_model_entry test helpers drop their api_base parameter — Phase B moves api_base onto ProviderKey, and the matrix harness now builds a fresh PK with the wiremock URI on every test. - Three test sites that still passed an extra api_base argument updated to the single-arg helper signature.
moonming
added a commit
that referenced
this pull request
May 7, 2026
…102) * feat(model): split provider_config inline into ProviderKey reference Realigns the standalone Model schema with the AISIX-Cloud control plane's normalised shape — the projection cp-api has been waiting for since PRD-09b §6 (the comment in mustMarshalModelKV calls this out as "Phase 2 swaps to {model, provider_key_id} with DP-side join, but requires Model.provider_config refactor across 26 DP files which is a separate PR" — that's this PR). Old shape (pre-#95 + this PR): { name, model: "<provider>/<id>", provider_config: { api_key, api_base } } New shape: { display_name, provider, model_name, provider_key_id } Where provider_key_id references a ProviderKey row (introduced as a top-level resource in #95) carrying secret + api_base. Routing models keep the same `routing` block but drop the upstream-config triple — the router resolves a target Model and dispatches against THAT model's provider_key_id. Why - One ProviderKey, many Models. Rotating the upstream secret used to require rewriting every Model row that embedded it; now it's a single PUT against the ProviderKey. - AISIX-Cloud parity. cp-api already has a `ProviderKey` table; managed-mode DPs need this shape to consume what cp-api projects into kine. - Snapshot-table integrity. The DP can validate at load time that every Model.provider_key_id resolves to a ProviderKey in the same snapshot, instead of carrying inline secrets it can't cross-check. Changes by area aisix-core - Model: replaced { name, model, provider_config } with { display_name, provider: Option<Provider>, model_name: Option<String>, provider_key_id: Option<String> }. Routing models set `routing` and leave the upstream triple as None. - Removed ProviderConfig struct entirely. - JSON Schema: oneOf encodes the direct-vs-routing XOR (direct ⇒ all three of provider/model_name/provider_key_id required; routing ⇒ all three forbidden). - Resource::name() now returns &display_name; ApiKey.allowed_models matches against the same field (already did, just renamed). aisix-gateway - BridgeContext gains `provider_key: Arc<ProviderKey>`. Constructor signature is now `new(request_id, model, provider_key)`. aisix-provider-{openai,anthropic,gemini,deepseek} - Bridge helpers (resolve_base / api_key / upstream_model) take `&BridgeContext` and read from ctx.provider_key + ctx.model rather than the now-gone provider_config. aisix-proxy - New `dispatch.rs` resolves both Model and ProviderKey from the snapshot before each per-endpoint handler builds BridgeContext. - Every endpoint (chat / completions / embeddings / messages / responses / rerank / images / audio / passthrough) updated to use the new resolver — no more inline `model.provider_config.api_key`. - 422 with a clear error envelope when a Model references a provider_key_id that isn't in the snapshot. Tests + fixtures - Every fixture across the workspace updated to the new JSON shape (~30 files: aisix-admin, aisix-cache, aisix-ratelimit, aisix-proxy, aisix-gateway, aisix-server, aisix-guardrails, aisix-etcd). Verified - `cargo fmt --all --check` clean - `cargo clippy --workspace --tests -- -D warnings` clean - `cargo test --workspace` green (520+ tests, 0 failures) Cross-repo follow-up - AISIX-Cloud's `mustMarshalModelKV` (internal/cpapi/resources/handlers.go) needs to switch from writing the inline `provider_config` shape to the new `{display_name, provider, model_name, provider_key_id}` shape. That's tracked separately and lands in AISIX-Cloud. * test: migrate Phase B fixtures — etcd_integration + e2e smoke The Phase B Model restructure commit landed the lib changes but the test fixtures in crates/aisix-admin/tests/etcd_integration.rs and tests/e2e/src/cases/smoke.test.ts still posted the old {name, model:"openai/...", provider_config:{...}} shape. Both surfaces fail in CI with the schema's "Additional properties are not allowed" rejection. - etcd_integration.rs: models_round_trip_through_real_etcd and loader_picks_up_every_admin_write switched to {display_name, provider, model_name, provider_key_id} - smoke.test.ts: now posts a ProviderKey first, then references its id from the Model — matches the production flow the dashboard drives. Adds AdminClient.createProviderKey for the test harness. * ci: kick the CI again — webhook missed 4d35529 * ci: trigger re-run for 4d35529 (webhook missed) * fix(messages): port PR #100 cross-provider /v1/messages to Phase B Model PR #100 (cross-provider /v1/messages — Anthropic protocol over non-Anthropic upstreams) landed on main with the pre-Phase-B Model API: model.provider() (method call), gemini_model(name, api_base) helpers, etc. After rebasing Phase B on top of #100, the Anthropic matrix tests + cross_provider_dispatch all stop compiling. This commit ports the survivors: - cross_provider_dispatch: switched to model.provider field access, picks up provider_key via dispatch::resolve_provider_key, threads it through BridgeContext::new(req_id, model, pk). - gemini_model / deepseek_model / anthropic_model_entry test helpers drop their api_base parameter — Phase B moves api_base onto ProviderKey, and the matrix harness now builds a fresh PK with the wiremock URI on every test. - Three test sites that still passed an extra api_base argument updated to the single-arg helper signature. * fix(supervisor): incremental watch must mirror every resource kind The supervisor's `apply_put`, `apply_delete`, and `clone_snapshot` helpers only handled `models` + `api_keys` — Phase B's ProviderKey and #97's Guardrail / CachePolicy / ObservabilityExporter were silently no-ops. Admin writes for those four resources landed in etcd fine, but the watch event got dropped and the proxy snapshot never updated, so dispatch saw a Model whose `provider_key_id` pointed at thin air. Smoke test #102 hit this: chat returned 500: bridge is misconfigured: model references unknown provider_key_id Fix is mechanical: extend the for-loops in apply_put + clone_snapshot and the match arms in apply_delete to cover every ResourceTable. Add `apply_put_propagates_every_resource_kind` + the matching delete test as forcing functions — any future resource type added to AisixSnapshot fails this test until the supervisor is updated. Verified - cargo fmt --all --check clean - cargo clippy --workspace --tests -- -D warnings clean - cargo test --workspace — 548 passed, 0 failed (was 546 + 2 new) * test(e2e): poll for snapshot readiness instead of fixed 500ms sleep The smoke test's `chat completion forwards to mock upstream` case intermittently fails on CI with `unknown provider_key_id` even though `a Model + ApiKey written via Admin API are visible to /v1/models` passes immediately before. The fixed-time `waitConfigPropagation()` times out in 500ms; on slower CI runners only the Model row makes it into the snapshot inside that window, while the ProviderKey row the Model references arrives a beat later — long enough for the chat call to look up `provider_key_id` and miss. waitConfigPropagation now accepts an optional `condition` callback that polls a positive readiness probe on a 50ms cadence with a 5s deadline. The smoke test uses two such probes: - After the Admin writes, poll /v1/models for the Model id (covers the Model row's propagation as before). - Before the chat assertion, poll the chat path itself, retrying as long as the response carries the `unknown provider_key_id` config error. That's the only signal that captures the *complete* snapshot state (Model + ProviderKey + ApiKey), since the proxy doesn't expose ProviderKey directly. The upstream-was-hit assertion still passes because both probe and the real call land on `/v1/chat/completions`. Local repro stays green; CI now has 5s of headroom for the second- event race instead of the old 0ms past the fixed sleep.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes the symmetry gap on the proxy's protocol-conversion surface.
Before:
/v1/chat/completionsaccepted any upstream (OpenAI / Anthropic / Gemini / DeepSeek), but/v1/messagesrejected anything that wasn't an Anthropic upstream with 422.After: both endpoints support all four upstreams. Clients pick the protocol that fits their SDK; the gateway translates.
Implementation pattern
Lifted from LiteLLM's experimental_pass_through adapter:
transformation.py→ request parser + non-streaming response rendererstreaming_iterator.py→ SSE state-machine pattern (message_start → content_block_* → message_delta → message_stop)Trimmed to text content blocks (tool_use / image / thinking blocks land in a follow-up — current behavior skips them silently on parse).
New surface (
aisix-provider-anthropic)parse_inbound_request(body) → ChatFormat— foldssysteminto a leading system message, concatenates text content blocks, surfaces unrecognized keys viaextrachat_response_into_anthropic_json(resp, alias) → Value— non-streaming response rendererAnthropicSseEncoder+AnthropicSseEvent— state machine for the streaming SSE event sequenceAnthropicInboundErrorfor 400-class translation errorsHandler change (
aisix-proxy::messages)Branches on
model.provider:ChatFormatcan't round-trip)cross_provider_dispatchparses →Hub.get(provider)→bridge.chatorbridge.chat_stream→ re-encode to Anthropic JSON / SSEThe response
modelfield echoes the operator alias (my-claude-alias) rather than the upstream id (gpt-4o), so callers see a stable identifier across upstream swaps.Streaming uses
async-streamto pump bridge chunks through the SSE encoder. Upstream errors surface asevent: errorSSE frames so Anthropic SDKs raise rather than silently truncating.Tests
wire.rs: parser (system shapes / unknown role / missing model / content blocks string vs array / extra keys), response encoder (shape + every finish_reason mapping), SSE encoder (first-chunk bootstrap / mid-stream deltas / finish trio / force-close / finish-without-content)messages.rs: non-streaming + streaming, both with a wiremock OpenAI upstream and full Anthropic-shape assertions on the response(Replaces the obsolete
non_anthropic_model_returns_400pin.)Dependency change
aisix-provider-anthropicmoves from[dev-dependencies]to[dependencies]inaisix-proxy/Cargo.toml. Proxy is the only consumer of the new public translation surface; other providers stay behind the Bridge trait.Docs
/v1/messagesreflects "any upstream"docs/api-proxy.md§4.5 expanded with the two-path explanationcrates/aisix-proxy/src/messages.rsfile-header comment rewrittenTest plan
cargo fmt --all --checkcleancargo clippy --workspace --tests -- -D warningscleancargo test --workspacegreen (full suite)Summary by CodeRabbit
New Features
Documentation
Tests