feat(provider-anthropic): AnthropicBridge with /v1/messages + typed SSE - #8
Conversation
Second concrete Bridge implementation.
- wire.rs: Anthropic /v1/messages request/response types, split_system()
that pulls leading system messages into a top-level `system` field
(Anthropic has no system role), text-block concatenation on response
decode, max_tokens fallback (Anthropic rejects requests without it),
typed SseEvent enum covering message_start / content_block_delta /
message_delta / message_stop plus a catch-all for the rest, and
StreamState that carries message id/model across chunks.
- bridge.rs: transport mirrors OpenAiBridge in shape but uses x-api-key
+ anthropic-version headers instead of Bearer, posts to
{base}/v1/messages, and runs the stream through StreamState so only
content-bearing or finish-reason-bearing events become ChatChunks.
- stop_reason mapping: end_turn → Stop, max_tokens → Length, tool_use →
ToolCalls, unknowns → Other(str). Tool role in request → Config error
(surfacing upstream incompatibility cleanly rather than silently
dropping).
- ANTHROPIC_VERSION pinned to 2023-06-01 with a builder escape hatch.
19 new unit tests covering wire translation (system split, tool role
reject, max_tokens default, stop_reason mapping, stream event variants,
StreamState), plus wiremock-backed bridge tests: happy path, 400
pass-through, malformed body decode, deadline timeout, missing api_key,
tool role, streaming happy path with typed SSE events, streaming 500
before stream starts, resolve_base trailing-slash.
There was a problem hiding this comment.
Pull request overview
Adds a new provider implementation (aisix-provider-anthropic) that implements the gateway Bridge trait against Anthropic’s Claude Messages API (POST /v1/messages), including translation between the gateway’s OpenAI-shaped ChatFormat and Anthropic’s request/response shapes plus typed SSE streaming support.
Changes:
- Introduces Anthropic wire types and translation utilities (system-message collapsing, stop-reason mapping, typed stream events + stream state).
- Adds
AnthropicBridgewith reqwest transport, Anthropic-specific headers,/v1/messagesendpoint handling, and SSE-to-ChatChunkstreaming. - Updates crate exports and dependencies to support streaming implementation and tests.
Reviewed changes
Copilot reviewed 4 out of 5 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| crates/aisix-provider-anthropic/src/wire.rs | Adds Anthropic request/response models, system splitting, stop-reason mapping, typed SSE event parsing, and stream state/chunk translation. |
| crates/aisix-provider-anthropic/src/bridge.rs | Implements Bridge for Anthropic including non-streaming + streaming transport, SSE decoding, and wiremock-backed tests. |
| crates/aisix-provider-anthropic/src/lib.rs | Updates crate docs and exports AnthropicBridge + constants; declares modules. |
| crates/aisix-provider-anthropic/Cargo.toml | Adds runtime deps needed for streaming (tokio, futures, async-stream, bytes) plus test deps. |
| Cargo.lock | Locks new dependency graph entries for the Anthropic provider crate. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| if s.len() <= n { | ||
| s.to_string() | ||
| } else { | ||
| format!("{}…", &s[..n]) |
There was a problem hiding this comment.
truncate() slices strings by byte index (&s[..n]), which can panic if the upstream body contains multi-byte UTF-8 and n lands mid-codepoint. Prefer truncating on a char boundary (e.g., using char_indices() to find the last valid boundary <= n) to avoid a potential crash while handling error responses.
| format!("{}…", &s[..n]) | |
| let end = s | |
| .char_indices() | |
| .map(|(idx, _)| idx) | |
| .take_while(|&idx| idx <= n) | |
| .last() | |
| .unwrap_or(0); | |
| format!("{}…", &s[..end]) |
| return; | ||
| } | ||
| } | ||
| } |
There was a problem hiding this comment.
build_chunk_stream() never calls decoder.finish() when the upstream closes the connection. If the final SSE event is not terminated by \n\n, it will remain buffered and be dropped, potentially losing the last delta/finish chunk. Consider flushing decoder.finish() after the while loop (similar to OpenAiBridge) and emitting any remaining event.
| } | |
| } | |
| for event in decoder.finish() { | |
| let SseEvent::Data(payload) = event else { continue }; | |
| let parsed: AnthropicStreamEvent = serde_json::from_str(&payload) | |
| .map_err(|e| BridgeError::UpstreamDecode(e.to_string()))?; | |
| state.update(&parsed); | |
| if let Some(c) = state.to_chunk(&parsed) { | |
| yield c; | |
| } | |
| if StreamState::is_terminal(&parsed) { | |
| return; | |
| } | |
| } |
| //! when the client didn't set one, but log the fallback so operators | ||
| //! can tune the default if desired. |
There was a problem hiding this comment.
Module docs say the max_tokens default fallback is logged, but this file currently just applies unwrap_or(DEFAULT_MAX_TOKENS) with no tracing call. Either add the log at the point the default is chosen (so operators can see it), or update the docs to match actual behavior.
| //! when the client didn't set one, but log the fallback so operators | |
| //! can tune the default if desired. | |
| //! when the client didn't set one so requests remain valid. |
| async fn chat_stream( | ||
| &self, | ||
| req: &ChatFormat, | ||
| ctx: &BridgeContext, | ||
| ) -> Result<ChatChunkStream, BridgeError> { | ||
| let model = ctx.model.as_ref(); | ||
| let base = resolve_base(model); | ||
| let key = api_key(model)?; | ||
| let upstream = upstream_model(model)?; | ||
|
|
||
| let (system, messages) = | ||
| split_system(req).map_err(|e| BridgeError::Config(e.to_string()))?; | ||
| let body = build_request(req, upstream, system, messages, true); | ||
| let url = format!("{base}/v1/messages"); | ||
| let client = self.client.clone(); | ||
| let api_version = self.api_version; | ||
| let started = Instant::now(); | ||
| let request_id = ctx.request_id.clone(); | ||
|
|
||
| let resp = with_deadline(ctx.deadline, started, async move { | ||
| client | ||
| .post(&url) | ||
| .header("x-api-key", key) | ||
| .header("anthropic-version", api_version) | ||
| .header(header::CONTENT_TYPE, "application/json") | ||
| .header(header::ACCEPT, "text/event-stream") | ||
| .header("x-aisix-request-id", &request_id) | ||
| .json(&body) | ||
| .send() | ||
| .await | ||
| .map_err(|e| BridgeError::Transport(e.to_string())) | ||
| }) | ||
| .await?; | ||
|
|
||
| let status = resp.status(); | ||
| if !status.is_success() { | ||
| return Err(map_http_error(status, resp).await); | ||
| } | ||
|
|
||
| let byte_stream = resp.bytes_stream(); | ||
| let stream = build_chunk_stream(byte_stream); | ||
| Ok(Box::pin(stream)) | ||
| } |
There was a problem hiding this comment.
BridgeContext::deadline is documented as a deadline for the entire upstream call, but chat_stream() only applies it to the initial .send(); consuming the SSE body can run indefinitely after the deadline has elapsed. Consider enforcing the remaining deadline while reading the byte stream (e.g., track started + deadline in build_chunk_stream and return BridgeError::Timeout once exceeded, or wrap stream.next() with tokio::time::timeout).
…ped reads, redact 5xx message, Vertex content-type guard Five concrete fixes from the Copilot inline review on PR #323. Two stale comments (#3, #4 — already fixed in commit 3) are skipped. **#1+#7 — Azure OpenAI-compatible code preservation.** Azure's envelope omits `error.type` and carries only `error.code`. The bridge previously put the upstream code into `view.kind` and left `view.code` as `None`. For OpenAI-compat tokens Azure inherits unchanged (e.g. `rate_limit_exceeded`), this meant downstream OpenAI clients received `error.type=rate_limit_exceeded` but `error.code=null` — exactly the SDK-retry break issue #322 is about. Fix: - Azure parser populates BOTH `view.kind` AND `view.code` from the upstream `error.code` field. - `render_openai_envelope`'s AzureOpenAI branch now prefers the translation-table-derived code (so explicit Azure tokens like `DeploymentNotFound` → `model_not_found` still win), falling back to `view.code` for OpenAI-compat pass-through. **#2 — Drain the response stream after hitting the cap.** `read_body_capped` previously broke out of the read loop the moment `limit` bytes were buffered. With reqwest/hyper that leaves unread bytes in the response and prevents connection reuse — during a burst of upstream errors the gateway would churn TCP connections instead of recycling the keep-alive pool. Fix: keep iterating the stream, discarding chunks past the cap. Memory stays bounded by `limit`. **#5 — Redact upstream `error.message` on 5xx.** The 5xx branch of `render_bridge_upstream_envelope` was forwarding `BridgeError::UpstreamStatus.message` verbatim — which for OpenAI / Anthropic comes from the parsed upstream `error.message`. Upstream 5xx bodies routinely embed operator-internal detail (engine names, shard ids, queue depth). Fix: on 5xx, emit a canned `"upstream returned {status}"` message; the full upstream body remains in operator logs via tracing. **#6 — Stale "follow-up" comment.** The docstring on `render_bridge_upstream_envelope` claimed cross-wire translation would ship in a follow-up, but it already shipped in commit 2. Rewrite the comment to describe current behaviour (4xx → `error_translate`; 5xx → canned envelope; `Unknown` wire → legacy generic envelope). **#8 — Content-type guard on Vertex (and Azure, while at it).** `capture_upstream_error_http` already gates serde parsing on `Content-Type: application/json` so a 64 KB HTML error page from a fronting WAF doesn't waste CPU on a doomed JSON parse. The Vertex and Azure bridges call serde directly because they need a custom parse path (canned message for redaction) — same guard now applies. Promoted `content_type_is_json` and added a `response_is_json` helper to the gateway's public surface; both bridges call it before `parse_*_error_*`. New tests: - `upstream_openai_5xx_with_json_envelope_collapses_and_redacts_message` pins the 5xx redaction (asserts `engine offline` / `shard 47` / `engine_overloaded` don't reach the customer envelope). - `chat_429_preserves_openai_compatible_code_for_sdk_retry` (Azure) pins that `parsed.code` carries the OpenAI-compat upstream code. - `chat_400_non_json_body_skips_envelope_parse` (Azure) and `chat_gemini_non_json_body_skips_envelope_parse` (Vertex) pin the new content-type guard. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Summary
Second concrete `Bridge` against the gateway trait — Anthropic's
Claude Messages API.
`split_system()` pulls leading system messages into Anthropic's
top-level `system` field, `max_tokens` default fallback covers
clients that omit the (required) field, and a typed `AnthropicStreamEvent`
enum covers `message_start` / `content_block_delta` /
`message_delta` / `message_stop` with a catch-all `Other`.
`StreamState` carries message id/model across chunks so ChatChunk
tagging stays consistent.
`x-api-key` + `anthropic-version` headers and posts to
`{base}/v1/messages`. Stream body pipes through `StreamState` so only
content-bearing or finish-reason-bearing events become ChatChunks.
`tool_use → ToolCalls`, unknowns → `Other`. Tool role → `Config` error.
Test plan