Skip to content

feat(provider-anthropic): AnthropicBridge with /v1/messages + typed SSE - #8

Merged
moonming merged 1 commit into
mainfrom
feat/provider-anthropic
Apr 17, 2026
Merged

feat(provider-anthropic): AnthropicBridge with /v1/messages + typed SSE#8
moonming merged 1 commit into
mainfrom
feat/provider-anthropic

Conversation

@moonming

Copy link
Copy Markdown
Collaborator

Summary

Second concrete `Bridge` against the gateway trait — Anthropic's
Claude Messages API.

  • wire.rs — Anthropic-specific request/response types.
    `split_system()` pulls leading system messages into Anthropic's
    top-level `system` field, `max_tokens` default fallback covers
    clients that omit the (required) field, and a typed `AnthropicStreamEvent`
    enum covers `message_start` / `content_block_delta` /
    `message_delta` / `message_stop` with a catch-all `Other`.
    `StreamState` carries message id/model across chunks so ChatChunk
    tagging stays consistent.
  • bridge.rs — transport mirrors `OpenAiBridge` but sends
    `x-api-key` + `anthropic-version` headers and posts to
    `{base}/v1/messages`. Stream body pipes through `StreamState` so only
    content-bearing or finish-reason-bearing events become ChatChunks.
  • `stop_reason` mapping: `end_turn → Stop`, `max_tokens → Length`,
    `tool_use → ToolCalls`, unknowns → `Other`. Tool role → `Config` error.
  • `ANTHROPIC_VERSION` pinned to `2023-06-01` with builder escape hatch.

Test plan

  • 19 new unit tests (10 wiremock-backed, 9 wire/translation)
    • happy path + 400 + malformed body + deadline + missing api_key
    • tool role → Config error
    • streaming happy path with typed SSE events
    • streaming 500 before stream starts
    • `split_system` merges/interleaves/rejects-tool
    • max_tokens default application
    • stop_reason mapping for all documented variants
  • `cargo test --workspace` — 137 tests pass
  • `cargo clippy --all-targets -- -D warnings` clean
  • `cargo fmt --check` clean
  • CI green across all 6 jobs

Second concrete Bridge implementation.

- wire.rs: Anthropic /v1/messages request/response types, split_system()
  that pulls leading system messages into a top-level `system` field
  (Anthropic has no system role), text-block concatenation on response
  decode, max_tokens fallback (Anthropic rejects requests without it),
  typed SseEvent enum covering message_start / content_block_delta /
  message_delta / message_stop plus a catch-all for the rest, and
  StreamState that carries message id/model across chunks.
- bridge.rs: transport mirrors OpenAiBridge in shape but uses x-api-key
  + anthropic-version headers instead of Bearer, posts to
  {base}/v1/messages, and runs the stream through StreamState so only
  content-bearing or finish-reason-bearing events become ChatChunks.
- stop_reason mapping: end_turn → Stop, max_tokens → Length, tool_use →
  ToolCalls, unknowns → Other(str). Tool role in request → Config error
  (surfacing upstream incompatibility cleanly rather than silently
  dropping).
- ANTHROPIC_VERSION pinned to 2023-06-01 with a builder escape hatch.

19 new unit tests covering wire translation (system split, tool role
reject, max_tokens default, stop_reason mapping, stream event variants,
StreamState), plus wiremock-backed bridge tests: happy path, 400
pass-through, malformed body decode, deadline timeout, missing api_key,
tool role, streaming happy path with typed SSE events, streaming 500
before stream starts, resolve_base trailing-slash.
Copilot AI review requested due to automatic review settings April 17, 2026 06:23

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a new provider implementation (aisix-provider-anthropic) that implements the gateway Bridge trait against Anthropic’s Claude Messages API (POST /v1/messages), including translation between the gateway’s OpenAI-shaped ChatFormat and Anthropic’s request/response shapes plus typed SSE streaming support.

Changes:

  • Introduces Anthropic wire types and translation utilities (system-message collapsing, stop-reason mapping, typed stream events + stream state).
  • Adds AnthropicBridge with reqwest transport, Anthropic-specific headers, /v1/messages endpoint handling, and SSE-to-ChatChunk streaming.
  • Updates crate exports and dependencies to support streaming implementation and tests.

Reviewed changes

Copilot reviewed 4 out of 5 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
crates/aisix-provider-anthropic/src/wire.rs Adds Anthropic request/response models, system splitting, stop-reason mapping, typed SSE event parsing, and stream state/chunk translation.
crates/aisix-provider-anthropic/src/bridge.rs Implements Bridge for Anthropic including non-streaming + streaming transport, SSE decoding, and wiremock-backed tests.
crates/aisix-provider-anthropic/src/lib.rs Updates crate docs and exports AnthropicBridge + constants; declares modules.
crates/aisix-provider-anthropic/Cargo.toml Adds runtime deps needed for streaming (tokio, futures, async-stream, bytes) plus test deps.
Cargo.lock Locks new dependency graph entries for the Anthropic provider crate.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

if s.len() <= n {
s.to_string()
} else {
format!("{}…", &s[..n])

Copilot AI Apr 17, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

truncate() slices strings by byte index (&s[..n]), which can panic if the upstream body contains multi-byte UTF-8 and n lands mid-codepoint. Prefer truncating on a char boundary (e.g., using char_indices() to find the last valid boundary <= n) to avoid a potential crash while handling error responses.

Suggested change
format!("{}…", &s[..n])
let end = s
.char_indices()
.map(|(idx, _)| idx)
.take_while(|&idx| idx <= n)
.last()
.unwrap_or(0);
format!("{}…", &s[..end])

Copilot uses AI. Check for mistakes.
return;
}
}
}

Copilot AI Apr 17, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

build_chunk_stream() never calls decoder.finish() when the upstream closes the connection. If the final SSE event is not terminated by \n\n, it will remain buffered and be dropped, potentially losing the last delta/finish chunk. Consider flushing decoder.finish() after the while loop (similar to OpenAiBridge) and emitting any remaining event.

Suggested change
}
}
for event in decoder.finish() {
let SseEvent::Data(payload) = event else { continue };
let parsed: AnthropicStreamEvent = serde_json::from_str(&payload)
.map_err(|e| BridgeError::UpstreamDecode(e.to_string()))?;
state.update(&parsed);
if let Some(c) = state.to_chunk(&parsed) {
yield c;
}
if StreamState::is_terminal(&parsed) {
return;
}
}

Copilot uses AI. Check for mistakes.
Comment on lines +17 to +18
//! when the client didn't set one, but log the fallback so operators
//! can tune the default if desired.

Copilot AI Apr 17, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Module docs say the max_tokens default fallback is logged, but this file currently just applies unwrap_or(DEFAULT_MAX_TOKENS) with no tracing call. Either add the log at the point the default is chosen (so operators can see it), or update the docs to match actual behavior.

Suggested change
//! when the client didn't set one, but log the fallback so operators
//! can tune the default if desired.
//! when the client didn't set one so requests remain valid.

Copilot uses AI. Check for mistakes.
Comment on lines +194 to +236
async fn chat_stream(
&self,
req: &ChatFormat,
ctx: &BridgeContext,
) -> Result<ChatChunkStream, BridgeError> {
let model = ctx.model.as_ref();
let base = resolve_base(model);
let key = api_key(model)?;
let upstream = upstream_model(model)?;

let (system, messages) =
split_system(req).map_err(|e| BridgeError::Config(e.to_string()))?;
let body = build_request(req, upstream, system, messages, true);
let url = format!("{base}/v1/messages");
let client = self.client.clone();
let api_version = self.api_version;
let started = Instant::now();
let request_id = ctx.request_id.clone();

let resp = with_deadline(ctx.deadline, started, async move {
client
.post(&url)
.header("x-api-key", key)
.header("anthropic-version", api_version)
.header(header::CONTENT_TYPE, "application/json")
.header(header::ACCEPT, "text/event-stream")
.header("x-aisix-request-id", &request_id)
.json(&body)
.send()
.await
.map_err(|e| BridgeError::Transport(e.to_string()))
})
.await?;

let status = resp.status();
if !status.is_success() {
return Err(map_http_error(status, resp).await);
}

let byte_stream = resp.bytes_stream();
let stream = build_chunk_stream(byte_stream);
Ok(Box::pin(stream))
}

Copilot AI Apr 17, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BridgeContext::deadline is documented as a deadline for the entire upstream call, but chat_stream() only applies it to the initial .send(); consuming the SSE body can run indefinitely after the deadline has elapsed. Consider enforcing the remaining deadline while reading the byte stream (e.g., track started + deadline in build_chunk_stream and return BridgeError::Timeout once exceeded, or wrap stream.next() with tokio::time::timeout).

Copilot uses AI. Check for mistakes.
@moonming
moonming merged commit f6f77bf into main Apr 17, 2026
10 checks passed
@moonming
moonming deleted the feat/provider-anthropic branch April 17, 2026 06:28
moonming added a commit that referenced this pull request May 18, 2026
…ped reads, redact 5xx message, Vertex content-type guard

Five concrete fixes from the Copilot inline review on PR #323. Two
stale comments (#3, #4 — already fixed in commit 3) are skipped.

**#1+#7 — Azure OpenAI-compatible code preservation.**
Azure's envelope omits `error.type` and carries only `error.code`.
The bridge previously put the upstream code into `view.kind` and
left `view.code` as `None`. For OpenAI-compat tokens Azure inherits
unchanged (e.g. `rate_limit_exceeded`), this meant downstream OpenAI
clients received `error.type=rate_limit_exceeded` but
`error.code=null` — exactly the SDK-retry break issue #322 is about.

Fix:
- Azure parser populates BOTH `view.kind` AND `view.code` from the
  upstream `error.code` field.
- `render_openai_envelope`'s AzureOpenAI branch now prefers the
  translation-table-derived code (so explicit Azure tokens like
  `DeploymentNotFound` → `model_not_found` still win), falling back
  to `view.code` for OpenAI-compat pass-through.

**#2 — Drain the response stream after hitting the cap.**
`read_body_capped` previously broke out of the read loop the moment
`limit` bytes were buffered. With reqwest/hyper that leaves unread
bytes in the response and prevents connection reuse — during a burst
of upstream errors the gateway would churn TCP connections instead
of recycling the keep-alive pool. Fix: keep iterating the stream,
discarding chunks past the cap. Memory stays bounded by `limit`.

**#5 — Redact upstream `error.message` on 5xx.**
The 5xx branch of `render_bridge_upstream_envelope` was forwarding
`BridgeError::UpstreamStatus.message` verbatim — which for OpenAI /
Anthropic comes from the parsed upstream `error.message`. Upstream
5xx bodies routinely embed operator-internal detail (engine names,
shard ids, queue depth). Fix: on 5xx, emit a canned
`"upstream returned {status}"` message; the full upstream body
remains in operator logs via tracing.

**#6 — Stale "follow-up" comment.**
The docstring on `render_bridge_upstream_envelope` claimed cross-wire
translation would ship in a follow-up, but it already shipped in
commit 2. Rewrite the comment to describe current behaviour
(4xx → `error_translate`; 5xx → canned envelope; `Unknown` wire →
legacy generic envelope).

**#8 — Content-type guard on Vertex (and Azure, while at it).**
`capture_upstream_error_http` already gates serde parsing on
`Content-Type: application/json` so a 64 KB HTML error page from a
fronting WAF doesn't waste CPU on a doomed JSON parse. The Vertex
and Azure bridges call serde directly because they need a custom
parse path (canned message for redaction) — same guard now applies.
Promoted `content_type_is_json` and added a `response_is_json`
helper to the gateway's public surface; both bridges call it before
`parse_*_error_*`.

New tests:
- `upstream_openai_5xx_with_json_envelope_collapses_and_redacts_message`
  pins the 5xx redaction (asserts `engine offline` / `shard 47` /
  `engine_overloaded` don't reach the customer envelope).
- `chat_429_preserves_openai_compatible_code_for_sdk_retry` (Azure)
  pins that `parsed.code` carries the OpenAI-compat upstream code.
- `chat_400_non_json_body_skips_envelope_parse` (Azure) and
  `chat_gemini_non_json_body_skips_envelope_parse` (Vertex) pin the
  new content-type guard.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants