DP can't reach dp-manager /dp/budget_check (transport error) → chats fail-close to 429; breaks all DP-mediated AISIX-Cloud e2e

[moonming]

Symptom

The DP (ghcr.io/api7/ai-gateway:dev) cannot connect to dp-manager's /dp/* REST surface. Every /v1/chat/completions fails-closed to 429 before reservation/dispatch (latency_ms=1), each one preceded by:

WARN aisix_proxy::budget: budget_check failed; falling back to cache or fail_mode
     api_key_id=<uuid>
     error=error sending request for url (https://dpm:7944/dp/budget_check?api_key_id=<uuid>)

/dp/heartbeat and /dp/telemetry (same dp-manager REST surface, https://dpm:7944/dp/*) fail identically. The kine/config path still works — the model resolves (the response is 429, a budget decision, not a 404 model-not-found) — so it is specifically the dp-manager REST client (budget/heartbeat/telemetry) that can't establish a connection, not the snapshot/watch path. error sending request is a transport-level failure (connect/TLS), not an HTTP error response.

Impact

All DP-mediated e2e in api7/AISIX-Cloud are red — guardrails-live, adapter-vertex/bedrock-live, dp-*-live, telemetry-cross-org-isolation, the Azure Content Safety specs, etc. One e2e-playwright run: 73 failed, 3990 budget_check failed warns, 45 "DP never accepted a clean prompt (pipeline not live)". AISIX-Cloud main CI has been red since ~2026-06-06T06:07Z.

Repro

1. Bring up the AISIX-Cloud e2e stack (deploy/compose.yml + compose.e2e.yml), boot a DP via the dp-harness.
2. POST /v1/chat/completions with a fresh api key (no budget configured).
3. → 429; DP logs show the budget_check failed + error sending request for url (https://dpm:7944/dp/budget_check) above.

(A chat with no budget on the key still hits budget_check and fail-closes — so this is purely the DP↔dp-manager REST reachability, independent of any budget config.)

Suspect window (needs a bisect)

The same e2e was green when the :dev image was built from 881c2e2 (#528) and ran at 2026-06-06T04:21Z; it is red on the newer :dev built from the subsequent commits:

• 9ad522e — feat(obs): object_store sink (feat(obs): object_store sink — one S3/GCS/Azure exporter on the shared pipeline #531)
• 31a6a80 — feat(obs): capture request/response content for full-mode SLS exporters (feat(obs): capture request/response content for full-mode SLS exporters #532)

These are the observability-sink framework series (#528 → #531 → #532). An architectural change to the DP's pipeline / shared HTTP-client setup is the leading suspect for breaking the dp-manager REST client, but this needs a bisect of those commits to confirm. (One AISIX-Cloud main run at 06:07Z went red on the older 881c2e2 :dev — possibly a separate flake; the persistent failures are all on the newer obs-sink :dev.)

A dp-manager-side change cannot be fully ruled out, but the kine/config path working while the REST client fails points at the DP-side REST/mTLS client.

Cross-ref

Surfaced by api7/AISIX-Cloud#707 (real Azure Content Safety integration test) and the existing AISIX-Cloud DP-mediated e2e suite. The Azure test's pre-flight "pipeline live?" poll caught it before any false guardrail signal.

[coderabbitai]

🔗 Related PRs

api7/AISIX-Cloud#373 - fix(dashboard): lock DP install script before cert issuance [merged]
api7/AISIX-Cloud#571 - test: Add DeepSeek observability CI coverage [merged]
api7/AISIX-Cloud#658 - fix(dashboard): retry idempotent GETs through transient cp-api blips (#211) [merged]
#494 - fix: harden upstream-error handling — forward 429 Retry-After + UTF-8-safe truncation (#144, #420) [merged]
api7/AISIX-Cloud#676 - fix(e2e): stop every DP when the first dumpLogs throws (#675) [merged]

---

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

• Create Plan

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

[moonming]

Verified fixed by #536. ghcr.io/api7/ai-gateway:dev rebuilt from af43b3c (pin HTTP/1.1 on the dp-manager mTLS REST clients):

• AISIX-Cloud azure-cs-real-chain — which reproduced this as a budget_check→429 fail-close on the pre-guardrail sanity poll — now passes 3/3 against the real Azure resource (run 27064045888).
• e2e-playwright + e2e-go on AISIX-Cloud PR fix(passthrough): emit UsageEvents (success + failure) with PK attribution #707 also went green on the same rebuilt image (they were red here solely due to this regression).

Root cause matched the diagnosis exactly: the mTLS REST client negotiating HTTP/2 → transport error under contention → fail-close to 429. Closing as fixed by #536.