Skip to content

fix(managed): pin HTTP/1.1 on the dp-manager mTLS REST clients - #536

Merged
nic-6443 merged 1 commit into
mainfrom
fix/dp-dpm-http1-only
Jun 7, 2026
Merged

fix(managed): pin HTTP/1.1 on the dp-manager mTLS REST clients#536
nic-6443 merged 1 commit into
mainfrom
fix/dp-dpm-http1-only

Conversation

@jarvis9443

@jarvis9443 jarvis9443 commented Jun 7, 2026

Copy link
Copy Markdown
Contributor

Problem

In managed mode the DP talks to dp-manager over a single mTLS port (:7944) that cmux-multiplexes two protocols: gRPC (kine/etcd, HTTP/2) and the REST surface (/dp/heartbeat, /dp/budget_check, /dp/telemetry). cmux dispatches each connection by its negotiated ALPN protocol.

Since the observability cloud-sink work (object_store + Aliyun SLS), Cargo.lock pulls reqwest's http2 feature into the workspace, and Cargo feature unification turns it on for every reqwest client — including the DP→dp-manager mTLS REST clients. Those clients then started advertising h2 in ALPN, so cmux handed their REST POSTs to the gRPC handler and every request failed with error sending request. The DP then fail-closes budget checks (429 budget_exceeded) and drops heartbeats/telemetry. The etcd/gRPC client is a separate tonic h2 connection, so it kept working — which is why the symptom is "gRPC up, REST down" on the same port.

Fix

Pin the two DP→dp-manager mTLS clients to HTTP/1.1 with .http1_only() (crates/aisix-server/src/heartbeat.rs — also used by the budget client — and crates/aisix-server/src/telemetry.rs). ALPN then offers only http/1.1, so cmux routes them to the REST mux. No behavior change to the etcd/gRPC path.

Testing

The regression is exercised by the control-plane e2e (AISIX-Cloud e2e-go, the TestGuardrail* suite), which boots a real DP from the :dev image against a real dp-manager: it fails (DP never serves; 429 budget_exceeded) with the buggy image and passes with this fix. Verified locally by building the fixed image and running TestGuardrailAliyunBlocksAndWireShape — fails before, passes after (2.8s vs a 3-min timeout). A pure DP-side unit test isn't practical here (the behavior is ALPN negotiation against a cmux server); the cross-repo e2e is the guard.

Unblocks the per-attempt telemetry CP PR (api7/AISIX-Cloud#710), whose e2e-go is currently red solely due to this.

Summary by CodeRabbit

  • Bug Fixes
    • Fixed heartbeat and telemetry POST request delivery failures. Ensured correct HTTP protocol configuration to prevent request routing issues and improve system reliability.

dp-manager multiplexes gRPC (kine/etcd, HTTP/2) and the REST surface
(/dp/heartbeat, /dp/budget_check, /dp/telemetry) on a single TLS port
via cmux, which dispatches by the negotiated ALPN protocol. The
observability cloud-sink crates (object_store + Aliyun SLS) pull
reqwest's `http2` feature, and Cargo feature unification applied it to
every reqwest client in the workspace — so the DP's mTLS REST clients
began advertising `h2` in ALPN. cmux then routed those REST POSTs to
the gRPC handler and every request failed with 'error sending request',
making the DP fail-close on budget checks (429 budget_exceeded) and
drop telemetry/heartbeats.

Force these two clients to HTTP/1.1 with .http1_only() so ALPN stays at
http/1.1 and cmux routes them to the REST mux. The etcd/gRPC client is a
separate tonic h2 connection and is unaffected.
@coderabbitai

coderabbitai Bot commented Jun 7, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 6684c69d-f4e5-424c-9a6e-532970e664ba

📥 Commits

Reviewing files that changed from the base of the PR and between 6bc88ed and f6344b9.

📒 Files selected for processing (2)
  • crates/aisix-server/src/heartbeat.rs
  • crates/aisix-server/src/telemetry.rs

📝 Walkthrough

Walkthrough

This PR fixes ALPN-based request misrouting in two outbound client configurations. Heartbeat and telemetry clients now explicitly pin HTTP/1.1 transport via .http1_only(), with inline documentation explaining how HTTP/2 advertisement previously caused cmux to route REST requests to the gRPC handler instead of the intended REST endpoint.

Changes

HTTP/1.1 Enforcement in Client Builders

Layer / File(s) Summary
Force HTTP/1.1 in heartbeat and telemetry clients
crates/aisix-server/src/heartbeat.rs, crates/aisix-server/src/telemetry.rs
Heartbeat and telemetry reqwest client builders now call .http1_only() with explanatory comments describing how ALPN/HTTP2 unification previously caused cmux to misroute /dp/* REST traffic to the gRPC/etcd handler, resulting in POST failures.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~8 minutes

🚥 Pre-merge checks | ✅ 4
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately describes the main change: pinning HTTP/1.1 on dp-manager mTLS REST clients to fix cmux misrouting. It is specific, concise, and directly related to the primary objective of the changeset.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch

Comment @coderabbitai help to get the list of available commands and usage tips.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants