fix(managed): pin HTTP/1.1 on the dp-manager mTLS REST clients - #536
Merged
Conversation
dp-manager multiplexes gRPC (kine/etcd, HTTP/2) and the REST surface (/dp/heartbeat, /dp/budget_check, /dp/telemetry) on a single TLS port via cmux, which dispatches by the negotiated ALPN protocol. The observability cloud-sink crates (object_store + Aliyun SLS) pull reqwest's `http2` feature, and Cargo feature unification applied it to every reqwest client in the workspace — so the DP's mTLS REST clients began advertising `h2` in ALPN. cmux then routed those REST POSTs to the gRPC handler and every request failed with 'error sending request', making the DP fail-close on budget checks (429 budget_exceeded) and drop telemetry/heartbeats. Force these two clients to HTTP/1.1 with .http1_only() so ALPN stays at http/1.1 and cmux routes them to the REST mux. The etcd/gRPC client is a separate tonic h2 connection and is unaffected.
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (2)
📝 WalkthroughWalkthroughThis PR fixes ALPN-based request misrouting in two outbound client configurations. Heartbeat and telemetry clients now explicitly pin HTTP/1.1 transport via ChangesHTTP/1.1 Enforcement in Client Builders
Estimated code review effort🎯 2 (Simple) | ⏱️ ~8 minutes 🚥 Pre-merge checks | ✅ 4✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
Comment |
This was referenced Jun 8, 2026
6 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
In managed mode the DP talks to dp-manager over a single mTLS port (
:7944) that cmux-multiplexes two protocols: gRPC (kine/etcd, HTTP/2) and the REST surface (/dp/heartbeat,/dp/budget_check,/dp/telemetry). cmux dispatches each connection by its negotiated ALPN protocol.Since the observability cloud-sink work (object_store + Aliyun SLS),
Cargo.lockpulls reqwest'shttp2feature into the workspace, and Cargo feature unification turns it on for every reqwest client — including the DP→dp-manager mTLS REST clients. Those clients then started advertisingh2in ALPN, so cmux handed their REST POSTs to the gRPC handler and every request failed witherror sending request. The DP then fail-closes budget checks (429 budget_exceeded) and drops heartbeats/telemetry. The etcd/gRPC client is a separate tonic h2 connection, so it kept working — which is why the symptom is "gRPC up, REST down" on the same port.Fix
Pin the two DP→dp-manager mTLS clients to HTTP/1.1 with
.http1_only()(crates/aisix-server/src/heartbeat.rs— also used by the budget client — andcrates/aisix-server/src/telemetry.rs). ALPN then offers onlyhttp/1.1, so cmux routes them to the REST mux. No behavior change to the etcd/gRPC path.Testing
The regression is exercised by the control-plane e2e (
AISIX-Cloude2e-go, theTestGuardrail*suite), which boots a real DP from the:devimage against a real dp-manager: it fails (DP never serves;429 budget_exceeded) with the buggy image and passes with this fix. Verified locally by building the fixed image and runningTestGuardrailAliyunBlocksAndWireShape— fails before, passes after (2.8s vs a 3-min timeout). A pure DP-side unit test isn't practical here (the behavior is ALPN negotiation against a cmux server); the cross-repo e2e is the guard.Unblocks the per-attempt telemetry CP PR (api7/AISIX-Cloud#710), whose
e2e-gois currently red solely due to this.Summary by CodeRabbit