Repository navigation
fix(manager): close idle inbound connections - #822
Merged
Merged
Conversation
axum::serve never closes an idle keep-alive connection, so any client that opens connections and forgets them pins hyper's per-connection buffers in the manager for the life of the process. Serve through hyper-util's builder with a timer and a header read timeout, which hyper also applies between requests, so a connection that sends nothing for 90s is closed. Above the load balancer's 60s idle timeout so it never reuses a connection the manager just closed. Refs ALIEN-1152 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
|
…ection Protocol auto-detection waited for a connection's first byte with no timeout, so a client that connected and sent nothing was never closed, and idle HTTP/2 connections had no timeout at all. Nothing speaks HTTP/2 to the manager. The test now reads whole responses and also covers the silent-connection case. Refs ALIEN-1152 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Member
Author
|
@greptileai re-review: switched to HTTP/1-only serving so silent and idle connections are both closed; test reads full responses and covers the silent case. |
…er-idle-timeout # Conflicts: # crates/alien-manager/Cargo.toml
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
axum::servesets no timer, so it never closes an idle keep-alive connection. Any client that opens connections and forgets them pins hyper's per-connection read buffers in the manager forever. In production that was ~8,400 idle connections and ~1.6Gi per replica after three hours, from a client bug (the demo's per-tick OTLP exporters, fixed separately). The manager should not depend on every client being well behaved.Linear: ALIEN-1152
What
server.rsserves connections through hyper-util'sauto::Builderwith a tokio timer andheader_read_timeout. hyper applies that timeout between requests on a keep-alive connection too, so a connection that sends no request for 90s is closed. That's above the ALB's 60s idle timeout, which AWS requires so the balancer never reuses a connection the target has just closed. Upgrades (debug WebSockets) are still served viaserve_connection_with_upgrades, and accept errors (fd exhaustion) are logged and retried asaxum::servedid.Two new crate dependencies, both already in the workspace:
hyperandhyper-util.Validation
idle_keep_alive_connection_is_closed_after_the_timeout: raw TCP to a server with a 300ms idle timeout. Two requests inside the window are served on the same connection (keep-alive still works); then the client goes silent and the server closes the socket (read returns EOF). It fails onmain's behavior.cargo test -p alien-manager --lib: 291 passed.cargo fmt --checkclean. Clippy via CI.After deploy,
ls /proc/1/fd | wc -lon a manager pod should stay in the low hundreds even with a leaking client.🤖 Generated with Claude Code