Skip to content

fix(manager): close idle inbound connections - #822

Merged
alongubkin merged 4 commits into
mainfrom
alon/alien-1152-manager-idle-timeout
Oct 2, 2026
Merged

alongubkin merged 4 commits into
mainfrom
alon/alien-1152-manager-idle-timeout

Conversation

@alongubkin

Copy link
Copy Markdown
Member

Why

axum::serve sets no timer, so it never closes an idle keep-alive connection. Any client that opens connections and forgets them pins hyper's per-connection read buffers in the manager forever. In production that was ~8,400 idle connections and ~1.6Gi per replica after three hours, from a client bug (the demo's per-tick OTLP exporters, fixed separately). The manager should not depend on every client being well behaved.

Linear: ALIEN-1152

What

server.rs serves connections through hyper-util's auto::Builder with a tokio timer and header_read_timeout. hyper applies that timeout between requests on a keep-alive connection too, so a connection that sends no request for 90s is closed. That's above the ALB's 60s idle timeout, which AWS requires so the balancer never reuses a connection the target has just closed. Upgrades (debug WebSockets) are still served via serve_connection_with_upgrades, and accept errors (fd exhaustion) are logged and retried as axum::serve did.

Two new crate dependencies, both already in the workspace: hyper and hyper-util.

Validation

  • New test idle_keep_alive_connection_is_closed_after_the_timeout: raw TCP to a server with a 300ms idle timeout. Two requests inside the window are served on the same connection (keep-alive still works); then the client goes silent and the server closes the socket (read returns EOF). It fails on main's behavior.
  • cargo test -p alien-manager --lib: 291 passed.
  • cargo fmt --check clean. Clippy via CI.

After deploy, ls /proc/1/fd | wc -l on a manager pod should stay in the low hundreds even with a leaking client.

🤖 Generated with Claude Code

axum::serve never closes an idle keep-alive connection, so any client that opens connections and forgets them pins hyper's per-connection buffers in the manager for the life of the process. Serve through hyper-util's builder with a timer and a header read timeout, which hyper also applies between requests, so a connection that sends nothing for 90s is closed. Above the load balancer's 60s idle timeout so it never reuses a connection the manager just closed.

Refs ALIEN-1152

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@greptile-apps

greptile-apps Bot commented Oct 2, 2026 •

Copy link
Copy Markdown

RetriggerConfidence Score: 5/5

[High risk] Replaces the HTTP server implementation with custom connection handling.

The PR appears safe to merge based on the reviewed changes.

Summary

The PR replaces the manager’s server loop with an HTTP/1 server configured to close idle inbound connections, and adds coverage for silent and idle keep-alive sockets.

Reviews (2) · Last reviewed commit: "fix(manager): serve HTTP/1 only so the i..."

Comment thread crates/alien-manager/src/server.rs Outdated
Comment thread crates/alien-manager/src/server.rs
Comment thread crates/alien-manager/src/server.rs
…ection

Protocol auto-detection waited for a connection's first byte with no timeout, so a client that connected and sent nothing was never closed, and idle HTTP/2 connections had no timeout at all. Nothing speaks HTTP/2 to the manager. The test now reads whole responses and also covers the silent-connection case.

Refs ALIEN-1152

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@alongubkin

Copy link
Copy Markdown
Member Author

@greptileai re-review: switched to HTTP/1-only serving so silent and idle connections are both closed; test reads full responses and covers the silent case.

alongubkin and others added 2 commits October 2, 2026 13:58
…er-idle-timeout

# Conflicts:
#	crates/alien-manager/Cargo.toml
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@alongubkin
alongubkin merged commit 122514e into main Oct 2, 2026
21 checks passed
@alongubkin
alongubkin deleted the alon/alien-1152-manager-idle-timeout branch October 2, 2026 18:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant