fix: prevent SQS connection churn and blocking DNS

[lilienblum]

Concurrent SQS acknowledgements repeatedly open new connections because the shared HTTP client retains only four idle connections and successful responses are dropped before their bodies finish. DNS lookups for those connections run in Tokio's blocking pool.

Drain successful response bodies so HTTP connections can return to the pool. Give the SQS transport a bounded pool of 512 idle connections per host with a 30-second idle timeout, and a per-client async DNS resolver with TTL caching. The DNS change is scoped to SQS. The shared response-body fix also applies to other clients using send_no_response.

Validation:

• Seven targeted tests pass, including signed SQS acknowledgement bursts with streamed metadata, truncated-body errors, and a retrying mutation that makes exactly one request after successful headers.

• Counterfactual checks fail with the old idle pool limit, the blocking DNS resolver, or discarded response bodies.

• Built the release native addon and embedded it in a standalone Bun 1.4.2 executable with seven JavaScript workers and two queue receivers. With the source and addon files absent from disk, approximately 29,000 acknowledgement requests with fragmented responses used 512 connections and seven Tokio threads, with no DNS worker stacks.

• Validated the exact PR head as an embedded native addon in a compiled Bun 1.4.2 Linux ARM64 application on real AWS: seven Workers, a seven-core CPU quota, two SQS receivers with four receive(10) loops each, 400 in-flight messages per receiver, individual acknowledgements, cached HTTP checks, Worker RPC, and SQLite updates. All data and infrastructure were synthetic and isolated.

• The native baseline collapsed to 5,648 achieved HTTP RPS with 874,687 transport failures in 180 seconds, 98 Tokio threads, and 2.70 mean CPU cores. The fixed native build sustained 15,000 HTTP RPS over 360 seconds, approximately 1,500 push attempts/sec, seven Tokio threads, and 5.51 mean CPU cores. Independent SQS metrics show approximately 1,517 deletions/sec during five complete minutes.

• Every timeout and overload in that fixed run occurred in its first 30 seconds. The remaining 330 seconds had 4,950,047 requests, zero timeouts or overloads, and 13 connection errors. The same-binary SDK control also had startup timeouts and overloads; its following 150 seconds had 2,250,024 successful requests. The SDK processed approximately 630 queue deletions/sec, so its queue throughput is not equivalent to native throughput.

• Repeated the fixed build for ten minutes with default runtime settings: 9,000,090 requests at 15,000 RPS, 901,689 push attempts, seven Tokio threads throughout, 5.54 mean CPU cores, no restart, no CPU throttling, and no receive/ack warnings. AWS independently measured 1,503 deletions/sec over eight complete minutes. All 11,394 timeouts and 43 overloads occurred in the first 30 seconds; the following 570 seconds handled 8,550,085 requests with zero timeouts or overloads and one connection reset. The maximum per-window p99 upper bound after startup was 201 ms. Per-window request totals match the complete report.

This validates the sustained transport regression, not a zero-error cold-start guarantee or an exact reproduction of an unavailable original dataset/load command. No runtime GC/JIT override or receiver rewrite is included. Those diagnostic alternatives did not establish a repeatable complete improvement.

Body-read failures after successful headers surface as non-retryable HTTP_RESPONSE_BODY_READ_FAILED, preserving the received HTTP status and preventing replay of accepted mutations. The async resolver reads system DNS configuration and hosts entries; OS-specific NSS plugins are not used.

[enclave-ai]

Enclave skipped this draft pull request. It will run automatically when you mark the PR as Ready for review.

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-07T16:14:33.504160Z	9695760	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[greptile-apps]

Confidence Score: 5/5

[High risk] Adds DNS resolver and connection pooling to SQS client.

The current head appears safe to merge; the earlier mutation-replay issue is fixed.

What we checked:

• Accepted mutations are not replayed: handle_no_response returns an error with retries disabled. The default retry check uses that flag, so it stops rather than sending the mutation again.

Summary

The PR drains successful response bodies so connections can be reused. SQS gets a larger idle connection pool and its own async DNS resolver.

• The current head fixes the earlier mutation-replay finding. Body-read failures after successful headers are non-retryable and retain the received status.
• The added regression test checks that a truncated success response causes exactly one mutation request.
• No new actionable issues or repository-rule violations were found. Tests were inspected, not run during this review.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[SQS request] --> B[Per-client async DNS resolver]
  B --> C[HTTP connection pool]
  C --> D[Receive response headers]
  D --> E[Drain successful response body]
  E -->|Complete| F[Return connection to pool]
  E -->|Read fails| G[Return non-retryable error]

Loading

_{Reviews (2) · Last reviewed commit: "fix: prevent replay after successful res..." · Reviewed by Greptile}

[lilienblum]

@greptileai review the current head f5b92939f4cc50262e9c0d30430f1832e94a9131. The post-success mutation replay finding is fixed and covered by a wire-level regression test.

[alongubkin]

Real AWS / Linux ARM64 validation of f5b9293 is in progress. The transport changes materially reduce connection churn and keep Tokio at seven threads, but I cannot yet call the mixed HTTP + SQS workload fully fixed.

With a standalone Bun 1.4.2 executable, seven serving Workers, a seven-CPU limit, two receivers × four long polls, 400 pending messages per receiver, individual acknowledgments, a nonempty real SQS backlog, and 15,000 HTTP requests/second:

• A normal managed native deployment with this head still produced 8,903 HTTP timeouts and 1,131 overload responses out of 2,925,027 requests in three minutes. No receive/ack errors; Tokio stayed at seven threads.
• A six-minute SDK reference completed all 5,625,056 HTTP requests successfully, with zero timeouts and queue errors.
• Replacing the receiver's repeated Promise.race(pending) backpressure with one shared capacity signal did not clear the failure: 7,408 timeouts and 198 overloads out of 5,625,056 requests. This is not established as the root cause.

The current HTTP handler deliberately repeats SQLite reads and JSON parsing to calibrate CPU load; it is heavier than a cached application handler. I am now validating the actual cached-handler / routed-write flow with synthetic data before drawing a workload-specific conclusion. No debugger-attached run is included in the numbers above.

The accepted-response body-read error is correctly non-retryable, preserving the distinction between a failed response read and a mutation that is safe to replay. The body-drain, burst reuse, and no-replay tests were also rerun successfully.

[alongubkin]

Completed real AWS mixed-load validation of f5b9293 using a compiled Bun 1.4.2 Linux ARM64 application, seven Workers, individual SQS acknowledgements, and synthetic data.

The native baseline reproduced the sustained regression: Tokio threads reached 98, mean CPU fell to 2.70 cores, and achieved HTTP throughput fell to 5,648 RPS. This PR kept Tokio at seven threads and restored both HTTP and queue throughput.

The final ten-minute repeat, with default runtime settings, served 9,000,090 requests at 15,000 RPS while processing approximately 1,503 push attempts/sec. Independent AWS metrics confirmed approximately 1,503 queue deletions/sec. There were no restarts, receive/ack warnings, or CPU throttling.

All 11,394 timeouts and 43 overloads occurred in the first 30 seconds. The next 570 seconds served 8,550,085 requests with zero timeouts or overloads and one connection reset; per-window p99 was at most 201 ms. A separate six-minute repeat likewise had no timeouts or overloads after its first 30 seconds. The SDK reference also showed startup failures, while processing substantially fewer queue messages.

The receiver rewrite failed load acceptance and was discarded. GC/JIT overrides did not establish a repeatable complete improvement and are not part of the fix. The existing transport change is the supported fix for the sustained regression; these results do not establish a zero-error cold-start guarantee or correctness for every future workload. The original dataset/load command was unavailable, so this is a controlled realistic reproduction. PR validation details are updated accordingly.

[alongubkin]

Validated the exact head on real AWS/Linux ARM64 in a compiled Bun 1.4.2 application. The baseline reproduced thread growth and sustained throughput collapse; the fixed build sustained 15k HTTP RPS and approximately 1.5k independently measured SQS deletions/sec across a ten-minute repeat, with seven Tokio threads and no receive/ack warnings or restarts. The transport change and nonretryable successful-header body-read error handling are correct for this regression. Startup HTTP failures remain and are documented; no zero-error cold-start guarantee is implied. No additional receiver or runtime-flag change is justified by the tests.