Repository navigation
The gRPC Kestrel test owns its ports — closes the FreePort() TOCTOU race (#2379) - #2427
Conversation
…ace (#2379) `FreePort()` bound a socket to `:0`, read the port the kernel assigned and then RELEASED it, leaving the port owned by nobody until Kestrel's own bind. Anything else on the box — another shard's process, an ephemeral outbound connection — could take it in that window, and Kestrel then failed to start with `IOException: Failed to bind to address http://127.0.0.1:NNNNN: address already in use`. Narrow on an idle machine, wide on a loaded CI runner, which is why it read as a flake while being a plain ownership bug. It redded #2364, whose diff is three workflow YAMLs and a Documentation.Test guard. The cure is structural, not a retry (AGENTS.md → "No band-aids": never raise a bound to make a race less likely — a retry loop here still races, just less often). `LoopbackPort.Reserve()` binds AND listens and never closes: the kernel refuses every rival bind for as long as it lives. Kestrel does not open a second socket for that port either — it is handed THIS one through `SocketTransportOptions.CreateBoundListenSocket`, so discovery and use are the same socket and there is nothing in between to lose. Binding `:0` and reading the port back from `IServerAddressesFeature` is not available here: the test's whole subject is `Grpc:TrustedPort` bound from configuration BEFORE the host starts, so the number has to exist up front. `ListenLocalhost` → `Listen(IPAddress.Loopback, …)`: the former binds the IPv4 AND IPv6 loopback, i.e. two sockets for one port number, and only one of them can be the reserved one. The clients dial `127.0.0.1` explicitly, which is also the address `GrpcOptions.TrustedPort` documents. `LoopbackPortReservationTest` runs the losing interleaving deterministically instead of waiting for a loaded runner to produce it: a rival grabs the port at exactly the wrong instant. Both halves are load-bearing — the rival must be REFUSED (with a released port it wins), and Kestrel must bind that port WHILE the reservation is still held (only possible because it adopts the reserved socket; drop the handover and the reservation is what blocks Kestrel). Verified on macOS 25.5.0 / .NET 10.0.301, Release + `-warnaserror`, 0 warnings: - regressed `Reserve()` to the old release-after-discovery shape ⇒ the new test fails in 17 ms with `Expected: AddressAlreadyInUse / Actual: null` (the rival took the port), and a standalone probe of the same shape reproduces the issue's exact text: `IOException: Failed to bind to address http://127.0.0.1:56067: address already in use.` - with the fix: 20/20 iterations of the new test and 5/5 of the full 12-test project pass. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
This PR fixes a deterministic test flake in the gRPC/Kestrel integration tests by eliminating a TOCTOU race in port selection: the test now reserves (binds and keeps) loopback ports and hands the already-owned sockets directly to Kestrel, ensuring no other process can steal the ports between “find” and “bind”.
Changes:
- Replaces the old “find a free port then release it” helper with a
LoopbackPortreservation that keeps ownership until Kestrel shuts down, and configures Kestrel to adopt the reserved sockets viaSocketTransportOptions.CreateBoundListenSocket. - Updates Kestrel binding and client dialing to use IPv4 loopback explicitly (
Listen(IPAddress.Loopback, ...)+127.0.0.1) to avoid dual-stackListenLocalhostambiguity. - Adds a regression test that forces the losing interleaving (rival bind attempt) to pin the ownership behavior deterministically.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| test/MeshWeaver.Hosting.Grpc.Test/MeshGrpcTrustedKestrelTest.cs | Reserves loopback ports, hands sockets to Kestrel to avoid TOCTOU, switches to explicit IPv4 loopback, and adds a deterministic regression test for port ownership. |
| src/MeshWeaver.Documentation/Data/WhatsNew/2026-08-26-grpc-test-no-longer-loses-its-port.md | Adds a What’s New entry documenting the test-flake fix and the new deterministic guard. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
|
Filed the two sibling sites named under Not in scope as #2436 — both are the same bug class, both need a different structural fix ( |
Test Results (shard 0)616 tests 612 ✅ 13m 24s ⏱️ Results for commit b1d3b4e. |
Test Results (shard 3) 9 files 9 suites 5m 19s ⏱️ Results for commit b1d3b4e. |
Test Results (shard 5) 9 files 9 suites 6m 44s ⏱️ Results for commit b1d3b4e. |
Test Results (shard 2)2 878 tests 2 775 ✅ 8m 19s ⏱️ Results for commit b1d3b4e. |
Test Results (shard 1)2 036 tests 1 843 ✅ 9m 19s ⏱️ Results for commit b1d3b4e. |
Test Results (shard 4)2 195 tests 2 193 ✅ 13m 35s ⏱️ Results for commit b1d3b4e. |
Test Results 53 files 53 suites 56m 43s ⏱️ Results for commit b1d3b4e. |
Closes #2379.
The race
MeshGrpcTrustedKestrelTest.FreePort()bound a socket to:0, read the port the kernel assigned, and then released it. Kestrel bound that number later. Between the release and the bind the port was owned by nobody, so anything else on the machine — another test, another CI shard's process, an ephemeral outbound connection — could take it, and Kestrel then failed to start:Narrow window on an idle box, wide one on a loaded runner — which is why it reads as a flake while being a plain ownership bug. It redded #2364, whose diff is three workflow YAMLs and one Documentation.Test guard.
The fix — structural, not a retry
LoopbackPort.Reserve()binds and listens and never closes: the kernel refuses every rival bind for as long as it lives. Kestrel does not open a second socket for that port either — it is handed this one viaSocketTransportOptions.CreateBoundListenSocket, so discovery and use are the same socket and there is nothing in between to lose.No retry loop, no wider range, no sleep — each of those only makes the window smaller (AGENTS.md → "No band-aids": never raise a bound to make a race less likely).
The issue's option 1 — bind
:0and read the port back fromIServerAddressesFeature— is not available here: the test's whole subject isGrpc:TrustedPortbound from configuration before the host starts, so the number has to exist up front. Option 2 (SO_REUSEADDRalongside) and option 3 (reserving from the OS ephemeral range) are both declined for the reasons the issue gives.ListenLocalhost→Listen(IPAddress.Loopback, …): the former binds the IPv4 and IPv6 loopback, i.e. two sockets for one port number, and only one of them can be the reserved one. The clients now dial127.0.0.1explicitly — also the addressGrpcOptions.TrustedPortdocuments (http://127.0.0.1:{TrustedPort}).The test fails without the fix — deterministically
LoopbackPortReservationTest.Kestrel_binds_a_reserved_port_no_rival_can_takeruns the losing interleaving on purpose rather than waiting for a loaded runner to produce it. Both halves are load-bearing:Evidence (macOS 25.5.0, .NET 10.0.301,
-c Release -warnaserror, 0 warnings / 0 errors):Reserve()temporarily regressed to the old release-after-discovery shape ⇒ the new test fails in 17 ms:Assert.Equal() Failure — Expected: AddressAlreadyInUse, Actual: null(the rival took the port). No load needed.IOException: Failed to bind to address http://127.0.0.1:56067: address already in use.Not in scope
Two other sites reserve-then-release the same way, and neither can use this cure:
test/MeshWeaver.Graph.Test/TestOgServer.cs:43—HttpListenercannot be handed a socket;test/MeshWeaver.Persistence.Test/StorageAndDb.cs:10— hands the port to Azurite, an external process.Both need a different fix; filing separately rather than widening this one. (
MeshGrpcLiveTestalready does the right thing — it binds:0, because it does not need the number up front.)🤖 Generated with Claude Code