Repository navigation
IoPool: admission is taken on the subscriber's thread, so accepted work is never silently discarded (#4555) - #4568
Conversation
…ver lost (#4555) Three of the four entry points deferred their whole prologue to the ThreadPool (SubscribeOn) and took their admission region INSIDE it. Between a caller's Subscribe() returning and that prologue running, the leaf was counted by nothing — not _gateUsers, not _inFlight, not CurrentlyWaiting. The caller believed the work was queued; the pool did not know it existed. Drain() waits while anything is outstanding and gives it a grace. A leaf in that window is outstanding to nobody, so the drain never enters the grace, cancels the pool token, and the leaf then arrives to find itself cancelled — too late even to be counted in LeavesCancelledAfterGrace, which is captured before the cancel. Measured on main, subscribing one leaf and draining immediately: 50 of 50 for Invoke and 50 of 50 for InvokeStream lost the work, terminated the subscriber with a cancellation, and reported LeavesCancelledAfterGrace == 0. Accepted work discarded in silence, which is what #3291 forbids. Invoke, InvokeStream and SubscribeThroughPool's setup leaf now enter the region synchronously on the subscriber's thread — what InvokeBlocking has always done — and the leaf claims it when its prologue starts. The claim is a CAS with three states because only one of the two may release: a leaf that has started still touches _gate and _poolCts, so an unsubscribe releasing the region under it would reopen the hole #2146 closed; a leaf that never started is released by the unsubscribe, so a dropped subscription cannot leak a region and park Disposed. Unchanged: the work still runs on a pool thread (only the admission moved), a refusal still leaves on a pool thread (#4530), and the region still ends with the SETUP LEAF for SubscribeThroughPool rather than with the long-lived subscription. New cost: Disposed now waits for a leaf accepted but not yet started, which is the correct direction and is bounded by the ThreadPool reaching the prologue. IoPoolAcceptedWorkIsAccountedTest parks a prologue on the new OnLeafPrologueStarting seam — so the leaf is inside the window by construction rather than by racing a ThreadPool hop — and asserts both halves of the contract: the drain spends its grace, and the leaf it has to cancel is counted. Red on main for both entry points at "Expected 0 to be greater than 0". Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
🔵 Needs a closer look
It changes core IoPool concurrency/admission semantics on hot entry points, so it warrants final human review despite good targeted test coverage.
Pull request overview
This PR fixes a teardown-accounting hole in IoPool where work could be “accepted” (the caller’s Subscribe() returned) but still be invisible to Drain() until the ThreadPool ran the leaf prologue, allowing drains to cancel that work without it being counted or named (issue #4555). It moves the admission region to be taken synchronously on the subscriber’s thread (while still running the actual work on the pool thread), adds a small CAS-based ownership protocol to ensure the region is released exactly once, and pins the behavior with a deterministic regression test plus documentation.
Changes:
- Take the IoPool admission region synchronously at subscribe time for
Invoke,InvokeStream, andSubscribeThroughPool’s setup leaf, eliminating the “Subscribe-returned-but-unaccounted” window. - Add a test seam (
OnLeafPrologueStarting) and a regression test that deterministically holds a leaf in the window and assertsDrain()spends grace and counts the cancellation. - Document the “admission on subscriber thread” rule and the rationale/protocol in
ControlledIoPooling.md.
File summaries
| File | Description |
|---|---|
| test/MeshWeaver.Hosting.Test/IoPoolAcceptedWorkIsAccountedTest.cs | Adds a deterministic regression test for #4555 using the new prologue seam. |
| src/MeshWeaver.Mesh.Contract/Threading/IoPool.cs | Moves admission to the subscriber thread and implements single-release ownership (leaf vs unsubscribe) for affected entry points. |
| src/MeshWeaver.Documentation/Data/Architecture/ControlledIoPooling.md | Documents the admission-thread rule, the failure mode, and the CAS protocol used to fix it. |
Review details
- Files reviewed: 3/3 changed files
- Comments generated: 0
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Test Results (shard 0)264 tests 264 ✅ 2m 12s ⏱️ Results for commit adf3a79. |
Test Results (shard 1)533 tests 533 ✅ 1m 11s ⏱️ Results for commit adf3a79. |
Test Results (shard 3)1 397 tests 1 397 ✅ 3m 5s ⏱️ Results for commit adf3a79. |
Test Results (shard 4) 2 files 2 suites 5m 6s ⏱️ Results for commit adf3a79. |
Test Results (shard 5) 4 files 4 suites 5m 33s ⏱️ Results for commit adf3a79. |
Test Results (shard 2)1 764 tests 1 764 ✅ 11m 46s ⏱️ Results for commit adf3a79. |
Test Results 13 files 13 suites 28m 55s ⏱️ Results for commit adf3a79. |
Fixes #4555
The window, measured before anything changed
Three of the four entry points deferred their whole prologue to the ThreadPool (
SubscribeOn) and took their admission region inside it. Between a caller'sSubscribe()returning and that prologue running, the leaf was counted by nothing — not_gateUsers, not_inFlight, notCurrentlyWaiting.Drain()waits while anything is outstanding and gives it a grace. A leaf in that window is outstanding to nobody, so the drain never enters the grace, cancels the pool token, and the leaf then arrives to find itself cancelled — too late even to be counted inLeavesCancelledAfterGrace, which is captured before the cancel.Scratch probe on current
main(50 attempts each, subscribe then drain immediately):Every attempt lost the work, terminated the subscriber with a cancellation, and reported
LeavesCancelledAfterGrace == 0. That is accepted work discarded in silence — what #3291 exists to forbid (teardown lets accepted work finish and NAMES what it had to stop).Fix — where the region is taken, not a new counter
Invoke,InvokeStreamandSubscribeThroughPool's setup leaf now enter the region synchronously on the subscriber's thread (whatInvokeBlockinghas always done), and the leaf claims it when its prologue starts:The CAS is the whole protocol, and it is needed because only one of the two may release: a leaf that has started still touches
_gateand_poolCts, so an unsubscribe releasing the region under it would reopen the hole #2146 closed; a leaf that never started must be released by the unsubscribe, or a dropped subscription would leak a region and parkDisposedforever. No new gate, no new counter, no new primitive.Unchanged: the work still runs on a pool thread (only the admission moved — the
SubscribeOnis untouched), refusals still leave off the subscriber's thread, and the region still ends with the setup leaf forSubscribeThroughPool, never with the long-lived subscription.New cost, stated plainly:
Disposednow waits for a leaf that has been accepted but whose prologue has not run, where before it could complete and leave that leaf to be refused on arrival. That is the correct direction — the pool waits for work it accepted — and it is bounded by the ThreadPool reaching the prologue.Proof
IoPoolAcceptedWorkIsAccountedTestparks a prologue on a newOnLeafPrologueStartingseam, so the leaf is inside the window by construction rather than by racing a ThreadPool hop, then asserts both halves of the contract: the drain spends its grace, and the leaf it has to cancel is counted.main+ the seam only (so the control isolates the admission change), test project rebuilt → red for both entry points atExpected 0 to be greater than 0. Fix restored, rebuilt → green.SpinUntilmeasures its budget on a clock myStopwatchdid not start. It now compares against half the grace — the discrimination is microseconds vs 1.2 s, so it never needed the precision.Verification (local)
dotnet build -c Release -warnaserror: 0 warnings, 0 errors forMeshWeaver.Mesh.Contract,MeshWeaver.Hosting.Test,MeshWeaver.Hosting.Orleans.Test.MeshWeaver.Hosting.Test: 795 passed, 0 failed (1 pre-existing unrelated skip).IoPoolDrainGraceTest): 64/64.MeshWeaver.Documentation.Test: 508/508.Coordination
Rebased onto current
main, which carries #4556 (merged). This diff does not touchDrain(),TryEnterGateRegionor the_admissionsCompletedcounter that PR added — it only moves when a region is entered, which is what #4556's own comment identified as the remaining half ("this removes the masking, not the whole window").Notes
Doc/Architecture/ControlledIoPooling→ "ADMISSION is taken on the subscriber's thread, so 'accepted' means one thing".internal); no i18n keys. NoPairs-with:,Implementers:orMirror-sync:.🤖 Generated with Claude Code