Skip to content

IoPool: admission is taken on the subscriber's thread, so accepted work is never silently discarded (#4555) - #4568

Merged
meshweaver-cloud[bot] merged 1 commit into
mainfrom
fix/4555-admission-window
Sep 17, 2026
Merged

meshweaver-cloud[bot] merged 1 commit into
mainfrom
fix/4555-admission-window

Conversation

@rbuergi

@rbuergi rbuergi commented Sep 17, 2026

Copy link
Copy Markdown
Contributor

Fixes #4555

The window, measured before anything changed

Three of the four entry points deferred their whole prologue to the ThreadPool (SubscribeOn) and took their admission region inside it. Between a caller's Subscribe() returning and that prologue running, the leaf was counted by nothing — not _gateUsers, not _inFlight, not CurrentlyWaiting.

Drain() waits while anything is outstanding and gives it a grace. A leaf in that window is outstanding to nobody, so the drain never enters the grace, cancels the pool token, and the leaf then arrives to find itself cancelled — too late even to be counted in LeavesCancelledAfterGrace, which is captured before the cancel.

Scratch probe on current main (50 attempts each, subscribe then drain immediately):

Invoke         work completed=0/50   cancelled-and-NAMED=0   cancelled-and-SILENT=50
InvokeStream   work completed=0/50   cancelled-and-NAMED=0   cancelled-and-SILENT=50

Every attempt lost the work, terminated the subscriber with a cancellation, and reported LeavesCancelledAfterGrace == 0. That is accepted work discarded in silence — what #3291 exists to forbid (teardown lets accepted work finish and NAMES what it had to stop).

Fix — where the region is taken, not a new counter

Invoke, InvokeStream and SubscribeThroughPool's setup leaf now enter the region synchronously on the subscriber's thread (what InvokeBlocking has always done), and the leaf claims it when its prologue starts:

if (!TryEnterGateRegion())
    return RefuseOffSubscriber(observer);        // a refusal still leaves on a POOL thread (#4530)

var regionOwner = 0;                             // 0 unclaimed · 1 the leaf · 2 an unsubscribe
// prologue:    CAS(1,0) != 0  ⇒ already released, touch nothing, throw
// unsubscribe: CAS(2,0) == 0  ⇒ the leaf never started, release here

The CAS is the whole protocol, and it is needed because only one of the two may release: a leaf that has started still touches _gate and _poolCts, so an unsubscribe releasing the region under it would reopen the hole #2146 closed; a leaf that never started must be released by the unsubscribe, or a dropped subscription would leak a region and park Disposed forever. No new gate, no new counter, no new primitive.

Unchanged: the work still runs on a pool thread (only the admission moved — the SubscribeOn is untouched), refusals still leave off the subscriber's thread, and the region still ends with the setup leaf for SubscribeThroughPool, never with the long-lived subscription.

New cost, stated plainly: Disposed now waits for a leaf that has been accepted but whose prologue has not run, where before it could complete and leave that leaf to be refused on arrival. That is the correct direction — the pool waits for work it accepted — and it is bounded by the ThreadPool reaching the prologue.

Proof

IoPoolAcceptedWorkIsAccountedTest parks a prologue on a new OnLeafPrologueStarting seam, so the leaf is inside the window by construction rather than by racing a ThreadPool hop, then asserts both halves of the contract: the drain spends its grace, and the leaf it has to cancel is counted.

  • Negative control: main + the seam only (so the control isolates the admission change), test project rebuilt → red for both entry points at Expected 0 to be greater than 0. Fix restored, rebuilt → green.
  • One test-precision fix on the way: the elapsed assertion first compared against the whole grace and missed by 0.6 ms, because SpinUntil measures its budget on a clock my Stopwatch did not start. It now compares against half the grace — the discrimination is microseconds vs 1.2 s, so it never needed the precision.

Verification (local)

Coordination

Rebased onto current main, which carries #4556 (merged). This diff does not touch Drain(), TryEnterGateRegion or the _admissionsCompleted counter that PR added — it only moves when a region is entered, which is what #4556's own comment identified as the remaining half ("this removes the masking, not the whole window").

Notes

  • Recorded in Doc/Architecture/ControlledIoPooling → "ADMISSION is taken on the subscriber's thread, so 'accepted' means one thing".
  • What's New: skipped deliberately — internal teardown accounting, no user-visible change.
  • No public surface added or removed (the seam is internal); no i18n keys. No Pairs-with:, Implementers: or Mirror-sync:.

🤖 Generated with Claude Code

…ver lost (#4555)

Three of the four entry points deferred their whole prologue to the ThreadPool
(SubscribeOn) and took their admission region INSIDE it. Between a caller's
Subscribe() returning and that prologue running, the leaf was counted by nothing —
not _gateUsers, not _inFlight, not CurrentlyWaiting. The caller believed the work
was queued; the pool did not know it existed.

Drain() waits while anything is outstanding and gives it a grace. A leaf in that
window is outstanding to nobody, so the drain never enters the grace, cancels the
pool token, and the leaf then arrives to find itself cancelled — too late even to
be counted in LeavesCancelledAfterGrace, which is captured before the cancel.
Measured on main, subscribing one leaf and draining immediately: 50 of 50 for
Invoke and 50 of 50 for InvokeStream lost the work, terminated the subscriber with
a cancellation, and reported LeavesCancelledAfterGrace == 0. Accepted work
discarded in silence, which is what #3291 forbids.

Invoke, InvokeStream and SubscribeThroughPool's setup leaf now enter the region
synchronously on the subscriber's thread — what InvokeBlocking has always done —
and the leaf claims it when its prologue starts. The claim is a CAS with three
states because only one of the two may release: a leaf that has started still
touches _gate and _poolCts, so an unsubscribe releasing the region under it would
reopen the hole #2146 closed; a leaf that never started is released by the
unsubscribe, so a dropped subscription cannot leak a region and park Disposed.

Unchanged: the work still runs on a pool thread (only the admission moved), a
refusal still leaves on a pool thread (#4530), and the region still ends with the
SETUP LEAF for SubscribeThroughPool rather than with the long-lived subscription.
New cost: Disposed now waits for a leaf accepted but not yet started, which is the
correct direction and is bounded by the ThreadPool reaching the prologue.

IoPoolAcceptedWorkIsAccountedTest parks a prologue on the new
OnLeafPrologueStarting seam — so the leaf is inside the window by construction
rather than by racing a ThreadPool hop — and asserts both halves of the contract:
the drain spends its grace, and the leaf it has to cancel is counted. Red on main
for both entry points at "Expected 0 to be greater than 0".

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Copilot AI lite review requested due to automatic review settings September 17, 2026 06:56

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

It changes core IoPool concurrency/admission semantics on hot entry points, so it warrants final human review despite good targeted test coverage.

Pull request overview

This PR fixes a teardown-accounting hole in IoPool where work could be “accepted” (the caller’s Subscribe() returned) but still be invisible to Drain() until the ThreadPool ran the leaf prologue, allowing drains to cancel that work without it being counted or named (issue #4555). It moves the admission region to be taken synchronously on the subscriber’s thread (while still running the actual work on the pool thread), adds a small CAS-based ownership protocol to ensure the region is released exactly once, and pins the behavior with a deterministic regression test plus documentation.

Changes:

  • Take the IoPool admission region synchronously at subscribe time for Invoke, InvokeStream, and SubscribeThroughPool’s setup leaf, eliminating the “Subscribe-returned-but-unaccounted” window.
  • Add a test seam (OnLeafPrologueStarting) and a regression test that deterministically holds a leaf in the window and asserts Drain() spends grace and counts the cancellation.
  • Document the “admission on subscriber thread” rule and the rationale/protocol in ControlledIoPooling.md.
File summaries
File Description
test/MeshWeaver.Hosting.Test/IoPoolAcceptedWorkIsAccountedTest.cs Adds a deterministic regression test for #4555 using the new prologue seam.
src/MeshWeaver.Mesh.Contract/Threading/IoPool.cs Moves admission to the subscriber thread and implements single-release ownership (leaf vs unsubscribe) for affected entry points.
src/MeshWeaver.Documentation/Data/Architecture/ControlledIoPooling.md Documents the admission-thread rule, the failure mode, and the CAS protocol used to fix it.
Review details
  • Files reviewed: 3/3 changed files
  • Comments generated: 0
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@github-actions

Copy link
Copy Markdown
Contributor

Test Results (shard 0)

264 tests   264 ✅  2m 12s ⏱️
  1 suites    0 💤
  1 files      0 ❌

Results for commit adf3a79.

@github-actions

Copy link
Copy Markdown
Contributor

Test Results (shard 1)

533 tests   533 ✅  1m 11s ⏱️
  1 suites    0 💤
  1 files      0 ❌

Results for commit adf3a79.

@github-actions

Copy link
Copy Markdown
Contributor

Test Results (shard 3)

1 397 tests   1 397 ✅  3m 5s ⏱️
    2 suites      0 💤
    2 files        0 ❌

Results for commit adf3a79.

@github-actions

Copy link
Copy Markdown
Contributor

Test Results (shard 4)

    2 files      2 suites   5m 6s ⏱️
1 763 tests 1 763 ✅ 0 💤 0 ❌
1 764 runs  1 764 ✅ 0 💤 0 ❌

Results for commit adf3a79.

@github-actions

Copy link
Copy Markdown
Contributor

Test Results (shard 5)

    4 files      4 suites   5m 33s ⏱️
2 271 tests 2 271 ✅ 0 💤 0 ❌
2 275 runs  2 275 ✅ 0 💤 0 ❌

Results for commit adf3a79.

@github-actions

Copy link
Copy Markdown
Contributor

Test Results (shard 2)

1 764 tests   1 764 ✅  11m 46s ⏱️
    3 suites      0 💤
    3 files        0 ❌

Results for commit adf3a79.

@github-actions

Copy link
Copy Markdown
Contributor

Test Results

   13 files     13 suites   28m 55s ⏱️
7 992 tests 7 992 ✅ 0 💤 0 ❌
7 997 runs  7 997 ✅ 0 💤 0 ❌

Results for commit adf3a79.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

IoPool: a leaf between Subscribe() and its ThreadPool prologue is invisible to Drain()'s grace

2 participants