[jeremydmiller]

Follow-up to #3796, as discussed on merge.

The problem

DrainWaitForPrefetch completes the consumer's BatchingChannel inside RabbitMqListener.StopAsync. But StopAsync is not always a terminal shutdown — RequeueContinuation.cs:47-49 stops the listener inline from the handler pipeline, and the background PauseAsync then stops the same consumer again before disposing it. So a durable listener genuinely sees two StopAsync calls against one channel.

I measured what a JasperFx BatchingChannel actually does after Complete() rather than assuming:

call after Complete()	behavior
TriggerBatch()	no-op, no throw
Complete()	no-op, no throw
WaitForCompletionAsync()	no-op, no throw
PostAsync(item)	accepted and silently discarded

Nothing throws — so the duplicate stop is harmless but not free. The real problem is the last row. A delivery landing between Complete() and Dispose's latch is posted into a completed channel and silently swallowed: it never reaches the receiver and is never acked, so the broker redelivers it on channel close. That is precisely the redelivery DrainWaitForPrefetch exists to prevent, reached through a silent path.

The fix

DrainBatchedDeliveriesAsync now latches before completing, so such a delivery is rejected-with-requeue through the guard already at the top of HandleBasicDeliverAsync instead of vanishing. Safe because the drain only runs after cancel-ok, and the broker sends no further deliveries on a cancelled consumer.

The drain is also idempotent now, so the second stop doesn't re-run the cancel-ok wait against an already-drained channel, and Dispose skips a TriggerBatch that would do nothing.

For reference, SqsListener sidesteps this ordering entirely — its StopAsync calls only TriggerBatch, with Complete reserved for the dispose-time flushPendingDeletesAsync, and a comment naming the paused-listener case. The Rabbit drain has to complete earlier than dispose in order to beat LatchReceiver, so it tracks the state instead.

On the test

drain_wait_for_prefetch_repeated_stop pins the double-stop sequence end to end. Being straight about it: it is a regression guard, not a red-first repro — I verified it passes both with and without the fix, because the channel tolerates the duplicate calls. What it protects is that everything still drains when StopAsync runs twice, which a naive "just guard the second Complete()" fix would break. The silent-swallow window itself needs a delivery to land inside a sub-millisecond race, which I didn't think was worth building a flaky test around.

Unrelated flake, partially addressed — folded in

multi_tenancy_through_virtual_hosts.send_message_to_a_specific_tenant waited only for the request to reach node two, then asserted on the response arriving back at main. A local full-suite trace caught the session completing in the same millisecond node two Sent the response, so SingleRecord had nothing to find. It now waits for the response too.

Two caveats, because this file carries a standing GH-3763 note I should have read before calling it fixed:

• That note records a prior inference from CI run 30856898284 that "the request never produced its response". My trace shows the opposite — the response was produced, the session just stopped before it landed. I've updated the note so the file doesn't argue with itself.
• The note's real subject — suspected interference from the fixture's fixed queue names (Queue1, multi_response, global_response, multi_incoming) in the shared default vhost — is untouched by this. If it bites, the symptom changes from "no messages of type MultiTenantResponse" to a 15s timeout.

So: one contributing race identified and removed, not a chronic flake eliminated.

Run	Result
Rabbit suite, before	501/502 and 498/499 — same test failing
Rabbit suite, after	502/502 — one clean run, which is evidence, not proof
dotnet build wolverine.slnx -c Release -f net9.0	clean

Included here rather than split out because it failed two consecutive full-suite runs and would otherwise redden this PR's own CI.

Not addressed here

RequeueContinuation's inline StopAsync() can now block a handler for up to DurabilitySettings.DrainTimeout (default 30s) when the flag is on, waiting for cancel-ok. It is not a deadlock — the batch flush persists and enqueues rather than waiting on the handler pipeline — but it is latency on a path whose own comments explain it was carefully built not to wait. Worth a separate look at whether that call site should opt out of the drain.

🤖 Generated with Claude Code

https://claude.ai/code/session_01JG8Un6iNeyXECKJk3jo5uC