Cloud probe budget test fails when a starved thread pool fires the budget timer late #4741

Claude posting for Erik Darling

DarlingCloudIdentityProbeTests.ProbeAsync_NeitherCloudResponds_ReturnsNone_WithinItsOwnBudget failed twice on 2026-09-29. It checks that the probe returns in under 2000 ms when no metadata endpoint answers.

The probe code has not changed since #4330. The test uses a fake handler, so no network is involved. It fails near 2 s, not at random lengths.

What the code does

• DarlingCloudIdentityProbe.ProbeAsync(HttpMessageHandler, CancellationToken) makes one linked token source and calls CancelAfter(ProbeBudget), where ProbeBudget is 200 ms.
• The EC2 attempt and then the Azure attempt both use that same token. So running the two clouds one after the other does not add up past the budget.
• The fake handler waits on Task.Delay(Timeout.InfiniteTimeSpan, cancellationToken), so it ends only when that token is cancelled.

Suspected causes

1. Thread-pool starvation. CancelAfter fires from a timer callback. That callback needs a free thread-pool thread, and so do the continuations after it. When other tests in the same run block pool threads, the pool adds new threads slowly. A delay of close to 2 s fits that.
2. A second wait that the probe's token does not bound. Places to look: a retry, TryEc2Async, TryAzureAsync, an HttpClient default. A first read found none, but this needs a careful check.

Steps

1. Write the test first. Block a set number of pool threads, then run the budget test. Show that it goes past 2 s the same way CI did.
2. Read TryEc2Async, TryAzureAsync and ReadCappedValueAsync for any wait that budget.Token does not bound.
3. If starvation is the cause, change the test, not the product. One way is to assert that the handler saw its own cancellation within the budget plus some slack. The other is to move the test into a collection that does not run in parallel.
4. If the probe has a real second wait, fix the product. Add a Fixed entry to the CHANGELOG.
5. Once the cause is known, rename this issue after it.