Repository navigation
Defer the probe's query legs: a subscribe-time throw was killing the test host - #717
Merged
Merged
Conversation
…test host
Root cause of the CI "exit=2 with an all-green trx" shard failures — pinned
from the failing run's own logs, not inferred:
[FATAL ERROR] System.ObjectDisposedException
at AutofacServiceProvider.GetService ← disposed scope
at MeshService.CaptureContext (MeshService.cs:62)
at MeshService.Query[T] (MeshService.cs:257)
at MeshNodeCompilationService.DirectSourceProbe b__2 (:618)
at CombineLatest.Run ← SubscribeRaw ← DelaySubscription ← ThreadPool
The probe's per-leg `mesh.Query(...)` lambda is evaluated inside CombineLatest's
SUBSCRIPTION, and the probe's subscription is SCHEDULED (DelaySubscription → a
ThreadPool tick). When the mesh is disposed between scheduling and firing,
CaptureContext throws synchronously on the dead Autofac scope — and Rx routes a
synchronous throw during a Producer's subscribe to the SUBSCRIBE CALLER, not to
OnError, so neither the leg's .Catch nor the probe's outer .Catch ever sees it.
The caller is the scheduler → unhandled on the ThreadPool → xUnit v3's
AppDomain handler reports "[FATAL ERROR]", waits for the run to finish
(finishEvent.Wait() — which is why the summary prints all green, Errors: 0),
then Environment.Exit(2). Our TeardownStragglerCapturer never sees it: xUnit's
handler subscribed first and exits the process before later handlers run.
This is why the exit=2 transients began exactly when #690 made this probe run
on every compile (before, it was a delay-subscribed fallback that almost never
fired): #708 attempt 1 (AI.Test + Acme.Test, both compile-heavy), attempt 2
green on the identical commit (won the timer race), #716 (base includes #690).
Fix: wrap each leg in Observable.Defer. Defer's FACTORY exceptions ARE
forwarded to OnError, so the throw lands in the leg's existing .Catch and the
leg degrades to empty — the exact behaviour the probe already intends for a
faulted leg ("NEVER win the race with an EMPTY set… stay silent").
No deterministic in-repo repro: it needs the mesh container to die between the
probe's schedule and its tick, a window the test base's drain now also narrows
(#715). The CI FATAL stack above is the repro; the fix is the Rx-contract
correction that makes the window harmless.
Verified: CompileLeafStabilityTest 2/2 + CompileSourceSnapshotWedgeTest 1/1
(the stalled-snapshot path), Graph builds clean under -c Release -warnaserror.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
This PR fixes a teardown-time reliability issue in the Graph compilation source probe where a scheduled subscription could run after the mesh/DI scope is disposed, causing a synchronous ObjectDisposedException to escape Rx error handling and crash the xUnit host (CI exit=2 despite all-green TRX).
Changes:
- Wrap each per-query “leg” inside
DirectSourceProbewithObservable.Deferso any synchronous exceptions thrown during subscription are surfaced viaOnErrorand handled by the existing per-leg.Catch(...)fallback. - Add an in-code explanation of the Rx contract nuance (subscribe-time throws escaping to the subscribe caller) and why per-leg deferral is required even though the outer probe is already delayed.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Contributor
Test Results (shard 2)1 129 tests 1 026 ✅ 5m 10s ⏱️ Results for commit a1c2433. |
Contributor
Contributor
Test Results (shard 0)959 tests 953 ✅ 4m 44s ⏱️ Results for commit a1c2433. |
Contributor
Contributor
Contributor
Contributor
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Root cause of the CI
exit=2with an all-green trx shard failures — pinned from the failing run's own logs (#716's run), not inferred:Mechanism
The probe's per-leg
mesh.Query(...)lambda is evaluated insideCombineLatest's subscription, and the probe's subscription is scheduled (DelaySubscription→ a ThreadPool tick). When the mesh is disposed between scheduling and firing,CaptureContextthrows synchronously on the dead Autofac scope — and Rx routes a synchronous throw during aProducer's subscribe to the subscribe caller, not toOnError, so neither the leg's.Catchnor the probe's outer.Catchever sees it. The caller is the scheduler → unhandled on the ThreadPool → xUnit v3'sAppDomainhandler reports[FATAL ERROR], waits for the run to finish (finishEvent.Wait()— which is why the summary prints all green,Errors: 0), thenEnvironment.Exit(2).TeardownStragglerCapturernever sees it: xUnit's handler subscribed first and exits the process before later handlers run.This is why the
exit=2transients began exactly when #690 made this probe run on every compile (previously a delay-subscribed fallback that almost never fired): #708 attempt 1 (AI.Test+Acme.Test, both compile-heavy), attempt 2 green on the identical commit (won the timer race), #716 (base includes #690, its own diff is fixture-only).Fix
Wrap each leg in
Observable.Defer.Defer's factory exceptions are forwarded toOnError, so the throw lands in the leg's existing.Catchand the leg degrades to empty — the exact behaviour the probe already intends for a faulted leg ("NEVER win the race with an EMPTY set… stay silent").No deterministic in-repo repro: it needs the container to die between the probe's schedule and its tick — a window #715's construction drain also narrows. The CI FATAL stack above is the repro; the fix is the Rx-contract correction that makes the window harmless.
Verification
CompileLeafStabilityTest2/2 +CompileSourceSnapshotWedgeTest1/1 (the stalled-snapshot path Compile discovery: read sources directly instead of through the live synced query #690 exists for)MeshWeaver.Graphbuilds clean under-c Release -warnaserrorNo What's New entry: internal reliability fix, no user-visible change.
Related: #690 (made the window hot), #708/#716 (the runs it redded), #613 (the caught-straggler class is adjacent but distinct — this one was the unhandled escape).
🤖 Generated with Claude Code