Repository navigation
GetMeshNode timeout: probe the target hub too, not just the reader - #663
Merged
Merged
Conversation
The first CI occurrence with the loud timeout proved the reader is innocent: Reader RunLevel=Started, Queue(buffer=0,deferred=0,exec=0), no Executing clause at all, its only outstanding work our own read. So the request left the reader, was routed, and the owner produced no reply in 60s. What the message could not distinguish is WHY: 'owner never activated' vs 'owner answered and the reply was lost'. This probes the target with GetHostedHub(address, HostedHubCreation.Never) — a pure dictionary lookup that must never activate the hub as a side effect of diagnosing it (RoutingServiceBase uses the same probe) — and reports one of three outcomes: no local hub (never activated here, or owned by another silo), the target's own RunLevel/queue/ executing/pending snapshot (activated but stuck, and on what), or 'this hub itself' for a self-read. Either of the first two closes the diagnosis on the next occurrence. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Improves GetMeshNode timeout diagnostics so a stalled read can better distinguish “reader is idle” from “target hub never activated / is stuck,” helping narrow down activation vs. reply-loss scenarios in CI and production.
Changes:
- Extend the
GetMeshNodetimeout message to include both reader-hub and target-hub diagnostic snapshots. - Add test assertions ensuring the surfaced timeout message includes
Reader:andTarget:labels.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| test/MeshWeaver.Hosting.Monolith.Test/GetMeshNodeTimeoutSurfacingTest.cs | Adds assertions that timeout messages label reader vs. target diagnostics. |
| src/MeshWeaver.Mesh.Contract/MeshNodeStreamExtensions.cs | Adds target-hub probing to the timeout message alongside existing reader diagnostics. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+1431
to
+1444
| string targetState; | ||
| try | ||
| { | ||
| targetState = string.Equals(hub.Address.ToString(), path, StringComparison.Ordinal) | ||
| ? "Target: this hub itself." | ||
| : hub.GetHostedHub(new Address(path), HostedHubCreation.Never) is { } owner | ||
| ? $"Target: {owner.GetPendingRequestDiagnostics()}" | ||
| : $"Target: NO LOCAL HUB at '{path}' — it never activated here (or is owned " | ||
| + "by another silo), so no reply was ever going to be produced."; | ||
| } | ||
| catch (Exception targetEx) | ||
| { | ||
| targetState = $"<target diagnostics unavailable: {targetEx.GetType().Name}>"; | ||
| } |
Contributor
Contributor
Test Results (shard 2)1 129 tests 1 026 ✅ 5m 33s ⏱️ Results for commit 0956091. |
Contributor
Contributor
Test Results (shard 5)1 138 tests 1 137 ✅ 6m 4s ⏱️ Results for commit 0956091. |
Contributor
Contributor
Contributor
This was referenced Jul 27, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #661, driven by what its first CI occurrence actually said.
What #661's diagnostic proved
Verbatim from the trx of run 30242205261 (
ThreadAgentIntegrationTest.cs:280— theReadNodecall itself, 60.49 s):The reader is exonerated, precisely:
RunLevel=Started, all queues empty, and noExecuting(...)clause at all — so it wasn't congested, blocked or starved, and its only outstanding work was our own read. The request left the reader, was routed, and the owner produced no reply in 60 s.What the message can't yet distinguish is why: owner never activated versus owner answered and the reply was lost. Those need completely different fixes.
The change
The timeout branch now probes the target as well, via
GetHostedHub(address, HostedHubCreation.Never)— a pure dictionary lookup that must never activate the hub as a side effect of diagnosing it (RoutingServiceBasealready uses exactly this probe). The message becomesReader: … Target: …with three outcomes:Target: NO LOCAL HUB at 'X' — it never activated here (or is owned by another silo), so no reply was ever going to be produced.Target: Hub X RunLevel=… Queue(…) Executing(…) PendingCallbacks=…→ activated but stuck, and on what.Target: this hub itself.→ self-read, so no false "no local hub" claim.Either of the first two closes the diagnosis on the next occurrence.
Two assertions added to
GetMeshNodeTimeoutSurfacingTest(Reader:/Target:). Verified locally: Release-p:CIRun=true -warnaserrorclean;MeshWeaver.AI.Test889/894 (5 skipped, 0 failed);Hosting.Monolith.Test398/400 (2 skipped, 0 failed).Where this points
The timing profile in that run puts the money on cold activation, not a dropped reply: the two sibling tests in the same class were already 5–10× slower than a healthy run before the failing one executed (11.69 s and 3.72 s vs 2.09 s and 0.83 s). Every
[Fact]builds a fresh mesh with a fresh assembly-store root, so the class activatesACME/ProductLaunch— anACME/Projectnode requiring a Roslyn compile — three times over, on a 4-core runner. That also explainsSwitchThread_IsolatesConversationStatefailing the same way on a different PR: same class, same path, whichever method draws the short straw pays the cost.No What's New entry — diagnostics only, no user-visible behaviour change.
🤖 Generated with Claude Code