Repository navigation
fix(mesh): a registration probe no longer reports itself as a transient ERROR - #3004
Conversation
…nt ERROR
Every mesh start wrote one Error-level line per static NodeType whose HubConfiguration
installs an Activity Control Plane:
Error MeshWeaver.Kernel.Hub.KernelContainer: ActivityControlPlane subscription faulted
on content-type-registration/65936dcbcfa348bd921eb49885dea848 — re-establishing
<< InvalidOperationException: Failed to create stream
Nothing was wrong with the mesh. That address is the boot-time content-type sweep's own
throwaway probe (ContentTypeRegistration.ProbeRegister), so the line was the sweep
reporting itself — at the severity operators alert on, once per swept type, forever.
ROOT CAUSE: a transient node probe was running the node control plane.
AsTransientNodeProbe states outright that a probe must not, and MeshDataSource honours it
(SubscribeToOwnDeletionInit returns early on a probe). But the ACP is installed by the
ADOPTER — KernelContainer and every Activity-shaped NodeType, from their own
WithInitialization — so a guard placed per-adopter is a guard the next adopter will not
have, and the ACP went on being installed on every probe. There it subscribes to the hub's
own MeshNode stream; the probe is built startDataSources:false, so that reduce has no
started data source and Workspace.ReduceLocalStream threw a causeless
InvalidOperationException("Failed to create stream"). SubscribeWithReEstablish's three
terminal classifiers (HubDisposingException on this address, a deserialization
MeshNodeStreamException, a routing NotFound) all miss a bare InvalidOperationException, so
it fell through to "genuinely transient": LogError plus a 1 s re-establish timer armed
against a hub already being disposed — the #991 TimerQueue-root shape, safe today only by
the ordering accident that RegisterForDisposal disposes a late registrant immediately.
Same root, second face: an Activity NodeType that ALSO declares a content type got far
enough for the reduce to construct a SynchronizationStream, whose constructor always calls
GetHostedHub(sync/{id}, Always) — into the probe's own disposal, re-creating exactly the
"Rejecting hosted hub creation … during disposal" warning that startDataSources:false was
introduced to remove.
THE FIX — at the seams, never per adopter, and never a quieter log level.
1. WatchControlPlane / WatchSubmission install nothing on a hub carrying the
TransientNodeProbe marker. This is the shared seam every ACP adopter goes through.
2. MeshNodeStreamHandle.Subscribe answers a probe's OWN-node stream with an EMPTY stream.
This is the THIRD own-node read seam and the last one that was not saying "there is no
node here": GetMeshNodeOutcome answers Absent (#2468) and MeshNodeStreamCache.
GetStreamRaw answers empty (#2894). It covers watchers that compose their own source
rather than going through (1) — BuildNodeType's claim arbiter is the live example, and
it was the remaining sync/ sub-hub after (1) alone.
3. ProbeRegister mints its address from TransientProbeAddresses instead of a literal, and
IsProbeAddress learns the prefix. The registration probe was a transient probe by every
other measure while the one guard keyed off the ADDRESS did not know it existed: a read
of one of its paths died on "No node found at 'content-type-registration/…'" — #2894,
re-created by one string literal.
4. ReduceLocalStream's exception names the reference and the owning hub. A diagnostic that
names no reference, no owner and no cause is why three deliberate classifiers missed it.
Deliberately NOT the fix: downgrading the Error. It would have left the re-establish timer
armed and the sub-hub still being created, and "transient" keeps meaning what it meant —
the node is alive and will come back.
EVIDENCE. ContentTypeProbeControlPlaneTest starts a real monolith mesh, registers two
Activity-shaped static NodeTypes (one bare like `kernel`, one declaring a content type),
and captures the mesh's own ILoggerFactory through an ILoggerProvider registered via
MeshBuilder.ConfigureServices — what a deployment's log sink receives, not test output.
Ordering is a positive signal, not a hope: each NodeType reports from inside its own
WithInitialization once WatchControlPlane has RETURNED, and SubscribeWithReEstablish
establishes synchronously, so by then the line has been written or provably will not be.
On origin/main, both facts fail — reproducing the reported line verbatim, twice (from the
platform's own `kernel` type and from the test's), plus 3 sync/-into-disposal warnings.
With (1) only: the Error fact passes, 1 warning remains (the claim arbiter). With (1)+(2):
both pass. The third fact was falsified by reverting (3) alone: it fails with
"DeliveryFailureException: No node found at 'content-type-registration/…'".
Closes #2990
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SD7aiLSo59xng2TzU32xEa
There was a problem hiding this comment.
Pull request overview
This PR fixes a noisy boot-time false alarm where transient probe hubs (e.g., content-type-registration/{guid}) inadvertently ran Activity Control Plane (ACP) watchers and reported their own missing own-node stream as a transient Error, including re-establish timers on a hub being disposed.
Changes:
- Skip installing ACP watchers (
WatchControlPlane/WatchSubmission) when the hub is marked as aTransientNodeProbe. - Make a transient probe’s own-node stream (
workspace.GetMeshNodeStream()subscribe path) complete empty instead of throwing, and add the content-type registration probe prefix toTransientProbeAddresses. - Add a regression test and update docs/What’s New to explain the probe contract and the removed boot-time error.
Reviewed changes
Copilot reviewed 10 out of 10 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| test/MeshWeaver.Graph.Test/ContentTypeProbeControlPlaneTest.cs | Adds regression tests that assert no ACP fault logs / no sync/ sub-hub creation on registration probes. |
| src/MeshWeaver.Mesh.Contract/MeshNodeStreamExtensions.cs | Completes a probe’s own-node stream as empty on subscribe to avoid false transient faults. |
| src/MeshWeaver.Mesh.Contract/ActivityControlPlaneExtensions.cs | Adds a shared guard to avoid installing ACP watchers on transient probe hubs. |
| src/MeshWeaver.Hosting/MeshNodeStreamCache.cs | Updates commentary to reflect the “three seams” model for own-address reads. |
| src/MeshWeaver.Graph/ContentTypeRegistrationSweep.cs | Mints probe addresses from TransientProbeAddresses instead of a literal prefix. |
| src/MeshWeaver.Documentation/Data/WhatsNew/2026-09-02-a-clean-start-no-longer-reports-an-error.md | Adds a What’s New entry describing the removed boot-time error noise. |
| src/MeshWeaver.Documentation/Data/Architecture/TransientNodeProbes.md | Documents the registration probe and the three own-address read seams and watcher rules. |
| src/MeshWeaver.Documentation/Data/Architecture/ActivityControlPlane.md | Documents probe behavior and links to transient probe rules. |
| src/MeshWeaver.Data/Workspace.cs | Improves ReduceLocalStream failure diagnostic to include reference + owning hub. |
| src/MeshWeaver.Data/TransientNodeProbe.cs | Adds ContentTypeRegistrationProbePrefix and extends IsProbeAddress. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Test Results (shard 3)586 tests 586 ✅ 52s ⏱️ Results for commit 2748c3f. |
Test Results (shard 0)216 tests 216 ✅ 2m 29s ⏱️ Results for commit 2748c3f. |
Test Results (shard 5) 3 files 3 suites 1m 16s ⏱️ Results for commit 2748c3f. |
Test Results (shard 2)529 tests 527 ✅ 1m 31s ⏱️ Results for commit 2748c3f. |
Test Results (shard 1)419 tests 419 ✅ 1m 49s ⏱️ Results for commit 2748c3f. |
Test Results (shard 4)1 690 tests 1 690 ✅ 1m 51s ⏱️ Results for commit 2748c3f. |
Test Results 14 files 14 suites 9m 50s ⏱️ Results for commit 2748c3f. |
|
Review finding acted on in #3007 — it arrived after auto-merge had already taken this PR, so it lands as a follow-up rather than a push here. |
Fixes #2990.
The symptom
Every mesh start wrote one Error-level line per static NodeType whose
HubConfigurationinstalls an Activity Control Plane:
content-type-registration/{guid}is the boot-time content-type sweep's own throwaway probe, sothe line was that sweep reporting itself — at the severity operators alert on, once per swept type,
on every boot.
Verified root cause — issue candidate (a): the probe was running the node control plane
AsTransientNodeProbestates outright that a probe must not run the per-node control plane, andMeshDataSourcehonours it (SubscribeToOwnDeletionInitreturns early on a probe). But the ACP isinstalled by the adopter —
KernelContainerand every Activity-shaped NodeType, from their ownWithInitialization— so a guard placed per-adopter is a guard the next adopter will not have, andthe ACP went on being installed on every probe.
There it subscribes to the hub's own MeshNode stream. The probe is built
startDataSources: false,so that reduce has no started data source and
Workspace.ReduceLocalStreamthrew a causelessInvalidOperationException("Failed to create stream").SubscribeWithReEstablish's three terminalclassifiers all miss a bare
InvalidOperationException, so it fell through to genuinelytransient:
LogError+ a 1 s re-establish timer armed against a hub already being disposed (the#991
TimerQueue-root shape, safe today only by the ordering accident thatRegisterForDisposaldisposes a late registrant immediately).
Same root, second face. An Activity NodeType that also declares a content type got far enough
for the reduce to construct a
SynchronizationStream, whose constructor always callsGetHostedHub(sync/{id}, Always)— into the probe's own disposal, re-creating exactly theRejecting hosted hub creation … during disposalwarning thatstartDataSources: falsewasintroduced to remove.
The fix — at the seams, never per adopter, and never a quieter log level
WatchControlPlane/WatchSubmissioninstall nothing on a hub carrying theTransientNodeProbemarkerMeshNodeStreamHandle.Subscribeanswers a probe's own-node stream with an empty streamGetMeshNodeOutcomeanswersAbsent(#2468),MeshNodeStreamCache.GetStreamRawanswers empty (#2894). Covers watchers that compose their own source rather than going through (1) —BuildNodeType's claim arbiter is the live example, and was the remainingsync/sub-hub after (1) aloneProbeRegistermints its address fromTransientProbeAddresses;IsProbeAddresslearns the prefixNo node found at 'content-type-registration/…', i.e. #2894 re-created by one string literalReduceLocalStream's exception names the reference and the owning hub🚨 Deliberately not the fix: downgrading the Error. That would have left the re-establish timer
armed and the sub-hub still being created. "Transient" keeps meaning exactly what it meant — the
node is alive and will come back.
Test that fails before and passes after
test/MeshWeaver.Graph.Test/ContentTypeProbeControlPlaneTest.csstarts a real monolith mesh(
MonolithMeshTestBase, no mocking), registers two Activity-shaped static NodeTypes — one bare likethe platform's own
kerneltype, one declaring a content type — and captures the mesh's ownILoggerFactorythrough anILoggerProviderregistered viaMeshBuilder.ConfigureServices: whata deployment's log sink receives, not test output.
Ordering is a positive signal, not a hope. Each NodeType reports into an
AsyncSubjectfrom insideits own
WithInitialization, onceWatchControlPlanehas returned — andSubscribeWithReEstablishestablishes synchronously, withMeshNodeStreamHandle.Subscribereporting a failed
AcquireStreamthroughOnErroron the calling thread. By the time the signalarrives the line has been written or provably will not be. (An assertion on a log's absence with no
such signal passes whenever it runs first — which is most of the time.)
…LogsNoControlPlaneFault…OpensNoSyncSubHubOnAProbeStreamCache_ReadOfARegistrationProbeAddress_CompletesEmptyorigin/mainkerneltype, one from the test's)sync/-into-disposal warningsDeliveryFailureException: No node found at 'content-type-registration/…'The third row's third column was falsified independently by reverting only (3) and re-running.
Docs
Doc/Architecture/TransientNodeProbesgains the registration probe in its producer table, thethird own-address read seam, and a section on the own-node watchers with the boot-time Error it
removed;
Doc/Architecture/ActivityControlPlanegains the probe rule, a cross-link, and the missingfourth fault class its table never listed. What's New entry:
Category: Fix.🤖 Generated with Claude Code
https://claude.ai/code/session_01SD7aiLSo59xng2TzU32xEa