Skip to content

fix(mesh): a registration probe no longer reports itself as a transient ERROR - #3004

Merged
rbuergi merged 1 commit into
mainfrom
fix/2990-probe-does-not-arm-a-reestablish
Sep 1, 2026
Merged

rbuergi merged 1 commit into
mainfrom
fix/2990-probe-does-not-arm-a-reestablish

Conversation

@rbuergi

@rbuergi rbuergi commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Fixes #2990.

The symptom

Every mesh start wrote one Error-level line per static NodeType whose HubConfiguration
installs an Activity Control Plane:

Error MeshWeaver.Kernel.Hub.KernelContainer: ActivityControlPlane subscription faulted on
content-type-registration/65936dcbcfa348bd921eb49885dea848 — re-establishing
   << InvalidOperationException: Failed to create stream

content-type-registration/{guid} is the boot-time content-type sweep's own throwaway probe, so
the line was that sweep reporting itself — at the severity operators alert on, once per swept type,
on every boot.

Verified root cause — issue candidate (a): the probe was running the node control plane

AsTransientNodeProbe states outright that a probe must not run the per-node control plane, and
MeshDataSource honours it (SubscribeToOwnDeletionInit returns early on a probe). But the ACP is
installed by the adopter — KernelContainer and every Activity-shaped NodeType, from their own
WithInitialization — so a guard placed per-adopter is a guard the next adopter will not have, and
the ACP went on being installed on every probe.

There it subscribes to the hub's own MeshNode stream. The probe is built startDataSources: false,
so that reduce has no started data source and Workspace.ReduceLocalStream threw a causeless
InvalidOperationException("Failed to create stream"). SubscribeWithReEstablish's three terminal
classifiers all miss a bare InvalidOperationException, so it fell through to genuinely
transient
: LogError + a 1 s re-establish timer armed against a hub already being disposed (the
#991 TimerQueue-root shape, safe today only by the ordering accident that RegisterForDisposal
disposes a late registrant immediately).

Same root, second face. An Activity NodeType that also declares a content type got far enough
for the reduce to construct a SynchronizationStream, whose constructor always calls
GetHostedHub(sync/{id}, Always) — into the probe's own disposal, re-creating exactly the
Rejecting hosted hub creation … during disposal warning that startDataSources: false was
introduced to remove.

The fix — at the seams, never per adopter, and never a quieter log level

# Change Why here
1 WatchControlPlane / WatchSubmission install nothing on a hub carrying the TransientNodeProbe marker the shared seam every ACP adopter goes through
2 MeshNodeStreamHandle.Subscribe answers a probe's own-node stream with an empty stream the THIRD own-node read seam, and the last one not saying "there is no node here" — GetMeshNodeOutcome answers Absent (#2468), MeshNodeStreamCache.GetStreamRaw answers empty (#2894). Covers watchers that compose their own source rather than going through (1) — BuildNodeType's claim arbiter is the live example, and was the remaining sync/ sub-hub after (1) alone
3 ProbeRegister mints its address from TransientProbeAddresses; IsProbeAddress learns the prefix the registration probe was a transient probe by every other measure while the one guard keyed off the ADDRESS did not know it existed — a read of one of its paths died on No node found at 'content-type-registration/…', i.e. #2894 re-created by one string literal
4 ReduceLocalStream's exception names the reference and the owning hub issue candidate (b): a diagnostic naming no reference, no owner and no cause is why three deliberate classifiers missed it

🚨 Deliberately not the fix: downgrading the Error. That would have left the re-establish timer
armed and the sub-hub still being created. "Transient" keeps meaning exactly what it meant — the
node is alive and will come back
.

Test that fails before and passes after

test/MeshWeaver.Graph.Test/ContentTypeProbeControlPlaneTest.cs starts a real monolith mesh
(MonolithMeshTestBase, no mocking), registers two Activity-shaped static NodeTypes — one bare like
the platform's own kernel type, one declaring a content type — and captures the mesh's own
ILoggerFactory
through an ILoggerProvider registered via MeshBuilder.ConfigureServices: what
a deployment's log sink receives, not test output.

Ordering is a positive signal, not a hope. Each NodeType reports into an AsyncSubject from inside
its own WithInitialization, once WatchControlPlane has returned — and
SubscribeWithReEstablish establishes synchronously, with MeshNodeStreamHandle.Subscribe
reporting a failed AcquireStream through OnError on the calling thread. By the time the signal
arrives the line has been written or provably will not be. (An assertion on a log's absence with no
such signal passes whenever it runs first — which is most of the time.)

Build …LogsNoControlPlaneFault …OpensNoSyncSubHubOnAProbe StreamCache_ReadOfARegistrationProbeAddress_CompletesEmpty
origin/main FAIL — 2 Error records, the reported line verbatim (one from the platform's kernel type, one from the test's) FAIL — 3 sync/-into-disposal warnings FAIL — DeliveryFailureException: No node found at 'content-type-registration/…'
with (1) only PASS FAIL — 1 warning left (the claim arbiter) —
with (1)+(2)+(3) PASS PASS PASS

The third row's third column was falsified independently by reverting only (3) and re-running.

Docs

Doc/Architecture/TransientNodeProbes gains the registration probe in its producer table, the
third own-address read seam, and a section on the own-node watchers with the boot-time Error it
removed; Doc/Architecture/ActivityControlPlane gains the probe rule, a cross-link, and the missing
fourth fault class its table never listed. What's New entry: Category: Fix.

🤖 Generated with Claude Code

https://claude.ai/code/session_01SD7aiLSo59xng2TzU32xEa

…nt ERROR

Every mesh start wrote one Error-level line per static NodeType whose HubConfiguration
installs an Activity Control Plane:

    Error MeshWeaver.Kernel.Hub.KernelContainer: ActivityControlPlane subscription faulted
    on content-type-registration/65936dcbcfa348bd921eb49885dea848 — re-establishing
    << InvalidOperationException: Failed to create stream

Nothing was wrong with the mesh. That address is the boot-time content-type sweep's own
throwaway probe (ContentTypeRegistration.ProbeRegister), so the line was the sweep
reporting itself — at the severity operators alert on, once per swept type, forever.

ROOT CAUSE: a transient node probe was running the node control plane.

AsTransientNodeProbe states outright that a probe must not, and MeshDataSource honours it
(SubscribeToOwnDeletionInit returns early on a probe). But the ACP is installed by the
ADOPTER — KernelContainer and every Activity-shaped NodeType, from their own
WithInitialization — so a guard placed per-adopter is a guard the next adopter will not
have, and the ACP went on being installed on every probe. There it subscribes to the hub's
own MeshNode stream; the probe is built startDataSources:false, so that reduce has no
started data source and Workspace.ReduceLocalStream threw a causeless
InvalidOperationException("Failed to create stream"). SubscribeWithReEstablish's three
terminal classifiers (HubDisposingException on this address, a deserialization
MeshNodeStreamException, a routing NotFound) all miss a bare InvalidOperationException, so
it fell through to "genuinely transient": LogError plus a 1 s re-establish timer armed
against a hub already being disposed — the #991 TimerQueue-root shape, safe today only by
the ordering accident that RegisterForDisposal disposes a late registrant immediately.

Same root, second face: an Activity NodeType that ALSO declares a content type got far
enough for the reduce to construct a SynchronizationStream, whose constructor always calls
GetHostedHub(sync/{id}, Always) — into the probe's own disposal, re-creating exactly the
"Rejecting hosted hub creation … during disposal" warning that startDataSources:false was
introduced to remove.

THE FIX — at the seams, never per adopter, and never a quieter log level.

1. WatchControlPlane / WatchSubmission install nothing on a hub carrying the
   TransientNodeProbe marker. This is the shared seam every ACP adopter goes through.
2. MeshNodeStreamHandle.Subscribe answers a probe's OWN-node stream with an EMPTY stream.
   This is the THIRD own-node read seam and the last one that was not saying "there is no
   node here": GetMeshNodeOutcome answers Absent (#2468) and MeshNodeStreamCache.
   GetStreamRaw answers empty (#2894). It covers watchers that compose their own source
   rather than going through (1) — BuildNodeType's claim arbiter is the live example, and
   it was the remaining sync/ sub-hub after (1) alone.
3. ProbeRegister mints its address from TransientProbeAddresses instead of a literal, and
   IsProbeAddress learns the prefix. The registration probe was a transient probe by every
   other measure while the one guard keyed off the ADDRESS did not know it existed: a read
   of one of its paths died on "No node found at 'content-type-registration/…'" — #2894,
   re-created by one string literal.
4. ReduceLocalStream's exception names the reference and the owning hub. A diagnostic that
   names no reference, no owner and no cause is why three deliberate classifiers missed it.

Deliberately NOT the fix: downgrading the Error. It would have left the re-establish timer
armed and the sub-hub still being created, and "transient" keeps meaning what it meant —
the node is alive and will come back.

EVIDENCE. ContentTypeProbeControlPlaneTest starts a real monolith mesh, registers two
Activity-shaped static NodeTypes (one bare like `kernel`, one declaring a content type),
and captures the mesh's own ILoggerFactory through an ILoggerProvider registered via
MeshBuilder.ConfigureServices — what a deployment's log sink receives, not test output.
Ordering is a positive signal, not a hope: each NodeType reports from inside its own
WithInitialization once WatchControlPlane has RETURNED, and SubscribeWithReEstablish
establishes synchronously, so by then the line has been written or provably will not be.

On origin/main, both facts fail — reproducing the reported line verbatim, twice (from the
platform's own `kernel` type and from the test's), plus 3 sync/-into-disposal warnings.
With (1) only: the Error fact passes, 1 warning remains (the claim arbiter). With (1)+(2):
both pass. The third fact was falsified by reverting (3) alone: it fails with
"DeliveryFailureException: No node found at 'content-type-registration/…'".

Closes #2990

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SD7aiLSo59xng2TzU32xEa
Copilot AI lite review requested due to automatic review settings September 1, 2026 19:23
@github-actions
github-actions Bot enabled auto-merge September 1, 2026 19:23
@rbuergi
rbuergi disabled auto-merge September 1, 2026 19:27

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes a noisy boot-time false alarm where transient probe hubs (e.g., content-type-registration/{guid}) inadvertently ran Activity Control Plane (ACP) watchers and reported their own missing own-node stream as a transient Error, including re-establish timers on a hub being disposed.

Changes:

  • Skip installing ACP watchers (WatchControlPlane / WatchSubmission) when the hub is marked as a TransientNodeProbe.
  • Make a transient probe’s own-node stream (workspace.GetMeshNodeStream() subscribe path) complete empty instead of throwing, and add the content-type registration probe prefix to TransientProbeAddresses.
  • Add a regression test and update docs/What’s New to explain the probe contract and the removed boot-time error.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
test/MeshWeaver.Graph.Test/ContentTypeProbeControlPlaneTest.cs Adds regression tests that assert no ACP fault logs / no sync/ sub-hub creation on registration probes.
src/MeshWeaver.Mesh.Contract/MeshNodeStreamExtensions.cs Completes a probe’s own-node stream as empty on subscribe to avoid false transient faults.
src/MeshWeaver.Mesh.Contract/ActivityControlPlaneExtensions.cs Adds a shared guard to avoid installing ACP watchers on transient probe hubs.
src/MeshWeaver.Hosting/MeshNodeStreamCache.cs Updates commentary to reflect the “three seams” model for own-address reads.
src/MeshWeaver.Graph/ContentTypeRegistrationSweep.cs Mints probe addresses from TransientProbeAddresses instead of a literal prefix.
src/MeshWeaver.Documentation/Data/WhatsNew/2026-09-02-a-clean-start-no-longer-reports-an-error.md Adds a What’s New entry describing the removed boot-time error noise.
src/MeshWeaver.Documentation/Data/Architecture/TransientNodeProbes.md Documents the registration probe and the three own-address read seams and watcher rules.
src/MeshWeaver.Documentation/Data/Architecture/ActivityControlPlane.md Documents probe behavior and links to transient probe rules.
src/MeshWeaver.Data/Workspace.cs Improves ReduceLocalStream failure diagnostic to include reference + owning hub.
src/MeshWeaver.Data/TransientNodeProbe.cs Adds ContentTypeRegistrationProbePrefix and extends IsProbeAddress.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread src/MeshWeaver.Data/Workspace.cs Resolved
@github-actions

github-actions Bot commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Test Results (shard 3)

586 tests   586 ✅  52s ⏱️
  2 suites    0 💤
  2 files      0 ❌

Results for commit 2748c3f.

@github-actions

github-actions Bot commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Test Results (shard 0)

216 tests   216 ✅  2m 29s ⏱️
  1 suites    0 💤
  1 files      0 ❌

Results for commit 2748c3f.

@github-actions

github-actions Bot commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Test Results (shard 5)

    3 files      3 suites   1m 16s ⏱️
1 218 tests 1 026 ✅ 192 💤 0 ❌
1 219 runs  1 027 ✅ 192 💤 0 ❌

Results for commit 2748c3f.

@github-actions

github-actions Bot commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Test Results (shard 2)

529 tests   527 ✅  1m 31s ⏱️
  4 suites    2 💤
  4 files      0 ❌

Results for commit 2748c3f.

@github-actions

github-actions Bot commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Test Results (shard 1)

419 tests   419 ✅  1m 49s ⏱️
  1 suites    0 💤
  1 files      0 ❌

Results for commit 2748c3f.

@github-actions

github-actions Bot commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Test Results (shard 4)

1 690 tests   1 690 ✅  1m 51s ⏱️
    3 suites      0 💤
    3 files        0 ❌

Results for commit 2748c3f.

@github-actions

github-actions Bot commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Test Results

   14 files     14 suites   9m 50s ⏱️
4 658 tests 4 464 ✅ 194 💤 0 ❌
4 659 runs  4 465 ✅ 194 💤 0 ❌

Results for commit 2748c3f.

@rbuergi
rbuergi merged commit 812f7dd into main Sep 1, 2026
27 checks passed
@rbuergi

rbuergi commented Sep 1, 2026

Copy link
Copy Markdown
Contributor Author

Review finding acted on in #3007 — it arrived after auto-merge had already taken this PR, so it lands as a follow-up rather than a push here. GetStream(params Type[] types) now names the resolved collections and the hub, matching ReduceLocalStream's contract.

@rbuergi
rbuergi deleted the fix/2990-probe-does-not-arm-a-reestablish branch October 10, 2026 13:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ActivityControlPlane watcher reports the boot-time content-type probe as a transient ERROR and arms a re-establish

2 participants