[rbuergi]

Refs #1549.

#1549's original defect (per-node hubs silently forwarding for 2 h 20 m) was fixed by #1990 / 5cd54d3f2 and verified on the deployed portal in August. The issue stayed open because later, unrelated release failures kept folding into it. This PR stops that folding.

Root cause

• fix(#1549): a release failure says WHICH failure, in the template #4932 gave each NodeTypeReleaseFailure class its own message template.
• But the incident watcher (MeshWeaver.Plugins StructuralLogIncidentIdentity.ComputeSiteFold) folds every fingerprint of one log site into one issue, and a site is category + EventId + exception type.
• NodeTypeRecompileExtensions.LogReleaseRefusal logged every class under one category, EventId 0 and no exception, so every class was the same site.
• As a result, NO NODE EXISTS (2026-09-24) and OWNING HUB REACHED NO VERDICT (2026-09-25 18:55:27Z) arrived on Per-node hubs stopped answering on memex-cloud for 2h20m — the NodeType release burst is the symptom that surfaced it #1549 as "a different fingerprint of the same log site 48a398f36ff176dd", taking it to 459 occurrences of causes it was never about.

Fix

LogReleaseRefusal logs each class under ReleaseFailureEventId(failure) = an explicit, append-only code per class (ReleaseFailureEventIds, 7400–7415), named NodeTypeRelease<Class>.

• The codes are never the enum ordinal (review), so inserting a class cannot renumber a deployed site.
• An undeclared value takes Unclassified's id, the same way it takes that class's template.
• Templates and levels are unchanged.
• The 7400 range is unused in core; the disposal verdicts use 73xx.

Test

test/MeshWeaver.Graph.Test/ReleaseFailureClassIsInTheTemplateTest.cs → EveryFailureClass_LogsUnderAnEventIdOfItsOwn.

• It asserts that no class logs under id 0, that the ids are distinct per class, and that each id's name is its class.
• Negative control: with the ids removed from the calls, the test fails, because every class logs under the one shared id.
• The class has 10 tests and all pass. MeshWeaver.Graph and MeshWeaver.Graph.Test build clean with -c Release -warnaserror.

Effect on incidents

Each class now opens its own incident site; ids are permanent explicit codes 7400–7415, pinned by TheEventIdOfEveryClass_IsPinned. The classes that are not defects (NodeMissing, CompileDenied, OwnerRecycling) stay at Error. Deciding their levels is a separate question, as the remarks in LogReleaseRefusal say.

Doc: Doc/Architecture/ReleaseFailureClasses → "…and its own EVENT ID, because a recurrence folds by SITE".

Pairs-with: none — no public surface removed (one internal helper and one internal const added).

🤖 Generated with Claude Code

[Copilot]

🟡 Changes recommended

Implicit enum ordinals can renumber or reuse deployed incident-site IDs when failure classes are inserted.

1 open finding

• Use explicit stable codes for failure class site IDs · New

What changed in this PR

Separates NodeType release failures into distinct incident sites using per-class event IDs.

Changes:

• Adds event IDs to release-refusal logs.
• Tests event-ID uniqueness and naming.
• Documents incident-site folding behavior.

File	Description
src/​MeshWeaver.Graph/​NodeTypeRecompileExtensions.cs	Assigns event IDs to failure logs.
test/​MeshWeaver.Graph.Test/​ReleaseFailureClassIsInTheTemplateTest.cs	Verifies IDs are nonzero and distinct.
src/​MeshWeaver.Documentation/​Data/​Architecture/​ReleaseFailureClasses.md	Documents event-ID-based separation.

🧠 Review effort: Balanced

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[rbuergi]

Agreed: an id is a deployed incident site, so it can't be derived from an ordinal. Fixed in 7d673cc.

• ReleaseFailureEventIds is an explicit, append-only mapping from each class to a permanent code in 7400–7415 (Unclassified=7400 … HostLeaving=7415). These are the values the derived scheme would have produced, so nothing that has already shipped moves. An unlisted or undeclared value logs under Unclassified's id, the same way it takes that class's template.
• TheEventIdOfEveryClass_IsPinned pins every class to its exact code and fails if any enum member has no entry. A class inserted mid-enum therefore can't take a historical site, and a new class has to be given its own code. EveryFailureClass_LogsUnderAnEventIdOfItsOwn still checks that the ids are distinct.
• Negative control: I removed HostLeaving's row from the mapping, and both id tests fail. The suite passes 11/11 with the row restored, and the build is clean with -c Release -warnaserror.
• The doc (ReleaseFailureClasses) now says the codes are permanent and append-only, and that they are never the ordinal.

[github-actions]

Test Results

   14 files   -    10     14 suites   - 10   40m 49s ⏱️ - 8m 38s
9 680 tests  - 1 468  9 680 ✅  - 1 275  0 💤  - 193  0 ❌ ±0 
9 693 runs   - 1 468  9 693 ✅  - 1 275  0 💤  - 193  0 ❌ ±0 

Results for commit 3adb660. ± Comparison against base commit 1007a35.

This pull request removes 1501 and adds 9 tests. Note that renamed tests count towards both.

   --- End of inner exception stack trace ---
   --- End of inner exception stack trace ---, expected: True)
   --- End of inner exception stack trace ---, isDenial: True)
 ---> (Inner Exception #1) MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [pg] did not emit an Initial within the query fan-in's 16s bound for query 'nodeType:NodeType' (user 'system'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.<---
 ---> (Inner Exception #1) MeshWeaver.Messaging.Hub.Test.InfrastructureFaultTest+ProviderException (0x80004005): Failed to connect to 10.42.18.4:5432<---
 ---> (Inner Exception #1) System.ArgumentException: Value does not fall within the expected range.<---
 ---> (Inner Exception #1) System.InvalidOperationException: boom<---
 ---> (Inner Exception #1) System.InvalidOperationException: source B is misconfigured<---
 ---> (Inner Exception #1) System.Net.Sockets.SocketException (0xFFFDFFFF): Name or service not known<---
…

Memex.Portal.Shared.Test.InstanceIdRulesMatchTheRegistryTest ‑ TheSetupHostAgreesWithTheRegistry(candidate: "8c610fb3-f168-46a0-96a0-0413dfbb26f8")
Memex.Portal.Shared.Test.SessionDenialIsAnAnswerTest ‑ OnlyAVerdictReadsAsADenial(shape: "the same verdict nested, as a late denial dispatch"···, failure: System.InvalidOperationException: write failed
 ---> System.UnauthorizedAccessException: Access denied
   --- End of inner exception stack trace ---, isDenial: True)
MeshWeaver.Compiler.Pipeline.Test.SourcesWatcherStallBackoffTest ‑ The_production_classifier_recognises_a_stall_through_every_wrapping(because: "a stall as an aggregate's FIRST member", fault: System.AggregateException: One or more errors occurred. (Query provider(s) [pg] did not emit an Initial within the query fan-in's 16s bound for query 'nodeType:NodeType' (user 'system'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.)
 ---> MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [pg] did not emit an Initial within the query fan-in's 16s bound for query 'nodeType:NodeType' (user 'system'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.
   --- End of inner exception stack trace ---, expected: True)
MeshWeaver.Compiler.Pipeline.Test.SourcesWatcherStallBackoffTest ‑ The_production_classifier_recognises_a_stall_through_every_wrapping(because: "a stall as an aggregate's SECOND member", fault: System.AggregateException: One or more errors occurred. (The operation has timed out.) (Query provider(s) [pg] did not emit an Initial within the query fan-in's 16s bound for query 'nodeType:NodeType' (user 'system'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.)
 ---> System.TimeoutException: The operation has timed out.
   --- End of inner exception stack trace ---
 ---> (Inner Exception #1) MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [pg] did not emit an Initial within the query fan-in's 16s bound for query 'nodeType:NodeType' (user 'system'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.<---
, expected: True)
MeshWeaver.Compiler.Pipeline.Test.SourcesWatcherStallBackoffTest ‑ The_production_classifier_recognises_a_stall_through_every_wrapping(because: "a stall nested in an aggregate inside a wrapper", fault: System.InvalidOperationException: outer
 ---> System.AggregateException: One or more errors occurred. (Value does not fall within the expected range.) (Query provider(s) [pg] did not emit an Initial within the query fan-in's 16s bound for query 'nodeType:NodeType' (user 'system'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.)
 ---> System.ArgumentException: Value does not fall within the expected range.
   --- End of inner exception stack trace ---
 ---> (Inner Exception #1) MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [pg] did not emit an Initial within the query fan-in's 16s bound for query 'nodeType:NodeType' (user 'system'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.<---

   --- End of inner exception stack trace ---, expected: True)
MeshWeaver.Compiler.Pipeline.Test.SourcesWatcherStallBackoffTest ‑ The_production_classifier_recognises_a_stall_through_every_wrapping(because: "a stall wrapped as InnerException", fault: System.InvalidOperationException: outer
 ---> MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [pg] did not emit an Initial within the query fan-in's 16s bound for query 'nodeType:NodeType' (user 'system'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.
   --- End of inner exception stack trace ---, expected: True)
MeshWeaver.Compiler.Pipeline.Test.SourcesWatcherStallBackoffTest ‑ The_production_classifier_recognises_a_stall_through_every_wrapping(because: "an aggregate of unrelated faults", fault: System.AggregateException: One or more errors occurred. (The operation has timed out.) (Value does not fall within the expected range.)
 ---> System.TimeoutException: The operation has timed out.
   --- End of inner exception stack trace ---
 ---> (Inner Exception #1) System.ArgumentException: Value does not fall within the expected range.<---
, expected: False)
MeshWeaver.Graph.Test.ReleaseFailureClassIsInTheTemplateTest ‑ EveryFailureClass_LogsUnderAnEventIdOfItsOwn
MeshWeaver.Graph.Test.ReleaseFailureClassIsInTheTemplateTest ‑ TheEventIdOfEveryClass_IsPinned

♻️ This comment has been updated with latest results.