[rbuergi]

Doc only. Adds the investigation behind the 2026-10 silo-churn issues to Doc/Architecture/ReadingASiloEviction, so the next reader does not have to re-derive it from Loki.

Refs #6432, #6395, #6429, #6392, #6394.

What the page now says

• A second eviction shape: a silo wedged from boot. It shows no memory pressure and no thread-pool stall. About 30 s after boot the silo's own hosted client gets no answer from messagehub/*/_Activity/import-manifest placed on the same silo. This held for 3 of 3 evicted pods (hrdhq on memex 10-09; 5j9gx and gvt4n on memex-cloud 10-10) and for none of the healthy new pods in those rolls.
• Threads are consumed, not starved. poolThreads goes from 20 to 55 while about 500 work items complete per 10 s. A healthy pod in the same roll runs on 16 threads at about 47 000.
• Why eviction takes 8 to 20 minutes. Orleans 10 answers pings in SiloConnection.HandlePingMessage, inside the receive loop. Everything else goes through MessageCenter.ReceiveMessage and Catalog.GetOrCreateActivation on that same loop. A silo with a blocked activation or dispatch path therefore keeps passing probes while answering nothing else.
• The scatter of downstream incidents it files: Portal hub stream-subscription attach times out during silo churn, latching the circuit into cross-process-routing-disabled #6429, Grain-call cancellation delivery times out against departing silos — cancellation batches not propagated during silo lifecycle events #6392, Layout area rendering fails with DeliveryFailureException when placement of the tenant messagehub grain times out #6394 and Memory stream queue dequeue times out (30s) against a churn-affected remote silo #6431.
• What is still open. The blocking call itself needs the FailFast heap dump, which takes cluster access.

The evidence is in the governed Logs actions Ops/Actions/w11a-* on the control instance, and in the comment on #6432.

Why there is no code change

The blocking call has not been identified. Widening probe tolerance or adding a watchdog would be a band-aid; the existing page already argues against #6428-style tolerance.

Checks

• dotnet build src/MeshWeaver.Documentation/MeshWeaver.Documentation.csproj -c Release -warnaserror passes: 0 warnings, 0 errors.
• The change adds no internal links.

Mirror-sync: none — no catalog keys changed.

🤖 Generated with Claude Code

[Copilot]

🟡 Changes recommended

Some correlated incidents are presented as causally proven despite insufficient evidence.

1 open finding

• Do not conflate correlated incidents with confirmed target timeouts · New

What changed in this PR

Documents the boot-time silo wedge investigation and its diagnostic fingerprint.

Changes:

• Adds boot, thread-pool, and eviction indicators.
• Extends the investigation procedure and dump guidance.
• Relates observed downstream timeout incidents.

File	Description
src/​MeshWeaver.Documentation/​Data/​Architecture/​ReadingASiloEviction.md	Adds the boot-wedge investigation and diagnostic procedure.

🧠 Review effort: Balanced

---

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

🟡 Changes recommended

The thread-pool counters are presented as proving permanent blocking, although they establish only thread injection and reduced throughput.

1 open finding

• Avoid concluding threads never return from counters alone · New

1 resolved since last review

• Do not conflate correlated incidents with confirmed target timeouts

🧠 Review effort: Balanced

[github-actions]

Test Results

    4 files   -     20      4 suites   - 20   8m 37s ⏱️ - 44m 35s
1 222 tests  - 10 392  1 222 ✅  - 10 199  0 💤  - 193  0 ❌ ±0 
1 226 runs   - 10 402  1 226 ✅  - 10 209  0 💤  - 193  0 ❌ ±0 

Results for commit 5be9adc. ± Comparison against base commit 108ad86.

This pull request removes 10416 tests.

   --- End of inner exception stack trace ---
   --- End of inner exception stack trace ---, expected: True)
   --- End of inner exception stack trace ---, isDenial: True)
 ---> (Inner Exception #1) MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [pg] did not emit an Initial within the query fan-in's 16s bound for query 'nodeType:NodeType' (user 'system'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.<---
 ---> (Inner Exception #1) MeshWeaver.Messaging.Hub.Test.InfrastructureFaultTest+ProviderException (0x80004005): Failed to connect to 10.42.18.4:5432<---
 ---> (Inner Exception #1) System.ArgumentException: Value does not fall within the expected range.<---
 ---> (Inner Exception #1) System.InvalidOperationException: boom<---
 ---> (Inner Exception #1) System.InvalidOperationException: source B is misconfigured<---
 ---> (Inner Exception #1) System.Net.Sockets.SocketException (0xFFFDFFFF): Name or service not known<---
…

♻️ This comment has been updated with latest results.

[Copilot]

🟡 Changes recommended

The new guidance conflicts with existing ping/configuration documentation and incorrectly claims no governed dump action exists.

3 open findings

• Reconcile contradictory Ping execution models · New
• Correct membership timeout configuration and #6395 rationale · New
• Use the governed AnalyzeDump path for dump analysis · New

1 resolved since last review

• Avoid concluding threads never return from counters alone

🧠 Review effort: Balanced

[Copilot]

🟡 Changes recommended

The documentation retains a contradictory incident classification and assumes a crash dump always exists.

2 open findings

• Document crash-dump preflight failures and missing artifacts · New
• Clarify IAmAlive does not prove grain calls were served · New

3 resolved since last review

• Correct membership timeout configuration and #6395 rationale
• Reconcile contradictory Ping execution models
• Use the governed AnalyzeDump path for dump analysis

🧠 Review effort: Balanced

[Copilot]

🟡 Changes recommended

The dump-log instruction and stale PR description could misdirect incident investigation.

1 open finding

• Use the final crash-dump outcome to determine whether a dump was written · New

2 resolved since last review

• Clarify IAmAlive does not prove grain calls were served
• Document crash-dump preflight failures and missing artifacts

Previously missed (1)

In code that hasn't changed since last review

Update PR description to remove obsolete cluster-access requirement

src/​MeshWeaver.Documentation/​Data/​Architecture/​ReadingASiloEviction.md:214

The PR description still says the required heap dump “takes cluster access,” while this corrected procedure says the governed AnalyzeDump action needs no direct cluster access. Update the PR description so its investigation summary does not preserve the obsolete operational requirement.

🧠 Review effort: Balanced

[Copilot]

🟡 Changes recommended

Several operational statements remain ambiguous or inconsistent with the documented evidence and retention-script behavior.

2 open findings

• Clarify retention errors versus the canonical dump-arming verdict · New
• Use terminal arming outcome instead of arbitrary retention errors · New

1 resolved since last review

• Use the final crash-dump outcome to determine whether a dump was written

Previously missed (3)

In code that hasn't changed since last review

Limit no-response claim to traffic on affected receive loops

src/​MeshWeaver.Documentation/​Data/​Architecture/​ReadingASiloEviction.md:185

“Answers nothing else” is stronger than this mechanism and contradicts the preceding counter evidence that some work continues. Receive loops that have not encountered the blocking message can still process traffic (including the pings used to explain the delayed eviction); only traffic on affected loops is shown to receive no answer. Narrow this sentence so the page does not turn selective dispatch blockage into a total-silo claim.

This issue also appears on line 216 of the same file.

Remove obsolete cluster-access requirement from the PR description

src/​MeshWeaver.Documentation/​Data/​Architecture/​ReadingASiloEviction.md:214

The PR description still says the FailFast dump “takes cluster access,” while this updated section correctly routes analysis through the governed AnalyzeDump action and explicitly says no direct cluster access is needed. Update the PR body to match the reviewed procedure so the investigation handoff does not preserve the obsolete operational requirement.

Update obsolete investigation plan in the remaining-unknowns paragraph

src/​MeshWeaver.Hosting.Orleans/​ClusterMembershipTolerance.cs:27

The later paragraph now establishes that #6395 was a boot-wedged silo, but the existing “What is not proven” paragraph still says the cause must be settled from CPU/GC metrics and the 20:43 probe window. This page’s new evidence already rules out that stall shape; what remains unknown is the blocking activation/dispatch call, which requires the armed FailFast dump. Update that paragraph so this class comment does not preserve the superseded investigation plan.

🧠 Review effort: Balanced

[Copilot]

🔵 Needs a closer look

Operational documentation and the PR description still contain conflicting or overly narrow dump-diagnostic guidance.

0 open findings

2 resolved since last review

• Use terminal arming outcome instead of arbitrary retention errors
• Clarify retention errors versus the canonical dump-arming verdict

Previously missed (4)

In code that hasn't changed since last review

Update README to inspect the final crash-dump status line

src/​MeshWeaver.Documentation/​Data/​Architecture/​DebuggingNativeCrashes.md:152

This correction is still contradicted by deploy/aks/README.md:1389-1390, which tells operators to read the first [crash-dumps] line. Since the script emits retention lines before its terminal arming result, that operational path can still misclassify the dump state. Update the README to use the last line of the relevant run as well.

Preserve the actual reason for HELD dump status

src/​MeshWeaver.Documentation/​Data/​Architecture/​OnboardingNewEnvironment.md:279

HELD is not limited to insufficient free space: the script also emits it when numeric settings are unusable or df cannot report free space (crash-dump-retention.sh:74-78,125-127). Naming only storage pressure can send onboarding toward the wrong repair; preserve the reason carried by the actual log line.

Correct PR description about FailFast dump access

src/​MeshWeaver.Documentation/​Data/​Architecture/​ReadingASiloEviction.md:214

The PR description still says the FailFast heap dump “takes cluster access,” while this change correctly documents that AnalyzeDump performs the read through the governed control-instance action with no direct cluster access. Update the description so the durable review context does not preserve the obsolete operational requirement.

Clarify that the remaining unknown is the blocking call

src/​MeshWeaver.Hosting.Orleans/​ClusterMembershipTolerance.cs:27

This now settles #6395 as a boot-wedged silo, but the same summary's “What is not proven” paragraph still says the unanswered probes need CPU/GC metrics and peer probe lines to settle. The new investigation already supplies those signals; what remains unknown is the blocking call, which requires the FailFast dump. Update that paragraph so it no longer sends the next investigation down the superseded path.

🧠 Review effort: Balanced