[rbuergi]

Refs Systemorph/MeshWeaver.Plugins#2799 · follows #6405

Root cause

At pod boot, reads of dynamic content happen before those types are registered on the replica. On memex-cloud pod 884964bb7-6gv59, 132 of 134 stayed an untyped JsonElement lines were written at 19:48:37.3. The pre-warmer only started at 19:48:37.8, and the registration-only pass finished at 19:52:50. The readers are the process's own boot work, not traffic: a Posts query, the standing watches on Ops/Status/* and Ops/Watch/fleet-watch, and, on the current 3.1.10365 generation, the */_Install/* install records (Ops/Actions/sevm-20261010-2799-bootorder-cloud on control). Holding readiness back would not reach these readers, because they run before ApplicationStarted. Moving the pass ahead of the barrier would bring back the 414-type boot cost that #1660 removed. #6405 only deferred the warning. A query read in that window stayed untyped until the next change to it, because a query has no late re-type.

Fix

MeshNodeStreamCache.GetStream(path, options) and GetQuery(id, options, …) now handle content that stays untyped and names a NodeType differently: the emission waits for ContentTypeOnDemandRegistration.EnsureRegistered(nodeType) and is typed after that.

• EnsureRegistered reads the type's record as system, with the same 30 s budget the pass uses for its enumeration. It then runs DynamicContentTypeRegistrar.RegisterType, which is the pass's own route for one type: the record's claimed build, identity-checked existing bytes (refetched from the shipped bundle when they are missing), a configuration load from the existing assembly, and one transient probe. It never compiles and never writes the record. A type with no usable build answers NotBaked at once and the read degrades as before.
• Each type gets one attempt per process, coalesced through an instance PromiseCache, on the FileSystem IO pool. Only a Faulted attempt is released for retry. Order is preserved (Concat). Content that types cleanly is answered at once.
• The barrier concern does not apply: store writes are atomic (A NodeType assembly can be loaded mid-write — 'format is invalid' parks the type, and the emit path skips AtomicFileWrite #1387), and only bytes whose MVID matches the published build are loaded.
• The service is registered by AddDynamicTypePreWarming. Hosts without it, including test hosts, keep the old behaviour. Boot untyped-content warnings settle with the registration pass; a closed type set fetches no bundle #6405's deferred warning stays as the diagnostic for anything still untyped.
• This amends policy dynamic-content-type-registration-pass, which said "Never from a read seam". The register row now says: never the enrichment (compiling) path from a read seam; the registration-only route may run on demand for the one type being read. This was the maintainer's directive for ci: Build and Test fires on merge_group, so a merge queue can be enabled (#2412) #2799, option (c). Flagged here so it can be vetoed.

Tests

AReadAtBootIsTypedNotUntypedTest (MeshWeaver.Hosting.Test). Setup: the type is baked without hub activation, and the instance is written straight to the store, so nothing registers the type as a side effect.

• Negative control: the same read without the service, which is current main, returns a JsonElement.
• Query and stream seams: both return typed content on the first emission. The NodeType record's build stamp does not change.
• No-build type: answers NotBaked and nothing gets compiled.
• Seam check: with the seam's wiring removed, the query case fails with "Did not expect value to be of type JsonElement".

Local runs, Release:

• Hosting.Test: 1177/1178 passed (1 skipped).
• Documentation.Test: 759/759 passed (guards and doc links).
• -warnaserror builds of Hosting, Graph, Hosting.Test and Documentation.Test are clean.

Deploy

No recycle needed: this takes effect at process start. To verify after the next roll of memex-cloud and memex, count |~ "(?i)stayed an untyped JsonElement" in each replica's boot window. Before this change it was about 130 per boot; it should now be about 0. Each type a boot reader needed should log one ContentTypeOnDemandRegistration: <type> → Registered line.

Pairs-with: none — adds a public type and internal members only; removes nothing public.
Implementers: none — no interface member added.
Mirror-sync: none — no i18n catalog change.

🤖 Generated with Claude Code

[Copilot]

🟡 Changes recommended

Exact-path query staleness can cache an incorrect verdict, while concurrent registration can exhaust the filesystem pool.

2 open findings

• Cache incomplete NodeType reads as NotBaked · New
• Serialize NodeType loads to avoid exhausting the filesystem pool · New

What changed in this PR

Ensures boot-time reads can register baked dynamic content types on demand and return typed content immediately.

Changes:

• Adds process-scoped on-demand type registration.
• Integrates registration into stream and query read seams.
• Adds integration tests and updates architecture policy/documentation.

File	Description
test/​MeshWeaver.Hosting.Test/​AReadAtBootIsTypedNotUntypedTest.cs	Tests typed boot reads and no-build behavior.
src/​MeshWeaver.Hosting/​MeshNodeStreamCache.cs	Adds on-demand typing to streams and queries.
src/​MeshWeaver.Hosting/​DynamicTypePreWarmerHostedService.cs	Registers the new service.
src/​MeshWeaver.Hosting/​DynamicContentTypeRegistrar.cs	Exposes single-type registration.
src/​MeshWeaver.Hosting/​ContentTypeOnDemandRegistration.cs	Implements coalesced registration attempts.
src/​MeshWeaver.Hosting/​ContentDegradationRegistry.cs	Updates degradation guidance.
src/​MeshWeaver.Graph/​ContentTypeRegistrationSweep.cs	Documents permitted read-seam registration.
src/​MeshWeaver.Documentation/​Data/​Architecture/​PolicyNotProse.md	Amends registration policy.
src/​MeshWeaver.Documentation/​Data/​Architecture/​DynamicContentTypeRegistration.md	Documents behavior and verification.

🧠 Review effort: Balanced

---

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[rbuergi]

Agreed, fixed in 0b84fbc. The record is now read from the STORE (IStorageAdapter.Read), which is the authority, never from a query, so a stale or incomplete snapshot can no longer be cached as a NotBaked verdict. It is also never read through the NodeType's own stream, whose first touch can drive the compiling enrichment path. A null from the store means the record genuinely does not exist, and that is a verdict. A fault or an expired RecordReadBudget is Faulted, which is released, so the next read retries.

[rbuergi]

Agreed, fixed in 0b84fbc. The distinct unresolved NodeTypes of one result set are now registered one at a time, in ordinal order (OrderBy + Concat), so a broad query never fans out concurrent assembly loads on the FileSystem pool. The emission still waits for the whole set, and each type's attempt stays coalesced across readers by the PromiseCache.

[github-actions]

Test Results

   14 files   -    10     14 suites   - 10   41m 32s ⏱️ - 7m 47s
9 900 tests  - 1 467  9 900 ✅  - 1 274  0 💤  - 193  0 ❌ ±0 
9 914 runs   - 1 467  9 914 ✅  - 1 274  0 💤  - 193  0 ❌ ±0 

Results for commit 0b84fbc. ± Comparison against base commit 9809c5e.

This pull request removes 1501 and adds 10 tests. Note that renamed tests count towards both.

   --- End of inner exception stack trace ---
   --- End of inner exception stack trace ---, expected: True)
   --- End of inner exception stack trace ---, isDenial: True)
 ---> (Inner Exception #1) MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [pg] did not emit an Initial within the query fan-in's 16s bound for query 'nodeType:NodeType' (user 'system'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.<---
 ---> (Inner Exception #1) MeshWeaver.Messaging.Hub.Test.InfrastructureFaultTest+ProviderException (0x80004005): Failed to connect to 10.42.18.4:5432<---
 ---> (Inner Exception #1) System.ArgumentException: Value does not fall within the expected range.<---
 ---> (Inner Exception #1) System.InvalidOperationException: boom<---
 ---> (Inner Exception #1) System.InvalidOperationException: source B is misconfigured<---
 ---> (Inner Exception #1) System.Net.Sockets.SocketException (0xFFFDFFFF): Name or service not known<---
…

Memex.Portal.Shared.Test.InstanceIdRulesMatchTheRegistryTest ‑ TheSetupHostAgreesWithTheRegistry(candidate: "98af544b-03f6-49d1-bba4-567307994f79")
Memex.Portal.Shared.Test.SessionDenialIsAnAnswerTest ‑ OnlyAVerdictReadsAsADenial(shape: "the same verdict nested, as a late denial dispatch"···, failure: System.InvalidOperationException: write failed
 ---> System.UnauthorizedAccessException: Access denied
   --- End of inner exception stack trace ---, isDenial: True)
MeshWeaver.Compiler.Pipeline.Test.SourcesWatcherStallBackoffTest ‑ The_production_classifier_recognises_a_stall_through_every_wrapping(because: "a stall as an aggregate's FIRST member", fault: System.AggregateException: One or more errors occurred. (Query provider(s) [pg] did not emit an Initial within the query fan-in's 16s bound for query 'nodeType:NodeType' (user 'system'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.)
 ---> MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [pg] did not emit an Initial within the query fan-in's 16s bound for query 'nodeType:NodeType' (user 'system'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.
   --- End of inner exception stack trace ---, expected: True)
MeshWeaver.Compiler.Pipeline.Test.SourcesWatcherStallBackoffTest ‑ The_production_classifier_recognises_a_stall_through_every_wrapping(because: "a stall as an aggregate's SECOND member", fault: System.AggregateException: One or more errors occurred. (The operation has timed out.) (Query provider(s) [pg] did not emit an Initial within the query fan-in's 16s bound for query 'nodeType:NodeType' (user 'system'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.)
 ---> System.TimeoutException: The operation has timed out.
   --- End of inner exception stack trace ---
 ---> (Inner Exception #1) MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [pg] did not emit an Initial within the query fan-in's 16s bound for query 'nodeType:NodeType' (user 'system'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.<---
, expected: True)
MeshWeaver.Compiler.Pipeline.Test.SourcesWatcherStallBackoffTest ‑ The_production_classifier_recognises_a_stall_through_every_wrapping(because: "a stall nested in an aggregate inside a wrapper", fault: System.InvalidOperationException: outer
 ---> System.AggregateException: One or more errors occurred. (Value does not fall within the expected range.) (Query provider(s) [pg] did not emit an Initial within the query fan-in's 16s bound for query 'nodeType:NodeType' (user 'system'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.)
 ---> System.ArgumentException: Value does not fall within the expected range.
   --- End of inner exception stack trace ---
 ---> (Inner Exception #1) MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [pg] did not emit an Initial within the query fan-in's 16s bound for query 'nodeType:NodeType' (user 'system'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.<---

   --- End of inner exception stack trace ---, expected: True)
MeshWeaver.Compiler.Pipeline.Test.SourcesWatcherStallBackoffTest ‑ The_production_classifier_recognises_a_stall_through_every_wrapping(because: "a stall wrapped as InnerException", fault: System.InvalidOperationException: outer
 ---> MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [pg] did not emit an Initial within the query fan-in's 16s bound for query 'nodeType:NodeType' (user 'system'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.
   --- End of inner exception stack trace ---, expected: True)
MeshWeaver.Compiler.Pipeline.Test.SourcesWatcherStallBackoffTest ‑ The_production_classifier_recognises_a_stall_through_every_wrapping(because: "an aggregate of unrelated faults", fault: System.AggregateException: One or more errors occurred. (The operation has timed out.) (Value does not fall within the expected range.)
 ---> System.TimeoutException: The operation has timed out.
   --- End of inner exception stack trace ---
 ---> (Inner Exception #1) System.ArgumentException: Value does not fall within the expected range.<---
, expected: False)
MeshWeaver.Hosting.Test.AReadAtBootIsTypedNotUntypedTest ‑ AQueryRead_OfABuiltButUnregisteredType_IsTypedOnItsFirstEmission
MeshWeaver.Hosting.Test.AReadAtBootIsTypedNotUntypedTest ‑ AReadOfATypeWithNoBuild_DegradesAsBefore_AndCompilesNothing
MeshWeaver.Hosting.Test.AReadAtBootIsTypedNotUntypedTest ‑ AStreamRead_OfABuiltButUnregisteredType_IsTypedOnItsFirstEmission