CI: one Roslyn Emit poisons the whole process — every later compile NREs in Cci.MetadataWriter.GetConsolidatedTypeParameters, shard ends exit=124

Split from #612 — a SECOND, distinct defect observed in PR #884 run 31153582625 shard 2 (2026-08-07), NOT explained by the empty-source-snapshot race fixed there.

Evidence

Every compile in that test process failed with CompilationError = Object reference not set to an instance of an object. — including CodeEditRecompileTest's deliberately-invalid BrokenType, whose failure should be a Roslyn diagnostic, i.e. the NRE strikes PRE-Roslyn. Source discovery was healthy (IsDirty=True) and release triggers were handled (fresh LastCompilationActivityPath minted per retry). The wedge cascaded across classes on the shard (OverlaySelfHealInstanceRecycle → LinkedInTelemetryImport → PandasExplorer → CodeEditRecompile → ReleaseRequestWatcherHighWater), consistent with a process-wide condition.

Why it is currently undiagnosable, and what changed

Every sink reduced the fault to .Message — no stack anywhere. The #612 fix adds: exception TYPE + full stack on the activity log for non-Roslyn aborts, {TypeName}: {Message} on the node's CompilationError, and the exception object on the terminal LogWarning. The next CI occurrence will name its throw site — triage should start from that.

Probed and excluded (locally green)

DOTNET_ENVIRONMENT=Development + Logging__LogLevel__Default=Warning
#if CIRun-gated code (none exists)

Remaining suspects: Linux-only or load/saturation-dependent null-race in the pre-Roslyn compile pipeline.

🤖 Generated with Claude Code

Recurrence sweep: NO recurrence found — and one correction to the premise

Swept CI for a post-diagnostics recurrence so the next person doesn't redo it.

What was searched

The #612 diagnostics landed in 5ef3a89f9 (merged 0a0118582, 2026-08-07 07:19 UTC), so only runs created after that timestamp can carry them.

Window: 2026-08-07 07:19 UTC → 2026-08-08 19:00 UTC (MeshWeaver Build and Test, workflow dotnet-test.yml).
Runs in window: 185. Of those, 58 had a final conclusion of failure/cancelled, and 10 were re-runs (run_attempt = 2).
Jobs actually read: every Run tests (shard N) job that did not succeed, across all attempts — including attempt 1 of the 10 re-run runs. The process-wide symptom in this issue always reds or wedges its shard, so a green shard cannot hide it.

Known limitation, stated deliberately: a single isolated compile NRE could survive on a green shard. CodeEditRecompileTest asserts only failedDef.CompilationError.Should().NotBeNullOrEmpty() for the deliberately-broken type, which an NRE satisfies just as well as a Roslyn diagnostic. So "0 hits" rules out a recurrence of the cascade, not every possible single NRE. Tightening that assertion to require a Roslyn diagnostic (e.g. a CS#### code) would turn the silent case into a red test.
Grep: CompilationError = , Object reference not set, NullReferenceException.
Volume: 108 shard-job logs read. Every one was re-fetched and confirmed non-empty — 0 unfetchable, so no result here is a failed download masquerading as "clean". Of those 108, 43 actually ran MeshWeaver.Hosting.Monolith.Test (where CodeEditRecompileTest lives), i.e. 43 shard-jobs where the symptom had somewhere to appear.

Result: 0 occurrences.

The grep was calibrated on the original occurrence

Run 31153582625 (the one in the issue) ends green — shard 2 failed on attempt 1 and was re-run. Attempt 1's shard-2 job log carries the signature 5×, one per NodeType:

CompilationError = Object reference not set to an instance of an object.

on SelfHeal…, ToolRetryType, StaleBarrierType, BrokenType, CodeEditType — i.e. the same grep on the same sink does find it when it is there. Note the shape: the run's final state was success, so a sweep of run-level conclusions alone would have missed it. Hence the all-attempts pass above.

⚠️ Correction: the next occurrence will NOT name its throw site

The issue says "the next CI occurrence will name its throw site". It will not — only one of the three sinks #612 added is observable in CI.

#612 added:

CompilationError = "{TypeName}: {Message}" on the node — NodeTypeCompilationHelpers.cs:1535-1552
Compile aborted by {FullName}:\n{stack} on the ActivityLog node — NodeTypeCompilationHelpers.cs:1450-1456
logger.LogWarning(outcome.Error, …) — the exception object, NodeTypeCompilationHelpers.cs:1545

Only (1) reaches CI, and only because the test's assertion dump prints the whole MeshNode. (2) lives on a MeshNode that is never uploaded, and (3) goes to ILogger, which no CI artifact captures. Evidence:

The pre-#612 code already had logger?.LogWarning("Compile failure for {HubPath}: {Error}", …). In the attempt-1 log above, where five compiles failed, Compile failure for appears 0 times.
A full MeshWeaver.Hosting.Monolith.Test .trx (443 KB) contains 1 StdOut section (a skip reason) and 0 [COMPILE-TRACE] lines — although HandleDispatchCompile logs [COMPILE-TRACE] … running compile INLINE at Information on every compile.

So a recurrence today yields CompilationError = NullReferenceException: Object reference not set to an instance of an object. — the exception class, no frames. Better than before, but not the throw site.

Concrete fix, if we want the stack: a CI-retained sink already exists. MonolithMeshTestBase.cs:286 writes a phase trace to /tmp/meshweaver-test-trace.log, and dotnet-test.yml:589 copies it to collected-logs/_meshweaver-test-trace.log and uploads it — it survives even the exit=124 wedge case, where no .trx is written at all. Appending the non-Roslyn compile abort ({FullName} + stack, already assembled at NodeTypeCompilationHelpers.cs:1455) to that same file would put frames in front of the next triager.

Two things found while verifying

A second, stale compile terminal-write path still has the un-enriched code. NodeTypeCompileActivityHandler.cs:428 still reads
```
var errorSummary = outcome.Error?.Message ?? …   // bare .Message — the pre-#612 shape
```
It is dead: no production code posts a RunCompileRequest — the only things referencing it are the record declaration, the type-registry entry, the ActivityNodeType.WithHandler wiring, and a deliberately-❌ WRONG example in AsynchronousCalls.md. NodeTypeCompilationHelpers.cs:1142-1155 documents why — the cross-hub dispatch was replaced by an inline RunCompile because a just-created _Activity hub is not yet routable. Harmless at runtime, but it is a triage trap: grepping CompilationError = errorSummary lands on the un-enriched dead copy first.
No banned static state in the compile path. NodeTypeCompileParkRegistry (which caches a terminal error and replays it for parked types — the one thing in this path shaped like "bad state that sticks") is a DI singleton resolved from hub.ServiceProvider, i.e. mesh-scoped, dying with the mesh. The two static readonly fields in MeshNodeCompilationService (_references, BuiltInGeneratorPaths) are write-once constant lookups.

Status

Not reproduced, so not fixed — no speculative change made. The symptom is rare: exactly one documented occurrence (31153582625), and 185 runs / 58 red runs since the diagnostics landed have not produced another. Recommend leaving this open and, when it recurs, starting from the CompilationError type name — after routing the stack somewhere CI keeps.

Cross-link: #618 may be the same root cause

Flagging the structural sibling explicitly, because the two are close enough that whoever picks up either should read both.

#618 — "Orleans test host: corrupt/evicted DynamicNode assembly poisons every TestCluster.DeployAsync" — is the same shape as this issue:

	#890 (this)	#618
Trigger	a compile	a compile
Blast radius	every compile in the process	every silo build in the process
Cascade	across unrelated test classes on the shard	across unrelated test classes in the project
Onset	one bad state, then permanent	one bad state, then permanent
Timing	load / saturation dependent	load / saturation dependent

Both are "one compile puts a process-wide facility into a bad state, and it stays there" — not a per-test logic bug. #618 already has its mechanism identified (a dynamically-compiled NodeType assembly that is corrupt on disk or whose collectible ALC was evicted, leaving a dangling Assembly that Orleans' ReferencedAssemblyProvider still enumerates); #890 has only the symptom, because the stack never reached CI.

#618 is being worked right now from the ALC-lifetime side. If that work lands and this stops recurring, that is evidence they were one defect — an ALC/assembly-lifetime fault reached #618 through Orleans' assembly enumeration and reached #890 through the compile pipeline, which would explain why #890's NRE strikes pre-Roslyn and is process-wide.

So, concretely: do not root-cause #890 in isolation before checking #618's outcome. And when #890 next recurs, the first thing to read is the exception TYPE — if it is not NullReferenceException, or its stack points into assembly loading / ALC teardown rather than the compile's own code, treat it as #618's family first.

Correction: #618 is a SEPARATE defect — same shape, different mechanism

This supersedes the shared-root-cause speculation in my previous comment. I wrote there that #890 and #618 "may share a root cause" and advised not to root-cause #890 before #618's outcome. That was wrong, and following it would have stalled this issue behind an unrelated fix.

#618's mechanism has since been established, and it is specific to it:

AssemblyLoadContext.Unload() is asynchronous by contract. The assembly stays in
AppDomain.CurrentDomain.GetAssemblies() until the LoaderAllocator is collected, and Orleans'
assembly scan takes no pin — so it enumerates an assembly that is on its way out.

That is an enumerability-after-unload problem in Orleans' assembly scan. It requires Orleans enumerating loaded assemblies, which the #890 compile abort does not do — and #890 has been seen on the Monolith shard, with no Orleans silo build involved. The two share a shape (one compile puts a process-wide facility into a bad state; the cascade crosses unrelated test classes) and nothing else.

So: do not merge these two issues, and do not block either on the other. Concretely:

Fixing Orleans test host: corrupt/evicted DynamicNode assembly poisons every TestCluster.DeployAsync (TypeLoadException 'format is invalid') — whole-project cascade #618 should not be expected to make CI: one Roslyn Emit poisons the whole process — every later compile NREs in Cci.MetadataWriter.GetConsolidatedTypeParameters, shard ends exit=124 #890 stop recurring. If someone observes that and infers the issues were one, that inference is unsupported.
When CI: one Roslyn Emit poisons the whole process — every later compile NREs in Cci.MetadataWriter.GetConsolidatedTypeParameters, shard ends exit=124 #890 next recurs, read the exception TYPE and its stack (now that they will actually reach CI — PR A compile fault's stack now reaches CI — #890 could never have been diagnosed #966). Treat it as CI: one Roslyn Emit poisons the whole process — every later compile NREs in Cci.MetadataWriter.GetConsolidatedTypeParameters, shard ends exit=124 #890's own defect unless the stack genuinely lands in assembly loading, and even then check it against Orleans test host: corrupt/evicted DynamicNode assembly poisons every TestCluster.DeployAsync (TypeLoadException 'format is invalid') — whole-project cascade #618's specific mechanism rather than assuming.

The one thing worth carrying across is the diagnostic lesson, not the cause: both were expensive because a process-wide fault left no frames in any artifact CI retains.

Merge pull request #966 from Systemorph/fix/890-compile-abort-diagnos…

Recommend CLOSE — the sink is verified end-to-end in CI artifacts, and a 253-run sweep is clean

Two things had to be true to close this: (1) a recurrence must now be diagnosable, and (2) it must not be recurring. I checked (1) first, on downloaded artifacts rather than by reading code, because a close resting on an unverified diagnostic would repeat this issue's original mistake — it sat for a month waiting for evidence that could never arrive.

1. The sink genuinely reaches CI — verified on artifacts, not on code reading

The chain on current main:

XUnitFileLogger.Log calls TestTraceLog.AppendFault(category, level, message, exception) for every record with exception is not null && level >= Warning, placed before the two filters that hid it (null output helper / !IsInTestMethod()) — src/MeshWeaver.Fixture/XUnitFileOutputHelper.cs.
TestBase does ClearProviders() and registers only XUnitFileLoggerProvider (src/MeshWeaver.Fixture/TestBase.cs:34-40), so every framework ILogger in a test process goes through it.
The compile terminal path passes the exception object: NodeTypeCompilationHelpers.cs:1544 — logger?.LogWarning(outcome.Error, "Compile failure for {HubPath}: {Error}", hubPath, errorSummary).
TestTraceLog.Path = Path.GetTempPath()/meshweaver-test-trace.log; MonolithMeshTestBase.cs:290/316 writes its phase trace to the same handle; .github/workflows/dotnet-test.yml:588-589 copies it to collected-logs/_meshweaver-test-trace.log, and the Upload artifact: shard test results step (if: always(), retention 15 days) ships collected-logs/.

The part that actually settles it — I downloaded the artifacts:

17 post-A compile fault's stack now reaches CI — #890 could never have been diagnosed #966 runs, 102 shard artifacts. 102/102 contain _meshweaver-test-trace.log (2.2–7.4 MB each), carrying 148,993 [FAULT] records with full Exception.ToString() including /home/runner/work/MeshWeaver/…:line N frames.
Control, pre-A compile fault's stack now reaches CI — #890 could never have been diagnosed #966 (run 31271444339, shard 2): the trace log is there but it is 543 KB with 0 [FAULT] lines. The difference is exactly A compile fault's stack now reaches CI — #890 could never have been diagnosed #966.
The compile path specifically is demonstrated, not assumed. Those artifacts carry 50 Compile failure for … fault records — 34 real compile aborts from the deliberate wedge types, with stacks:

[FAULT] [Warning] [MeshWeaver.Graph.CompileWatcher] Compile failure for TestData/SnapshotWedgeType: TimeoutException: Source snapshot …
System.TimeoutException: Source snapshot for 'TestData/SnapshotWedgeType' did not emit within 5s …
   at MeshWeaver.Data.Serialization.SynchronizationStream`1..ctor(…) in /home/runner/work/MeshWeaver/MeshWeaver/src/MeshWeaver.Data/Serialization/SynchronizationStream.cs:line 951

…plus 16 from #966's own probe (TestTraceLogFaultTest, category MeshWeaver.Graph.Configuration.MeshNodeCompilationService) — i.e. the guard test fires on every CI run and its record is in the uploaded artifact, not just green in a trx.

So a recurrence today writes NullReferenceException and its throw site into a file CI keeps for 15 days. The premise this issue was parked on now actually holds.

Also confirmed landed (99d95ed): CodeEditRecompileTest requires CompilationError to MatchRegex(@"CS\d{4}"), closing the hole where an isolated pre-Roslyn NRE passed silently as "some error occurred".

Two residual limits, stated so nobody is surprised later:

A per-process cap of 1000 fault records (TestTraceLog.MaxFaultRecords). Two processes hit it in every run I sampled — both on shard 3 (AgentSelectionMatchedButUnbuildableTest, ~90 s in; GroupPermissionTests) — logging further fault records suppressed after 1000 in this process. A compile fault arriving after the 1000th record in the same process would be dropped. Neither process runs a compile test today, and CI: one Roslyn Emit poisons the whole process — every later compile NREs in Cci.MetadataWriter.GetConsolidatedTypeParameters, shard ends exit=124 #890's symptom is that every compile fails from the start (so the first record names it) — but the noise floor is high: 2348 of shard 3's faults are teardown TargetInvocationException from CompileStateMirror / HubDisposingException.
"Survives exit=124" is structural, not observed. The collect and upload steps are if: always(), but no shard in the 1301 logs I read actually hit the wall-clock cap, so I could not confirm it from a live example.

2. Sweep: 0 recurrences, at a sensitivity the previous sweep did not have

Window 2026-08-07T07:00Z → 2026-08-09T06:00Z (47 h, dotnet-test.yml): 253 runs, 11 of them with a second attempt — all attempts read (attempt 1 and 2 for all 11, 6 shard logs each, 12/12 indexed per run).
1349 shard jobs listed → 1301 logs fetched and grepped. Every fetched log confirmed non-empty (smallest 21 KB); 0 empty-after-filter. Not read: 32 skipped (already-green-tree reuse — ran nothing) and 16 unfetchable, of which 15 are zero-duration cancelled matrix placeholders whose name never expanded (Run tests (shard ${{ matrix.shard }}), BlobNotFound, ran nothing). The 16th (job 93148159108, shard 2 of cancelled run 31275033601) ran 5 min before cancellation and its blob is gone — its five sibling shards were also cancelled early (21–31 KB, no test project completed). One genuinely unreadable log out of 1301.
Sensitivity: 420 of the 1301 shard logs actually ran MeshWeaver.Hosting.Monolith.Test (where CodeEditRecompileTest lives), across 200 distinct runs; 206 ran MeshWeaver.Graph.Test.
Grep calibrated against the original occurrence: the same patterns run over run 31153582625 attempt 1 shard 2 return 5 hits (CompilationError = Object reference not set… on SelfHeal…, ToolRetryType, StaleBarrierType, BrokenType, CodeEditType). It finds the signature when it is there. (One trap worth recording: the workflow's own source contains LookupCompilationError, so the command-echo lines must be stripped or every log self-matches.)
Result: 2 hits in 1301 logs, neither is CI: one Roslyn Emit poisons the whole process — every later compile NREs in Cci.MetadataWriter.GetConsolidatedTypeParameters, shard ends exit=124 #890.
- 31171377098 — the literal string inside a test name: MeshPluginTest.Get_InstanceOfBrokenNodeType_WrapsResponseWithCompilationError.
- 31278893197 — a real NRE, but a layout render fault, not a compile one: LayoutAreaHost: Rendering failed for area ProfitabilityTable/2 on FutuRe/Analysis → System.NullReferenceException at SegmentedArrayBuilder.Expand ← Workspace.cs:209 GetStream. A different defect; worth its own issue if it recurs.
Plus a trace-log sweep of the post-A compile fault's stack now reaches CI — #890 could never have been diagnosed #966 window — sensitivity the job-log grep can never have, because the trace log records faults that red no test at all. 102 shard trace logs, 148,993 fault records, classified by logger category. NRE-bearing records occur in exactly two categories: MeshWeaver.Layout.Composition.LayoutAreaHost (102) and MeshWeaver.AccessContext (88, all teardown-shaped). Compile faults whose exception is an NRE: 0.

Combined with the earlier 185-run sweep, that is 47 h of continuous CI, with the tightened assertion in place for the last of it, and no sign of the defect.

Recommendation

Close as dormant-and-now-diagnosable — with the reasoning stated explicitly, because the reasoning is the whole justification:

The close is not earned by "we didn't see it again". One documented occurrence of a load-dependent race would stay dormant far longer than 47 h, and a dormant bug with no way to diagnose it should stay open — which is exactly why this issue was right to stay open until now. It is earned by the fact that diagnosability is now verified against real CI artifacts, and that nothing testable remains here: every remaining hypothesis needs a fresh occurrence, and a fresh occurrence will now arrive carrying its throw site.

Reopen trigger, so this stays falsifiable: any [FAULT] … Compile failure for …: NullReferenceException in collected-logs/_meshweaver-test-trace.log, or any CompilationError that is not a CS#### diagnostic. The first lands in a retained artifact; the second reds CodeEditRecompileTest.

If you would rather keep it open, the only evidence-based argument I can offer is the pair of residual limits above (the 1000-record per-process cap, and exit=124 survival being structural rather than observed). Neither touches the compile path today. I don't think either outweighs closing.

And nothing here supports merging with #618 — consistent with the correction above. The enumerability-after-unload family appears nowhere near these compile faults, and the compile faults CI does produce are all TimeoutException from the deliberate wedge tests.

🤖 Generated with Claude Code

Closing — dormant, and now genuinely diagnosable

The close is earned by verified diagnosability, not by absence of recurrence. A dormant bug with no way to diagnose it should stay open; that was this issue's own trap, and it is now fixed.

The premise in this issue was wrong

It said "the next CI occurrence will name its throw site". It would not have. Of #612's three sinks, two never reach CI: ActivityLog nodes are not artifacts, and nothing captures ILogger output. Only CompilationError survived — incidentally, because a test's assertion dump prints the whole MeshNode — and it carries no stack. So this issue was waiting for evidence that could not arrive.

The sink is now verified end-to-end, on downloaded artifacts

Chain on current main: XUnitFileLogger.Log → TestTraceLog.AppendFault(...) for any exception != null && level >= Warning, placed before the two filters that previously hid it; TestBase.cs:34-40 registers only that provider; the compile terminal path passes the exception object (NodeTypeCompilationHelpers.cs:1544); dotnet-test.yml:588-589 copies /tmp/meshweaver-test-trace.log into collected-logs/ and uploads it if: always().

Empirically, not by reading code:

102/102 shard artifacts across 17 post-A compile fault's stack now reaches CI — #890 could never have been diagnosed #966 runs carry the trace log (2.2–7.4 MB), with 148,993 [FAULT] records including full stacks.
Pre-A compile fault's stack now reaches CI — #890 could never have been diagnosed #966 control (run 31271444339): same file, 543 KB, 0 [FAULT] lines.
The compile path specifically is demonstrated, not assumed: 50 Compile failure for … records, 34 of them real aborts with /home/runner/...:line N frames.
A compile fault's stack now reaches CI — #890 could never have been diagnosed #966's CS\d{4} tightening of CodeEditRecompileTest also landed (99d95eda5), closing the hole where an isolated NRE could pass silently as "some error occurred".

The sweep, with stated sensitivity

Window 2026-08-07T07:00Z → 2026-08-09T06:00Z: 253 runs, 11 with a second attempt, all attempts read. 1349 shard jobs → 1301 logs fetched, all confirmed non-empty. Exactly one genuinely lost log (a cancelled shard on a superseded run whose siblings ran nothing). 420 logs actually ran MeshWeaver.Hosting.Monolith.Test, across 200 runs — so the absence has real sensitivity.

Zero occurrences. The two grep hits were a string inside a test name and a LayoutAreaHost render NRE at Workspace.cs:209 (a different defect). The trace-log sweep adds sensitivity a job-log grep cannot have: of 148,993 fault records classified by category, NRE-bearing ones appear only in LayoutAreaHost (102) and AccessContext (88) — compile faults with an NRE: 0.

Two methodology notes worth keeping: the workflow's own source contains LookupCompilationError, so command-echo lines must be stripped or every log self-matches; and the grep was calibrated against the original occurrence (31153582625 attempt 1 shard 2 → 5 hits), so "0 hits" has a proven detector behind it rather than an assumed one.

Residual limits, tracked not buried

TestTraceLog silently drops faults past 1000 per process — the only CI-reaching diagnostic sink has an invisible ceiling #982 — TestTraceLog caps fault records at 1000 per process, and two shard-3 processes hit that cap in every sampled run. A recurrence arriving as fault Reading a document is not reviewing it — the redline moved into the version comparison #1001 would not be captured, which partially weakens the property this close rests on.
"Survives an exit=124 wedge" is structural, not observed — no shard in 1301 logs hit the wall-clock cap.

If this recurs, the trace log should now name the throw site. Reopen with that stack.

Recurred — and this time it named its throw site. The premise was wrong: the NRE is INSIDE Roslyn, during `Emit`.

Reopening. New occurrence: PR #1359, run
31649727371,
shard 2, 2026-08-12 23:18 UTC. Same shape as the original — every compile in the process fails
with a bare NRE, BrokenType included, and the shard then cascades to
exit=124 TIMEOUT (8m wall-clock cap hit):

CodeEditRecompileTest.FailedCompile_PreservesErrorLogAndDoesNotCreateRelease   [FAIL]
CodeEditRecompileTest.CodeEdit_ExplicitRelease_IsUpToDate_RecompilesOnSourceChange   [FAIL]
CodeEditRecompileTest.FirstCompileError_LatchedEmptySourceQuery_FreshTriggerMustRecompileFromDirectRead   [FAIL]
CodeEditRecompileTest.PressingCompileButton_SetsRequestedReleaseAt_AndProducesNewRelease   [FAIL]
CodeEditRecompileTest.IsDirty_FlipsTrueOnSourceEdit_FalseAfterCompile   [FAIL]
CodeEditRecompileTest.NodeType_RequestedReleasePath_PinsToHistoricalRelease   [FAIL]
CompileActivityLogTest.SuccessfulCompile_PopulatesCompiledSourcesSnapshot   [FAIL]
CompileActivityLogTest.SuccessfulCompile_ReportsActivityLogWithSourceQueriesAndMatchedPaths   [FAIL]
[CI] MeshWeaver.Hosting.Monolith.Test (part 2/2) exit=124 TIMEOUT

The stack, which this issue asked for and did not expect to get

The closing comment predicted a recurrence would yield the exception class only, no frames.
It now yields frames — a stack sink evidently landed since — and they settle the question:

[Warning] [MeshWeaver.Graph.CompileWatcher] Compile failure for TestData/BrokenType:
  NullReferenceException: Object reference not set to an instance of an object.
Exception: System.NullReferenceException: Object reference not set to an instance of an object.
   at Microsoft.CodeAnalysis.CSharp.Symbols.NamedTypeSymbol.Microsoft.Cci.ITypeDefinitionMember.get_ContainingTypeDefinition()
   at Microsoft.Cci.MetadataWriter.<GetConsolidatedTypeParameters>g__getConsolidatedTypeParameters|104_0(ITypeDefinition typeDef, ITypeDefinition owner)
   at Microsoft.Cci.MetadataWriter.<GetConsolidatedTypeParameters>g__getConsolidatedTypeParameters|104_0(ITypeDefinition typeDef, ITypeDefinition owner)
   at Microsoft.Cci.MetadataWriter.GetConsolidatedTypeParameters(ITypeDefinition typeDef)
   at Microsoft.Cci.FullMetadataWriter.CreateIndicesForNonTypeMembers(ITypeDefinition typeDef)
   at Microsoft.Cci.MetadataWriter.<CreateIndicesForModule>g__createIndices|102_0(ITypeDefinition typeDef, Queue`1 typesToIndex)
   at Microsoft.Cci.MetadataWriter.CreateIndicesForModule()
   at Microsoft.Cci.MetadataWriter.CreateIndices()
   at Microsoft.Cci.MetadataWriter.BuildMetadataAndIL(…)
   at Microsoft.Cci.PeWriter.WritePeToStream(…)
   at Microsoft.CodeAnalysis.Compilation.SerializePeToStream(…)
   at Microsoft.CodeAnalysis.Compilation.SerializeToPeStream(…)

Two premises in the issue body are now falsified:

"the NRE strikes PRE-Roslyn" — it does not. It strikes deep inside Roslyn, in the PE
metadata writer, during Emit. There is no MeshWeaver frame anywhere on the stack.
"BrokenType's failure should be a Roslyn diagnostic" — BrokenType's source is invalid,
but the crash is in the emit phase, which runs after binding. The reason BrokenType looks
like it "should not have reached Roslyn" is that it reached a later Roslyn phase than
expected, not an earlier one.

So the remaining suspect list should be rewritten. This is not a null-race in our pre-Roslyn
pipeline. GetConsolidatedTypeParameters recursing on ContainingTypeDefinition and hitting null
is a Roslyn symbol-graph inconsistency at emit time — the shapes that produce it are a
compilation being emitted while its symbol table is mutated or shared across threads, or a
CSharpCompilation/MetadataReference set reused across emits. Worth checking specifically:
whether MeshNodeCompilationService can emit the same Compilation (or share a
CSharpCompilation/reference set) from more than one emit concurrently under shard-2 load.

Not caused by the PR it surfaced on

PR #1359's diff at that commit is progress-log batching plus a file-publication change in
MeshWeaver.Hosting.Persistence (write temp + rename instead of writing the final name directly).
Neither can reach Roslyn's PE writer; the same eight tests pass locally in 18 s on that exact tree,
and every other shard on that run is green.

On the "0 recurrences" sweep

The sweep was sound but its window closed 2026-08-08. This is 2026-08-12 — so the correct reading
is "did not recur for a day and a half", not "fixed". The sweep comment's own suggestion still
stands and is now more clearly worth doing: CodeEditRecompileTest accepts any non-empty
CompilationError for BrokenType, so an isolated occurrence on an otherwise-green shard is
still invisible. Requiring a CS#### code there would turn the silent case red.

Sweep: this is 14 occurrences and rising, not a dormant one-off — and it is essentially the whole `Hosting.Monolith.Test` timeout population

Reopening the framing, not just the issue. The picture above (one occurrence, dormant, close-and-wait) is wrong in a way that matters for whoever picks this up, and three of the issue's own premises are falsified below.

Diagnostics to make the next occurrence answer the question: #1378.

1. Frequency: 14 occurrences, 2026-08-07 → 08-12

Swept 1,471 dotnet-test.yml runs / 6,814 testResults-shard* artifacts (all six shards, all attempts), plus an independent second detector over 7,165 shard job logs (CompilationError = …Object reference not set), both calibrated against the known occurrence before being trusted — run 31649727371 shard 2 yields its 12 NREs on both.

#	UTC	Run	Shard	Branch	NREs	`exit=124`	last classes before 1st NRE
1	08-07 06:20	31153582625	s2 att1	fix/674-clrmd-vacuous-pass	5*	yes	LateNackReenqueue › OverlaySelfHealInstanceRecycle
2	08-09 13:45	31316661302	s1	fix/whatsnew-frontmatter-main-red	8	yes	DynamicGraphIntegration › NodeAlcUnloadTeardownOrdering › MeshNodeLanguageService
3	08-10 22:28	31438448649	s2	fix/filing-reask-respects-explicit-request	12	yes	OverlaySelfHealInstanceRecycle › SubscribeRequestIdentityRouting › CodeEditRecompile
4	08-10 23:00	31440642821	s1	fix/1183-healthcheck-cancellation	13	no	AgentPickerProjectionPartitioned › MeshNodeStreamCacheIdleRelease › CompileSingleDriverConsistency
5	08-11 17:16	31516695942	s2	test/robust-invariant-assertions	14	yes	OwnerDisposalNack › KernelScriptMemoryLeak › CodeEditRecompile
6	08-11 20:57	31535550702	s4	fix/1214-torn-snapshot-regression-retraction	6	no	IncrementalUpdate › PluginDependencyOrder › SelfTypedRootInstall
7	08-12 08:55	31580606990	s4	fix/1250-live-supplement-rls-bypass	2	no	FreshMeshPartitionProvisioning › CommercialPackageAuthorization › StaticShadowedInstall
8	08-12 09:10	31581721627	s2	fix/1200-storm-key-path-discrimination	12	yes	MeshNodeStreamCacheTransientBreaker › KernelReplChaining › CodeEditRecompile
9	08-12 11:37	31592675453	s2	main (push)	11	yes	CreateLayoutAreaIntegration › SamplesGraphData › LinkedInPullActions
10	08-12 14:46	31608622132	s1	fix/edu-nav-unified-query	15	yes	GetDataRequestPropagation › ActivationBacklogFifo › NodeAlcUnloadTeardownOrdering
11	08-12 16:05	31615857313	s2	fix/1300-thread-teardown-classification	16	yes	UpsertInnerCreateObservation › RecursiveDeleteAuthoritative › CessionLayoutArea
12	08-12 16:06	31615938816	s2	fix/1314-log-incident-ingest-declared-identity	18	yes	OwnerDisposalNack › UpsertInnerCreateObservation › CompileLeafStability
13	08-12 16:28	31617758554	s2	fix/1311-synced-query-cache-ignores-query-set	20	yes	ContentNarrowingPersistence › ExportDocumentScriptRelay › CessionLayoutArea
14	08-12 23:07	31649727371	s2	fix/quadratic-activity-log-writes	12	yes	LinkedInPublishService › NodeTypeAssemblyLeak › CodeEditRecompile

* job-log-only (pre-instrumentation) — this is the occurrence the issue was filed on.

The rate is climbing. Normalised against shard artifacts that actually ran Hosting.Monolith.Test: 0.32% → 0.35% → 0.73% → 1.43% across Aug 9/10/11/12; per 1,000 Compile failure for events, 3.2 → 8.4. Twelve different PR branches plus a push to main — a main-carried, load-dependent defect, not a branch defect.

2. It is wearing a timeout costume

10 of the 14 ended exit=124. Conversely, of every exit=124 shard in the sweep: 10 are this defect, 3 are pre-instrumentation Hosting.Monolith.Test timeouts (one of them confirmed to be this), and 1 is an unrelated Cosmos.Test timeout. So "Hosting.Monolith.Test is a flaky project that times out" is almost entirely one defect.

The mechanism is visible in the traces and settles the causal direction the issue left open: once poisoned, every subsequent NodeType compile NREs and takes ~25–50 s instead of ~1 s, so the project blows its wall-clock cap. In run 31649727371 there are zero test-class transitions between 23:19:27 and 23:25:04 — six CodeEditRecompileTest tests each burning ~50 s — and the 8-minute cap fires at 23:26:05. exit=124 is a consequence, not a cause.

Worth checking whether #1368's exit=124 sightings belong here.

3. Not Monolith-specific — and there is a cheaper repro target

Occurrences 6 and 7 are on shard 4, running PluginCatalog / PluginImage / Content.Test. That is a far smaller surface than Monolith and it reproduced twice — the better place to try for a local repro.

No precursor class is necessary either: CodeEditRecompile ×4, NodeAlcUnloadTeardownOrdering ×2, OwnerDisposalNack ×2, UpsertInnerCreateObservation ×2, DynamicGraphIntegration ×2, CessionLayoutArea ×2, plus ContentCollectionReference, SamplesGraphData, CompileLeafStability, NodeTypeAssemblyLeak, KernelScriptMemoryLeak. alc=1–6, asm=123–164 at onset — no threshold. And in all 13 artifact-detected traces the compile NRE is the first NullReferenceException in the process: it is not downstream of an earlier fault.

4. Why the 2026-08-09 close looked safe — a third variant of "green on absent evidence"

Not carelessness. The sweep that justified the close was measuring an instrument that had barely been installed:

The primary signature cannot exist in any artifact before A compile fault's stack now reaches CI — #890 could never have been diagnosed #966 (037624314, merged 2026-08-08 21:59Z) — 0 of 2,111 pre-merge artifacts contain a single CompileWatcher line, against 3,614 of 4,703 after.
The closing sweep's window (08-07T07:00 → 08-09T06:00) therefore had roughly 8 instrumented hours out of 47.
Occurrence Test #2 landed 08-09 13:45 — hours after that window closed.

So "0 recurrences in 253 runs" was true and meant almost nothing. The lesson is the familiar one in a new costume: a sweep's sensitivity is a property of when the sink started existing, not of how many runs you read.

5. Three premises in the issue body are falsified

"No MeshWeaver frame present" is an artifact of truncation, not evidence. TestTraceLog.MaxExceptionChars = 2500; a .NET stack prints deepest-frame-first, the Roslyn emit frames consume the whole budget, and the record is cut off mid-Compilation.Emit — exactly where our frames begin. We still do not know which emit call site threw. A fault record was truncated exactly where its useful frames begin (#890 diagnostics) #1378 fixes this generally (elide the middle, keep both ends).
"The NRE strikes PRE-Roslyn" — no. Verified against Roslyn source (Compilation.cs:3030-3037): if (HasUnsuppressedErrors(diagnostics)) success = false; then if (success) SerializeToPeStream(...). The metadata writer is reached only on a clean bind. BrokenType logged two proper CompilationExceptions with CS diagnostics at 23:18:46.709/.745 and then NRE'd on a third compile 130 ms later whose source bound cleanly — so the NRE'ing compile was never compiling the broken source.
"Dormant / one documented occurrence" — see §1.

6. What the NRE actually is (most of an upstream report already)

From verbatim Roslyn source (5.6.0), MetadataWriter.GetConsolidatedTypeParameters:

INestedTypeDefinition nestedTypeDef = typeDef.AsNestedTypeDefinition(Context);   // returns `this` ONLY if ContainingType != null
if (nestedTypeDef != null && nestedTypeDef.InheritsEnclosingTypeTypeParameters)  // C#: hardcoded true
    result = getConsolidatedTypeParameters(nestedTypeDef.ContainingTypeDefinition, owner);   // <-- NRE

AsNestedTypeDefinitionImpl guards on ContainingType != null; ContainingTypeDefinition then does AdaptedNamedTypeSymbol.ContainingType.GetCciAdapter() and is preceded by Debug.Assert((object)AdaptedNamedTypeSymbol.ContainingType != null). In a Release build NamedTypeSymbol is the adapter and GetCciAdapter() => this.

So the same symbol's ContainingType read non-null and then null microseconds later — a Roslyn-internal invariant violation that Roslyn itself asserts cannot happen. If the #1378 canary implicates shared state, this is a dotnet/roslyn report and the paragraph above is most of the write-up.

7. Ruled out — by experiment, so nobody repeats it

A probe replicating GetDefaultReferences() verbatim plus the Monolith test bin (467 references — the real size) and the exact Emit(dll, pdb, xmlDoc, PortablePdb, DocumentationMode.Diagnose) shape, with depth-3 nested generics, nested lambdas, iterators, async and collection expressions, under DOTNET_PROCESSOR_COUNT=2. 881 emits, 0 failures:

hypothesis	test	result
static `MetadataReference` set collected/finalized	GC storms between emits	clean
ALC unload poisons emit	load + unload + reflect, sequential	clean
ALC unload / compacting GC racing in-flight emits	emit threads + GC thread + ALC churn + starved pool	361 emits, clean
concurrent emits sharing the static reference list	8 threads	clean
same `CSharpCompilation` emitted concurrently	8 threads, one instance	clean
`BoundLeg` cancelling an emit poisons shared PE symbols	106 aborted mid-flight, interleaved with clean emits	clean

Also excluded by inspection: source generators never run in these tests (no MeshWeaver.BusinessRules.Generator.dll in the test bin ⇒ BuiltInGeneratorPaths empty; SourceGeneratorLoader holds no shared state regardless); no shared CSharpCompilation between the LSP/speculative path and emit; no CI GC/JIT env overrides. CompileToReleaseAsync has zero non-bin callers (dead code), so the live path is EmitCompilationToDirectory.

Local repro attempts, both green and therefore both null: the CI-ordered 17-test subset under 2 CPUs (24 s), and the full 636-test Monolith project under 2 CPUs (7 min).

8. Onset is unrecoverable — say it plainly

Confirmed present 2026-08-07 06:20Z, which is before the instrumentation existed; Aug 5 artifacts and job logs are past retention. Roslyn was bumped 5.3.0 → 5.6.0 on 2026-08-02 (ee42d005a) and both the first occurrence and every later one are post-bump — but with the pre-Aug-7 evidence destroyed, that correlation is neither confirmed nor excluded. It should not be carried forward as a live suspicion, only as an untested one.

(Unrelated, noted so nobody re-discovers it as a suspect: Directory.Packages.props:166 still pins Microsoft.CodeAnalysis.Scripting at 5.3.0 while the rest of the set is 5.6.0. It is inert — Scripting.Common resolves to 5.6.0 and that is what ships.)

Next step

Land #1378, then wait for the first occurrence carrying a canary verdict — at 1.4% per run that should be within a day. The verdict closes off half the search space either way:

canary=OK → the fault is specific to that compilation's inputs → dump its generated source.
canary=THREW → process-wide Roslyn/reference state is poisoned → bisect the reference set → upstream.

🤖 Generated with Claude Code

Merge pull request #1378 from Systemorph/fix/890-readable-compile-fau…

The #1378 canary caught its first CI occurrence — run 31704966824 (PR #1426, shard 1, 2026-08-13 13:38–13:41 UTC), and it names the throw site as promised:

System.NullReferenceException: Object reference not set to an instance of an object.
   at Microsoft.CodeAnalysis.CSharp.Symbols.NamedTypeSymbol.Microsoft.Cci.ITypeDefinitionMember.get_ContainingTypeDefinition()

Canary verdict on the same failure: "a trivial emit against the SAME reference set fails too ⇒ process-wide Roslyn/reference state is POISONED, not this compilation's inputs (bisect the reference set)" — i.e. the NRE is inside Roslyn's emit against a shared symbol/reference, consistent with a collectible-ALC'd or torn MetadataReference surviving in a process-wide reference set. Cascade in this occurrence: NodeTypeAlcSharedWithInstancesTest → MeshNodeCompilationIntegrationTest (×2) → DynamicTypePreWarmerTest (×5) — everything compile-shaped on the shard after the first hit, matching the 2026-08-07 pattern.

The PR's diff (claim-election ordering + a grant window; no compile-pipeline surface) doesn't reach this code; the same suites are green locally on the branch. Triage can now start from NamedTypeSymbol.get_ContainingTypeDefinition + the reference-set bisect the canary suggests.

🤖 Generated with Claude Code

The canary fired, 95 minutes after #1378 merged — verdict: `canary=THREW`. Process-wide state is poisoned; half the search space is now closed.

#1378 merged 2026-08-13T12:04:10Z. The first instrumented occurrence landed 13:38:45Z. The canary was built to be falsifiable in one shot, and it was.

Run 31704966824, attempt 1, shard 1, branch fix/1424-claim-election, project MeshWeaver.Hosting.Monolith.Test (part 1/2), pid 3731. 20 canary records, all byte-identical, all the same verdict, verbatim:

canary=THREW NullReferenceException: Object reference not set to an instance of an object. — a trivial
emit against the SAME reference set fails too ⇒ process-wide Roslyn/reference state is POISONED, not
this compilation's inputs (bisect the reference set)

Never canary=OK, never canary=DIAGNOSTICS. So per the contract this issue set up:

canary=OK → the fault is specific to that compilation's inputs → dump its generated source.
canary=THREW → process-wide Roslyn/reference state is poisoned → bisect the reference set → upstream.

The generated-source half is dead. Nobody should spend time dumping a node's generated source, and type/AlcShareProbe's inputs are not interesting — a trivial depth-3 nested generic (MwEmitCanary<T>.Inner<U>.Leaf<V>) built fresh against the same MetadataReference instances threw the same NRE, in the same frame.

The truncation fix also worked — our frames are present now

The old head-cut kept the least useful end of the stack. The record now carries the full chain, and it reaches all the way into our code:

System.NullReferenceException: Object reference not set to an instance of an object.
   at Microsoft.CodeAnalysis.CSharp.Symbols.NamedTypeSymbol.Microsoft.Cci.ITypeDefinitionMember.get_ContainingTypeDefinition()
   at Microsoft.Cci.MetadataWriter.<GetConsolidatedTypeParameters>g__getConsolidatedTypeParameters|104_0(...)
   at Microsoft.Cci.MetadataWriter.<GetConsolidatedTypeParameters>g__getConsolidatedTypeParameters|104_0(...)
   at Microsoft.Cci.FullMetadataWriter.CreateIndicesForNonTypeMembers(ITypeDefinition typeDef)
   at Microsoft.Cci.MetadataWriter.CreateIndicesForModule()  … at Microsoft.Cci.PeWriter.WritePeToStream(…)
   at Microsoft.CodeAnalysis.Compilation.SerializeToPeStream(…)  at …Compilation.Emit(…)
   at MeshWeaver.Graph.Configuration.MeshNodeCompilationService.EmitCompilationToDirectory(…) MeshNodeCompilationService.cs:line 2101
   at …EmitToDiskWithRetry(…) :2283   at …CompileToDiskAsync(…) :2053   at …CompileAsyncCore(…) :1948
   at …RunCompileAndEvict(…) :1866    at …OnThreadPool… :74

The live emit call site is EmitCompilationToDirectory — as expected, not CompileToReleaseAsync (still dead code).

Shape of the occurrence

Onset: first NRE at 13:38:45.024, 293 ms into NodeTypeAlcSharedWithInstancesTest (INIT_START 13:38:44.731), on its very first compile. The three classes immediately before it were all ALC-heavy: NodeTypeRecompileAlcLeakTest, NodeAlcUnloadTeardownOrderingTest, NodeTypeReleaseTest (which compiled clean).
Total, no recovery: every subsequent emit in the process failed — 20 records over 17 distinct node paths across unrelated classes and partitions (type/AlcShareProbe, type/Story, type/Post, ACME/Project, Doc/…/Cession ×3, TestData/*, 5× PreWarmTest*), spanning 6 m 41 s.
Then a wedge, not an OOM: after the last NRE the process emitted only MEM_WATCHDOG managed=82MiB rss=1034MiB heartbeats — flat memory, zero test progress — until the 8 m cap killed it. exit=124 is a consequence, as §2 of the sweep above says.
Process-scoped: the next test host on the same runner (pid 4716) was clean.

The 17 paths narrow it further than the canary alone can

The canary is built against faulted.References, which is the static set plus that compile's own dynamic additions (MeshNodeCompilationService.cs:1904-1909). Taken alone it cannot say which half is poisoned. But the poisoning survived 17 different compilations with different dynamic reference additions, in different partitions, for 6+ minutes. The only thing all 17 share is the intersection:

// MeshNodeCompilationService.cs:179
private static readonly IReadOnlyList<MetadataReference> _references = GetDefaultReferences();

a process-wide static set of MetadataReference instances created via MetadataReference.CreateFromFile (mmap, lazy read), plus the Roslyn AssemblyMetadata / symbol caches that hang off those instances. That is where the bisect should start.

Worth stating plainly, because an earlier comment in this issue classified it as safe: _references was written off as one of "two static readonly fields … write-once constant lookups". It is write-once as a list, but the objects in it are not constants — PortableExecutableReference holds Metadata that is IDisposable and lazily memory-mapped, and Roslyn caches derived assembly/symbol tables against those instances. It is a mutable process-wide cache wearing static readonly clothing, which is exactly the shape AGENTS.md warns is both the likely mechanism here and a rule violation.

Note this does not contradict §7's ruled-out table. Those probes exercised a replica of the reference set under synthetic GC/ALC churn and stayed clean; what is now established is that in the real process the shared state does get poisoned. The probe's null result means the synthetic reproduction was not faithful, not that the shared state is innocent.

Sensitivity, stated honestly

	count
`dotnet-test.yml` runs created ≥ 2026-08-13T11:55Z (sweep at 13:45Z)	22
genuinely instrumented (tested tree contains `6dba926fc`)	19
shard artifacts downloaded and grepped	126
`_meshweaver-test-trace.log` examined	126 / 126

Instrumentation was resolved per tested commit, not per timestamp — for pull_request runs by pulling the real refs/pull/N/merge sha out of the checkout step and comparing the base, since a PR branch that has not merged main since 12:04Z carries no canary however late its run started. Three runs were excluded on exactly that basis. This is the correction the 2026-08-09 close needed and did not have: a sweep's sensitivity is a property of when the sink started existing.

Detector calibrated before being trusted: the same grep over the known pre-canary occurrence (run 31649727371, shard 2) returns its 12 Compile failure for … NullReferenceException records and 87 MetadataWriter|GetConsolidatedTypeParameters lines, and correctly returns 0 canary= there.

Second detector: across all 126 artifacts there is exactly one exit=124, and it is this same shard. So there is no likely-but-unconfirmed second occurrence in the window — every timeout is accounted for. An exhaustive grep for Microsoft.Cci.MetadataWriter / GetConsolidatedTypeParameters / canary= across every file in all 126 artifacts returns that one trace log and nothing else.

1 in 19 instrumented runs is consistent with (if anything above) the 1.4 %/run estimate in §1 — but n = 19, so treat it as "still roughly 1 %", not as a rate increase.

Two runs were still in flight at sweep time and are unmeasured, not clean: 31704966824 attempt 2 (re-run, shard 1 in progress) and 31706908171.

Next step

Bisect GetDefaultReferences() — the static set — rather than any node's source. The most direct experiment is to make the reference set per-mesh instead of process-wide and see whether the poisoning still crosses test classes; that is both the diagnostic and, if it holds, the fix, and it moves the code onto the right side of the no-static-state rule either way. If the poison survives a per-mesh reference set, the remaining shared state is Roslyn-internal and §6 of the sweep above is already most of the dotnet/roslyn write-up.

Not merged with #618, #1387 or #1412 — consistent with the corrections already in this thread.

Title corrected, the reference set is now per-mesh — and Roslyn's own source says the reference set is probably not the poison

Three things in this comment: the title fix (asked for explicitly), the change I am shipping and what it measurably costs, and a new piece of evidence that weakens the hypothesis the change is built on. I am stating the last one plainly because the change is a fix by hypothesis, and the evidence moved after I started.

1. Title

Was: "…every compile in the shard (incl. pre-Roslyn BrokenType) fails 'Object reference not set…'".
Now: "one Roslyn Emit poisons the whole process — every later compile NREs in Cci.MetadataWriter.GetConsolidatedTypeParameters, shard ends exit=124".

"pre-Roslyn" has been falsified twice in this thread and the old title kept sending triage to the pre-Roslyn pipeline. The new one names the frame and the exit=124 costume, which is what a triager actually needs.

2. What shipped

MeshNodeCompilationService._references — private static readonly IReadOnlyList<MetadataReference> — is gone. The set is now CompilationReferenceSet, a mesh-scoped singleton registered next to MeshNodeCompilationService, lazily materialized so a mesh that never compiles never builds one.

The reclassification is the point. That field was twice defended as "a write-once constant lookup", which is the sanctioned exemption in NoStaticState.md. The list is write-once; its elements are not. A PortableExecutableReference owns lazily materialized, IDisposable metadata and Roslyn caches derived assembly/symbol tables against that instance.

It is not disposed at mesh teardown and must not be: a compile can still be in flight when a mesh dies, and disposing metadata underneath one is a use-after-free on exactly the object graph whose corruption we are chasing.

The guard that lets this recur is now closed. NoStaticCollectionsTest matched only concrete collection types, so a static readonly IReadOnlyList<MetadataReference> was invisible to it — that is why the review call went the way it did. A new guard keys on the element type, so no declared-type spelling (array, ImmutableArray, IReadOnlyList, Lazy<>, plain field) can hide a Roslyn reference in static state again. I calibrated it by reintroducing the exact old field and confirming it reds.

The two surviving static holders are KernelScriptReferences.Materialized / SharedSnapshot (the kernel script path, a different compilation surface). They are deliberate — they exist because re-materializing per session leaked ~200 MiB/session — and are now explicitly allowlisted rather than incidentally invisible.

3. What it costs — measured, not asserted

macOS, .NET 10, Roslyn 5.6, the 470-reference set a Hosting.Monolith.Test process really has. Synthetic first (50 iterations, one emit each):

	shared set	per-mesh set
peak RSS delta	+310 MiB	+1076 MiB
per iteration	10.9 ms	83.7 ms

≈ 15 MiB and ~73 ms per reference set. It is a high-water mark, not a leak — five rounds of 50 sets plateau (937 → 1042 → 1058 → 1083 → 1078 MiB) rather than growing, so the native metadata is reused. But it is reclaimed only on finalization, which the GC has no pressure signal for.

Then the real thing, same tests both sides, all passing both sides:

slice	baseline (main)	with the change	delta
`CodeEditRecompileTest` (11 tests)	662 MiB / 18.4 s	1012 MiB / 19.6 s	+53% RSS, +7% time
`~Compile\|~NodeType` (133 tests)	1071 MiB / 137 s	1701 MiB / 152 s	+59% RSS, +11% time

Against the test host's 4 GiB MEM_PRESSURE / 6 GiB MEM_CRITICAL_FAILFAST, and the 1034 MiB the instrumented occurrence was sitting at. It should not FailFast, but it eats real headroom. Caveat: these are macOS numbers. On Linux the metadata is RssFile (mmap'd pages the kernel can reclaim under pressure), so the CI magnitude is untested — the direction holds, the size may not.

4. 🚨 The evidence moved: the reference set is probably not the poison

I pulled the exact Roslyn the portal runs — Microsoft.CodeAnalysis.CSharp 5.6.0 resolves to dotnet/roslyn@c0573ed0a7dc3e3b4d2e70da47f97cc51a35524f per the nuspec — and read the two sites on the NRE's frame.

The guard is ContainingType != null, confirming §6's reading (NamedTypeSymbolAdapter.cs:169-181):

private Cci.INestedTypeDefinition AsNestedTypeDefinitionImpl(PEModuleBuilder moduleBeingBuilt)
{
    if ((object)AdaptedNamedTypeSymbol.ContainingType != null &&
        AdaptedNamedTypeSymbol.IsDefinition &&
        AdaptedNamedTypeSymbol.ContainingModule == moduleBeingBuilt.SourceModule)
        return this;
    return null;
}

and in a Release Roslyn AdaptedNamedTypeSymbol => this and GetCciAdapter() => this, so get_ContainingTypeDefinition (:864-873) reduces to return this.ContainingType.GetCciAdapter();. The NRE means ContainingType read null on the same object that just read non-null. So far, so §6.

But the third clause is new information, and it points away from the reference set. ContainingModule == moduleBeingBuilt.SourceModule admits only a source symbol of the compilation being emitted. For the canary that is MwEmitCanary<T>.Inner<U>.Leaf<V> — symbols created milliseconds earlier from a freshly parsed tree. A PE symbol arriving from a MetadataReference cannot be the object that goes null. And ContainingType for such a symbol is _containingSymbol as NamedTypeSymbol over a readonly field set in the constructor (SourceMemberContainerSymbol.cs:192,235,788) — the as is the only way a non-null field reads null, and isinst returns null rather than crashing when the referent's method table is unreadable or reused.

And Roslyn has no cross-emit state left to blame. FullMetadataWriter is constructed per emit and its indices are instance fields; the only statics in MetadataWriter.cs are constants. AssemblyMetadata.CachedSymbols is a weak list of assembly symbols. The ObjectPool hypothesis (a double-Free() aliasing a pooled ArrayBuilder permanently — and note Roslyn's double-free assert is [Conditional("DEBUG")], so it would be silent in Release) does not fit this frame either: GetConsolidatedTypeParameters allocates new List<IGenericTypeParameter>(), uses no pooled builder, and no Symbol is ever pooled.

Also worth recording as a null result: GetConsolidatedTypeParameters returns 2 hits across all of GitHub, both of them this repo (#890, #1368). No dotnet/roslyn issue has ever reported this frame. The nearest dotnet/runtime GC-hole cluster (#129682 → PR #130321) is Windows-only, post-.NET-10, GCStress-only — not this.

So: leg 1 of the canary proved the fault is process-wide. It did not prove the fault is in the reference set, and the guard clause above is an argument that it is not.

5. So the canary now has a second leg that settles it

ProbeSharedEmitState no longer stops at "the shared set fails too". When leg 1 fails it re-runs the same source against a brand-new, minimal reference set that has never been handed to Roslyn:

canary=REFERENCES — pristine emits, shared does not ⇒ the poison really does travel with the reference instances, and this change is on the right axis.
canary=BELOW-ROSLYN — neither emits ⇒ brand-new references plus freshly parsed source still cannot emit, so nothing about the reference set explains it. The broken state is under Roslyn (CLR heap / JIT / GC) and no reference-set change can fix it.

It is calibrated: a test asserts that an unusable (empty) reference set yields canary=REFERENCES with pristine:OK, so the BELOW-ROSLYN branch is falsifiable rather than the answer to everything.

⚠️ Sweep vocabulary changed. Detectors grepping the literal canary=THREW will now miss; the token survives inside shared:THREW …. Grep canary= and read the leading verdict.

6. Honest status

Shipping on hypothesis. I did not find a deterministic provocation, and I do not believe one exists on the reference-set axis — §4 is an argument that the reference set cannot supply the null directly. What I have instead is (a) a real rule violation removed and permanently guarded, (b) the cost of removing it measured rather than hand-waved, and (c) an armed experiment that makes the next occurrence answer the remaining question in one shot instead of another sweep.

If the next occurrence reads canary=BELOW-ROSLYN, the search moves off Roslyn entirely, and the sharpest next probes are: a core dump (DOTNET_DbgEnableMiniDump=1), and a run with DOTNET_TieredCompilation=0 / DOTNET_TieredPGO=0 / DOTNET_ReadyToRun=0. The reason tiering is worth a shot: for the NRE to fire on fresh symbols on every emit for 6+ minutes, the guard read and the getter read must diverge repeatedly — that is a property of the compiled code at those two sites, not of one corrupted object. A bad guarded devirtualization installed at tier-1 is permanent for the process, process-wide, deterministic in one frame, starts abruptly once the method tiers up, and is gone in the next process. That fits every observation in this thread, including the flat-memory wedge.

DOTNET_HeapVerify=1 remains available as a sharper detector (measured non-trivial on a release runtime: 485 ms → 1914 ms on an identical GC-heavy loop, same GC counts).

docs(#890): the fault is ACQUIRED and then DEEPENS — both measured, n…

docs(#890): state the test-start count as what was RECORDED, with its…

docs(#890): a FOURTH occurrence, a four-day denominator, and a measur…

docs(#890): the cleanest control there is — same head, same set, re-r…

Reopened because the same process-wide failure recurred on 2026-09-28 in MeshWeaver.Plugins run 36413384318, job 108903097444. The first PROCESS CANNOT EMIT (#890) record is at 11:22:43Z in LinkedInProfileLayoutAreaTest: shared and pristine canaries throw NullReferenceException at NamedTypeSymbol...get_ContainingTypeDefinition, while compiler=PRIVATE-COPY-EMITS. Later unrelated NodeType compiles fail the same way; the host test ends at exit=124. This matches the four prior private-copy readings recorded above, and the root cause remains unresolved.

PR #5854 adds the next measurement: repeated emits inside one held private Roslyn copy on the first failure. It is diagnostic work, not a fix. The Plugins publication run is red; do not treat a later clean retry as proof this is gone.

Merge pull request #6200 from Systemorph/test/stand-in-compile-probes…

test(#890): pin the ROOT props half — mw-plugin-test and mw-combo-ver…

exp(#890): round 2 — the fix build, TieredPGO from runtimeconfig only

exp(#890): round 2 — list the tar before grepping it (pipefail)

Merge pull request #6254 from Systemorph/fix/890-jit-drops-isinst-tie…

docs(#890): round 2 — the fix build itself, 0 of 14 poisoned; why its…

Merge pull request #6264 from Systemorph/docs/890-round2-fix-build

Upstream report filed: dotnet/runtime#135442 (Tier1 Dynamic PGO drops an isinst inside a GDV-guarded inline, linux-x64 .NET 10.0.12). Our workaround is TieredPGO=false (core #6254, Plugins #3114).

CI: one Roslyn Emit poisons the whole process — every later compile NREs in Cci.MetadataWriter.GetConsolidatedTypeParameters, shard ends exit=124 #890

Description

Evidence

Why it is currently undiagnosable, and what changed

Probed and excluded (locally green)

Activity

rbuergi commented on Aug 8, 2026

Recurrence sweep: NO recurrence found — and one correction to the premise

What was searched

The grep was calibrated on the original occurrence

⚠️ Correction: the next occurrence will NOT name its throw site

Two things found while verifying

Read next to this

Status

rbuergi commented on Aug 8, 2026

Cross-link: #618 may be the same root cause

rbuergi commented on Aug 8, 2026

Correction: #618 is a SEPARATE defect — same shape, different mechanism

rbuergi commented on Aug 9, 2026

Recommend CLOSE — the sink is verified end-to-end in CI artifacts, and a 253-run sweep is clean

1. The sink genuinely reaches CI — verified on artifacts, not on code reading

2. Sweep: 0 recurrences, at a sensitivity the previous sweep did not have

Recommendation

rbuergi commented on Aug 9, 2026

Closing — dormant, and now genuinely diagnosable

The premise in this issue was wrong

The sink is now verified end-to-end, on downloaded artifacts

The sweep, with stated sensitivity

Residual limits, tracked not buried

rbuergi commented on Aug 12, 2026

Recurred — and this time it named its throw site. The premise was wrong: the NRE is INSIDE Roslyn, during Emit.

The stack, which this issue asked for and did not expect to get

Not caused by the PR it surfaced on

On the "0 recurrences" sweep

rbuergi commented on Aug 13, 2026

Sweep: this is 14 occurrences and rising, not a dormant one-off — and it is essentially the whole Hosting.Monolith.Test timeout population

1. Frequency: 14 occurrences, 2026-08-07 → 08-12

2. It is wearing a timeout costume

3. Not Monolith-specific — and there is a cheaper repro target

4. Why the 2026-08-09 close looked safe — a third variant of "green on absent evidence"

5. Three premises in the issue body are falsified

6. What the NRE actually is (most of an upstream report already)

7. Ruled out — by experiment, so nobody repeats it

8. Onset is unrecoverable — say it plainly

Next step

rbuergi commented on Aug 13, 2026

rbuergi commented on Aug 13, 2026

The canary fired, 95 minutes after #1378 merged — verdict: canary=THREW. Process-wide state is poisoned; half the search space is now closed.

The truncation fix also worked — our frames are present now

Shape of the occurrence

The 17 paths narrow it further than the canary alone can

Sensitivity, stated honestly

Next step

rbuergi commented on Aug 13, 2026

Title corrected, the reference set is now per-mesh — and Roslyn's own source says the reference set is probably not the poison

1. Title

2. What shipped

3. What it costs — measured, not asserted

4. 🚨 The evidence moved: the reference set is probably not the poison

5. So the canary now has a second leg that settles it

6. Honest status

64 remaining items

rbuergi commented on Sep 28, 2026

rbuergi commented on Oct 8, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Recurred — and this time it named its throw site. The premise was wrong: the NRE is INSIDE Roslyn, during `Emit`.

Sweep: this is 14 occurrences and rising, not a dormant one-off — and it is essentially the whole `Hosting.Monolith.Test` timeout population

The canary fired, 95 minutes after #1378 merged — verdict: `canary=THREW`. Process-wide state is poisoned; half the search space is now closed.