MeshWeaver#890 recurred 3x in 30h with the leg-5 reading it was closed waiting for — reopen-or-new-path call needed

MeshWeaver#890 (CI: one Roslyn Emit poisons the whole process; shard ends exit=124) is CLOSED since 2026-09-13. It has now occurred three times inside thirty hours, and each occurrence carries the exact reading that issue said would close a question. The reopen-versus-new-path decision belongs to whoever owns the fix, which is why this is a triage filing and not a reopen.

OCCURRENCES. All three are MeshWeaver.Plugins, workflow Plugin Catalog CI, job portal-hosts (Hosting.Monolith.Test), all exit 124 with no verdict:

2026-09-21 13:36Z, run 35606133491, job 106355181093, branch fix/incident-fold-dedupe, platform set 3.0.0-ci.9088 (core 88cd4b7). canary=BELOW-ROSLYN x40, dissect=READS-HEALTHY x40, flat=SAME-FRAME@NamedTypeSymbol.Microsoft.Cci.ITypeDefinitionMember.get_ContainingTypeDefinition x40, compiler=PRIVATE-COPY-EMITS x40.
2026-09-21 22:41Z, run 35661949332, job 106544668720, branch fix/hosting-bake-probe, platform set 3.0.0-ci.9108 (core b1aa8a9). Same canary/dissect x58; flat=EMITS for the first SIX failures and flat=SAME-FRAME for the following FIFTY-TWO, in one process; compiler=PRIVATE-COPY-EMITS x58.
2026-09-22 11:37Z, run 35722501930, job 106742555061, branch fix/ai-stream-cancel (PR 2231), platform set 3.0.0-ci.9168 (core 620a489). Same canary/dissect x58, flat=EMITS x58, compiler=PRIVATE-COPY-EMITS x58.

WHAT THE READINGS SETTLE. compiler=PRIVATE-COPY-EMITS means a second Roslyn, loaded from fresh bytes into its own collectible context and never executed in this process before, emits the very source the shared copy cannot, in the same process on the same CLR and heap. So BELOW-ROSLYN is void on all three and the dotnet/runtime venue is wrong. And flat= turns out to be a PHASE, not a property of an occurrence: occurrence 2 read both values in one process, shallow first. That closes the 2026-09-08 vs 2026-09-10 contradiction recorded on the issue (one defect at two depths) and retires confined-to-GetConsolidatedTypeParameters-recursion as a general claim. Of the three candidates leg 5 leaves open (the image, its mapping, the native code produced for it), only the third can get worse while a process runs.

NEXT MEASUREMENT, as the issue already prescribes: repeat the private emit until it tiers up. If it then fails, the fault is in what tiering produces; if it never does, it is the shared image or its mapping. This does NOT yet exclude a corrupted static reachable only from the MetadataWriter call site.

WHAT IS NOT THE CAUSE. Not a pull request: three unrelated branches. Not the platform ceiling: three different sets, and the diff between core 6b3fda2 (a set that ran the same suite green, 813 of 813, run 35700505749) and core 620a489 touches nothing under src/MeshWeaver.Compiler or src/MeshWeaver.Compiler.Pipeline, with Microsoft.CodeAnalysis.CSharp 5.9.0 and global.json unchanged.

RATE WITH ITS DENOMINATOR. Since 2026-09-21T00:00Z the unit ran 102 times across the workflow: 75 success, 22 cancelled (superseded pushes), 2 running, 3 failure, and all three failures are this defect. All 102 are pull_request runs: Hosting.Monolith.Test was not selected on main once in that window (main ran Kernel.Test, network-133, Json.Test and others instead), so it passes on main is a zero denominator, not a green, and a single re-run cannot supply the control either.

USER IMPACT. Each occurrence kills an otherwise-healthy pull request run with no verdict, and the red reads as the PR's own until someone spends an hour proving otherwise.

EVIDENCE AND WRITE-UP. Issue comment: github.com//issues/890#issuecomment-5777968513. Committed to Doc/Architecture/NodeTypeCompilation in pull request github.com//pull/5211. Attribution note on the affected PR: github.com/Systemorph/MeshWeaver.Plugins/pull/2231#issuecomment-5777974872.

WHAT I DID NOT ESTABLISH. Whether this is a regression of the fix that closed 890 or a second path to the same symptom; whether a successful emit preceded the first poisoned one in occurrence 3 (successful compiles are not logged, so the evidence is indirect); and the mapping-versus-native-code split, which the tier-up repeat is the experiment for.

Origin: https://memex.systemorph.com/rbuergi/Feedback/890-roslyn-emit-poison-three-occurrences-2026-09-22

filed by the triage agent on behalf of rbuergi via systemorph-com

Two amendments to the body above — it was minted from the filing a few minutes before both were established.

1. The rate is a floor, and the floor now states what it could not see. A cancelled shard can
carry the defect (filtering a population on its conclusion is the exact mistake MeshWeaver#890's
2026-09-12 correction records), so the 22 cancelled executions were read rather than assumed: 18 logs
clean of both canary=BELOW-ROSLYN and PROCESS CANNOT EMIT, 4 with no retrievable log. 3 of 96
determined, 6 undetermined.

2. The "next measurement" as stated could not have been carried out, and this is the part worth
carrying forward. Both #890 and the leg's own printed residual said "repeat the private emit until it
tiers up". PrivateRoslynCopy.Emit creates its own collectible AssemblyLoadContext per invocation
and unloads it in finally (src/MeshWeaver.Compiler/PrivateRoslynCopy.cs:85, :113-115), so N calls
are N cold copies — the same tier-0 code measured N times, tiering nothing. The experiment needs
ONE copy held loaded while the emit is repeated inside it, with the unload deferred to the end: a
second entry point, not a loop at the existing one. Caught by the automatic review on
#5211, which corrects it in the XML doc, in the message an
operator actually reads, and in Doc/Architecture/NodeTypeCompilation.

That PR also carries the occurrence table, the flat= phase finding, and the four dead ends (platform
set, main as a control, the exit 124 arithmetic, the straggler capture). It does not attempt a fix:
the reopen-versus-new-path call on #890, and the tier-up entry point, both belong to whoever owns that
issue.

A fourth occurrence, and a four-day denominator. Widening the sweep to 2026-09-19 found one more,
which also strengthens the attribution: the sets now span four, not three.

occurrence	run / job	platform set (core)	branch	`flat=`	`compiler=`
2026-09-19 08:22Z	`35430730563` / `105867058225`	`3.0.0-ci.8963` (`31be311dc`)	`fix/4740-plugins-runner-timeouts`	`EMITS` ×52	`PRIVATE-COPY-EMITS` ×52
2026-09-21 13:36Z	`35606133491` / `106355181093`	`3.0.0-ci.9088` (`88cd4b7b4`)	`fix/incident-fold-dedupe`	`SAME-FRAME@…` ×40	×40
2026-09-21 22:41Z	`35661949332` / `106544668720`	`3.0.0-ci.9108` (`b1aa8a9e5`)	`fix/hosting-bake-probe`	`EMITS` ×6 then `SAME-FRAME@…` ×52	×58
2026-09-22 11:37Z	`35722501930` / `106742555061`	`3.0.0-ci.9168` (`620a4893a`)	`fix/ai-stream-cancel`	`EMITS` ×58	×58

Denominator. 201 executions of portal-hosts (Hosting.Monolith.Test) since 2026-09-19T00:00Z: 153
success, 39 cancelled, 1 running, 8 failure — 4 of the 8 are this defect; the other 4 carry no
canary= and no exit 124 at all. 🚨 All 201 are pull_request runs. The unit was not selected on
main once in four days, so the absence of a main control is a zero denominator over the wider window
too, not just the narrow one.

Core's own CI over the same days is a measured negative, and a weak one: 24 MeshWeaver Build and Test runs since 2026-09-20, 13 failed jobs between them, none carrying PROCESS CANNOT EMIT,
canary=BELOW-ROSLYN or exit 124. That localises the observation to the Plugins Monolith unit; it
does not exonerate core, because 13 failures is thin and core's suites do not drive in-process NodeType
compilation at that volume.

And the fault is ACQUIRED, then progressive — both measured. In the newest occurrence
OverlaySelfHealInstanceRecycleTest.OverlaidInstance_SelfRecycles_WhenTypeCompilesGreen passed at
12:52:07.654Z, and that test asserts a NodeType reaching CompilationStatus.Ok with a usable build
whose instance then renders a marker — 33 seconds before the first poisoned emit at 12:52:40Z. So
nothing arrived broken. The 22:41Z occurrence supplies the other half inside one process: flat=EMITS
for six failures, then flat=SAME-FRAME for fifty-two. That is the shape a tier-up produces and not
the shape a bad file produces, which is what makes the (corrected) tier-up experiment worth running.

All of it is in #5211.

The cleanest control arrived on its own, and it is worth recording as the canonical one for this
failure mode.

Attempt 2 of run 35722501930 re-ran the same job (106780475056, 14:07:57Z) with head
353b8137b unchanged and platform mount: set 3.0.0-ci.9168 (core 620a4893a) unchanged, on the
same runner pool, and reported Test run summary: Passed! total: 813, failed: 0 with zero
canary=BELOW-ROSLYN, in 14 minutes against the 900 s cap. Attempt 1 of that same run had 58 canary
blocks and died at exit 124.

One comparison excludes the pull request's diff, the platform set and the content of the suite at
once — none of them changed between the attempts.

🚨 And it names why this defect is expensive to read. A re-run clears it. To whoever re-runs, that
reads as a flake; to whoever only saw attempt 1, it reads as the pull request's fault. It is neither —
it is a process that stopped being able to emit, and the next process that did not. The instrument that
tells the difference is the emit canary in the job log: a control compilation failing alongside the
real ones. Without that line, "some in-mesh compiles NRE'd" reads as bad content in those NodeTypes.

Recorded in #5211.

Classification: same defect as #890, NOT a new path — and #890 was never closed on a fix, so this is not a regression either. Keep THIS issue as the live tracker; leave #890 closed as the instrument history.

Why, from #890's own record:

CI: one Roslyn Emit poisons the whole process — every later compile NREs in Cci.MetadataWriter.GetConsolidatedTypeParameters, shard ends exit=124 #890 was closed 2026-09-13T09:47Z (completed) when leg 5 of the canary — the pristine-compiler control, feat: leg 5 of the #890 canary — the pristine-COMPILER control (compiler=PRIVATE-COPY-EMITS / THREW) #4165 — was built. Its closing condition, stated on 2026-09-13, was "the reading on an occurrence": an instrument, not a remedy. No change to Roslyn's use, the reference set or the emit path was merged against it after that.
All four occurrences here carry the same signature CI: one Roslyn Emit poisons the whole process — every later compile NREs in Cci.MetadataWriter.GetConsolidatedTypeParameters, shard ends exit=124 #890 was about (PROCESS CANNOT EMIT (#890), canary=BELOW-ROSLYN, dissect=READS-HEALTHY, exit 124 on portal-hosts (Hosting.Monolith.Test)), and the leg-5 reading CI: one Roslyn Emit poisons the whole process — every later compile NREs in Cci.MetadataWriter.GetConsolidatedTypeParameters, shard ends exit=124 #890 was waiting for (compiler=PRIVATE-COPY-EMITS) on every one. A "different path sharing the fingerprint" would need a reading that disagrees somewhere; none does.
So the reopen-vs-new question resolves to: the defect is unfixed and live; its diagnosis advanced (BELOW-ROSLYN void; the fault is in the shared Roslyn copy's image, its mapping, or the native code tiering produces for it; flat= is a phase). Reopening CI: one Roslyn Emit poisons the whole process — every later compile NREs in Cci.MetadataWriter.GetConsolidatedTypeParameters, shard ends exit=124 #890 would bury that under 40 days of superseded readings; this issue carries the current state and fix(#890): the first four compiler= readings — and the tier-up instruction they made testable, which was wrong #5211 committed it to Doc/Architecture/NodeTypeCompilation.

The next step, precisely (from the 2026-09-22 amendment — the old instruction could not be executed): a second entry point that holds ONE PrivateRoslynCopy loaded and repeats the emit inside it until it tiers up, deferring the unload to the end (PrivateRoslynCopy.Emit today creates and unloads a fresh collectible context per call, src/MeshWeaver.Compiler/PrivateRoslynCopy.cs:85, :113-115, so N calls are N cold copies). If the held copy starts failing after tier-up, the fault is in what tiering produces; if it never does, it is the shared image or its mapping. This is diagnosis work on the owner's instrument, not a fix I can make from the evidence here.

Not established: the rate since 2026-09-22 14:30Z (I did not re-sweep portal-hosts runs); whether a corrupted static reachable only from the MetadataWriter call site is excluded (the 2026-09-22 body already says it is not).

It now happens on main: two occurrences today in a new suite, both starting at the same test, about 6 % of runs of that leg.

Plugins run / job (attempt 1)	set (core)	canary / dissect / flat / compiler	first poisoned compile
35974400848 / 107555406130	3.0.0-ci.9287 (`bf5ac85`)	BELOW-ROSLYN ×43 / READS-HEALTHY / EMITS / PRIVATE-COPY-EMITS	`Widget/Thing` 08:45:39Z
35985209459 / 107589527570	3.0.0-ci.9296 (`8d3fd9f`)	×41, same readings	`Widget/Thing` 10:27:55Z

Both are portal-hosts (network-133 · leg 4/8) running MeshWeaver.PluginCatalog.Test on runtime 10.0.12 linux-x64 with Roslyn 5.9.0, and both went green on attempt 2. The stack matches every earlier occurrence: CreateIndicesForNonTypeMembers → getConsolidatedTypeParameters ×2 → NamedTypeSymbol.ITypeDefinitionMember.get_ContainingTypeDefinition.

Denominator: 2 of 33 executions of that leg since 09-23T00:00Z. The two newest main runs (36002604963, 35997990577) are red for other reasons and carry no canary.
Onset position: identical in both. It is the same test start in the leg's deterministic order, RetiredNodePruneTest.SharedSourceChange_… compiling Widget/Thing. Earlier emits in the same host had succeeded (ADependentsSource_CompilesAgainstTheTypeItsDependencyShips asserts Ok and passed about 2 minutes before). So the same workload poisons about one process in 16, which means something is decided per process.
Population harness, 0 occurrences: ≥600 short processes × 1,500 emits with the pipeline's options (osx-arm64, runtime 10.0.11), where about 36 were expected. The 800k-emit loop on 09-10 was a single process, so it was a single draw.
Ruled out as the trigger: core fix(compiler): keep NodeType compiles off the silo's shared ThreadPool #5327/fix(compiler): bound compile CPU on a dedicated-thread IIoPool lane #5375 (the CompileCpu lane, ConcurrentBuild off). The defect predates them by a month. Shared references and the shared Compilation are also out: legs 1 and 2 exclude them, and the private copy emits.
Hypothesis, not established: the tier-1 code of the guard's call path. AsNestedTypeDefinitionImpl's tier-1 inlines _containingSymbol as NamedTypeSymbol behind GDV plus a profile-guided cast expansion. Its shape varies per process, which fits per-process randomness, acquire-then-deepen, the cold private copy's immunity, and a consistent frame.
Instrument: Systemorph/MeshWeaver.Plugins#2357 writes DOTNET_JitDisasm listings of the six methods on that path into test-logs/jit-890-listing.log. The file is uploaded only when a suite fails. The next occurrence will therefore carry the exact native code that answered the guard. The write-up and how to read the listing are in docs(#5212): #890 on main — one onset position, a per-process harness null, and the JIT-listing capture #5663 (Doc/Architecture/NodeTypeCompilation).

Not established: whether production replicas reach this state. A memex Logs read of PROCESS CANNOT EMIT over 3 h returned 0, with no positive control. The 12 h and 3-day reads on memex and memex-cloud were refused by Loki (timeout, then 429). The harness null is arm64 only. In the 10:27 host, a TypeLoadException … format is invalid on BulkPack/Thing 51 s before onset is recorded but not concluded from: that text is also the known load-during-unload tear, and the 08:45 host does not show it.

Severity is a triage call I'm leaving open. At sev:M this now reddens main in about 6 % of runs of this leg.

docs(#5212): review — per-attempt reruns, 'not by workload alone', ad…

docs(#5212): full census — 3 of 87 leg executions, all three at the s…

Census correction to my comment above: the full sweep has now finished. It covers 87 executions of portal-hosts (network-133 · leg 4/8) from 09-23T00:00Z to 09-24T14:00Z, and 84 of them ran PluginCatalog.Test. 3 carry the canary (≈3.5 %), not 2 of 33. The third is run 35906530904 / job 107342449379 on 09-23 at 19:34Z, set 3.0.0-ci.9263 (core 9a61fed), readings BELOW-ROSLYN / READS-HEALTHY / flat=SAME-FRAME / PRIVATE-COPY-EMITS ×43. Its first poisoned compile is again Widget/Thing in RetiredNodePruneTest.SharedSourceChange_…, so all three occurrences start at the same onset test. #5663 carries the corrected table.

docs(#5212): the workflow-wide sweep — six occurrences in 31 h, four …

Workflow-wide sweep finished. I read every non-success job of every Plugin Catalog CI attempt from 09-23T06:46Z to 09-24T14:00Z (1,060 job logs). It finds six occurrences in 31 h, four of them on main: the three in network-133 · leg 4/8 (PluginCatalog.Test, all starting at Widget/Thing), plus three in network-133 · leg 1/8 (Hosting.Monolith.Test). Those are 35828369047 (main, TestData/UploadLegWedgeType), 35846873901 and 35858189010 (PRs, both starting at TestData/SatisfiedType). #5663 carries the list.

New attributable occurrence blocking the 9945 fleet arm: Plugins core-candidate run 37211768667, leg 1/12 job 111464402311 (Hosting.Monolith.Test), core 87422043 + Plugins fd3dbd1e. The host emitted successfully before the first poisoned compile at 2026-10-04 15:15:13 UTC (Systemorph/LinkedInProfile); then PROCESS CANNOT EMIT (#890) recurred across unrelated NodeTypes. Its first canary has shared/pristine NullReferenceException at NamedTypeSymbol.Microsoft.Cci.ITypeDefinitionMember.get_ContainingTypeDefinition, dissect=READS-HEALTHY, flat=EMITS, compiler=PRIVATE-COPY-EMITS. The leg kept running through many failing assertions, was canceled at 15:53:29, and uploaded a log but no .trx. The paired verdict correctly reports missingEvidence: 1, so arm-promoted-set refused 9945 at 16:15:12; production remains 9939. This is the established acquired process-wide emit failure, not evidence that the 9945 diff caused it. I am investigating the missing JIT capture/root cause; the gate must stay red until there is a valid verdict.

Candidate core-pair run 37211768667 exposed an evidence gap: the ordinary Plugins portal-host lane captures the targeted Roslyn JIT listing, but the core-candidate leg did not, and this canceled host left no .trx. Plugins PR #2873 now adds the same JIT capture to candidate/control leg artifacts and documents it. Local actionlint and workflow-shell parsing pass. The candidate verdict remains red on missing test evidence. This is instrumentation for the next attributable occurrence, not a root-cause fix or a reason to arm 9945; the PR itself is waiting on the required review/CI gate.

test(#5212): pin NO-BASELINE-MAPPED-DIFFERS; host= and baseline-cost …

Merge pull request #6220 from Systemorph/fix/5212-emit-image-integrit…

Another occurrence: core PR #6231, head 64317ea803, run job 112723124834 (shard 3, Memex.Portal.Shared.Test).

Failure: NullReferenceException at NamedTypeSymbol.Microsoft.Cci.ITypeDefinitionMember.get_ContainingTypeDefinition in MetadataWriter.PopulateNestedClassTableRows. The compile watcher logged PROCESS CANNOT EMIT (#890) with canary=BELOW-ROSLYN.
First poisoned compile: InstallNeverInheritsRepoCompileVerdictTest at 09:30:42Z, the 69th of 150 test classes in that process. After it, ModulesUpdateIndependentlyOfThePlatformTest (2 tests) failed with the same NRE.
Not the PR: the PR doesn't touch the compiler, and the same classes pass locally on the same head (7/7). Shard 3 passed on the branch's previous head.
Not established: what poisoned the process before 09:30:42Z, because successful compiles aren't logged.

PR #6231 shard 3 (2 of 2): what a local reproduction attempt established. It did NOT reproduce.

Scope. I set out to reproduce the 2-of-2 onset on PR #6231 locally, bisect the preceding classes, find the mechanism and fix it. None of the 11 local processes reproduced it, so there is no minimal repro and no fix in this comment. Below is what the attempt established and what it excluded. Some of the findings correct earlier statements on this issue.

1. The two CI processes

Jobs 112723124834 and 112729207042 ran the same tested tree: head 64317ea80 merged onto main@fa6bb901bf (fresh merge 6930b34bcf38). The tests ran Memex.Portal.Shared.Test part 1/2 (161 of 321 classes, every second class from the sorted list), with seed 1319993060, parallel mode = none, on .NET 10.0.12 linux-x64.
The method order is identical in both jobs (280 test methods, diffed). The first poisoned compile is VerdictPkg/Catalog in InstallNeverInheritsRepoCompileVerdictTest, the 70th class. The 69 classes before it are SettingsRowActionsActOnTheClickedRowTest (Fix chat composer losing focus when prompt ends with question mark #69) and earlier; the full list is reproducible from either log.
First canary reading, job 1, 09:30:42.611Z: canary=BELOW-ROSLYN, shared and pristine both THREW at NamedTypeSymbol…get_ContainingTypeDefinition, dissect=READS-HEALTHY, flat=SAME-FRAME, compiler=PRIVATE-COPY-EMITS, held=EMITS(64/64), image=INTACT (metadata, IL of in-scope methods and file hashes all unchanged), host=(.NET 10.0.12 linux-x64 AMD EPYC 7763, 4 cores, avx2, no avx512, tiering=defaults).
Between the last test that is known to have compiled successfully (DiagnosticsNameTheContentTypeAssemblyTest, 09:30:17) and the onset, neither log contains a TypeLoad, BadImageFormat, "format is invalid", unload or AccessViolation line.
The onset position is not tied to one preceding class. Across the three core occurrences the onset test differs: LinkVerdictPerPlatformTest (2.4 #57, job 111891768728), ModulesUpdateIndependentlyOfThePlatformTest (job 112147039747), and InstallNeverInherits… (Fix incorrect chat composer behavior when prompt contains slash character #70, today). SettingsRowActions… and PackageUninstallTest come before the onset in two of the three, but not in the 10-05 job. What all three share is the position: the first compile about 4 to 5 minutes into the process, after a stretch of classes that do not compile.

2. 🚨 Correction: CI's seed does not reproduce CI's order locally

xUnit v3 orders test collections by unique IDs that hash the assembly path, so :<seed> with the same class set runs in a different order on another machine. Measured: same seed, same 161 classes, and locally the first class was StandInCompileFailureNamesTheEmitCanaryTest where CI's was AFailedPackageNamesItsCauseTest. The 10-06 "full run with CI's seed, 0 failed" therefore did not run CI's order. To replay CI's exact order I used a local-only [assembly: TestCollectionOrderer] plus [assembly: TestMethodOrderer] that read the class and method order from the CI log. The resulting method order diffed empty against CI's.

3. Local attempts (all on the merged tree, built Release; 0 of 11 processes poisoned)

environment	runs	order	result
macOS arm64, .NET 10.0.11	1	CI class set, local order	1305 passed, no canary
linux-arm64 container, 10.0.12, 4 CPUs	1	exact CI order	380 passed, no canary
macOS x64 (Rosetta 2), 10.0.12	5	exact CI order (3 of the 5 with `DOTNET_PROCESSOR_COUNT=4`)	all clean
macOS x64 (Rosetta), `TC_CallCountThreshold=5`, `TC_CallCountingDelayMs=0`	1	exact CI order	clean
macOS x64 (Rosetta), full suite (321 classes)	1	default	2409 tests, 1 unrelated failure (no x64 SDK for `dotnet msbuild`), no canary
linux-x64 under qemu (`EnableWriteXorExecute=0`, which qemu needs)	1	exact CI order	380 passed, no canary

Two more harnesses also came back clean:

Standalone emit population on x64 (Rosetta): 160 processes × 300 emits. Each process uses a randomized mix of nested and generic types, lambdas (display classes), static lambdas, async and iterator state machines, anonymous types, records and LINQ, emitted with portable PDB and XML docs, with the flat canary after every emit. 0 poisoned. All four writer methods reach Tier1 in this harness.
Roslyn-copy probe (a local class placed just before the onset class): there is exactly one Microsoft.CodeAnalysis/.CSharp in the process, in the Default ALC, and no collectible ALC is alive at that point.

4. What `DOTNET_JitDisasmSummary` shows on x64 (new)

Under CI's exact order with default tiering, the emit path never leaves Tier0 locally. In 5 of 5 runs, MetadataWriter.PopulateNestedClassTableRows, getConsolidatedTypeParameters, FullMetadataWriter.CreateIndicesForNonTypeMembers and NamedTypeSymbol.AsNestedTypeDefinitionImpl were still at (Instrumented) Tier0 through class 71. So locally, class 70 runs unoptimized emit code. Whether CI's process had promoted these methods by then is unknown; nothing in a CI log records it.
When they do reach Tier1 locally (with the lowered threshold, in the full suite, and in the harness), every one is Tier1 with Synthesized PGO, and none poisons. The Tier1 listing of AsNestedTypeDefinitionImpl on x64 is correct. It does a GDV on SourceNamedTypeSymbol and inlines _containingSymbol as NamedTypeSymbol as null → null, MT == SourceNamedTypeSymbol → self, otherwise CORINFO_HELP_ISINSTANCEOFCLASS. That is the right null polarity for a top-level type.
This sharpens the codegen hypothesis without confirming it. If the cause is Tier1 code, it needs a Dynamic-PGO profile shape that none of the local runs produced. The onset falling on the first compile after a quiet stretch fits that: call counting and tier-up happen in idle windows, such as the four 500 ms NotEmit waits in SettingsRowActions…. That is a fit, not proof.

5. Excluded by reading the runtime source (release/10.0)

A stale collectible type in a PGO profile. JIT_ClassProfile32/64 records DEFAULT_UNKNOWN_HANDLE for collectible types, and the delegate and vtable probes skip collectible MethodDescs (jithelpers.cpp 1957–2189).
A stale CastCache entry after an unload. LoaderAllocator flushes it (CastCache::FlushCurrentCache(), loaderallocator.cpp:631).
A shared System.Reflection.Metadata or System.Collections.Immutable. PrivateRoslynCopy loads only Microsoft.CodeAnalysis and .CSharp privately and shares those two from Default, yet it emits. That clears them.
Corrupted pooled objects or other Roslyn static state do not explain flat=SAME-FRAME. PopulateNestedClassTableRows and the guard path use no pooled objects, and a member-less top-level class can only reach the frame through a guard misread. A corrupted static that is reachable only from the writer's call site is still not formally excluded.

6. Incidental finding, found along the way

A stock Linux container (fs.inotify.max_user_instances=128) fails 305 of 380 tests in this shard with IOException: The configured user limit (128) on the number of inotify instances has been reached. The trace runs ServiceSetup.CreateServiceCollection (test/MeshWeaver.Fixture/ServiceSetup.cs:37) → JsonConfigurationSource.Build → PhysicalFilesWatcher. That looks like one config file watcher per test that is never released. GitHub runners evidently allow more instances, so CI does not see it. Not triaged further.

7. Next step

This composition reproduced 2 of 2 on CI, so one CI run of shard 3 on this tree is decisive in a way 11 local processes were not. Run it with two arms:

(a) DOTNET_JitStdOutFile + DOTNET_JitDisasmSummary=1 + DOTNET_JitDisasm="PopulateNestedClassTableRows AsNestedTypeDefinitionImpl *getConsolidatedTypeParameters* CreateIndicesForNonTypeMembers", which shows whether a Tier1 version of these methods existed before 09:30:42 and what code it contained;
(b) DOTNET_TieredPGO=0, the split arm.

If (b) is clean and (a) shows a Dynamic-PGO Tier1 body with an inverted null branch, the venue is dotnet/runtime, and that listing is the report. If (a) shows the methods still at Tier0 at onset, the codegen hypothesis is dead and the search moves to process state.

I did not run that experiment: this task excluded CI polling.

Three more core occurrences on 2026-10-07, all shard 3 Memex.Portal.Shared.Test (part 1/2, 322 class args). They were reported as "caused by #6208 / the fresh-merge base". They are not.

run / job	event, tested tree	first bad reading	failing tests
37612813368 / 112766252743	push main `cbb3053100` (11:15Z), before #6208 merged	`canary=BELOW-ROSLYN` ×11	InstallNeverInherits…, ModulesUpdateIndependentlyOfThePlatformTest ×2
37637952561 / 112852969409	PR #6231 head `a9fa0f8dbb` fresh-merged onto main `cac0664de`	14:49:36Z `PROCESS CANNOT EMIT`, `canary=BELOW-ROSLYN` (~4m41s into the process)	the same 3
37637806820 / 112852220267	PR #6244 (workflow/doc only) fresh-merged onto main `cac0664de`	14:52:50Z CS0246 on a stand-in passed in the reference list, `canary=OK` (~3m57s in)	LinkVerdictPerPlatformTest ×1, ModuleLinkVersionTest ×11, then the same 3

New in the third one: the shallow phase reads canary=OK. The first 12 failures are stand-in compiles (StandInCompile.Emit) where the extra MetadataReference.CreateFromImage(...) is invisible (CS0246 'YamlDotNet' ×9, CS0246 'PerPlatformVerdictFromTheFuture', CS0234 'MeshWeaver.Test' ×2) while the canary, which compiles against the same reference set WITHOUT an extra image reference, still emits. 15 s later (14:53:05Z) the same process reads canary=BELOW-ROSLYN (shared: and pristine: both THREW NRE at MetadataWriter.<GetConsolidatedTypeParameters>). So the canary misses the onset; the "invisible extra reference" symptom comes first in this process.

Non-determinism, same inputs: both PR runs tested a fresh merge onto the same main cac0664de with nothing in their diffs that reaches the compiler, and LinkVerdictPerPlatformTest/ModuleLinkVersionTest passed in the #6231 process and failed in the #6244 process. Main's own push run of cac0664de (37625265553) passed shard 3.

Not established: what poisoned either process before the onset (successful compiles are not logged), and I did not attempt a local repro (the 11 null attempts above make it a poor use of time without the JIT/tiering capture from experiment/5212-jit-capture).

Three more core occurrences on 2026-10-07, all shard 3 Memex.Portal.Shared.Test (part 1/2, 322 class args). They were reported as "caused by #6208 / the fresh-merge base". They are not.

run / job	event, tested tree	first bad reading	failing tests
37612813368 / 112766252743	push main `cbb3053100` (11:15Z), before #6208 merged	`canary=BELOW-ROSLYN` ×11	InstallNeverInherits…, ModulesUpdateIndependentlyOfThePlatformTest ×2
37637952561 / 112852969409	PR #6231 head `a9fa0f8dbb` fresh-merged onto main `cac0664de`	14:49:36Z `PROCESS CANNOT EMIT`, `canary=BELOW-ROSLYN` (~4m41s into the process)	the same 3
37637806820 / 112852220267	PR #6244 (workflow/doc only) fresh-merged onto main `cac0664de`	14:52:50Z CS0246 on a stand-in passed in the reference list, `canary=OK` (~3m57s in)	LinkVerdictPerPlatformTest ×1, ModuleLinkVersionTest ×11, then the same 3

New in the third one: the shallow phase reads canary=OK. The first 12 failures are stand-in compiles (StandInCompile.Emit) where the extra MetadataReference.CreateFromImage(...) is invisible (CS0246 'YamlDotNet' ×9, CS0246 'PerPlatformVerdictFromTheFuture', CS0234 'MeshWeaver.Test' ×2) while the canary, which compiles against the same reference set WITHOUT an extra image reference, still emits. 15 s later (14:53:05Z) the same process reads canary=BELOW-ROSLYN (shared: and pristine: both THREW NRE at MetadataWriter.<GetConsolidatedTypeParameters>). So the canary misses the onset; the "invisible extra reference" symptom comes first in this process.

Non-determinism, same inputs: both PR runs tested a fresh merge onto the same main cac0664de with nothing in their diffs that reaches the compiler, and LinkVerdictPerPlatformTest/ModuleLinkVersionTest passed in the #6231 process and failed in the #6244 process. Main's own push run of cac0664de (37625265553) passed shard 3.

Not established: what poisoned either process before the onset (successful compiles are not logged), and I did not attempt a local repro (the 11 null attempts above make it a poor use of time without the JIT/tiering capture from experiment/5212-jit-capture).

Three more core occurrences on 2026-10-07, all shard 3 Memex.Portal.Shared.Test (part 1/2, 322 class args). They were reported as "caused by #6208 / the fresh-merge base". They are not.

run / job	event, tested tree	first bad reading	failing tests
37612813368 / 112766252743	push main `cbb3053100` (11:15Z), before #6208 merged	`canary=BELOW-ROSLYN` ×11	InstallNeverInherits…, ModulesUpdateIndependentlyOfThePlatformTest ×2
37637952561 / 112852969409	PR #6231 head `a9fa0f8dbb` fresh-merged onto main `cac0664de`	14:49:36Z `PROCESS CANNOT EMIT`, `canary=BELOW-ROSLYN` (~4m41s into the process)	the same 3
37637806820 / 112852220267	PR #6244 (workflow/doc only) fresh-merged onto main `cac0664de`	14:52:50Z CS0246 on a stand-in passed in the reference list, `canary=OK` (~3m57s in)	LinkVerdictPerPlatformTest ×1, ModuleLinkVersionTest ×11, then the same 3

New in the third one: the shallow phase reads canary=OK. The first 12 failures are stand-in compiles (StandInCompile.Emit) where the extra MetadataReference.CreateFromImage(...) is invisible (CS0246 'YamlDotNet' ×9, CS0246 'PerPlatformVerdictFromTheFuture', CS0234 'MeshWeaver.Test' ×2) while the canary, which compiles against the same reference set WITHOUT an extra image reference, still emits. 15 s later (14:53:05Z) the same process reads canary=BELOW-ROSLYN (shared: and pristine: both THREW NRE at MetadataWriter.<GetConsolidatedTypeParameters>). So the canary misses the onset; the "invisible extra reference" symptom comes first in this process.

Non-determinism, same inputs: both PR runs tested a fresh merge onto the same main cac0664de with nothing in their diffs that reaches the compiler, and LinkVerdictPerPlatformTest/ModuleLinkVersionTest passed in the #6231 process and failed in the #6244 process. Main's own push run of cac0664de (37625265553) passed shard 3.

Not established: what poisoned either process before the onset (successful compiles are not logged), and I did not attempt a local repro (the 11 null attempts above make it a poor use of time without the JIT/tiering capture from experiment/5212-jit-capture).

Three more core occurrences on 2026-10-07, all shard 3 Memex.Portal.Shared.Test (part 1/2, 322 class args). They were reported as "caused by #6208 / the fresh-merge base". They are not.

run / job	event, tested tree	first bad reading	failing tests
37612813368 / 112766252743	push main `cbb3053100` (11:15Z), before #6208 merged	`canary=BELOW-ROSLYN` ×11	InstallNeverInherits…, ModulesUpdateIndependentlyOfThePlatformTest ×2
37637952561 / 112852969409	PR #6231 head `a9fa0f8dbb` fresh-merged onto main `cac0664de`	14:49:36Z `PROCESS CANNOT EMIT`, `canary=BELOW-ROSLYN` (~4m41s into the process)	the same 3
37637806820 / 112852220267	PR #6244 (workflow/doc only) fresh-merged onto main `cac0664de`	14:52:50Z CS0246 on a stand-in passed in the reference list, `canary=OK` (~3m57s in)	LinkVerdictPerPlatformTest ×1, ModuleLinkVersionTest ×11, then the same 3

New in the third one: the shallow phase reads canary=OK. The first 12 failures are stand-in compiles (StandInCompile.Emit) where the extra MetadataReference.CreateFromImage(...) is invisible (CS0246 'YamlDotNet' ×9, CS0246 'PerPlatformVerdictFromTheFuture', CS0234 'MeshWeaver.Test' ×2) while the canary, which compiles against the same reference set WITHOUT an extra image reference, still emits. 15 s later (14:53:05Z) the same process reads canary=BELOW-ROSLYN (shared: and pristine: both THREW NRE at MetadataWriter.<GetConsolidatedTypeParameters>). So the canary misses the onset; the "invisible extra reference" symptom comes first in this process.

Non-determinism, same inputs: both PR runs tested a fresh merge onto the same main cac0664de with nothing in their diffs that reaches the compiler, and LinkVerdictPerPlatformTest/ModuleLinkVersionTest passed in the #6231 process and failed in the #6244 process. Main's own push run of cac0664de (37625265553) passed shard 3.

Not established: what poisoned either process before the onset (successful compiles are not logged), and I did not attempt a local repro (the 11 null attempts above make it a poor use of time without the JIT/tiering capture from experiment/5212-jit-capture).

Stopgap (explicit, temporary): while this is open, a core PR whose shard 3 log carries this issue's signature (canary=BELOW-ROSLYN / PROCESS CANNOT EMIT) is treated as a test-process (harness) death and its failed jobs are re-run ONCE — the poisoned process tells us nothing about the PR. Applied 2026-10-07 to #6244, #6231, #6248 (11 signature lines each). Root cause stays open here: experiment #6245 ran both arms (A: JIT capture, B: TieredPGO=0) clean once — no occurrence, so inconclusive — and is being re-run to catch an occurrence under capture.

Experiment #6245, run 37635375205 attempt 3: the JIT capture caught an occurrence

Arm A (JIT capture, default tiering) failed with this issue's signature in job 112878675728. Arm B (DOTNET_TieredPGO=0, job 112878675545) passed. I read the arm A artifact jit-capture-shard3-attempt3 (jit.txt, 50 MB) together with the job log and the .trx.

Capture coverage

DOTNET_JitStdOutFile appends; it does not overwrite. The one file holds 5 processes run one after another. Each starts with its own XunitAutoGeneratedEntryPoint:Main, and the summary numbering restarts at 1:. The Memex.Portal.Shared.Test part-1 process starts at its Main (file line 902), so the capture covers that process from its very first compile.

Timeline (log and trx timestamps; JIT events placed by their position among the test classes' own first-time Tier0 compiles)

when (UTC)	event
15:46:46.6	process starts its first test class
15:51:05.40–06.52	`ModuleLinkVersionTest` runs 11 stand-in emits, all pass. Its last new Roslyn JITs (`WithPublicSign`, `StrongNameKeys`) are the last first-time Roslyn compiles before the onset
15:51:06.5 → 15:51:15.1 (during `ModuleReloadTransientDownloadTest`)	background tier-up batch of the metadata writer (≈400 Tier1 compiles). The first one is `MetadataWriter:PopulateNestedClassTableRows() [Tier1 with Synthesized PGO]`
—	no first-time Roslyn JIT between the batch and the onset. That fits no emit running in between, but does not prove it
15:51:21.991	onset: `VerdictPkg/Catalog` in `InstallNeverInheritsRepoCompileVerdictTest`, NRE at `NamedTypeSymbol…get_ContainingTypeDefinition` ← `PopulateNestedClassTableRows`; `canary=BELOW-ROSLYN`, `flat=SAME-FRAME`, `compiler=PRIVATE-COPY-EMITS`, `held=EMITS(64/64)`, `image=INTACT`, `host=(.NET 10.0.12 linux-x64 INTEL XEON PLATINUM 8573C, avx2+avx512f, tiering=defaults)`
after the first canary	`getConsolidatedTypeParameters` reaches Tier1. The canary's private-copy legs show up as FullOpts compiles of the same methods

Tier of each method at the onset (shared copy, Default ALC)

method	at onset
`NamedTypeSymbol:AsNestedTypeDefinitionImpl`	Tier1 (PGO), compiled long before the onset (`fgCalledCount 190`). It is correct, see below
`FullMetadataWriter:CreateIndicesForNonTypeMembers`	Tier1 (PGO), long before. Not on the faulting path
`MetadataWriter:PopulateNestedClassTableRows`	Tier1 (PGO), `fgCalledCount 205`, 17 inlinees with PGO data. Compiled after the last successful emit; the next emit is the onset
`getConsolidatedTypeParameters`	Instrumented Tier0 (Tier1 only after the onset)

No OSR compile of any of the four. Each loop method shows two Instrumented Tier0 compiles of identical code size; I have not explained that.

The Tier1 `PopulateNestedClassTableRows` drops `isinst NamedTypeSymbol`

Roslyn 5.9.0 SourceMemberContainerTypeSymbol.ContainingType is ldfld _containingSymbol; isinst NamedTypeSymbol; ret, where the field is a NamespaceOrTypeSymbol. The Tier1 body inlines AsNestedTypeDefinitionImpl behind a GDV on method-table 0x7FB5D6EBAA60 and tests the field for null:

G_M000_IG15:  mov rax, 0x7FB5D6EBAA60 ; cmp qword ptr [r12], rax ; jne IG34   ; GDV on typeDef, else interface call
G_M000_IG09:  mov rax, gword ptr [r12+0x38]                                    ; _containingSymbol
              test rax, rax
              jne G_M000_IG22                                                  ; non-null NAMESPACE => "has ContainingType"
G_M000_IG22:  mov rcx, gword ptr [r12+0x38] ; ... ; call ContainingModule       ; == [module+0xB0] (SourceModule)
G_M000_IG27:  mov r13, r12                                                     ; return this => "nested"
G_M000_IG19:  call [NamedTypeSymbol:…get_ContainingTypeDefinition()]           ; ContainingType == null => NRE

Neither loop clone has a CORINFO_HELP_ISINSTANCEOFCLASS. For a top-level source type, _containingSymbol is its namespace, which is non-null. So the guard reads TRUE, the type is written as nested, and ContainingTypeDefinition throws. That is exactly flat=SAME-FRAME: a member-less top-level class is enough.

Control in the same process, on the same field and the same guessed class. The Tier1 getConsolidatedTypeParameters, compiled just after the onset, carries the same dropped cast in its inlined guard (IG39: mov rdi,[rbx+0x38]; test rdi,rdi; jne IG42). In the same method, another [x+0x38] as NamedTypeSymbol is expanded correctly: IG49: cmp [rax],0x7FB5D6EBAA60; je … / IG50: call CORINFO_HELP_ISINSTANCEOFCLASS. That second site explains the earlier occurrences that failed in the getConsolidatedTypeParameters frame, and the flat=EMITS → SAME-FRAME deepening: one method, then the next, tiers into the same defect. The standalone Tier1 AsNestedTypeDefinitionImpl guesses SynthesizedReadOnlyListEnumeratorTypeSymbol. For that class the container field is typed as a sealed NamedTypeSymbol subclass, so leaving the cast out there is legitimate.

The leg readings now have an explanation. The private copy lives in a collectible ALC. Collectible code is not tiered: every compile in it is FullOpts, with no PGO and no GDV. That is why compiler=PRIVATE-COPY-EMITS and held=EMITS(64/64) hold every time. It also means the held-copy experiment cannot test tier-up as designed.

Verdict

This points to the JIT: Tier1 code with Dynamic PGO that inlines a GDV-guarded callee and loses a class isinst. It matches the mechanism stated in dotnet/runtime#134193 (an isinst result inherits the input's non-null-ness), but that issue is about interfaces, so whether it is the same defect is for the runtime team. A report draft is ready, with the listing excerpts, the in-method control and the evidence chain. It has not been filed.

Proposed stopgap (not implemented)

Temporary stopgap: set DOTNET_TieredPGO=0 on the test shards. The root cause is the JIT miscompile above. The real fix is the runtime fix, or a runtime version that has it; this setting only avoids the trigger. Production portals compile NodeTypes on the same runtime and the Plugins occurrences show the same signature, so the portals are exposed too. That needs its own decision; this proposal covers test shards only.

Is "PGO off prevents it" statistically meaningful? Not yet.

Arm B has 1 clean run against this capture's 1 failure. Arm A failed in 1 of 3 attempts, and recent shard-3 runs fail about 30–50 % of the time. One clean arm proves nothing on its own. I recommend ≥10 paired runs (both arms, same commit and window):

If arm B is 0/10 and arm A is ≥4/10, Fisher one-sided p ≈ 0.04 (≈0.016 at 5/10).
Against a base rate of ≥0.3 alone, arm B 0/13 gives p < 0.01.

The disassembly already names the mechanism, so the arms are now confirmation, not discovery.

Not established

The method-table → class mapping is inferred, not observed. I took Roslyn 5.9.0 field offsets from a local runtime (osx-arm64 10.0.11). SourceNamedTypeSymbol._containingSymbol is at +0x38. PEModuleBuilder.SourceModule is at +0xB0, which matches the listing's [module+0xB0] compare. The release JIT's listing does not print class names.
That this Tier1 body was the code running at 15:51:21.991. It was compiled before the onset emit and after the last successful one, and nothing in between shows Roslyn activity. Code activation itself is not logged.
Why this process produced that profile or inline shape and others do not. The local runs never did.
No JitDump or checked-JIT confirmation. No local repro.

Filed upstream as dotnet/runtime#135350 (public, maintainer-approved). Searched dotnet/runtime first: no duplicate; dotnet/runtime#134193 (interface isinst inheriting the input's non-null-ness) is cited there as possibly related, not asserted as the same defect. The public body is the draft from the analysis comment, scrubbed of job ids, repo names and artifact names, with the TieredPGO=0 evidence stated as 1 clean run vs 1 failure against a ~30-50% base rate, and an explicit 'What is NOT established' section (no standalone repro, activation not logged, MT->class mapping inferred).

ci(#5212): DOTNET_TieredPGO=0 on the build job too (Roslyn compiler s…

Merge pull request #6268 from Systemorph/fix/5212-tieredpgo-off-env

Closing: the call this issue asked for is made, the cause is fixed upstream, and no failed test job since the mitigation carries the signature.

The call. Not a reopen of #890's fix and not a second path in our code: a .NET 10 JIT miscompile (Tier1 with dynamic PGO drops a class isinst in Roslyn's inlined ContainingType), captured in the 2026-10-07 experiment above.

Upstream. dotnet/runtime#135350 is closed. The runtime team's fix is dotnet/runtime#135363 ("JIT: Make GenTree::Compare more conservative around several node types"), merged to main on 2026-10-08. Their comment says they suspect it fixes this and could not confirm without a repro, so that link is their judgement and not a verified one.

Mitigation in this fleet (explicit and temporary — it avoids the trigger, the runtime fix is the cure):

ebdfeb349f (2026-10-07): <TieredPGO>false</TieredPGO> in Directory.Build.props and test/Directory.Build.props, so every host built from this tree runs with Tiered PGO off.
e5b965717f (2026-10-07): DOTNET_TieredPGO=0 in every emitting CI job and lane, and the chart renders it from the deployment record (the memex record carries DOTNET_TieredPGO: "0", read from a Restart plan rendered 2026-10-11).

Reading (REST, 2026-10-11 ~11:05Z). Failed MeshWeaver Build and Test runs created since 2026-10-07T18:00Z: 14, among at most 300 failed runs listed for that period (three pages). Their failed test jobs: 20. Job logs carrying PROCESS CANNOT EMIT or canary=BELOW-ROSLYN: 0 of 20. Positive control: the same grep over job 112766252743 (the 2026-10-07 occurrence on main) finds 11 lines, so the scan can see the signature. Before the mitigation the same shard failed with the signature three times on 2026-10-07 alone.

Not established:

I did not check whether three pages exhaust the period's failed runs, so 14 is a floor, not a census.
MeshWeaver.Plugins' own lanes were not scanned here.
No .NET servicing release carrying JIT: Make GenTree::Compare more conservative around several node types dotnet/runtime#135363 was identified. Removing the TieredPGO setting waits for one; that is a follow-up, not this issue.

_{Fleet issue wave, Claude Opus 5.5 (agent session).}

Navigation Menu

Repository navigation

MeshWeaver#890 recurred 3x in 30h with the leg-5 reading it was closed waiting for — reopen-or-new-path call needed #5212

6 remaining items

PR #6231 shard 3 (2 of 2): what a local reproduction attempt established. It did NOT reproduce.

1. The two CI processes

2. 🚨 Correction: CI's seed does not reproduce CI's order locally

3. Local attempts (all on the merged tree, built Release; 0 of 11 processes poisoned)

4. What `DOTNET_JitDisasmSummary` shows on x64 (new)

5. Excluded by reading the runtime source (release/10.0)

6. Incidental finding, found along the way

7. Next step

Experiment #6245, run 37635375205 attempt 3: the JIT capture caught an occurrence

Capture coverage

Timeline (log and trx timestamps; JIT events placed by their position among the test classes' own first-time Tier0 compiles)

Tier of each method at the onset (shared copy, Default ALC)

The Tier1 `PopulateNestedClassTableRows` drops `isinst NamedTypeSymbol`

Verdict

Proposed stopgap (not implemented)

Is "PGO off prevents it" statistically meaningful? Not yet.

Not established

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

MeshWeaver#890 recurred 3x in 30h with the leg-5 reading it was closed waiting for — reopen-or-new-path call needed #5212

Description

Activity

rbuergi commented on Sep 22, 2026

rbuergi commented on Sep 22, 2026

rbuergi commented on Sep 22, 2026

rbuergi commented on Sep 24, 2026

rbuergi commented on Sep 24, 2026

rbuergi commented on Sep 24, 2026

rbuergi commented on Sep 24, 2026

rbuergi commented on Oct 4, 2026

rbuergi commented on Oct 4, 2026

6 remaining items

rbuergi commented on Oct 7, 2026

rbuergi commented on Oct 7, 2026

PR #6231 shard 3 (2 of 2): what a local reproduction attempt established. It did NOT reproduce.

1. The two CI processes

2. 🚨 Correction: CI's seed does not reproduce CI's order locally

3. Local attempts (all on the merged tree, built Release; 0 of 11 processes poisoned)

4. What DOTNET_JitDisasmSummary shows on x64 (new)

5. Excluded by reading the runtime source (release/10.0)

6. Incidental finding, found along the way

7. Next step

rbuergi commented on Oct 7, 2026

rbuergi commented on Oct 7, 2026

rbuergi commented on Oct 7, 2026

rbuergi commented on Oct 7, 2026

rbuergi commented on Oct 7, 2026

rbuergi commented on Oct 7, 2026

Experiment #6245, run 37635375205 attempt 3: the JIT capture caught an occurrence

Capture coverage

Timeline (log and trx timestamps; JIT events placed by their position among the test classes' own first-time Tier0 compiles)

Tier of each method at the onset (shared copy, Default ALC)

The Tier1 PopulateNestedClassTableRows drops isinst NamedTypeSymbol

Verdict

Proposed stopgap (not implemented)

Is "PGO off prevents it" statistically meaningful? Not yet.

Not established

rbuergi commented on Oct 7, 2026

rbuergi commented on Oct 11, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

4. What `DOTNET_JitDisasmSummary` shows on x64 (new)

The Tier1 `PopulateNestedClassTableRows` drops `isinst NamedTypeSymbol`