A compile fault's stack now reaches CI — #890 could never have been diagnosed by rbuergi · Pull Request #966 · Systemorph/MeshWeaver

rbuergi · 2026-08-08T20:49:48Z

This does NOT fix the NRE — it makes the next occurrence diagnosable

Refs #890. No behaviour of the compile pipeline changes. #890 is an infrastructure fault that
aborts a compile before Roslyn runs, process-wide, cascading across test classes. It has been
observed once (run 31153582625) and did not recur in a sweep of every non-succeeding shard job
across all attempts of the 185 CI runs since #612's diagnostics landed — 108 job logs, 0 unfetchable,
0 hits (sweep detail).

The problem is that #890 as written can never be closed. It says "the next CI occurrence will
name its throw site". It will not.

Why the current diagnostics don't reach CI

#612 added three sinks for a non-Roslyn compile abort. Only one is observable in CI, and it is the
one without frames:

Sink	Reaches CI?
`CompilationError = "{TypeName}: {Message}"` on the node	✅ — only because a failing assertion dumps the whole `MeshNode`
`Compile aborted by {FullName}` + stack → ActivityLog node	❌ that MeshNode is never uploaded
`logger.LogWarning(outcome.Error, …)` — the exception object	❌ no CI artifact captures `ILogger`

Measured, not assumed:

The pre-CI wedge: CodeEditRecompileTest StaleBarrierType first compile ends Error, re-trigger not picked up (sub-case b) — shard timeout exit=124 #612 code already logged LogWarning("Compile failure for {HubPath}: {Error}"). In the
attempt-1 shard log of run 31153582625 — the occurrence this issue is filed on, where five
compiles failed — Compile failure for appears 0 times.
A full MeshWeaver.Hosting.Monolith.Test .trx (443 KB) contains 1 StdOut section (a skip
reason) and 0 [COMPILE-TRACE] lines, although HandleDispatchCompile logs one at
Information on every compile.

The cause is in XUnitFileLogger: it forwards to xUnit's ITestOutputHelper (which the trx logger
does not persist) and to a per-method file gated behind MESHWEAVER_TEST_FILE_LOGS (off in CI) —
and it drops the record entirely when no test method is active, which is exactly the wedge case
(fixture init, teardown, and every reactive continuation on a background scheduler).

What this PR does

1. A fault's stack reaches the one file CI keeps.
New MeshWeaver.Fixture/TestTraceLog owns the path and the write lock for
%TEMP%/meshweaver-test-trace.log — the file dotnet-test.yml copies into collected-logs/, and the
only one that survives an exit=124 shard kill, where no .trx is written at all.
XUnitFileLogger now appends Warning+ records that carry an exception (type + full stack) there,
before the output-helper / IsInTestMethod filters that were swallowing them.

Deliberately placed after IsEnabled: configured log levels are a production cost contract and
are not widened here. No Log* call's level changed.
MonolithMeshTestBase's phase trace moves onto the same lock — two writers with two locks would
interleave mid-line and corrupt both.
Bounded at 1000 records / 2500 chars. Sized off measurement (one deliberately-failing compile test
emits 15), and it bounds the artifact, not the diagnosis: the first faults in a cascade name the
cause.

2. A failed compile must carry a Roslyn diagnostic, not merely a non-empty string.
CodeEditRecompileTest asserted only CompilationError.Should().NotBeNullOrEmpty() — which an
infrastructure NRE satisfies exactly as well as a real diagnostic, so a single, non-cascading
occurrence passed silently on a green shard. It now requires CS#### (FormatCompileFailure always
writes "{d.Id} {d.Severity}"), turning the silent case into a red test. This is worth more than
the artifact — it converts a symptom we can only find by grepping logs into one CI reports itself.

3. Delete the dead pre-#612 compile path.
NodeTypeCompileActivityHandler still carried the bare outcome.Error?.Message that #612 replaced,
so a grep for the compile's terminal write landed on the un-enriched dead copy first — a triage
trap. Nothing posts RunCompileRequest: the cross-hub dispatch was replaced by inline RunCompile
(a just-created _Activity hub is not yet routable, so the request was dropped and the NodeType sat
at Compiling forever). Handler, request/response pair, handler wiring and type registrations are all
removed.

Verification

Solution builds -c Release -p:CIRun=true -warnaserror clean, after merging current main.
CodeEditRecompileTest 11/11; compile + release family 53/53; MeshWeaver.Graph.Test
850/850; MeshWeaver.Documentation.Test 19/19; new TestTraceLogFaultTest 2/2.
The sink was confirmed live, not just compiled: a real compile-failure run wrote
MeshWeaver.Graph.Configuration.CompilationException and
System.Reflection.TargetInvocationException with frames into the collected trace file.
The tightened assertion passes against a genuine Roslyn rejection, confirming CS#### is really
present (not merely assumed from reading FormatCompileFailure).

Notes

CI: one Roslyn Emit poisons the whole process — every later compile NREs in Cci.MetadataWriter.GetConsolidatedTypeParameters, shard ends exit=124 #890 stays open — the NRE itself is unfixed and unreproduced.
Orleans test host: corrupt/evicted DynamicNode assembly poisons every TestCluster.DeployAsync (TypeLoadException 'format is invalid') — whole-project cascade #618 is a SEPARATE defect — same shape (a compile puts a process-wide facility into a bad
state; the cascade crosses unrelated test classes), different mechanism. Orleans test host: corrupt/evicted DynamicNode assembly poisons every TestCluster.DeployAsync (TypeLoadException 'format is invalid') — whole-project cascade #618 is
enumerability-after-unload in Orleans' assembly scan: AssemblyLoadContext.Unload() is
asynchronous by contract, so the assembly stays in AppDomain.CurrentDomain.GetAssemblies() until
the LoaderAllocator is collected, and the scan takes no pin. CI: one Roslyn Emit poisons the whole process — every later compile NREs in Cci.MetadataWriter.GetConsolidatedTypeParameters, shard ends exit=124 #890 involves no Orleans assembly
enumeration and has been seen on the Monolith shard. Fixing Orleans test host: corrupt/evicted DynamicNode assembly poisons every TestCluster.DeployAsync (TypeLoadException 'format is invalid') — whole-project cascade #618 should not be expected to
stop CI: one Roslyn Emit poisons the whole process — every later compile NREs in Cci.MetadataWriter.GetConsolidatedTypeParameters, shard ends exit=124 #890, and neither issue should block the other.
What's New entry included: a failed build that aborts for an infrastructure reason now names the
fault instead of showing a bare message.

🤖 Generated with Claude Code

…iagnosed #890 waits for "the next occurrence to name its throw site". It cannot: of the three diagnostics #612 added for a non-Roslyn compile abort, only ONE is observable in CI, and it is the one WITHOUT frames. Measured, not assumed: * The pre-#612 code already logged `LogWarning("Compile failure for {HubPath}")`. In run 31153582625 attempt 1 — the occurrence the issue is filed on, where FIVE compiles failed — that string appears 0 times in the shard log. * A full Hosting.Monolith.Test .trx (443 KB) holds 1 StdOut section (a skip reason) and 0 [COMPILE-TRACE] lines, though HandleDispatchCompile logs one at Information on every compile. The cause is in XUnitFileLogger: it forwards to xUnit's ITestOutputHelper (which the trx logger does not persist) and to a per-method file gated behind MESHWEAVER_TEST_FILE_LOGS (off in CI) — and it drops the record entirely when no test method is active, which is exactly the wedge. So the activity-log entry (never uploaded) and the logged exception object both vanish, and all CI ever sees is the node's `CompilationError` string, because an assertion dump prints the MeshNode. A recurrence names the exception CLASS and nothing else. This does NOT fix the NRE. It makes the next occurrence diagnosable. 1. Faults reach the one file CI keeps. New Fixture/TestTraceLog owns the path and the write lock for %TEMP%/meshweaver-test-trace.log — the file the workflow copies to collected-logs/ and the only one that survives an exit=124 shard kill, where no .trx is written at all. XUnitFileLogger now appends Warning+ records that carry an exception (type + full stack) there, BEFORE the output-helper and IsInTestMethod filters. Deliberately AFTER IsEnabled: configured log levels are a production cost contract and are not widened. MonolithMeshTestBase's phase trace moves onto the same lock — two writers with two locks would interleave mid-line and corrupt both. Bounded at 1000 records / 2500 chars: one deliberately-failing compile test emits 15, so the budget is sized off measurement, and it bounds the ARTIFACT, not the diagnosis — the first faults in a cascade name the cause. 2. A failed compile must carry a Roslyn diagnostic, not just a non-empty string. CodeEditRecompileTest asserted only NotBeNullOrEmpty, which an infrastructure NRE satisfies exactly as well as a real diagnostic — a single non-cascading occurrence passed silently on a green shard. It now requires CS#### (FormatCompileFailure always writes "{d.Id} {d.Severity}"), turning the silent case red. This is worth more than the artifact. 3. Delete the dead pre-#612 compile path. NodeTypeCompileActivityHandler still carried the bare `outcome.Error?.Message` that #612 replaced, so grepping for the compile's terminal write landed on the un-enriched copy first. Nothing posts RunCompileRequest — the cross-hub dispatch was replaced by inline RunCompile because a just-created _Activity hub is not yet routable. Handler, request/response pair, handler wiring and type registrations all removed. Verified: solution builds -c Release -p:CIRun=true -warnaserror clean; CodeEditRecompileTest 11/11; compile+release family 53/53; Graph.Test 850/850; Documentation.Test 19/19; new TestTraceLogFaultTest 2/2 — and the sink was confirmed live, capturing CompilationException and TargetInvocationException with frames from a real compile-failure run. Refs #890. Possible shared root cause with #618 (see the issue comments). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Copilot

Pull request overview

Improves CI diagnosability for rare pre-Roslyn compile aborts (refs #890) by ensuring exception type + stack traces reach the one test log file that CI consistently retains, tightening a key compile failure assertion to distinguish infrastructure faults from real Roslyn diagnostics, and removing the dead compile activity-dispatch path that no longer runs.

Changes:

Add a shared TestTraceLog sink and route warning+ exception logs into the CI-collected trace file.
Tighten CodeEditRecompileTest to require a CS#### diagnostic code on expected Roslyn failures.
Remove the obsolete RunCompileRequest/Response contract and the unused activity-hub compile handler wiring; update related type-registry and docs/comments.

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
test/MeshWeaver.Observability.Test/TestTraceLogFaultTest.cs	Adds a regression test that pins “exception type + stack must reach CI-collected trace file”.
test/MeshWeaver.Hosting.Monolith.Test/CodeEditRecompileTest.cs	Strengthens the failure assertion to require a Roslyn `CS####` diagnostic code.
src/MeshWeaver.Mesh.Contract/NodeTypeCompileRequest.cs	Removes the dead `RunCompileRequest/Response` contract and leaves an explanatory note.
src/MeshWeaver.Hosting.Monolith.TestBase/MonolithMeshTestBase.cs	Routes phase tracing through the shared `TestTraceLog` writer/lock.
src/MeshWeaver.Graph/MeshNodeExtensions.cs	Removes type-registry entries for the deleted compile request/response types.
src/MeshWeaver.Graph/Configuration/NodeTypeCompileActivityHandler.cs	Deletes the unused activity-hub compile handler implementation.
src/MeshWeaver.Graph/Configuration/NodeTypeCompilationHelpers.cs	Updates doc comment to reflect the inline compile path (no cross-hub request).
src/MeshWeaver.Graph/Configuration/GraphConfigurationExtensions.cs	Updates comment to remove now-obsolete compile round-trip wording.
src/MeshWeaver.Graph/Configuration/ActivityNodeType.cs	Removes handler registration for the deleted `RunCompileRequest`.
src/MeshWeaver.Fixture/XUnitFileOutputHelper.cs	Adds early exception-to-trace-file logging before output-helper/test-method gating.
src/MeshWeaver.Fixture/TestTraceLog.cs	Introduces the shared trace file path + lock and fault-record appender with bounds.
src/MeshWeaver.Documentation/Data/WhatsNew/2026-08-08-compile-faults-name-their-cause.md	Adds a What’s New entry explaining improved fault reporting for infrastructure build failures.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Copilot review on #966: routing faults to the trace sink moved formatter(state, exception) ahead of the output-helper / IsInTestMethod early-returns, so records destined to be dropped were formatted anyway — precisely on the background-scheduler paths the change is about. Format at most once, and only where the result is consumed: eagerly for a fault (the sink needs it), otherwise not until the in-test-method path is known to be taken. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…-diagnosable

github-actions · 2026-08-08T21:23:38Z