Repository navigation
Orleans test host: corrupt/evicted DynamicNode assembly poisons every TestCluster.DeployAsync (TypeLoadException 'format is invalid') — whole-project cascade #618
Description
Non-deterministic, process-global flake in MeshWeaver.Hosting.Orleans.Test — independent of any single PR (observed on PR #607 run 30028943225 shard 2, but the branch is 132/132 green locally; latent in main).
Symptom
After ~3 min of green, a wall of failures (22 on that shard) all throwing the identical exception, all sharing one dynamic-assembly GUID 5af7f5d3886840588e9b1bb520f0481b, all at silo host build (before any grain activation / stream / classifier):
System.TypeLoadException: Could not load type
'MeshWeaver.Graph.Generated.type_OrleansInstanceType5af7f5d3…MeshNodeProviderAttribute'
from assembly 'DynamicNode_type_OrleansInstanceType5af7f5d3…' because the format is invalid.
at System.Reflection.RuntimeAssembly.IsDefined(...)
at Orleans.Serialization.Internal.ReferencedAssemblyProvider.AddAssembly / GetRelevantAssemblies
at Orleans.Serialization.ServiceCollectionExtensions.AddSerializer
at Orleans.Hosting.DefaultSiloServices.AddDefaultServices
… OrleansTestBase.InitializeAsync → TestCluster.DeployAsync
Mechanism
One earlier test compiles+loads a dynamically-generated NodeType assembly (DynamicNode_…) whose on-disk image is corrupt/truncated or whose collectible ALC is later evicted, leaving a dangling Assembly reference. Orleans' default AddSerializer enumerates all loaded assemblies and calls assembly.IsDefined(...) at every DeployAsync — so from that instant, one poisoned assembly fails every subsequent silo build in the process. Process-global assembly-load pollution → whole-project cascade, not a per-test logic bug.
Root-cause candidates
- Emit corruption at the disk-cache load path:
MeshNodeCompilationService.EmitToDiskWithRetry(~src/MeshWeaver.Graph/Configuration/MeshNodeCompilationService.cs:1451) producing/loading a truncated image (cf. the multiarch CopyRefAssembly corruption class, recurring 2026-07-23). - ALC-eviction leaving a dangling Assembly still enumerable by Orleans (
145414632"evict superseded ALCs on recompile"). Related to the FutuRe teardown ALC-unload SIGSEGV (CoreCLR heap corruption: an object's MethodTable word is zeroed; the GC faults walking it (upstream — not fixable here) #613) — same ALC-lifecycle family, different manifestation.
Fix direction
A corrupt/evicted dynamically-compiled NodeType assembly must never remain enumerable by Orleans' ReferencedAssemblyProvider — either guarantee the emitted image is complete before load (fail the compile, don't load a truncated dll) or ensure ALC eviction removes the Assembly from the domain's enumerable set before Orleans re-enumerates.
Not blocking
The branch under test is sound (132/132 green locally). This is a shared test-host hazard; a fresh CI run is expected green.
🤖 Generated with Claude Code
Activity
github-actions commentedon Jul 24, 2026
CD failed on main: run 30071266594 for commit
414eca9adeb3006f1dcbc983808cdbdaa2d2ef21.No deployable image was published for this merge - every self-updating install stays on the PREVIOUS image until a CD run goes green. Triage the failed job in the linked run, fix (or re-run if infrastructure), and close this issue once CD is green.
rbuergi commentedon Jul 24, 2026
Related occurrence (same family, different surface), main run 30072157879 (f462c60): exactly the two dynamic-compile tests (
CompiledArtifact_IsVisibleFromBothSilos,ColdStart_CompileViaGetCompilationPathRequest_Succeeds) failed fast withSuccess=falsecarrying a bare"Object reference not set to an instance of an object."(NRE message, no stack —NodeTypeContractHandler.Fail(null, ex.Message)drops the stack, a diagnosability gap worth fixing) while all 132 other Orleans tests passed. Not the literal TypeLoadException signature, but the same process-global compile-path poisoning pattern. Both tests green individually and 134/134 project-green locally on the same sha.Additional latent lead found during triage: the OWN-node stream reduces through the pathless
MeshNodeReference()legacy branch ofReduceToMeshNode(src/MeshWeaver.Graph/MeshDataSource.cs:1475+) where the sibling-satellite fix (pinned by MeshNodeReducePatchTest) was only applied to path-ful references — satellite-only patches during compile can still surface a sibling as the own-stream value.🤖 Generated with Claude Code
github-actions commentedon Jul 28, 2026
CD failed on main: run 30404078323 for commit
abfbe1383c0095973cd81b3f376d02c72acfd2c6.No deployable image was published for this merge - every self-updating install stays on the PREVIOUS image until a CD run goes green. Triage the failed job in the linked run, fix (or re-run if infrastructure), and close this issue once CD is green.
github-actions commentedon Aug 5, 2026
CD failed on main: run 31012108566 for commit
885225a021064528a819952fb941af06fd43cbca.No deployable image was published for this merge - every self-updating install stays on the PREVIOUS image until a CD run goes green. Triage the failed job in the linked run, fix (or re-run if infrastructure), and close this issue once CD is green.
github-actions commentedon Aug 7, 2026
CD failed on main: run 31132935227 for commit
75571eeaf0dda1116feb59ce822c91c4ae5eee8f.No deployable image was published for this merge - every self-updating install stays on the PREVIOUS image until a CD run goes green. Triage the failed job in the linked run, fix (or re-run if infrastructure), and close this issue once CD is green.
github-actions commentedon Aug 7, 2026
CD failed on main: run 31138803129 for commit
22fab9c8bb9a21f71ac4b4d87c0c93285c838b29.No deployable image was published for this merge - every self-updating install stays on the PREVIOUS image until a CD run goes green. Triage the failed job in the linked run, fix (or re-run if infrastructure), and close this issue once CD is green.
github-actions commentedon Aug 7, 2026
CD failed on main: run 31139761395 for commit
87f439200da85dcb9e3015e1495253353bc9e257.No deployable image was published for this merge - every self-updating install stays on the PREVIOUS image until a CD run goes green. Triage the failed job in the linked run, fix (or re-run if infrastructure), and close this issue once CD is green.
github-actions commentedon Aug 7, 2026
CD failed on main: run 31142331917 for commit
778ab30d0549a0f3e0b23ed24a448d7d1e09915a.No deployable image was published for this merge - every self-updating install stays on the PREVIOUS image until a CD run goes green. Triage the failed job in the linked run, fix (or re-run if infrastructure), and close this issue once CD is green.
github-actions commentedon Aug 7, 2026
CD failed on main: run 31143782207 for commit
fb70724b4bc350842b4d99fd58c039b901a12cd1.No deployable image was published for this merge - every self-updating install stays on the PREVIOUS image until a CD run goes green. Triage the failed job in the linked run, fix (or re-run if infrastructure), and close this issue once CD is green.
github-actions commentedon Aug 7, 2026
CD failed on main: run 31147515018 for commit
142e24c06d6751b072cfce1c2c989574f69bef85.No deployable image was published for this merge - every self-updating install stays on the PREVIOUS image until a CD run goes green. Triage the failed job in the linked run, fix (or re-run if infrastructure), and close this issue once CD is green.
rbuergi commentedon Aug 8, 2026
Shard 0 (the Orleans shard) produced two more unrelated-PR failures today, on diffs that cannot reach the Orleans host. Recording here because this issue already owns Orleans test-host instability.
DelegationCompletionTest.SubmitMessage_ResponseCellGetsTerminalText(a completion Notification that never arrived in 20 s). I verified the PR's change is provably inert for that test:PermissionEvaluatorshort-circuits (if (gates.Count == 0) return fromPolicy.DistinctUntilChanged();) and the harness declares no gate, so it takes the byte-identical pre-PR path. Re-ran unchanged → GREEN, and the PR merged.OrleansStaticRepoImportTest.Import_CreatesSpaceRoot_AndChildContent, on a diff that is test-only, in MeshWeaver.Query.Test. It cannot affect an Orleans import.Two different Orleans tests, two unrelated diffs, both shard 0, both passing on re-run or locally. That is the shape this issue describes (a poisoned/unstable TestCluster affecting whatever runs in it) rather than two independent product races.
For contrast, today's other two recurring flakes were NOT this shape and each got its own issue: #945 (CrossHubPatchAtomicityTest, shard 4, twice) and #946 (ObservableQueryIntegrationTests.ScopeDescendants, shard 1, four times — root-caused as a test asserting an exact emission count on a change feed, fixed in PR #955).
So of the four cross-PR flake sources seen today, one is fixed, one is analysed and open, and these two Orleans ones look like this issue. Worth checking whether shard 0's TestCluster lifecycle is the common factor.
rbuergi commentedon Aug 8, 2026
The #613 teardown-ordering fix (PR #967) does not fix this — and here is what this actually needs
Investigated while fixing #613 (node ALCs unloading before pooled work is drained). The two look like one bug — both are "a collectible node assembly touched after it should be gone" — but they are not, and the distinction points at the fix.
Why #967 doesn't cover it:
MessageHubGrain.OnDeactivateAsync's ownloadContext.Unload()— which already carries a documented KNOWN GAP for exactly this situation.AssemblyLoadContext.Unload()is asynchronous by contract: theAssemblystays reachable fromAppDomain.CurrentDomain.GetAssemblies()until the LoaderAllocator is actually collected. Orleans'ReferencedAssemblyProviderenumerates every loaded assembly at silo startup and callsRuntimeAssembly.IsDefinedon each — which is what throwsCould not load type '…MeshNodeProviderAttribute' … because the format is invalid. Unloading earlier or later does not change that the assembly is still enumerable in that window.NodeAssemblyLoadContext.Pin()already has a drain added for this exactTypeLoadException … format is invalidstring — but the pin only protects paths that take a pin. Orleans' assembly scan takes no pin, so it walks straight past that protection.So the fix #618 needs is coverage of that uncovered scan — either keeping the half-unloaded assembly out of the enumeration Orleans performs, or ensuring no silo starts while such an assembly is enumerable. That is a different change from the teardown ordering in #967, and should be tracked here rather than folded into it.
Observed impact, for prioritisation: this reddened PR #950's shard 0 with ten Orleans tests failing together, all from one
TypeLoadExceptionatTestClusterstartup. It re-greened on the next run after a merge with main, so it presents as a flake — but the ten names are a single incident, not ten independent failures. Anyone tallying flake frequency should count it once.