A recycled hub strands its live subscribers — StreamEndedEvent is suppressed and neither named recovery covers a recycle

Summary

MeshWeaver.FutuRe.Test.FutuReAnalysisTest.EuropeRe_AnnualReport_EmbeddedCharts_ShouldRenderViaPathResolution
waits 50 s for a render, and on main the whole project completes in 50 s. The margin is
approximately zero. It passes today because of how shard 3 happens to be packed, not because the
render is comfortably inside the budget.

Measured, 2026-08-27

Shard 3 on main (run 33112451022, green):

MeshWeaver.Hosting.Orleans.Test   exit=0 elapsed=221s
MeshWeaver.GitSync.Test           exit=0 elapsed=49s
MeshWeaver.FutuRe.Test            exit=0 elapsed=50s     ← the whole project, vs a 50s assertion
Memex.Portal.Shared.Test          exit=0 elapsed=24s
MeshWeaver.Hosting.Test           exit=0 elapsed=7s
MeshWeaver.Kernel.Test            exit=0 elapsed=0s
MeshWeaver.Reactive.Assertions.Test exit=0 elapsed=1s

The same shard on PR #2526 (run 33110530713, red) — that PR deletes test/MeshWeaver.AI.Test
and MeshWeaver.AI.Test.FakeCli and touches nothing under FutuRe, but removing two projects
re-packs every shard:

MeshWeaver.Hosting.Orleans.Test   exit=0 elapsed=265s     ← +44s vs main
MeshWeaver.FutuRe.Test            exit=1 TESTFAIL
MeshWeaver.Auth.Test              exit=0 elapsed=44s
MeshWeaver.Documentation.Test     exit=0 elapsed=11s
MeshWeaver.PluginImage.Test       exit=0 elapsed=0s
MeshWeaver.TestDomain             exit=0 elapsed=0s

MeshWeaver.Reactive.Assertions.ObservableAssertionException :
  Expected the observable to emit a value matching the predicate within 50s

Different neighbours, more contention on the shared runner (Orleans alone is 44 s longer), and the
assertion tips over. The PR did not break this test; it re-packed the shard and the test's absent
margin did the rest.

Why this is worth an issue rather than a shrug

It reads as a defect in whatever PR happens to re-pack shard 3. The moved test suites leave core (#2276 test emigration, deletion half) #2526 will now be debugged
for a failure it did not cause — and the next structural change to the test set inherits the same
false attribution. This is the [job-name-is-not-failure-identity] shape.
Green on main is not evidence of health here. A 50 s project against a 50 s assertion means
main is passing by luck, and the luck is a property of the current shard packing.
🚨 Raising the timeout is the forbidden fix. AGENTS.md is explicit: a timeout that trips
means something did not complete — find what. The question is why an embedded-chart render via
path resolution takes on the order of 50 seconds at all, not how much headroom to buy.

What to actually investigate

Where the ~50 s goes: NodeType compilation on first touch, path resolution round-trips, chart
rendering, or fixture bring-up. The test asserts a render via path resolution, so an obvious
first suspect is a per-node hub activation (or several) on the critical path.
Whether the wait can be made a wait on the actual condition rather than a wall-clock budget
(WritingTests.md → never sleep, filter the stream), and whether the render itself is doing
something serial that should be concurrent.
Whether other assertions in this project sit near their budget too — a project whose total
runtime equals one of its timeouts usually has more than one.

Not the fix

Adding retries, widening the 50 s, pinning FutuRe to a shard, or allow-listing it. Each hides the
same thing: a render that should be fast is not, and nobody knows why.

This is not a margin problem — it is a dropped terminal event

Measured while investigating a red on #2568 (which re-packs shards and so hits this more often).

Locally the render is ~2s, not ~50s. Full FutuRe suite under DOTNET_PROCESSOR_COUNT=4 (CI's
core count): 59/59 green, and the slowest test in the entire project is 4.7s —
EuropeRe_AnnualReport_EmbeddedCharts_ShouldRenderViaPathResolution is not even in the top five.
A 25× gap to CI with CPU held equal is not slowness, so "the budget is nearly exhausted by real
work" does not hold.

The CI trace shows the missed-observation signature, not a slow path (run 33150985489, shard 3):

07:38:58.227  TEST START
   …12s of real work…
07:39:05.574  [QUIESCE-OK] sync/TGR63DunCU6mjMmeVhjHAA: drained 0 callback(s)
07:39:10.578  warn  Dropping StreamEndedEvent for stream TGR63DunCU6mjMmeVhjHAA on hub cache
   …37.7 SECONDS OF TOTAL SILENCE…
07:39:48.326  TEST FAILED: …within 50s

Work bursts for 12s, then nothing at all for the remaining 37.7s. Per
.claude/skills/debug/SKILL.md, a burst-then-silence is a dropped reactive emission; a genuine
slow path or a lock shows continuous or mid-work-gapped activity.

The edge that breaks, in order:

sync/TGR63DunCU6mjMmeVhjHAA quiesces at 07:39:05 — 7s into a test that is still subscribed to
that stream. It drains 0 callbacks, so nothing signalled the waiter on the way out.
The StreamEndedEvent then arrives with its target hub already gone. DataExtensions waits its
5s grace for a sync hub to appear (src/MeshWeaver.Data/DataExtensions.cs:419-436) and, at
exactly +5.0s, drops it via LogStreamMessageDrop (:451).
The waiter is left with no value, no completion, and no error — the only thing that ever fires
is the assertion's own .Timeout(50s).

So the 50s is the timeout, not the work, and raising or lowering it changes nothing except how long
the shard sits there. The defect is that a stream's terminal event is dropped while a subscriber is
still waiting on it — the disposal path has nowhere to deliver the terminal state.

This likely also explains Systemorph/MeshWeaver.Plugins#1141 (same test, "times out on a RedirectControl"): a waiter that never
observes a terminal state reports whatever it last held.

Not proposing the fix here, but it is in the teardown ordering — a sync hub should not quiesce out
from under a live subscriber, or the drop path must fault the waiter instead of leaving it silent.
Suggest re-titling this issue away from "sits at its 50s assertion budget", which sent me (and would
send the next reader) looking for slowness.

Correction to my previous comment — the dropped StreamEndedEvent is NOT the cause

I posted the drop as the root cause. It is incidental: those drops are on other streams
(PVGcJx…, MyUh_r…, TGR63D…), not the render stream. The compile trace shows the actual edge,
and it is a type-registration race, not a delivery loss.

07:38:58.638  CompileWatcher: HandleDispatchCompile ENTERED on FutuRe/LocalAnalysis (INLINE)
07:39:01.734  Compiling assembly for FutuRe_LocalAnalysis (disk, 0 NuGet refs)
07:39:03.658  EnrichWithNodeType: compile still in flight after 5s — ACTIVATING COMPILE OVERLAY
07:39:03.729  ⚠️ MeshNodeTypeSource[FutuRe/EuropeRe/Analysis]: content discriminator '$type':
              'AnalysisContent' is not a registered type          ← the defect
07:39:04.948  Successfully compiled assembly for FutuRe/LocalAnalysis
07:39:05.220  Overlay self-heal: NodeType reached a usable build (version 17)
07:39:05.225  Slow-path typeStream emission
   …silence…
07:39:48.326  TEST FAILED (the 50s assertion budget)

The compile SUCCEEDS at 07:39:04.9 and the type becomes usable at 07:39:05.2 — 43 seconds before
the test gives up. So nothing is slow and nothing is lost. What happens is that the node's content
is read at 07:39:03.7, during the overlay window, when AnalysisContent is not yet a registered
discriminator. Per AGENTS.md the polymorphic converter degrades an unresolvable $type to a raw
JsonElement rather than throwing — so the content silently becomes untyped and the view renders
empty. It never re-materializes when the real type lands, which is why the remaining 43s are
silent: there is nothing left to wait for.

The EnrichWithNodeType overlay that activates "after 5s" is the trigger — it lets the read proceed
against an unregistered type instead of waiting for the build. That is a fail-closed-shaped fallback
producing a correct-looking wrong answer.

Why this is deterministic on #2568 and green on main. #2568 removes MeshWeaver.AI from
FrameworkBuildIdentity.ContentSurfaceAssemblies, which flips the framework identity and invalidates
every cached NodeType build. The compile above is therefore a cold ~6.3s Roslyn compile rather
than a cache hit, so it always crosses the 5s overlay threshold and always loses the race. On main
the assembly is cached, the compile is fast, and the window is rarely open. Shard packing only
changes how often main gets unlucky — it is not the mechanism.

🚨 This is not only a test problem. Any framework-identity bump recompiles every dynamic NodeType
fleet-wide, so the same window opens on every instance during that recompile — views rendering empty
until something re-reads them. That is worth fixing before the next identity flip ships, independent
of this test.

Root-caused, fixed in #2582 — and my second comment above was wrong

Correcting myself so nobody follows it: I claimed the cause was the compile-in-progress overlay
letting a read proceed against an unregistered AnalysisContent discriminator. It is not. That
warning is ambient noise (it appears on green runs of this suite too, as this issue already
suspected), and the overlay is a deliberate, working mechanism. The defect happens ~1.5 s later.

My first comment was closer: the terminal StreamEndedEvent really is the missing signal. But it
is not dropped — it is deliberately suppressed, and that distinction is the whole bug.

JsonSynchronizationStream.cs:1153-1180 announces StreamEndedEvent from the server stream's
disposal, and skips it whenever the owning hub is tearing down:

if (hub.RunLevel > MessageHubRunLevel.Started || hub is MessageHub { IsDisposing: true })
{
    logger.LogDebug("… not announcing the end of stream {StreamId} to {Subscriber} — the "
        + "subscriber recovers through the recycle re-arm / change-feed latch instead", …);
    return;
}

That suppression is correct on its own terms — a dying owner reaching up the hub tree resurrects the
Orleans activation it is retiring. The problem is the delegation in its own log line: neither
named recovery can fire for a self-heal recycle.

the recycle re-arm needs an in-flight SubscribeRequest to be NACKed — an established
subscription has none;
the change-feed latch needs a WRITE, and the same file states at line 635 that a recycle IS
NOT A WRITE.

Meanwhile StreamEndedEvent's own docstring (Messages.cs:289) already promises it fires for "the
owning hub tearing down — deactivation, recycle, restart". The gap between those two comments is
the bug.

It stayed hidden because only automatic recycles are exposed: MeshOperations.RecycleCore
publishes a MeshChangeEvent before its DisposeRequest (which fed the latch), and
RecycleLayoutArea is driven by the page shell. The overlay self-heal posts the dispose alone.

Reproduced deterministically as RecycleStrandsLiveSubscriberTest: a live layout-area
subscription plus a bare DisposeRequest, no write anywhere. On main it fails after a full 60 s
still holding ACTIVATION_1, with the hub visibly re-activating on its own 4.8 s in while the
subscriber is never told. 60 s timeout → 635 ms after the fix.

The fix is ordering, not a new mechanism: announce on the recycle's own turn while the hub is whole,
deliver after DisposalCompleted through the parent hub that outlives it. No bound raised, no
timer, poll, retry or watchdog.

Retitling this issue, because "sits at its 50s assertion budget" is what sent two readers looking
for slowness. There was never anything slow: the compile finished 43 seconds before the test gave
up.

Merge pull request #2582 from Systemorph/fix/overlay-recycle-strands-…

Navigation Menu

Repository navigation

A recycled hub strands its live subscribers — StreamEndedEvent is suppressed and neither named recovery covers a recycle #2551

Summary

Measured, 2026-08-27

Why this is worth an issue rather than a shrug

What to actually investigate

Not the fix

This is not a margin problem — it is a dropped terminal event

Correction to my previous comment — the dropped StreamEndedEvent is NOT the cause

Root-caused, fixed in #2582 — and my second comment above was wrong

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

A recycled hub strands its live subscribers — StreamEndedEvent is suppressed and neither named recovery covers a recycle #2551

Description

Summary

Measured, 2026-08-27

Why this is worth an issue rather than a shrug

What to actually investigate

Not the fix

Activity

rbuergi commented on Aug 28, 2026

This is not a margin problem — it is a dropped terminal event

rbuergi commented on Aug 28, 2026

Correction to my previous comment — the dropped StreamEndedEvent is NOT the cause

rbuergi commented on Aug 28, 2026

Root-caused, fixed in #2582 — and my second comment above was wrong

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions