Pre-warm one NodeType at a time — 4-at-once is the storm that keeps boot warming disabled by rbuergi · Pull Request #687 · Systemorph/MeshWeaver

rbuergi · 2026-07-28T04:24:10Z

The fix already existed; it was just unsafe to turn on

DynamicTypePreWarmer front-loads the cold per-type compile so the first visitor after a deploy doesn't wait for it — exactly the stall behind the 2026-07-27 outage. It ships disabled, and its own summary says why:

"enumerates EVERY NodeType across EVERY partition and activates + compiles each dynamic one — on a mesh with many partitions that is a boot-time subscribe/compile storm that starves the hubs (and, multi-silo, the Orleans membership probes — the 'I have been told I am dead' restart loop on memex, 2026-07-22)."

That storm is a concurrency choice, not an inherent property. DefaultMaxConcurrency was 4.

I reproduced the storm by accident, which is the evidence

On 2026-07-28 04:05 I triggered four compiles at memex in quick succession while gathering diagnostics — nothing to do with this warmer, but the identical load shape. Within minutes six plugin roots (Claims, Edu, Underwriting, Chess, Publish, Training) fell to the "did not settle" overlay and needed a scale-to-zero to recover.

Why concurrency buys nothing

The compiles already serialize on the Compile IoPool. So 4 adds only simultaneous cold activations — and activation is the expensive part. Issue #686 measured it directly: the same NodeType shape, same image, compiles in 109 ms with 0 ms discovery on a fresh mesh, versus 45.20 s of discovery on memex. The delta is cold cross-partition hubs — precisely what warming exists to pay off, and precisely what doing four at once turns into an outage.

Change

DefaultMaxConcurrency 4 → 1
2 s stagger between types, so back-to-back cold activations don't read as a burst even at concurrency 1

The sweep has no deadline and lazy compilation still covers correctness, so a trickle costs nothing. This is what makes PreWarm:DynamicTypes safe to actually enable — which is the real fix for a stall that only ever bites the first request after a deploy.

Testing

DefaultConcurrency_IsOne_AndTypesArePaced pins the value, because the damage only appears at production scale where no test runs. 2/2 DynamicTypePreWarmerTest pass.

Deploy note

Merging this changes nothing by itself — the warmer is still opt-in. Enabling it (PreWarm__DynamicTypes=true) is a separate, deliberate step, and should be done on one instance first with the roots watched.

🤖 Generated with Claude Code

…sabled The boot warm-up already exists and already solves the problem it was built for: front-load the cold per-type compile so the first visitor does not wait for it. It ships DISABLED, and its own summary says why — enumerating every NodeType and activating them together is "a boot-time subscribe/compile storm that starves the hubs", the 2026-07-22 "I have been told I am dead" restart loop. That storm is a concurrency choice, not an inherent property. DefaultMaxConcurrency was 4. I reproduced exactly that shape by accident on 2026-07-28 04:05 — four compiles fired at memex in quick succession while gathering diagnostics — and within minutes SIX plugin roots (Claims, Edu, Underwriting, Chess, Publish, Training) fell to the "did not settle" overlay and needed a scale-to-zero to recover. Concurrency buys nothing here: the compiles already serialize on the Compile IoPool, so 4 adds only simultaneous cold ACTIVATIONS — and activation is the expensive part. Issue #686 measured it: the same NodeType shape, same image, compiles in 109ms with 0ms discovery on a fresh mesh versus 45.20s of discovery on memex. The difference is cold cross-partition hubs, which is precisely what warming exists to pay off, and precisely what doing four at once turns into an outage. So: default concurrency 1, plus a 2s stagger between types so back-to-back cold activations do not read as a burst. The sweep has no deadline and lazy compilation still covers correctness, so a trickle costs nothing. This is what makes the warm-up safe to actually turn on (PreWarm:DynamicTypes) — which is the real fix for a stall that only ever bites the first request after a deploy. Pinned by DefaultConcurrency_IsOne_AndTypesArePaced, because the damage only appears at production scale where no test runs. 2/2 DynamicTypePreWarmerTest pass. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Copilot

Pull request overview

This PR aims to make DynamicTypePreWarmer safe to enable in production by preventing activation/compile storms during startup pre-warming of dynamic NodeTypes.

Changes:

Reduce default pre-warm concurrency (DefaultMaxConcurrency) from 4 to 1 to avoid concurrent cold activations.
Introduce a BetweenTypes pacing interval intended to stagger warm-ups over time.
Add a test that pins the intended default concurrency and validates the pacing interval bounds.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File	Description
`src/MeshWeaver.Hosting/DynamicTypePreWarmer.cs`	Lowers default concurrency and adds per-type pacing logic to avoid warm-up storms.
`test/MeshWeaver.Hosting.Monolith.Test/DynamicTypePreWarmerTest.cs`	Adds a regression test asserting default concurrency and the existence of a non-zero pacing interval.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+                            .Select((p, i) => WarmOne(workspace, accessService, p, budget, logger)
+                                .DelaySubscription(TimeSpan.FromTicks(BetweenTypes.Ticks * i)))


github-actions · 2026-07-28T04:39:11Z

Test Results (shard 3)

12 files ±0 12 suites ±0 4m 22s ⏱️ -3s
974 tests ±0 970 ✅ ±0 4 💤 ±0 0 ❌ ±0
1 037 runs ±0 1 033 ✅ ±0 4 💤 ±0 0 ❌ ±0