fix(probes): the bake gate holds READINESS only — it never fails the startup probe by rbuergi · Pull Request #5749 · Systemorph/MeshWeaver

rbuergi · 2026-09-26T08:11:23Z

What — Refs #5544, Refs #5704, Refs #5746

Maintainer decision (policy bake-gate-readiness-only, now in PolicyNotProse.md): the bake gate holds READINESS only and never fails the STARTUP probe.

Root cause (measured in #5746 / TheBakeGateOnlyStallsARoll.md):

The nodetype_bake verdict made /health red.
The startup probe (10 s × 1080 on memex) killed every container at the three-hour mark.
A restarted pod of the previous image could not pass that probe either.
Result: the control instance was down from 20:54Z to 04:07Z.

New rule: the startup probe proves only that the process booted. The gate's verdict is read by /ready alone. A refusal therefore stalls the roll: the pod stays alive and out of the Service, and nothing is killed.

Code (`memex/aspire/Memex.Portal.ServiceDefaults`, `src/MeshWeaver.Hosting`)

ProbeEndpoints.RollGateTag (roll-gate). ServiceDefaults.TagRollGates is a PostConfigure over every host's registrations. It tags the check named NodeTypeBakeGateExtensions.HealthCheckName (nodetype_bake, new const) as a roll gate and strips any live tag. The rule holds for the host's current, untagged registration in MeshWeaver.Plugins, so no Plugins change is needed. It takes effect in the next portal image built on this core.
/ready reads checks tagged ready plus roll gates. Its body now names every check that is not Healthy, so the kubelet's probe-failure event says why.
/health still runs and prints every check. A roll gate's reading always prints, with the suffix [roll gate: read by /ready only, never by the startup probe]. The status code and the word on line one are the startup verdict: the worst status over the non-roll-gate checks. That is a behaviour change behind an unchanged signature (shape 7). Readers that parse the body (Sample's healthDetail) still find the nodetype_bake: line.

Per-check decision (table in the doc)

Stays on the startup probe: db_version / DbVersionGate, PostgreSql, required_modules, process_progress.
Degraded-only and census checks: unchanged; they can never produce a 503.
Named as an open question, not decided here: required_modules has the same hazard shape when the registry is down for every image.

Chart (`deploy/helm`)

progressDeadlineSeconds adds probes.rollGate.bakeSeconds (1800) when PreWarm__GateReadiness is armed in the render. Rendered values: 900 with the gate off, 2700 with it armed.
Arming the gate no longer means raising probes.startup. Prose and comments are updated.
Invariant 10b now refuses two things for an armed gate: a readiness probe that is not on /ready, and a deadline shorter than startup budget + 600 s headroom + a cold bake. Negative control: bakeSeconds=0 produces the finding. The fixture no longer raises the startup budget, and its evidence check asserts /ready and a deadline of 2700.

Guards

RollGateReadinessOnlyTest, over real HTTP on the probe paths the chart ships. A refusing nodetype_bake, registered untagged as the host does, must give: startup 200, readiness 503 naming the gate, liveness 200, and a line on /health.
A live-tagged gate still cannot restart the pod.
Negative control: an ordinary Unhealthy check (db_version shape) still fails startup.
PreWarmGateReadinessGuard rewritten for the new prerequisites.

Docs

TheBakeGateOnlyStallsARoll gains "The gate holds readiness only" (wiring and the per-check table), and the old "startup probe is still the gate's only reader" caveat is replaced.
DeploymentAKS gains "Which probe reads what".
PolicyNotProse gains two rows: bake-gate-readiness-only (in force) and silos-pool-autoscaler-max (proposed, owed items named).

⚠️ #5746 (open) edits the same doc page, in a different section. Whichever PR lands second merges the other.

Verified locally

dotnet build -c Release -warnaserror is clean for: Memex.Portal.ServiceDefaults, MeshWeaver.Hosting, MeshWeaver.Documentation, Memex.Portal.Shared.Test, MeshWeaver.Documentation.Test.
Memex.Portal.Shared.Test: 2039 total, 0 failed.
MeshWeaver.Documentation.Test: 649 total, 0 failed.
check-chart-invariants.sh: all 16 combinations pass and all 6 refusal controls hold.

Companions: Systemorph/Memex#546 (silos pool autoscaler max 4 → 5) and the memex startup-budget PR.

Pairs-with: none — no public type or member is removed; one public const (NodeTypeBakeGateExtensions.HealthCheckName) and one (ProbeEndpoints.RollGateTag) are added.
Mirror-sync: none — no localization catalog key is added or changed.

🤖 Generated with Claude Code

…startup probe (#5544, #5704) Policy bake-gate-readiness-only. The nodetype_bake verdict made /health red, the startup probe (10 s x 1080 on memex) killed every container at the three-hour mark, and a restarted pod of the previous image could not pass it either: memex.systemorph.com was down 20:54Z-04:07Z. - ProbeEndpoints.RollGateTag ("roll-gate"). ServiceDefaults.TagRollGates (PostConfigure over every host registration) tags the check named NodeTypeBakeGateExtensions.HealthCheckName ("nodetype_bake") as a roll gate and strips any live tag — whatever the host registered. - /ready reads ready-tagged checks PLUS roll gates, and names every non-Healthy check in its body. - /health still runs and prints every check (a roll gate always prints, marked as read by /ready only); its status code and line one are the startup verdict, which excludes roll gates. - Chart: progressDeadlineSeconds adds probes.rollGate.bakeSeconds (1800) when the gate is armed; arming no longer raises probes.startup. Invariant 10b now requires an armed gate's reader to be the readinessProbe on /ready and the deadline to cover a cold bake; fixture updated. - RollGateReadinessOnlyTest pins it over real HTTP on the chart's paths, with a negative control (an ordinary Unhealthy check still fails startup). PreWarmGateReadinessGuard updated. - Docs: TheBakeGateOnlyStallsARoll (wiring + per-check table), DeploymentAKS, PolicyNotProse (bake-gate-readiness-only, silos-pool-autoscaler-max). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

…ss-only # Conflicts: # src/MeshWeaver.Documentation/Data/Architecture/TheBakeGateOnlyStallsARoll.md

Copilot

Copilot review overview

🟡 Changes recommended

Fix the chart’s opaque-source and startup-probe invariants, and reconcile the remaining operator documentation contradictions.

Review effort: Lite
Findings: 1 · 2

Open (3)

Opaque gate overrides can under-size the rollout deadline · New
Update Ready documentation to include roll-gate readiness · New
Update operator documentation for the new gate contract · New

What changed in this PR

Moves the NodeType bake gate to readiness-only handling so stalled rollouts keep pods alive while remaining out of service.

Changes:

Separates startup and readiness probe verdicts with roll-gate tagging.
Updates Helm rollout deadlines, fixtures, and chart invariants.
Adds HTTP coverage and updates operational documentation.

File	Summary
`test/MeshWeaver.Documentation.Test/PreWarmGateReadinessGuard.cs`	Updates chart prerequisite guards.
`test/Memex.Portal.Shared.Test/RollGateReadinessOnlyTest.cs`	Tests probe separation behavior.
`src/MeshWeaver.Hosting/NodeTypeBakeGate.cs`	Adds the gate health-check name contract.
`src/MeshWeaver.Documentation/Data/Architecture/TheBakeGateOnlyStallsARoll.md`	Documents readiness-only behavior.
`src/MeshWeaver.Documentation/Data/Architecture/PolicyNotProse.md`	Registers the policy.
`src/MeshWeaver.Documentation/Data/Architecture/DeploymentAKS.md`	Documents probe responsibilities.
`memex/aspire/Memex.Portal.ServiceDefaults/ServiceDefaults.cs`	Tags roll gates and separates health verdicts.
`memex/aspire/Memex.Portal.ServiceDefaults/ProbeEndpoints.cs`	Defines roll-gate endpoint semantics.
`deploy/helm/values.yaml`	Adds bake deadline configuration.
`deploy/helm/templates/memex-portal/deployment.yaml`	Applies the conditional rollout deadline.
`deploy/helm/templates/memex-portal/config.yaml`	Updates gate configuration guidance.
`deploy/aks/scripts/testdata/values.bake-gate-armed.yaml`	Updates the armed-gate fixture.
`deploy/aks/scripts/check-chart-invariants.sh`	Verifies armed-gate rendering.
`deploy/aks/scripts/check-chart-invariants.py`	Validates readiness, startup-probe, and deadline invariants.

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

rbuergi · 2026-09-26T08:22:18Z

+  # roll, which is what a refusing gate is. ⚠️ A gate armed by an env source this render cannot see
+  # (a Key Vault Secret, an inline env) does not add the bake term: arm it in
+  # config.memex_portal, where the render sees it.
+  progressDeadlineSeconds: {{ add (mul .Values.probes.startup.periodSeconds .Values.probes.startup.failureThreshold) 600 (ternary ((.Values.probes.rollGate).bakeSeconds | default 0) 0 (eq (toString .Values.config.memex_portal.PreWarm__GateReadiness) "true")) }}


Agreed, fixed in the follow-up commit. The bake term is no longer conditional. progressDeadlineSeconds = startup budget + 600 s + probes.rollGate.bakeSeconds in EVERY render, because arming can come from a source the render cannot see. Invariant 10b's deadline check now runs on every render as well. The trade-off: an ungated stuck roll is reported later. That is acceptable, because the deadline only reports and never kills.

rbuergi · 2026-09-26T08:22:19Z

+    /// <c>/ready</c> — the READINESS path. Checks tagged <see cref="ReadyTag"/> or
+    /// <see cref="RollGateTag"/>: the trivial process-up check, plus any roll gate the host
+    /// registered (the NodeType bake gate, <c>nodetype_bake</c>).


Fixed. The class summary's list now describes /health as the startup verdict with roll gates excluded from its status, and /ready as ready-tagged checks plus roll gates, including why a roll gate's 'no' is about the image.

rbuergi · 2026-09-26T08:22:20Z

+This table is correct for core's `main` and Plugins' `main` on the day it was written. A check added
+later that can answer Unhealthy must be placed deliberately: startup (`/health`, the container is
+killed) or roll gate (`/ready`, the roll stalls).


Fixed in this change. deploy/aks/README.md now names the readinessProbe on /ready as the gate's only reader and drops the 'REQUIRED with the gate' startup budget. SELF-UPDATE.md now says readiness, not the startupProbe on /health. values.aks.yaml no longer lists a raised startup budget as prerequisite 1, and its budget goes from 10 × 180 to 10 × 60 (plain boot; the gate is false there). A grep for gate-on-/health prose under deploy/aks finds only the historical 'until that change' sentences.

github-actions · 2026-09-26T08:19:22Z