Build instance rolls never converge: new memex-portal pod crash-loops at startup while the old pods keep serving #6793

Observed

FleetWatch's StuckRoll alarm on the build instance (Deployments/build, namespace build, Deployment memex-portal-deployment) is Open on the in-mesh issue Ops/Issue/build__StuckRoll since 2026-10-03 (825 versions — re-judged with every pass since), so the same shape has recurred on this instance for ~12 days. The current burst: the Deployment's template points at ReplicaSet memex-portal-deployment-79856cd7dc with image cr.meshweaver.cloud/memex-portal-ai:3.1.10869 — the same tag the 3 Ready pods already serve on the older ReplicaSet memex-portal-deployment-74c4f99bdc. The new ReplicaSet's single pod never becomes Ready: phase Running, /health at its pod IP refuses connections (HttpRequestException: Connection refused), restarts climbing to the crash-loop threshold within the first minutes of the roll. The alarm text states the Roll action that preceded it reported Done — the image was set successfully; this is what the cluster did with it.

Where and when

One namespace (build) of the fleet; no other instance shows this condition. Each roll on this instance since 2026-10-03 has left a new pod that crash-loops at startup while the old pods keep serving. Context read on the instance's own record (Deployments/build): the instance is retiring (2026-10-09: its build role moved to Deployments/control; Inventory → verified Backup → Teardown pending through the approved lane, status still Live, update policy Continuous), and its record changed on 2026-10-09 (inbox targets removed, HOSTING_RETIRE_VALUES retire-list, vaultValuesKeys declared). Whether the crash-loop began before or after exactly those record changes was not established.

Impact

No user-visible outage: the build portal keeps serving on its 3 old Ready pods, and the instance is staff-only with no live traffic (measured on its record, 2026-10-09: build queue off, worker endpoints 404). The build instance cannot complete any roll — it stays on its current pods indefinitely — and each roll leaves a crash-looping pod and re-fires a Critical alarm.

Not established

• Why the new pod exits. Its container / previous-container log is not in the incident and not in the mesh; nothing names the terminating exception. The restart cadence (threshold reached within the first minutes of the roll) indicates a startup crash rather than a slow readiness bake, but the faulting exception was not observed.
• Whether tag 3.1.10869 names the same digest the serving pods run — the alarm compares tags only; a re-published tag (new digest under the same tag) would look exactly like this.
• Whether any other fleet instance rolled 3.1.10869 cleanly (not read).
• Whether the condition still occurs beyond this burst (one alarm line at triage time).

Candidate causes and the reading that separates them

1. Startup failure in the build instance's current rendered configuration — a secret/env/value the new pod mounts that the still-serving pods only hold from an earlier render (the 2026-10-09 retirement changes are in this window). Predicts: the pod's previous-container log shows a config or secret failure at boot. Reading: that log for the new ReplicaSet's pod, and a diff of the rendered env/volumes of 79856cd7dc against the serving 74c4f99bdc.
2. A defect in the 3.1.10869 image that manifests only on build's startup path. Predicts: other instances' rolls to the same tag crash-loop too. Reading: any other fleet instance's roll history for 3.1.10869 (their Ops/Status readings and issue nodes).
3. A same-tag re-roll onto a re-published digest — the Roll set the tag the instance already serves, and the new digest crashes at startup. Predicts: the serving pods' imageID (digest) differs from the new ReplicaSet's. Reading: the two pods' imageID fields, or the registry's tag history for cr.meshweaver.cloud/memex-portal-ai:3.1.10869.

Adjacent, not duplicates: MeshWeaver.Plugins#2411 — the alarm sizes a roll on the startup budget alone, so a gated roll still baking can read as a false StuckRoll. This burst does not fit that shape: the pod restarts 5 times and refuses /health — a crash loop, not a bake; #2411's budget disagreement affects this alarm's wording but is not the cause here. Plugins#2403 / #2407 (a routed Roll stuck in Verify one generation) is a different failure — the Roll here reported Done.

Where to look

• Ops/Issue/build__StuckRoll — the in-mesh issue, 825 versions of per-pass evidence since 2026-10-03.
• Deployments/build — the instance record (retiring; the 2026-10-09 changes are the config-window candidate above).
• Hosting/Issue/Source/RollAlarms.cs (RollAlarms.Budget, RecordFactsOf) — the alarm's judgement and the tag-level "not taking over from" comparison.
• The new pod's own previous-container log — the one reading not available from the mesh.

Evidence

2026-10-15 04:33:51Z memex-portal-deployment-74c4f99bdc-7j7mv fail: FleetWatch[0]
      [FleetWatch] roll alarm StuckRoll on build: build: Stuck roll — 3.1.10869 is not taking over from 3.1.10869. Deployment memex-portal-deployment in namespace build: its template (ReplicaSet memex-portal-deployment-79856cd7dc) runs cr.meshweaver.cloud/memex-portal-ai:3.1.10869, while 3 ready pod(s) serve cr.meshweaver.cloud/memex-portal-ai:3.1.10869. New pod memex-portal-deployment-79856cd7dc-ldx5c has restarted 5 time(s) without becoming Ready (crash-loop threshold 5). Roll in flight since 2026-10-15 04:26:49Z; budget progress deadline 3 h 40 m (startup probe 10800 s + 600 s + bake window 1800 s) × 3 replica(s) = 11 h 0 m per roll. Pod memex-portal-deployment-79856cd7dc-ldx5c: phase Running, 5 restart(s), started 2026-10-15 04:26:49Z; readiness: /health unreachable at 10.244.6.189:8080 — HttpRequestException: Connection refused (10.244.6.189:8080). A Roll action that reported Done has only SET the image — this is what the cluster did with it. The remedy is the cause the pod names, not another roll and not a pin. Observed template 3.1.10869 — 0/1 new pod(s) Ready; serving 3.1.10869; expected every desired replica on 3.1.10869 within 11 h 0 m.
2026-10-15 04:34:24Z memex-portal-deployment-744b9866fb-rjdb9 fail: FleetWatch[0]
      [FleetWatch] roll alarm StuckRoll on build: build: Stuck roll — 3.1.10869 is not taking over from 3.1.10869. Deployment memex-portal-deployment in namespace build: its template (ReplicaSet memex-portal-deployment-79856cd7dc) runs cr.meshweaver.cloud/memex-portal-ai:3.1.10869, while 3 ready pod(s) serve cr.meshweaver.cloud/memex-portal-ai:3.1.10869. New pod memex-portal-deployment-79856cd7dc-ldx5c has restarted 5 time(s) without becoming Ready (crash-loop threshold 5). Roll in flight since 2026-10-15 04:26:49Z; budget progress deadline 3 h 40 m (startup probe 10800 s + 600 s + bake window 1800 s) × 3 replica(s) = 11 h 0 m per roll. Pod memex-portal-deployment-79856cd7dc-ldx5c: phase Running, 5 restart(s), started 2026-10-15 04:26:49Z; readiness: /health unreachable at 10.244.6.189:8080 — HttpRequestException: Connection refused (10.244.6.189:8080). A Roll action that reported Done has only SET the image — this is what the cluster did with it. The remedy is the cause the pod names, not another roll and not a pin. Observed template 3.1.10869 — 0/1 new pod(s) Ready; serving 3.1.10869; expected every desired replica on 3.1.10869 within 11 h 0 m.

This log site's previous ticket, #6343, was closed about a different fingerprint, and this fault fired after that close. It is left closed: an occurrence of this fingerprint is no evidence that the fault it was closed for came back. This issue now tracks the log site, and later fingerprints of it fold in here.

_{Opened automatically from Admin/_LogIncident/08aff1caa8dd5e93. Recurrences are folded into this issue rather than opening new ones.
It also stands for the whole log site 8ee9e0785e83a261: other fingerprints of this site fold in here as comments rather than opening tickets of their own.}