Repository navigation
Rollout of build to 3.0.0-ci.10150 never converges — the portal serves nothing and new pods never become ready #6343
Description
What is failing
The build deployment's portal (memex-portal-deployment, namespace build, build.meshweaver.cloud) is stuck mid-roll to 3.0.0-ci.10150 and has never converged: the roll is roughly 4× past its own 11-hour budget, zero ready pods serve the portal, and the fleet watch keeps sampling freshly-created new-generation pods that sit in phase Pending with /health connection-refused at their own address. The portal is effectively down, not slow.
Probable cause
The Roll action itself behaved as designed — image mirrored and present in the fleet registry, pull secret verified, database migration succeeded, image set, rollout handed to the control plane at generation 148, action Done (Ops/Actions/selfupdate-roll-build-3-0-0-ci-10150-bb46c809; per ApplyingIsNotRollingOut, Done means applied, not rolled out). What is broken is the cluster side of that hand-off: pods of the new ReplicaSet (584c896459) never reach a serving state. Moderate confidence on that shape; the exact mechanism is not established by the log line — the pod is Pending with zero restarts and connection refused at its pod IP, so the first things to check are pod events / kubectl describe: scheduling and resources (requests are 4 CPU / 8 GiB per pod against a 3-replica record with KEDA 3–7), image pull, or the application never opening port 8080 inside its startup gate. The status sample taken at hand-off time already read /health Degraded on the one ready new pod, so the new build was serving badly from the start.
Impact
build.meshweaver.cloud serves nothing — a primary fleet instance down, and the alarm is corroborated by the sibling open StalePortal alarm on the same deployment (Ops/Issue/build__StalePortal: newer promoted builds are not being taken), so fleet CD through the build instance is blocked while this stands. This is not the known StuckRoll false positive: the budget derivation in the alarm already includes the bake window (the Plugins#2411 fix is live) and the overrun is ~4× the budget, so the alarm is judged correct.
Where to look
- Durable incidents with the freshest samples:
Ops/Issue/build__StuckRollandOps/Issue/build__StalePortal. - The Roll action record and its operator log:
Ops/Actions/selfupdate-roll-build-3-0-0-ci-10150-bb46c809— all five steps green, hand-off atrollout_generation=148. - Alarm rules that fired:
Hosting/Issue/Source/RollAlarms.csandFleetWatch.cs(FileAlarm); design doc:Hosting/RollAlarms. - Not a duplicate of Plugins#2403 (closed): there the Roll action was stuck at "Verify one generation"; here the action completed and the Kubernetes rollout itself is stuck. Memex#628 (routed rolls of build always refused) is a different failure on the same deployment, worth knowing when working the build lane.
Evidence
| Fingerprint | cd94d5da0e0a278f |
| Category | FleetWatch |
| Severity | Error |
| Exception | HttpRequestException |
| Namespace | memex |
| Pods | memex-portal-deployment-69fdc8c44-hv4rk |
| Occurrences | 1 |
| First seen | 2026-10-09 07:00:25Z |
| Last seen | 2026-10-09 07:00:25Z |
| Routing | not determined — no configured route matches the category FleetWatch. This repository is the configured fallback, not a finding about who owns the fault; the category names the LOGGER, which may not be the subject. |
Recent log lines
2026-10-09 07:00:25Z memex-portal-deployment-69fdc8c44-hv4rk fail: FleetWatch[0]
[FleetWatch] roll alarm StuckRoll on build: build: Stuck roll — 3.0.0-ci.10150 is not taking over from nothing. Deployment memex-portal-deployment in namespace build: its template (ReplicaSet memex-portal-deployment-584c896459) runs cr.meshweaver.cloud/memex-portal-ai:3.0.0-ci.10150, while 0 ready pod(s) serve nothing. The roll has been in flight 1 d 16 h — past its budget of 11 h 0 m. Roll in flight since 2026-10-07 14:30:21Z; budget progress deadline 3 h 40 m (startup probe 10800 s + 600 s + bake window 1800 s) × 3 replica(s) = 11 h 0 m per roll. Pod memex-portal-deployment-584c896459-8zh6n: phase Pending, 0 restart(s), started 2026-10-09 06:58:59Z; readiness: /health unreachable at 10.244.2.84:8080 — HttpRequestException: Connection refused (10.244.2.84:8080). A Roll action that reported Done has only SET the image — this is what the cluster did with it. The remedy is the cause the pod names, not another roll and not a pin. Observed template 3.0.0-ci.10150 — 1/2 new pod(s) Ready; serving nothing; expected every desired replica on 3.0.0-ci.10150 within 11 h 0 m.
Opened automatically from Admin/_LogIncident/cd94d5da0e0a278f. Recurrences are folded into this issue rather than opening new ones.
It also stands for the whole log site 8ee9e0785e83a261: other fingerprints of this site fold in here as comments rather than opening tickets of their own.
Activity
rbuergi commentedon Oct 9, 2026
Verified on the running build instance, 2026-10-09 14:22–14:35Z. The rollout converges again; closing.
Readings
Ops/Status/buildon control.systemorph.com, sampled 14:22:12Z:notScraped: false, 3/3 ready,converged: true,generations: 1,warnings: []. All three replicas runcr.meshweaver.cloud/memex-portal-ai:3.0.0-ci.10319(ReplicaSetfd79c5f7c:25dhf,526tz,s4gmc), with 0 restarts and commit9704e993. Every replica reportsnodetype_bake: Healthy(146 already baked) andrequired_modules: Healthy.https://build.meshweaver.cloud/answers 200 and/api/versionanswers9704e993f50c…. Three probes at 14:34:58Z all returned that./healthisDegradedonly onbundle_adoption(Antigravity bundle carries no assemblies) and on one replica'spending_module_activation(MeshWeaver.AI, see below). Neither affects readiness.Ops/Issue/build__StuckRollno longer exists, andOps/Issue/build__StalePortalreads(resolved).Ops/Actions/selfupdate-roll-build-3-0-0-ci-10319-05d0cc8c) went from helm revision 10 at 14:04:41Z to 3/3 ready at 14:09:49Z, about 5 minutes against an 11 h budget.What held the 10150 roll (governed reads
Ops/Actions/logs-build-20261009-1425-crit-during-10150andOps/Actions/logs-build-20261009-1438-watchdog-only, namespacebuild, 2026-10-07 14:09Z to 2026-10-09 14:32Z)The pods of the new ReplicaSet were not stuck before start. Pod
memex-portal-deployment-584c896459-vln7xran and logged throughout. At 2026-10-07 19:27:04Z it logged[RebootWatchdog] WEDGED — SELF-REBOOT filed at Admin/_Reboot/20261007192704771-3b83874807b6: 3 critical NodeType(s) in compile Error for at least 20 min: Hosting/BugFixPool, Hosting/Coordination, Hosting/TriageStatus. By 2026-10-08 16:02:34Z it logged… a reboot is already in progress (Admin/_Reboot/20261007192704771…) … 17 critical NodeType(s) in compile Error.So the Hosting sources synced from Plugins main needed platform surface that the 10150 image did not have. This is the same class as #6195. The Hosting types stayed in Error on that image, and the replica's readiness gate kept the new pods out of service.
The image could not move forward because build's self-updater announced to the retired inbox (#5757). Two things cleared that:
Hosting__ReportTo: https://control.systemorph.com, applied by the approved ProvisionOps/Actions/provision-build-20261009-values-half(06:58Z, revision 5; it also composedhelm-values-buildfor AdoNet clustering,vault_values_bytes=633; step 13successfully rolled out, step 15answered 200);The cause is fixed, not masked. The instance now reports to control, and control has routed five rolls today that each converged. Since 07:00Z, the Hosting NodeTypes on build bake clean on every new replica.
Not established: the per-pod Kubernetes events for the one pod the alarm sampled as
Pending(584c896459-8zh6n, started 06:58:59Z). That pod belonged to the Provision's own rollout, which completed.Two incidental defects found while verifying, filed into triage (not this issue):
Ops/Actions/selfupdate-roll-build-3-0-0-ci-10322-e2fcb389, Failed at mirror:memex-portal-ai:3.0.0-ci.10322 does not exist. CD's arm job had loggednothing to armfor 10322. Code: MeshWeaver.PluginsHosting/Deployment/Source/SelfUpdateRouting.csRollInsteadOfRestart(…, controlLatestTag). Nothing destructive ran. Triage itemrbuergi/Feedback/selfupdate-roll-to-unarmed-control-tag-20261009.The HttpClient factory already has a registered client with the name 'WebSearchPlugin'. Because of that, one build replica keepspending_module_activation: MeshWeaver.AI. Triage itemrbuergi/Feedback/ai-websearch-httpclient-name-clash-live-swap-20261009.