A warm-up sweep faulted by a transient store refusal is never re-run — nodetype_bake held a memex pod unready ~74 min after Postgres recovered, until the pod was replaced

Observed

The nodetype_bake roll gate (check registered by the MeshWeaver.Plugins host under NodeTypeBakeGateExtensions.HealthCheckName when gateBake is armed) answered Unhealthy on the memex control-instance portal: the NodeType bake outcome was Faulted before the sweep could measure anything — the gate's own message says "the warm-up stream faulted before the sweep could finish". Each evaluation of the check returned in microseconds, i.e. the check is reading an already-faulted DynamicTypePreWarmer outcome, not compiling anything itself.

This refusal path is documented, deliberate gate behaviour: a Faulted sweep on an image that has NOT been admitted still refuses readiness unless PreWarm:AllowUnprovenBake is set (Doc/Architecture/TheBakeGateOnlyStallsARoll, § "What this does not cover"). The defect this ticket tracks is therefore whatever faulted the warm-up stream, and — if this build had already served — why the served-witness did not exempt it. The refusal itself is the gate working as designed.

Where and when

One memex portal pod, one fault, read repeatedly on the readiness cadence over roughly eleven seconds — three samples of a single faulted outcome, not three faults. Not established from the incident whether this pod was a first-of-a-new-build roll candidate or a restart of an already-serving image; the gate treats those two cases differently, and the evidence to tell them apart (the pod's platform build, the ReplicaSet it belongs to, whether any replica of that build was ever admitted) is not on the incident.

Impact

Small so far: one pod, one fault. But the documented consequence is that a refused pod stays alive and out of the Service indefinitely — under policy bake-gate-readiness-only nothing kills the pod anymore, and a Faulted sweep stays faulted until someone acts. If the input that faulted the warm-up stream is on every pod of this image, the rollout is fully stalled and will not self-heal. Whether a roll was actually blocked by this fault is not established.

Not established

Why the warm-up stream faulted. No exception type and no stack frame were captured on the incident, and no DynamicTypePreWarmer or BatchBake log line for this pod exists anywhere under Ops/Logs — searches for the pod's id and for faulted/warm-up lines returned zero. That absence is itself a reading worth explaining: the fault left no per-type compile evidence in the mesh's log sweep.
Whether the served-witness was read. Per the architecture doc, a Faulted sweep on an image that HAS served no longer refuses (the gate reads NodeTypeBakeGateState.ServedBefore directly), and a marker read that FAILS counts as "not served" — the strict reading. So either this build had genuinely never been admitted here, or the Admin/ServedPlatformBuilds/<build> row was missing/unreadable.
Whether the pod recovered, whether other replicas of the same image show the same fault, and whether the burst coincided with a roll, a restart, or a mesh change.
The gate message's restart advice. The message says "A restart re-runs the sweep", while the architecture doc states that under the readiness-only split a refused pod is no longer killed and a faulted sweep stays errored until a governed Restart of the instance or PreWarm:AllowUnprovenBake. Which text matches the deployed image is not established; the two disagree about what happens next.

Candidate causes and the reading that separates them

A real error inside the sweep (source discovery or a NodeType compile faulted the warm-up stream). Predicts per-type BatchBake … CompileError or DynamicTypePreWarmer lines from the same pod just before the gate refused. Reading: raw Loki stdout for this pod around the refusal window — the mesh's Ops/Logs sweep holds none, so start from Loki, not the mesh.
The image has served here but the witness read failed (row at Admin/ServedPlatformBuilds/<build> missing or unreadable through IStorageAdapter), so the strict reading refused a serving image. Predicts the marker row for this build is absent while the deployment history shows this build already admitted. Reading: the ServedPlatformBuilds row for this pod's platform build, and whether the gate's ServedBefore answer was "unreadable" rather than "false".
First boot of a never-admitted build (a genuine roll candidate) whose sweep really did fault — then the refusal is the gate working and the defect is the sweep fault alone. Reading: the pod's ReplicaSet and image tag, and whether any replica of that build has ever been admitted on this instance.

Known-issue check

Not a duplicate: the gate-refusal behaviour and the 2026-09-25/26 outage it caused are closed (#5725, #5749); the nearest filed items are MeshWeaver#5907 (empty bake report on closed-type-set hosts) and #6035 (bake census counting a record with no node), which are different symptoms. Nothing filed covers a Faulted warm-up sweep on nodetype_bake.

Where to look

Doc/Architecture/TheBakeGateOnlyStallsARoll — the refusal contract, the Faulted-sweep rules, and the served-witness (ServedBuildWitness, NodeTypeBakeGateState.ServedBefore).
The nodetype_bake check registration in the MeshWeaver.Plugins host (NodeTypeBakeGateExtensions, under if (gateBake)).
DynamicTypePreWarmer's sweep and its BaselineStamp.
The witness rows at Admin/ServedPlatformBuilds/<build>, read straight through IStorageAdapter.
RollGateReadinessOnlyTest — the guards that hold what the gate may and may not refuse.

Evidence


Fingerprint	`bde3da18f838e9ef`
Category	`Microsoft.Extensions.Diagnostics.HealthChecks.DefaultHealthCheckService`
Severity	Error
Namespace	`memex`
Pods	`memex-portal-deployment-65d6cb7648-97nnd`, `memex-portal-deployment-54486c96c4-msxhn`, `memex-portal-deployment-85cf49759f-pnbwl`
Occurrences	3359
First seen	2026-10-12 10:47:02Z
Last seen	2026-10-22 21:23:07Z
Routing	not determined — no configured route matches the category `Microsoft.Extensions.Diagnostics.HealthChecks.DefaultHealthCheckService`. This repository is the configured fallback, not a finding about who owns the fault; the category names the LOGGER, which may not be the subject.

Recent log lines

2026-10-12 17:04:57Z memex-portal-deployment-54486c96c4-msxhn fail: Microsoft.Extensions.Diagnostics.HealthChecks.DefaultHealthCheckService[103]
      Health check nodetype_bake with status Unhealthy completed after 0.0017ms with message 'NodeType bake FAULTED before it could verify this image — refusing readiness. A sweep that errored has verified nothing, and certifying an unmeasured bake is precisely what this gate exists to prevent. A restart re-runs the sweep; to serve regardless, set PreWarm:AllowUnprovenBake=true. bake NOT PROVEN — the sweep errored before it could verify this image: the warm-up stream faulted before the sweep could finish'
2026-10-12 17:05:07Z memex-portal-deployment-54486c96c4-msxhn fail: Microsoft.Extensions.Diagnostics.HealthChecks.DefaultHealthCheckService[103]
      Health check nodetype_bake with status Unhealthy completed after 0.0014ms with message 'NodeType bake FAULTED before it could verify this image — refusing readiness. A sweep that errored has verified nothing, and certifying an unmeasured bake is precisely what this gate exists to prevent. A restart re-runs the sweep; to serve regardless, set PreWarm:AllowUnprovenBake=true. bake NOT PROVEN — the sweep errored before it could verify this image: the warm-up stream faulted before the sweep could finish'
2026-10-12 17:05:17Z memex-portal-deployment-54486c96c4-msxhn fail: Microsoft.Extensions.Diagnostics.HealthChecks.DefaultHealthCheckService[103]
      Health check nodetype_bake with status Unhealthy completed after 0.0016ms with message 'NodeType bake FAULTED before it could verify this image — refusing readiness. A sweep that errored has verified nothing, and certifying an unmeasured bake is precisely what this gate exists to prevent. A restart re-runs the sweep; to serve regardless, set PreWarm:AllowUnprovenBake=true. bake NOT PROVEN — the sweep errored before it could verify this image: the warm-up stream faulted before the sweep could finish'
2026-10-12 17:05:27Z memex-portal-deployment-54486c96c4-msxhn fail: Microsoft.Extensions.Diagnostics.HealthChecks.DefaultHealthCheckService[103]
      Health check nodetype_bake with status Unhealthy completed after 0.0015ms with message 'NodeType bake FAULTED before it could verify this image — refusing readiness. A sweep that errored has verified nothing, and certifying an unmeasured bake is precisely what this gate exists to prevent. A restart re-runs the sweep; to serve regardless, set PreWarm:AllowUnprovenBake=true. bake NOT PROVEN — the sweep errored before it could verify this image: the warm-up stream faulted before the sweep could finish'
2026-10-12 17:05:33Z memex-portal-deployment-54486c96c4-msxhn fail: Microsoft.Extensions.Diagnostics.HealthChecks.DefaultHealthCheckService[103]
      Health check nodetype_bake with status Unhealthy completed after 0.0064ms with message 'NodeType bake FAULTED before it could verify this image — refusing readiness. A sweep that errored has verified nothing, and certifying an unmeasured bake is precisely what this gate exists to prevent. A restart re-runs the sweep; to serve regardless, set PreWarm:AllowUnprovenBake=true. bake NOT PROVEN — the sweep errored before it could verify this image: the warm-up stream faulted before the sweep could finish'
2026-10-12 17:05:35Z memex-portal-deployment-54486c96c4-msxhn fail: Microsoft.Extensions.Diagnostics.HealthChecks.DefaultHealthCheckService[103]
      Health check nodetype_bake with status Unhealthy completed after 0.0018ms with message 'NodeType bake FAULTED before it could verify this image — refusing readiness. A sweep that errored has verified nothing, and certifying an unmeasured bake is precisely what this gate exists to prevent. A restart re-runs the sweep; to serve regardless, set PreWarm:AllowUnprovenBake=true. bake NOT PROVEN — the sweep errored before it could verify this image: the warm-up stream faulted before the sweep could finish'
2026-10-12 17:05:43Z memex-portal-deployment-54486c96c4-msxhn fail: Microsoft.Extensions.Diagnostics.HealthChecks.DefaultHealthCheckService[103]
      Health check nodetype_bake with status Unhealthy completed after 0.001ms with message 'NodeType bake FAULTED before it could verify this image — refusing readiness. A sweep that errored has verified nothing, and certifying an unmeasured bake is precisely what this gate exists to prevent. A restart re-runs the sweep; to serve regardless, set PreWarm:AllowUnprovenBake=true. bake NOT PROVEN — the sweep errored before it could verify this image: the warm-up stream faulted before the sweep could finish'
2026-10-22 21:22:55Z memex-portal-deployment-85cf49759f-pnbwl fail: Microsoft.Extensions.Diagnostics.HealthChecks.DefaultHealthCheckService[103]
      Health check nodetype_bake with status Unhealthy completed after 0.0043ms with message 'NodeType bake FAULTED before it could verify this image — refusing readiness. A sweep that errored has verified nothing, and certifying an unmeasured bake is precisely what this gate exists to prevent. A restart re-runs the sweep; to serve regardless, set PreWarm:AllowUnprovenBake=true. bake NOT PROVEN — the sweep errored before it could verify this image: the warm-up stream faulted before the sweep could finish'
2026-10-22 21:22:57Z memex-portal-deployment-85cf49759f-pnbwl fail: Microsoft.Extensions.Diagnostics.HealthChecks.DefaultHealthCheckService[103]
      Health check nodetype_bake with status Unhealthy completed after 0.0048ms with message 'NodeType bake FAULTED before it could verify this image — refusing readiness. A sweep that errored has verified nothing, and certifying an unmeasured bake is precisely what this gate exists to prevent. A restart re-runs the sweep; to serve regardless, set PreWarm:AllowUnprovenBake=true. bake NOT PROVEN — the sweep errored before it could verify this image: the warm-up stream faulted before the sweep could finish'
2026-10-22 21:23:07Z memex-portal-deployment-85cf49759f-pnbwl fail: Microsoft.Extensions.Diagnostics.HealthChecks.DefaultHealthCheckService[103]
      Health check nodetype_bake with status Unhealthy completed after 0.0049ms with message 'NodeType bake FAULTED before it could verify this image — refusing readiness. A sweep that errored has verified nothing, and certifying an unmeasured bake is precisely what this gate exists to prevent. A restart re-runs the sweep; to serve regardless, set PreWarm:AllowUnprovenBake=true. bake NOT PROVEN — the sweep errored before it could verify this image: the warm-up stream faulted before the sweep could finish'

This log site's previous ticket, #5722, was closed about a different fingerprint, and this fault fired after that close. It is left closed: an occurrence of this fingerprint is no evidence that the fault it was closed for came back. This issue now tracks the log site, and later fingerprints of it fold in here.

_{Opened automatically from Admin/_LogIncident/bde3da18f838e9ef. Recurrences are folded into this issue rather than opening new ones.}
_{It also stands for the whole log site 5c0c3ddb141a07da: other fingerprints of this site fold in here as comments rather than opening tickets of their own.}

Folded in — a DIFFERENT fingerprint from the same log site.

This issue tracks the log site 5c0c3ddb141a07da (Microsoft.Extensions.Diagnostics.HealthChecks.DefaultHealthCheckService), and these occurrences belong to it. They carry their own fingerprint — the masking split one site across several — so before this they would have opened a SEPARATE ticket. One log site fanned out into 46 of those, which is why they fold here instead. Nothing is discarded: the distinguishing evidence is below, and splitting this back out is a person's call.


Folded fingerprint	`64d3e58d56547181`
Log site	`5c0c3ddb141a07da`
Category	`Microsoft.Extensions.Diagnostics.HealthChecks.DefaultHealthCheckService`
Severity	Error
Top frame	`Memex.Portal.Distributed.DbVersionHealthCheck.CheckHealthAsync(HealthCheckContext context, CancellationToken cancellationToken)`
Detail	`Health check db_version with status Unhealthy completed after {n}ms with message '{value}'`
Namespace	`memex`
Pods	`memex-portal-deployment-7b6649594-wtnwm`, `memex-portal-deployment-6485bffdff-cgd5r`, `memex-portal-deployment-56446674bc-7qdml`, `memex-portal-deployment-85995ff7b5-tlj4k`, `memex-portal-deployment-56446674bc-7l8cg`, `memex-portal-deployment-58c4469b85-jh7px`, `memex-portal-deployment-85cf49759f-pnbwl`, `memex-portal-deployment-58c4469b85-sfkq4`, `memex-portal-deployment-85cf49759f-77plc`, `memex-portal-deployment-5d997855c9-scc52`, `memex-portal-deployment-5d997855c9-p4t6q`, `memex-portal-deployment-5d997855c9-9ndvv`
Occurrences	23
First seen	2026-09-24 08:13:11Z
Last seen	2026-10-22 21:23:02Z
Already folded here	1 fingerprint(s)
Incident	`Admin/_LogIncident/64d3e58d56547181`
Triage called it	db_version health check flaps Unhealthy on transient in-cluster DNS (EAI_AGAIN) when opening a Postgres connector

What triage wrote about this fingerprint

What is failing

The db_version health check in the Memex portal (Memex.Portal.Distributed.DbVersionHealthCheck.CheckHealthAsync, DbVersionGate.cs) reported Unhealthy twice because its query against Postgres could not even open a connection: DNS resolution of the database host failed with SocketException (00000001, 11): Resource temporarily unavailable (EAI_AGAIN) from Dns.GetHostEntryOrAddressesCore while NpgsqlConnector.ConnectAsync tried to open a new connector. The check correctly surfaces the upstream failure; the portal code path itself is not the defect.

Probable cause

Transient cluster DNS unavailability (EAI_AGAIN is the resolver's "try again later" error) on the memex-portal pod at 2026-09-24 ~08:13 UTC — most likely a momentary CoreDNS/kube-dns hiccup or a pod-startup DNS race, not a portal bug. Medium confidence: the stack is unambiguous about what failed, but 2 occurrences 20 s apart on one pod do not pin down which resolver component was slow.

Impact

Minimal: 2 occurrences over 20 seconds on a single pod, nothing since. No user-visible failure is evident; at worst the pod's readiness briefly flapped because this check gates readiness. If these bursts recur, the operational concern is not the DB but in-cluster DNS.

Where to look

Portal check: Memex.Portal.Distributed/DbVersionGate.cs, DbVersionHealthCheck.CheckHealthAsync — consider whether the check should tolerate a single DNS failure (e.g. short retry or degraded rather than Unhealthy) so a DNS blip does not flap pod readiness.
CoreDNS / kube-dns logs around 2026-09-24T08:13Z and the pod's DNS readiness (NetworkPolicy ndots/resolv.conf) if it recurs.

Known related incidents

Same log site and health check previously triaged in Admin/_LogIncident/ba8ddb205fa7d458 (pgvector 58P01 and a connection timeout) — different underlying causes, but the same masking rule folds every distinct exception at this site onto one fingerprint; a per-exception-type case would keep them separable.

Still occurring: 23 more since the last update (23 total).

Last seen: 2026-10-22 21:23:02Z
Pods: memex-portal-deployment-7b6649594-wtnwm, memex-portal-deployment-6485bffdff-cgd5r, memex-portal-deployment-56446674bc-7qdml, memex-portal-deployment-85995ff7b5-tlj4k, memex-portal-deployment-56446674bc-7l8cg, memex-portal-deployment-58c4469b85-jh7px, memex-portal-deployment-85cf49759f-pnbwl, memex-portal-deployment-58c4469b85-sfkq4, memex-portal-deployment-85cf49759f-77plc, memex-portal-deployment-5d997855c9-scc52, memex-portal-deployment-5d997855c9-p4t6q, memex-portal-deployment-5d997855c9-9ndvv

Recent log lines

2026-10-22 21:23:00Z memex-portal-deployment-5d997855c9-p4t6q fail: Microsoft.Extensions.Diagnostics.HealthChecks.DefaultHealthCheckService[103]
      Health check db_version with status Unhealthy completed after 26.5262ms with message 'db_version check threw'
      Npgsql.NpgsqlException (0x80004005): Failed to connect to 10.42.18.5:5432
       ---> System.Net.Sockets.SocketException (111): Connection refused
         at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
         at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
         at Npgsql.Internal.NpgsqlConnector.ConnectAsync(NpgsqlTimeout timeout, CancellationToken cancellationToken)
         at Npgsql.Internal.NpgsqlConnector.ConnectAsync(NpgsqlTimeout timeout, CancellationToken cancellationToken)
         at Npgsql.Internal.NpgsqlConnector.RawOpen(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.Internal.NpgsqlConnector.<Open>g__OpenCore|209_0(NpgsqlConnector conn, String username, SslMode sslMode, GssEncryptionMode gssEncMode, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.Internal.NpgsqlConnector.Open(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.PoolingDataSource.OpenNewConnector(NpgsqlConnection conn, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.PoolingDataSource.<Get>g__RentAsync|33_0(NpgsqlConnection conn, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.NpgsqlConnection.<Open>g__OpenAsync|42_0(Boolean async, CancellationToken cancellationToken)
         at Npgsql.NpgsqlDataSourceCommand.ExecuteReader(Boolean async, CommandBehavior behavior, CancellationToken cancellationToken)
         at Npgsql.NpgsqlCommand.ExecuteScalar(Boolean async, CancellationToken cancellationToken)
…[truncated]
2026-10-22 21:23:00Z memex-portal-deployment-5d997855c9-9ndvv fail: Microsoft.Extensions.Diagnostics.HealthChecks.DefaultHealthCheckService[103]
      Health check db_version with status Unhealthy completed after 29.6861ms with message 'db_version check threw'
      Npgsql.NpgsqlException (0x80004005): Failed to connect to 10.42.18.5:5432
       ---> System.Net.Sockets.SocketException (111): Connection refused
         at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
         at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
         at Npgsql.Internal.NpgsqlConnector.ConnectAsync(NpgsqlTimeout timeout, CancellationToken cancellationToken)
         at Npgsql.Internal.NpgsqlConnector.ConnectAsync(NpgsqlTimeout timeout, CancellationToken cancellationToken)
         at Npgsql.Internal.NpgsqlConnector.RawOpen(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.Internal.NpgsqlConnector.<Open>g__OpenCore|209_0(NpgsqlConnector conn, String username, SslMode sslMode, GssEncryptionMode gssEncMode, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.Internal.NpgsqlConnector.Open(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.PoolingDataSource.OpenNewConnector(NpgsqlConnection conn, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.PoolingDataSource.<Get>g__RentAsync|33_0(NpgsqlConnection conn, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.NpgsqlConnection.<Open>g__OpenAsync|42_0(Boolean async, CancellationToken cancellationToken)
         at Npgsql.NpgsqlDataSourceCommand.ExecuteReader(Boolean async, CommandBehavior behavior, CancellationToken cancellationToken)
         at Npgsql.NpgsqlCommand.ExecuteScalar(Boolean async, CancellationToken cancellationToken)
…[truncated]
2026-10-22 21:23:02Z memex-portal-deployment-5d997855c9-scc52 fail: Microsoft.Extensions.Diagnostics.HealthChecks.DefaultHealthCheckService[103]
      Health check db_version with status Unhealthy completed after 77.3654ms with message 'db_version check threw'
      Npgsql.NpgsqlException (0x80004005): Failed to connect to 10.42.18.5:5432
       ---> System.Net.Sockets.SocketException (111): Connection refused
         at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
         at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
         at Npgsql.Internal.NpgsqlConnector.ConnectAsync(NpgsqlTimeout timeout, CancellationToken cancellationToken)
         at Npgsql.Internal.NpgsqlConnector.ConnectAsync(NpgsqlTimeout timeout, CancellationToken cancellationToken)
         at Npgsql.Internal.NpgsqlConnector.RawOpen(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.Internal.NpgsqlConnector.<Open>g__OpenCore|209_0(NpgsqlConnector conn, String username, SslMode sslMode, GssEncryptionMode gssEncMode, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.Internal.NpgsqlConnector.Open(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.PoolingDataSource.OpenNewConnector(NpgsqlConnection conn, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.PoolingDataSource.<Get>g__RentAsync|33_0(NpgsqlConnection conn, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
         at Npgsql.NpgsqlConnection.<Open>g__OpenAsync|42_0(Boolean async, CancellationToken cancellationToken)
         at Npgsql.NpgsqlDataSourceCommand.ExecuteReader(Boolean async, CommandBehavior behavior, CancellationToken cancellationToken)
         at Npgsql.NpgsqlCommand.ExecuteScalar(Boolean async, CancellationToken cancellationToken)
…[truncated]

_{Re-addressed by the current identity function: this incident inherited 34994e779e22d4d3 (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.}

Still occurring: 8 more since the last update (3367 total).

Last seen: 2026-10-22 21:24:05Z
Pods: memex-portal-deployment-65d6cb7648-97nnd, memex-portal-deployment-54486c96c4-msxhn, memex-portal-deployment-85cf49759f-pnbwl

Recent log lines

2026-10-22 21:23:55Z memex-portal-deployment-85cf49759f-pnbwl fail: Microsoft.Extensions.Diagnostics.HealthChecks.DefaultHealthCheckService[103]
      Health check nodetype_bake with status Unhealthy completed after 0.0047ms with message 'NodeType bake FAULTED before it could verify this image — refusing readiness. A sweep that errored has verified nothing, and certifying an unmeasured bake is precisely what this gate exists to prevent. A restart re-runs the sweep; to serve regardless, set PreWarm:AllowUnprovenBake=true. bake NOT PROVEN — the sweep errored before it could verify this image: the warm-up stream faulted before the sweep could finish'
2026-10-22 21:23:56Z memex-portal-deployment-85cf49759f-pnbwl fail: Microsoft.Extensions.Diagnostics.HealthChecks.DefaultHealthCheckService[103]
      Health check nodetype_bake with status Unhealthy completed after 0.0063ms with message 'NodeType bake FAULTED before it could verify this image — refusing readiness. A sweep that errored has verified nothing, and certifying an unmeasured bake is precisely what this gate exists to prevent. A restart re-runs the sweep; to serve regardless, set PreWarm:AllowUnprovenBake=true. bake NOT PROVEN — the sweep errored before it could verify this image: the warm-up stream faulted before the sweep could finish'
2026-10-22 21:24:05Z memex-portal-deployment-85cf49759f-pnbwl fail: Microsoft.Extensions.Diagnostics.HealthChecks.DefaultHealthCheckService[103]
      Health check nodetype_bake with status Unhealthy completed after 0.0078ms with message 'NodeType bake FAULTED before it could verify this image — refusing readiness. A sweep that errored has verified nothing, and certifying an unmeasured bake is precisely what this gate exists to prevent. A restart re-runs the sweep; to serve regardless, set PreWarm:AllowUnprovenBake=true. bake NOT PROVEN — the sweep errored before it could verify this image: the warm-up stream faulted before the sweep could finish'

Still occurring: 8 more since the last update (3367 total).

Last seen: 2026-10-22 21:24:05Z
Pods: memex-portal-deployment-65d6cb7648-97nnd, memex-portal-deployment-54486c96c4-msxhn, memex-portal-deployment-85cf49759f-pnbwl

Recent log lines

2026-10-22 21:23:55Z memex-portal-deployment-85cf49759f-pnbwl fail: Microsoft.Extensions.Diagnostics.HealthChecks.DefaultHealthCheckService[103]
      Health check nodetype_bake with status Unhealthy completed after 0.0047ms with message 'NodeType bake FAULTED before it could verify this image — refusing readiness. A sweep that errored has verified nothing, and certifying an unmeasured bake is precisely what this gate exists to prevent. A restart re-runs the sweep; to serve regardless, set PreWarm:AllowUnprovenBake=true. bake NOT PROVEN — the sweep errored before it could verify this image: the warm-up stream faulted before the sweep could finish'
2026-10-22 21:23:56Z memex-portal-deployment-85cf49759f-pnbwl fail: Microsoft.Extensions.Diagnostics.HealthChecks.DefaultHealthCheckService[103]
      Health check nodetype_bake with status Unhealthy completed after 0.0063ms with message 'NodeType bake FAULTED before it could verify this image — refusing readiness. A sweep that errored has verified nothing, and certifying an unmeasured bake is precisely what this gate exists to prevent. A restart re-runs the sweep; to serve regardless, set PreWarm:AllowUnprovenBake=true. bake NOT PROVEN — the sweep errored before it could verify this image: the warm-up stream faulted before the sweep could finish'
2026-10-22 21:24:05Z memex-portal-deployment-85cf49759f-pnbwl fail: Microsoft.Extensions.Diagnostics.HealthChecks.DefaultHealthCheckService[103]
      Health check nodetype_bake with status Unhealthy completed after 0.0078ms with message 'NodeType bake FAULTED before it could verify this image — refusing readiness. A sweep that errored has verified nothing, and certifying an unmeasured bake is precisely what this gate exists to prevent. A restart re-runs the sweep; to serve regardless, set PreWarm:AllowUnprovenBake=true. bake NOT PROVEN — the sweep errored before it could verify this image: the warm-up stream faulted before the sweep could finish'

Still occurring: 546 more since the last update (3913 total).

Last seen: 2026-10-22 22:37:05Z
Pods: memex-portal-deployment-65d6cb7648-97nnd, memex-portal-deployment-54486c96c4-msxhn, memex-portal-deployment-85cf49759f-pnbwl

Recent log lines

2026-10-22 22:36:47Z memex-portal-deployment-85cf49759f-pnbwl fail: Microsoft.Extensions.Diagnostics.HealthChecks.DefaultHealthCheckService[103]
      Health check nodetype_bake with status Unhealthy completed after 0.0019ms with message 'NodeType bake FAULTED before it could verify this image — refusing readiness. A sweep that errored has verified nothing, and certifying an unmeasured bake is precisely what this gate exists to prevent. A restart re-runs the sweep; to serve regardless, set PreWarm:AllowUnprovenBake=true. bake NOT PROVEN — the sweep errored before it could verify this image: the warm-up stream faulted before the sweep could finish'
2026-10-22 22:36:55Z memex-portal-deployment-85cf49759f-pnbwl fail: Microsoft.Extensions.Diagnostics.HealthChecks.DefaultHealthCheckService[103]
      Health check nodetype_bake with status Unhealthy completed after 0.0022ms with message 'NodeType bake FAULTED before it could verify this image — refusing readiness. A sweep that errored has verified nothing, and certifying an unmeasured bake is precisely what this gate exists to prevent. A restart re-runs the sweep; to serve regardless, set PreWarm:AllowUnprovenBake=true. bake NOT PROVEN — the sweep errored before it could verify this image: the warm-up stream faulted before the sweep could finish'
Task exception was never retrieved
future: <Task finished name='Task-3' coro=<MeshConnection._read_loop() done, defined at /usr/local/lib/python3.12/site-packages/meshweaver/connection.py:87> exception=<AioRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "failed to connect to all addresses; last error: UNKNOWN: ipv4:127.0.0.1:8082: Failed to connect to remote host: Connection refused"
	debug_error_string = "UNAVAILABLE:failed to connect to all addresses; last error: UNKNOWN: ipv4:127.0.0.1:8082: Failed to connect to remote host: Connection refused"
>>
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/meshweaver/connection.py", line 88, in _read_loop
    async for server_frame in self._call:
  File "/usr/local/lib/python3.12/site-packages/grpc/aio/_call.py", line 366, in _fetch_stream_responses
    await self._raise_for_status()
  File "/usr/local/lib/python3.12/site-packages/grpc/aio/_call.py", line 274, in _raise_for_status
    raise _create_rpc_error(
grpc.aio._call.AioRpcError: <AioRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "failed to connect to all addresses; last error: UNKNOWN: ipv4:127.0.0.1:8082: Failed to connect to remote host: Connection refused"
	debug_error_string = "UNAVAILABLE:failed to connect to all addresses; last error: UNKNOWN: ipv4:127.0.0.1:8082: Failed to co…[truncated]
2026-10-22 22:37:05Z memex-portal-deployment-85cf49759f-pnbwl fail: Microsoft.Extensions.Diagnostics.HealthChecks.DefaultHealthCheckService[103]
      Health check nodetype_bake with status Unhealthy completed after 0.0026ms with message 'NodeType bake FAULTED before it could verify this image — refusing readiness. A sweep that errored has verified nothing, and certifying an unmeasured bake is precisely what this gate exists to prevent. A restart re-runs the sweep; to serve regardless, set PreWarm:AllowUnprovenBake=true. bake NOT PROVEN — the sweep errored before it could verify this image: the warm-up stream faulted before the sweep could finish'

Triage: a RECOVERY DEFECT. The pod did not recover after the store came back. Kept open and retitled. Refs #7068.

The 2026-10-22 burst on this issue comes from the managed Postgres restart or failover described on #7068 (sessions cut at 21:22:51Z, new connections refused until about 21:23:34Z). What stayed wrong afterwards is this gate: its sweep faulted once on that transient refusal and was never run again, so the pod refused readiness until it was replaced.

Evidence (governed Logs actions on the control instance, memex namespace):

reading	action path
Pod `85cf49759f-pnbwl` started its warm-up at 21:21:49Z. The sweep FAULTED at 21:22:55Z on `Admin/Build`, inside the outage (this is #7074's line). It logged no further `DynamicTypePreWarmer` line in the rest of its life: 32 lines in total, the last at 21:24:24Z. No re-run and no "warm-up complete".	`Ops/Actions/i7066-prewarm-pnbwl-only-20261023`
`nodetype_bake` on that pod answered Unhealthy every ~10 s from 21:22:55Z until 22:37:05Z. At 22:37:10Z it logged `Application is shutting down...`	`Ops/Actions/i7066-pnbwl-last-hour-window-20261023`, `Ops/Actions/i7066-prewarm-pnbwl-20261023`
The pod was replaced, not recovered. The roll's post-run verification at 22:51:22Z lists `85cf49759f-77plc`, `85cf49759f-hcw4t`, … ready on 3.1.11417, and `pnbwl` is not among them. Same-image sibling `85cf49759f-77plc` had passed its sweep at 21:19:56Z, before the outage, and stayed ready throughout.	`Ops/Actions/selfupdate-roll-memex-3-1-11417-420ff7b3`

So the store was back about 21:23:34Z (#7068), but this pod stayed out of the Service for about 74 minutes, and the memex roll to 3.1.11417 could not converge until the pod went away. The gate's own message says "A restart re-runs the sweep". Under the readiness-only policy nothing restarts a refused pod, so a fault caused by a short store outage is held until something external replaces the pod.

The defect, restated: a Faulted warm-up sweep whose fault was a store-availability failure (Unavailable / connection refused) is treated as final. Neither the pre-warmer nor the gate re-runs the sweep once the store answers again.

Not established:

What terminated pnbwl at 22:37:10Z. It may have been the deployment controller, a probe, or something else. No InstanceAction for memex falls between 21:20Z and 23:30Z except the roll itself.
Whether the earlier occurrences on this fingerprint (first seen 2026-10-12, pods 65d6cb7648-97nnd and 54486c96c4-msxhn) were also store-availability faults. Their causes were not read here.
Whether the served-witness was consulted. This was a first boot of 3.1.11417 on a mid-roll pod, so the strict reading applied either way.

#7074 is the same event seen from the pre-warmer's side, and is folded in here.

refactor(prewarm): re-run the sweep on StoreReachability's evidence, …

fix(prewarm): answer review — aggregate-aware transience, live-only w…

test(prewarm): immutable accumulators in the #7066 tests (review)

fix(prewarm): re-run retracts a regression it measures usable; path-o…

fix(prewarm): re-run a sweep a transient store fault faulted, on the …

fix(faults): judge every branch from its own place; a text match yiel…

Navigation Menu

Repository navigation

A warm-up sweep faulted by a transient store refusal is never re-run — nodetype_bake held a memex pod unready ~74 min after Postgres recovered, until the pod was replaced #7066

Evidence

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

A warm-up sweep faulted by a transient store refusal is never re-run — nodetype_bake held a memex pod unready ~74 min after Postgres recovered, until the pod was replaced #7066

Description

Evidence

Activity

systemorph commented on Oct 22, 2026

systemorph commented on Oct 22, 2026

systemorph commented on Oct 22, 2026

systemorph commented on Oct 23, 2026

rbuergi commented on Oct 23, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions