Repository navigation
Portal boot fails when the PostgreSQL host does not resolve (memex) — duplicate of #3780 #3783
Closed as duplicate
Description
Duplicate of Systemorph/MeshWeaver#3780 — the same boot failure at a second log site. #3780 was filed from the silo-lifecycle fingerprint (Admin/_LogIncident/0b1877ae6c51353f) and is the inner failure: the Orleans MembershipTableManager aborting silo start at stage RuntimeGrainServices. This entry is the outer aggregate of the same boot — the .NET generic host giving up, with the portal's startup wrapper (RunOrExitOnCancelledStartup) observing the failed start and exiting the container. Close as a duplicate of #3780; this ticket exists so the second log site is traceable and links back.
What is failing
During the record-driven reconcile roll of the memex control instance (helm --wait upgrade launched about a minute before the failure), the roll's fresh surge pod failed to boot: the first PostgreSQL connection attempt could not resolve the database hostname — DNS resolution failed (Name or service not known out of Dns.GetHostEntryOrAddressesCore) before any TCP dial was attempted. The pod then cycled through repeated failed boots (the recurrence is carried on the sibling fingerprint above).
Probable cause — medium confidence
DNS resolution of the PostgreSQL host fails inside the boot window. This fingerprint adds no evidence beyond #3780, which names the two candidates and the one-command discriminator (getent hosts <db-host> from the pod): either a transient resolver/CoreDNS blip, or a wrong/default-derived connection-string value on this pod. The second is live here because the pod was created by a re-apply of the instance record moments earlier, and the connection strings reach the pod only through memex-portal-secrets (rendered from the Key Vault values half no repository holds) and memex-kv-secrets — no committed file states the host. Host resolves now → resolver blip; does not → bad rendered value.
Impact
A delayed roll, not an outage: the failing pod is one of two replicas, and the rollout is surge-first (maxUnavailable: 0), so the previously ready pods kept serving while the new pod cycled — the reconcile was still inside its helm --wait window when this incident fired. The exposure is that memex is the control instance (infrastructure control, accounting, home of the Deployments records): a persistent version of this failure stalls every roll of it, in the same "safe but stuck" shape as a gate-held rollout.
Where to look
- Systemorph/MeshWeaver#3780 — the canonical ticket for this boot: same pod, same second, full diagnosis path and discriminator.
Memex.Portal.Shared.CancelledStartup.RunOrExitOnCancelledStartup(Systemorph/Memex) — the startup wrapper that observes and rethrows the failed host start; the top application frame of this fingerprint.- Deployments/memex — the instance record;
ConnectionStrings__memexarrives viamemex-kv-secrets, and bothConnectionStrings__memexandConnectionStrings__orleansviavaultValuesKeysonmemex-portal-secrets(rendered fromhelm-values-memex, held in no repository). - Deployments/memex-reconcile-20260909-final — the reconcile that created the failing pod; its
helm upgrade --atomic --waitwas still waiting when this incident fired.
Evidence
| Fingerprint | 1c9c655198118379 |
| Category | Microsoft.Extensions.Hosting.Internal.Host |
| Severity | Error |
| Top frame | Memex.Portal.Shared.CancelledStartup.RunOrExitOnCancelledStartup(WebApplication app, ILogger logger) |
| Namespace | memex |
| Pods | memex-portal-deployment-85b6cfc9c9-p5d7p |
| Occurrences | 5 |
| First seen | 2026-09-09 02:36:24Z |
| Last seen | 2026-09-09 02:47:36Z |
Recent log lines
2026-09-09 02:36:24Z memex-portal-deployment-85b6cfc9c9-p5d7p fail: Microsoft.Extensions.Hosting.Internal.Host[11]
Hosting failed to start
Npgsql.NpgsqlException (0x80004005): Name or service not known
---> System.Net.Sockets.SocketException (00000005, 0xFFFDFFFF): Name or service not known
at System.Net.Dns.GetHostEntryOrAddressesCore(String hostName, Boolean justAddresses, AddressFamily addressFamily, Nullable`1 activityOrDefault)
at System.Net.Dns.<>c.<GetHostEntryOrAddressesCoreAsync>b__34_0(Object s, NameResolutionActivity activity)
at System.Net.Dns.<>c__DisplayClass40_0`1.<RunAsync>b__0(Task <p0>, Object <p1>)
at System.Threading.Tasks.ContinuationResultTaskFromTask`1.InnerInvoke()
at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
--- End of stack trace from previous location ---
at Npgsql.Internal.NpgsqlConnector.ConnectAsync(NpgsqlTimeout timeout, CancellationToken cancellationToken)
at Npgsql.Internal.NpgsqlConnector.ConnectAsync(NpgsqlTimeout timeout, CancellationToken cancellationToken)
at Npgsql.Internal.NpgsqlConnector.RawOpen(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
at Npgsql.Internal.NpgsqlConnector.<Open>g__OpenCore|209_0(NpgsqlConnector conn, String username, SslMode sslMode, GssEncryptionMode gssEncMode, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
at Npgsql.Internal.NpgsqlConnector.Open(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
at Npgsql.P…[truncated]
2026-09-09 02:39:53Z memex-portal-deployment-85b6cfc9c9-p5d7p fail: Microsoft.Extensions.Hosting.Internal.Host[11]
Hosting failed to start
Npgsql.NpgsqlException (0x80004005): Name or service not known
---> System.Net.Sockets.SocketException (00000005, 0xFFFDFFFF): Name or service not known
at System.Net.Dns.GetHostEntryOrAddressesCore(String hostName, Boolean justAddresses, AddressFamily addressFamily, Nullable`1 activityOrDefault)
at System.Net.Dns.<>c.<GetHostEntryOrAddressesCoreAsync>b__34_0(Object s, NameResolutionActivity activity)
at System.Net.Dns.<>c__DisplayClass40_0`1.<RunAsync>b__0(Task <p0>, Object <p1>)
at System.Threading.Tasks.ContinuationResultTaskFromTask`1.InnerInvoke()
at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
--- End of stack trace from previous location ---
at Npgsql.Internal.NpgsqlConnector.ConnectAsync(NpgsqlTimeout timeout, CancellationToken cancellationToken)
at Npgsql.Internal.NpgsqlConnector.ConnectAsync(NpgsqlTimeout timeout, CancellationToken cancellationToken)
at Npgsql.Internal.NpgsqlConnector.RawOpen(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
at Npgsql.Internal.NpgsqlConnector.<Open>g__OpenCore|209_0(NpgsqlConnector conn, String username, SslMode sslMode, GssEncryptionMode gssEncMode, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
at Npgsql.Internal.NpgsqlConnector.Open(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
at Npgsql.P…[truncated]
2026-09-09 02:41:31Z memex-portal-deployment-85b6cfc9c9-p5d7p fail: Microsoft.Extensions.Hosting.Internal.Host[11]
Hosting failed to start
Npgsql.NpgsqlException (0x80004005): Name or service not known
---> System.Net.Sockets.SocketException (00000005, 0xFFFDFFFF): Name or service not known
at System.Net.Dns.GetHostEntryOrAddressesCore(String hostName, Boolean justAddresses, AddressFamily addressFamily, Nullable`1 activityOrDefault)
at System.Net.Dns.<>c.<GetHostEntryOrAddressesCoreAsync>b__34_0(Object s, NameResolutionActivity activity)
at System.Net.Dns.<>c__DisplayClass40_0`1.<RunAsync>b__0(Task <p0>, Object <p1>)
at System.Threading.Tasks.ContinuationResultTaskFromTask`1.InnerInvoke()
at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
--- End of stack trace from previous location ---
at Npgsql.Internal.NpgsqlConnector.ConnectAsync(NpgsqlTimeout timeout, CancellationToken cancellationToken)
at Npgsql.Internal.NpgsqlConnector.ConnectAsync(NpgsqlTimeout timeout, CancellationToken cancellationToken)
at Npgsql.Internal.NpgsqlConnector.RawOpen(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
at Npgsql.Internal.NpgsqlConnector.<Open>g__OpenCore|209_0(NpgsqlConnector conn, String username, SslMode sslMode, GssEncryptionMode gssEncMode, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
at Npgsql.Internal.NpgsqlConnector.Open(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
at Npgsql.P…[truncated]
2026-09-09 02:43:51Z memex-portal-deployment-85b6cfc9c9-p5d7p fail: Microsoft.Extensions.Hosting.Internal.Host[11]
Hosting failed to start
Npgsql.NpgsqlException (0x80004005): Name or service not known
---> System.Net.Sockets.SocketException (00000005, 0xFFFDFFFF): Name or service not known
at System.Net.Dns.GetHostEntryOrAddressesCore(String hostName, Boolean justAddresses, AddressFamily addressFamily, Nullable`1 activityOrDefault)
at System.Net.Dns.<>c.<GetHostEntryOrAddressesCoreAsync>b__34_0(Object s, NameResolutionActivity activity)
at System.Net.Dns.<>c__DisplayClass40_0`1.<RunAsync>b__0(Task <p0>, Object <p1>)
at System.Threading.Tasks.ContinuationResultTaskFromTask`1.InnerInvoke()
at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
--- End of stack trace from previous location ---
at Npgsql.Internal.NpgsqlConnector.ConnectAsync(NpgsqlTimeout timeout, CancellationToken cancellationToken)
at Npgsql.Internal.NpgsqlConnector.ConnectAsync(NpgsqlTimeout timeout, CancellationToken cancellationToken)
at Npgsql.Internal.NpgsqlConnector.RawOpen(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
at Npgsql.Internal.NpgsqlConnector.<Open>g__OpenCore|209_0(NpgsqlConnector conn, String username, SslMode sslMode, GssEncryptionMode gssEncMode, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
at Npgsql.Internal.NpgsqlConnector.Open(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
at Npgsql.P…[truncated]
2026-09-09 02:47:36Z memex-portal-deployment-85b6cfc9c9-p5d7p fail: Microsoft.Extensions.Hosting.Internal.Host[11]
Hosting failed to start
Npgsql.NpgsqlException (0x80004005): Name or service not known
---> System.Net.Sockets.SocketException (00000005, 0xFFFDFFFF): Name or service not known
at System.Net.Dns.GetHostEntryOrAddressesCore(String hostName, Boolean justAddresses, AddressFamily addressFamily, Nullable`1 activityOrDefault)
at System.Net.Dns.<>c.<GetHostEntryOrAddressesCoreAsync>b__34_0(Object s, NameResolutionActivity activity)
at System.Net.Dns.<>c__DisplayClass40_0`1.<RunAsync>b__0(Task <p0>, Object <p1>)
at System.Threading.Tasks.ContinuationResultTaskFromTask`1.InnerInvoke()
at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
--- End of stack trace from previous location ---
at Npgsql.Internal.NpgsqlConnector.ConnectAsync(NpgsqlTimeout timeout, CancellationToken cancellationToken)
at Npgsql.Internal.NpgsqlConnector.ConnectAsync(NpgsqlTimeout timeout, CancellationToken cancellationToken)
at Npgsql.Internal.NpgsqlConnector.RawOpen(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
at Npgsql.Internal.NpgsqlConnector.<Open>g__OpenCore|209_0(NpgsqlConnector conn, String username, SslMode sslMode, GssEncryptionMode gssEncMode, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
at Npgsql.Internal.NpgsqlConnector.Open(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
at Npgsql.P…[truncated]
Opened automatically from Admin/_LogIncident/1c9c655198118379. Recurrences are folded into this issue rather than opening new ones.
Activity
rbuergi commentedon Sep 9, 2026
Duplicate of #3780, which now carries the consolidated record of this incident (the 02:35Z record-driven Reconcile, helm rev 44, portal pod unable to resolve its Postgres host) and a table saying what each report added — nothing from here is lost.
Closing as a duplicate only. The defect is NOT fixed: memex serving 200 today is explained by the rollback, not by anyone establishing why a record-driven Reconcile rendered an unresolvable Postgres hostname. #3780 stays open for that.
Note #3782 was deliberately NOT folded in — the
--atomic --wait 15mrollback of an upgrade that had already landed is a separate defect in the deploy path that would bite a healthy rollout too.systemorph commentedon Sep 14, 2026
Reopened — this fault is happening again.
Still occurring: 1 more since the last update (6 total).
memex-portal-deployment-85b6cfc9c9-p5d7p,memex-portal-deployment-77476b68-l4gw8Recent log lines
rbuergi commentedon Sep 15, 2026
Duplicate of #3780 — the cause, the two merged fixes (#4356, Systemorph/MeshWeaver.Plugins#1892) and the measurements are in #3780 (comment).
systemorph commentedon Sep 17, 2026
Reopened — this fault is happening again.
Still occurring: 26 more since the last update (32 total).
memex-portal-deployment-85b6cfc9c9-p5d7p,memex-portal-deployment-77476b68-l4gw8,memex-portal-deployment-78d795bf59-87c7xRecent log lines
rbuergi commentedon Sep 17, 2026
Re-closed — the reopen above was on stale evidence (#4577).
The recurrence bot reopened this at
2026-09-17T07:52:11Zunder the headline "this fault is happening again", but the occurrence its own comment cites is OLDER than the close it reverted:Last seen:2026-09-14 21:59:25Z2026-09-15T02:24:36ZSo the counter moved, not the clock — the fault's newest occurrence predates the close, which is the one thing "still happening" has to mean. Root cause and fix:
LogIncidentFiler.Reopenreopened on the sole condition that the issue was closed, while the comment-debt clock pays a final count for incidents whose fault has stopped (the fix, Systemorph/MeshWeaver.Plugins#2014, is written but NOT yet merged — it makes a reopen requireincident.LastSeen > issue.ClosedAt).This restores the close exactly as it stood; nothing is asserted about the defect itself. If the fault happens again, the next occurrence postdates this close and reopens the issue on evidence that means what it says.
One of 53 issues whose reopen was measured stale (129 bot reopens on 2026-09-17, 125 of them inside the 07:51-07:53Z sweep; 76 postdate their close and were left open). Measured and re-closed 2026-09-17 by Claude Opus 5.