Repository navigation
Catalog area render fails entirely on transient PostgreSQL connection timeout in PostgreSqlCrossSchemaQueryProvider.GetSchemasWithTableAsync #2876
Description
What is failing
Rendering of the Catalog layout area fails with an NpgsqlException when PostgreSqlCrossSchemaQueryProvider.GetSchemasWithTableAsync tries to open a PostgreSQL connection. The timeout occurs while Npgsql establishes a new connector (NpgsqlConnector.ConnectAsync / OpenNewConnector after a pool rent miss), i.e. the server could not be reached within the connect timeout — not a slow query. LayoutAreaHost surfaces this as a full area-render failure with no retry or graceful degradation.
Probable cause
High confidence on the mechanism: the connection pool had no idle connector and opening a fresh one timed out, so the database was briefly unreachable or too slow to connect from that pod (network blip, DB restart/unavailability, or pool exhaustion) around 05:26–05:27 UTC. Low confidence on which underlying condition — the log gives no database-side evidence, so the trigger (infra incident vs. pool misconfiguration) is unknown. Notably, a single transient connect timeout currently aborts the whole area render.
Impact
Low overall: 2 occurrences, one pod (memex-portal-deployment-9bf79b774-tk88b), within a 21-second window. Affected users would have seen the Catalog area fail to render during that window; if the condition recurs (DB outage), the same code path would fail every area render on every pod, since GetSchemasWithTableAsync runs at composition time.
Where to look
plugins-repo/src/MeshWeaver.Hosting.PostgreSql/PostgreSqlCrossSchemaQueryProvider.cs:269—GetSchemasWithTableAsync, which opens the connection; the only application frame in the trace.MeshWeaver.Layout.Composition.LayoutAreaHost— its render/error path logs the failure; assess whether a retry or degraded rendering is appropriate for transient connection failures.- Operationally: check PostgreSQL availability and pod-to-DB connectivity around 2026-08-31T05:26:51Z, and the Npgsql pool settings (Max Pool Size / Connect Timeout) in the
memex-clouddeployment.
Evidence
| Fingerprint | a7064759dd903d9d |
| Category | MeshWeaver.Layout.Composition.LayoutAreaHost |
| Severity | Error |
| Top frame | MeshWeaver.Hosting.PostgreSql.PostgreSqlCrossSchemaQueryProvider.GetSchemasWithTableAsync(String tableName, CancellationToken ct) |
| Namespace | memex-cloud |
| Pods | memex-portal-deployment-9bf79b774-tk88b |
| Occurrences | 2 |
| First seen | 2026-08-31 05:26:51Z |
| Last seen | 2026-08-31 05:27:12Z |
Recent log lines
2026-08-31 05:26:51Z memex-portal-deployment-9bf79b774-tk88b fail: MeshWeaver.Layout.Composition.LayoutAreaHost[0]
Rendering failed for area Catalog
Npgsql.NpgsqlException (0x80004005): The operation has timed out
---> System.TimeoutException: The operation has timed out.
at Npgsql.ThrowHelper.ThrowNpgsqlExceptionWithInnerTimeoutException(String message)
at Npgsql.Util.NpgsqlTimeout.Check()
at Npgsql.Util.NpgsqlTimeout.CheckAndGetTimeLeft()
at Npgsql.Internal.NpgsqlConnector.ConnectAsync(NpgsqlTimeout timeout, CancellationToken cancellationToken)
at Npgsql.Internal.NpgsqlConnector.RawOpen(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
at Npgsql.Internal.NpgsqlConnector.<Open>g__OpenCore|209_0(NpgsqlConnector conn, String username, SslMode sslMode, GssEncryptionMode gssEncMode, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
at Npgsql.Internal.NpgsqlConnector.<Open>g__OpenCore|209_0(NpgsqlConnector conn, String username, SslMode sslMode, GssEncryptionMode gssEncMode, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
at Npgsql.Internal.NpgsqlConnector.Open(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
at Npgsql.PoolingDataSource.OpenNewConnector(NpgsqlConnection conn, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
at Npgsql.PoolingDataSource.<Get>g__RentAsync|33_0(NpgsqlConnection conn, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
at Npgsql.NpgsqlConnection.<Open>g__OpenAsync|42_0(Boolean async, CancellationToken cancellationToken)
at Npgsql.NpgsqlDataSourceCommand.ExecuteReader(Boolean async, CommandBehavior behavior, CancellationToken cancellationToken)
at MeshWeaver.Hosting.PostgreSql.PostgreSqlCrossSchemaQueryProvider.GetSchemasWithTableAsync(String tableName, CancellationToken ct) in /home/runner/work/MeshWeaver/MeshWeaver/pl…[truncated]
2026-08-31 05:27:12Z memex-portal-deployment-9bf79b774-tk88b fail: MeshWeaver.Layout.Composition.LayoutAreaHost[0]
Rendering failed for area Catalog
Npgsql.NpgsqlException (0x80004005): The operation has timed out
---> System.TimeoutException: The operation has timed out.
at Npgsql.ThrowHelper.ThrowNpgsqlExceptionWithInnerTimeoutException(String message)
at Npgsql.Util.NpgsqlTimeout.Check()
at Npgsql.Util.NpgsqlTimeout.CheckAndGetTimeLeft()
at Npgsql.Internal.NpgsqlConnector.ConnectAsync(NpgsqlTimeout timeout, CancellationToken cancellationToken)
at Npgsql.Internal.NpgsqlConnector.RawOpen(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
at Npgsql.Internal.NpgsqlConnector.<Open>g__OpenCore|209_0(NpgsqlConnector conn, String username, SslMode sslMode, GssEncryptionMode gssEncMode, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
at Npgsql.Internal.NpgsqlConnector.Open(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
at Npgsql.PoolingDataSource.OpenNewConnector(NpgsqlConnection conn, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
at Npgsql.PoolingDataSource.<Get>g__RentAsync|33_0(NpgsqlConnection conn, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
at Npgsql.NpgsqlConnection.<Open>g__OpenAsync|42_0(Boolean async, CancellationToken cancellationToken)
at Npgsql.NpgsqlDataSourceCommand.ExecuteReader(Boolean async, CommandBehavior behavior, CancellationToken cancellationToken)
at MeshWeaver.Hosting.PostgreSql.PostgreSqlCrossSchemaQueryProvider.GetSchemasWithTableAsync(String tableName, CancellationToken ct) in /home/runner/work/MeshWeaver/MeshWeaver/plugins-repo/src/MeshWeaver.Hosting.PostgreSql/PostgreSqlCrossSchemaQueryProvider.cs:line 269
at MeshWeaver.Hosting.PostgreSql.PostgreSqlCrossSchemaQueryProvider.GetSchemasWithTableAsync(String tableName, CancellationToken…[truncated]
Opened automatically from Admin/_LogIncident/a7064759dd903d9d. Recurrences are folded into this issue rather than opening new ones.
Activity
rbuergi commentedon Sep 1, 2026
Before anyone adds a retry here: this is plausibly a SYMPTOM of #2640, and a retry would treat it as a cause
I came to this issue intending to add a bounded retry around the connection open, and stopped after measuring the surrounding system. Recording why, because "wrap it in a retry" is the obvious move and I think it is the wrong first one.
What the issue says the trigger might be
The body already lists the candidates and is honest that it cannot choose between them: "the connection pool had no idle connector and opening a fresh one timed out, so the database was briefly unreachable or too slow to connect from that pod (network blip, DB restart/unavailability, or pool exhaustion)". It explicitly marks this low confidence.
A measurement that makes pool exhaustion the leading candidate
Sampled memex-cloud's own
[CrossSchema] SLOWlines today (14:21Z,3.0.0-rc9.ci.7231):A steady stream of multi-second queries, each holding a pooled connector for its duration, is exactly the condition under which a fresh
OpenNewConnectorfinds no idle connector and races the connect timeout. That is the mechanism this issue observed, arriving from the direction the issue could not see from its own log.Why that changes the fix
GetSchemasWithTableAsync(PostgreSqlCrossSchemaQueryProvider.cs:252) is itself part of the fan-out machinery — it is the discovery step that enumerates which schemas carry a table, and it runs at composition time on the render path. So the same subsystem that is exhausting the pool is the one failing to get a connection from it.Adding a retry there would:
That is the shape of a band-aid: a bound raised to make a symptom pass, where the question is not "how do I get another connection" but "why are 1,030 queries holding connections for 2 s each to return nothing".
What I suggest instead
LayoutAreaHostcurrently treats every render exception identically, so a transient DB-connect fault and a genuine render defect are indistinguishable to whoever reads the log. Classifying it — and letting the area report "temporarily unavailable, retryable" rather than a hard failure — improves the operator's information without adding load. Note core A freshly uploaded content file 503s for minutes (10s timeout per request) before it starts serving — /api/content settle window #2901 landed today establishing the three-state pattern (granted / denied / undetermined-retryable) for exactly this kind of distinction.Not making the change; leaving the analysis so the next person does not spend the hour I did arriving at it.
docs(architecture): the elimination page points at what must NOT be e…
systemorph commentedon Sep 17, 2026
Reopened — this fault is happening again.
Still occurring: 9 more since the last update (11 total).
memex-portal-deployment-9bf79b774-tk88b,memex-portal-deployment-578b7bc84b-msvh5,memex-portal-deployment-58c88885f8-h5hfsRecent log lines
rbuergi commentedon Sep 18, 2026
Closing — the answer this issue converged on is in
origin/main, and the fault has been quiet 16 daysDormancy. Newest occurrence
Last seen: 2026-09-02 14:39:36Z(the bot's reopen comment above). Measured 2026-09-18T05:52Z that is 16 days quiet, with live log ingest (incidents updated 2026-09-18T05:42Z) and no portal roll in between.The suggestion recorded on this issue on 2026-09-01 — "prefer making the failure legible over making it disappear" — was implemented, and the retry it warned against was not added. Checked against
origin/mainat229930bf2f29845323e0f350e73829b1bdb4928e:LayoutAreaHostno longer treats every render exception identically. A store-unreachable fault now logs as what it is —— instead of
Rendering failed for area Catalog, which is the wording this ticket's reader-cost complaint was about, and line 956 gives the viewer a "temporarily unavailable / retryable" frame rather than the generic panel carrying the driver's text and the database host it could not reach.Pinned by a test that names this issue and states the boundary explicitly:
The load-side root cause this ticket was sequenced behind (#2640, the cross-schema fan-out that was holding the connectors) has its own census and elimination programme in
Doc/Architecture/CrossSchemaFanOutElimination, which is where any remaining connect-timeout work belongs — not here.GetSchemasWithTableAsyncitself lives in MeshWeaver.Plugins, so it is not core's to close either way.Closing as redundant: the question this ticket was reduced to — what does the area show when the retry is spent — is answered and tested. If the fault recurs, the recurrence bot reopens this issue with fresh evidence — the designed behaviour, and why closing a dormant incident is safe.