Repository navigation
Node-resolution reads in MessageHubGrain go silent for 30s when PostgreSqlPartitionedMeshQuery stalls, faulting Ops/Logs activations #5800
Description
What is failing
MessageHubGrain activations for freshly-created Ops/Logs/memex-cloud-* log-entry nodes fail with a System.TimeoutException because the node-resolution read never completes within 30 s — the activation source stays silent (not faulted, not empty). This is the log-landing pipeline: 124 log-entry activations faulted in a 7-second burst on one portal pod.
Probable cause
High confidence this is the same root cause as the QueryProviderStalledException sibling incident Admin/_LogIncident/23624cd171bed2bf (filed as #5393): MeshWeaver.Hosting.PostgreSql.PostgreSqlPartitionedMeshQuery stalls and never delivers an Initial to the query fan-in. Evidence:
- Both bursts hit the same grains (
Ops/Logs/memex-cloud-…), the same pod (memex-portal-deployment-555d9cd496-qntsh), and the same window (11:02:33–11:02:45 on 2026-09-27), interleaved line-by-line. - This incident's own diagnostic text states the silent timeout follows the fan-in's Initial-gate warning — which is exactly the sibling's
QueryProviderStalledExceptionnaming that provider as the one that did not deliver. - An absent node does not produce this signature; it faults fast with "No MeshNode resolvable", so the nodes themselves are not the problem.
So: the provider's read path stalls; some reads trip the 15 s fan-in Initial budget (explicit QueryProviderStalledException), the rest sit silent until the 30 s activation timeout. Whether it's genuine DB/partition-fan-out latency or a budget calibrated only against the in-memory monolith (per the referenced QueryFanInStallTerminal doc, as noted in #5393) is the open question.
Impact
The memex-cloud ops-log landing path was unavailable for the stall window: log entries could not be persisted and their hub activations faulted (retryable, no data corruption). This burst: 124 occurrences, ~7 s, one pod. But the provider-level pattern is recurring — the sibling incident counts 910 occurrences across 6 pods over 5 days — so this is one investigation, not a one-off.
Where to look
MeshWeaver.Hosting.PostgreSql.PostgreSqlPartitionedMeshQuery— theInitialemission path for path-scoped queries (path:"Ops/Logs/…").- The
MessageHubGrainactivation node-resolution read and the 30 s timeout it enforces — consumer side, do not widen the timeout; fix the provider. - Likely duplicate of PostgreSqlPartitionedMeshQuery stalls past the 15s query fan-in Initial budget, faulting MessageHubGrain activations (Admin/Build) #5393 — consider folding this into that provider-level ticket rather than fixing it separately.
Evidence
| Fingerprint | e8bc602f83d6724a |
| Category | MeshWeaver.Hosting.Orleans.MessageHubGrain |
| Severity | Error |
| Exception | System.TimeoutException |
| Namespace | memex-cloud |
| Pods | memex-portal-deployment-555d9cd496-qntsh |
| Occurrences | 124 |
| First seen | 2026-09-27 11:02:33Z |
| Last seen | 2026-09-27 11:02:40Z |
| Routing | not determined — no configured route matches the category MeshWeaver.Hosting.Orleans.MessageHubGrain. This repository is the configured fallback, not a finding about who owns the fault; the category names the LOGGER, which may not be the subject. |
Recent log lines
2026-09-27 11:02:33Z memex-portal-deployment-555d9cd496-qntsh fail: MeshWeaver.Hosting.Orleans.MessageHubGrain[0]
[ACTIVATE] Grain Ops/Logs/memex-cloud-1790189483100369660-97dc9b-r4nkg: activation faulted for Ops/Logs/memex-cloud-1790189483100369660-97dc9b-r4nkg
System.TimeoutException: Node resolution for 'Ops/Logs/memex-cloud-1790189483100369660-97dc9b-r4nkg' neither answered nor terminated within 30s. The activation source is SILENT — that is a stalled READ, and it says nothing about the node. An absent node does NOT reach here: it completes the source at once and faults with "No MeshNode resolvable for address …" instead. Look for the query fan-in's Initial-gate warning that lands BEFORE this line — it names the provider that did not deliver an Initial, which is the fault to fix.
2026-09-27 11:02:40Z memex-portal-deployment-555d9cd496-qntsh fail: MeshWeaver.Hosting.Orleans.MessageHubGrain[0]
[ACTIVATE] Grain Ops/Logs/memex-cloud-1790253074017208268-f8f895-vn49b: activation faulted for Ops/Logs/memex-cloud-1790253074017208268-f8f895-vn49b
System.TimeoutException: Node resolution for 'Ops/Logs/memex-cloud-1790253074017208268-f8f895-vn49b' neither answered nor terminated within 30s. The activation source is SILENT — that is a stalled READ, and it says nothing about the node. An absent node does NOT reach here: it completes the source at once and faults with "No MeshNode resolvable for address …" instead. Look for the query fan-in's Initial-gate warning that lands BEFORE this line — it names the provider that did not deliver an Initial, which is the fault to fix.
2026-09-27 11:02:40Z memex-portal-deployment-555d9cd496-qntsh fail: MeshWeaver.Hosting.Orleans.MessageHubGrain[0]
[ACTIVATE] Grain Ops/Logs/memex-cloud-1790253054012432896-f8f895-vn49b: activation faulted for Ops/Logs/memex-cloud-1790253054012432896-f8f895-vn49b
System.TimeoutException: Node resolution for 'Ops/Logs/memex-cloud-1790253054012432896-f8f895-vn49b' neither answered nor terminated within 30s. The activation source is SILENT — that is a stalled READ, and it says nothing about the node. An absent node does NOT reach here: it completes the source at once and faults with "No MeshNode resolvable for address …" instead. Look for the query fan-in's Initial-gate warning that lands BEFORE this line — it names the provider that did not deliver an Initial, which is the fault to fix.
2026-09-27 11:02:40Z memex-portal-deployment-555d9cd496-qntsh fail: MeshWeaver.Hosting.Orleans.MessageHubGrain[0]
[ACTIVATE] Grain Ops/Logs/memex-cloud-1790253037355079320-f8f895-vn49b: activation faulted for Ops/Logs/memex-cloud-1790253037355079320-f8f895-vn49b
System.TimeoutException: Node resolution for 'Ops/Logs/memex-cloud-1790253037355079320-f8f895-vn49b' neither answered nor terminated within 30s. The activation source is SILENT — that is a stalled READ, and it says nothing about the node. An absent node does NOT reach here: it completes the source at once and faults with "No MeshNode resolvable for address …" instead. Look for the query fan-in's Initial-gate warning that lands BEFORE this line — it names the provider that did not deliver an Initial, which is the fault to fix.
2026-09-27 11:02:40Z memex-portal-deployment-555d9cd496-qntsh fail: MeshWeaver.Hosting.Orleans.MessageHubGrain[0]
[ACTIVATE] Grain Ops/Logs/memex-cloud-1790253138776814593-f8f895-vn49b: activation faulted for Ops/Logs/memex-cloud-1790253138776814593-f8f895-vn49b
System.TimeoutException: Node resolution for 'Ops/Logs/memex-cloud-1790253138776814593-f8f895-vn49b' neither answered nor terminated within 30s. The activation source is SILENT — that is a stalled READ, and it says nothing about the node. An absent node does NOT reach here: it completes the source at once and faults with "No MeshNode resolvable for address …" instead. Look for the query fan-in's Initial-gate warning that lands BEFORE this line — it names the provider that did not deliver an Initial, which is the fault to fix.
2026-09-27 11:02:40Z memex-portal-deployment-555d9cd496-qntsh fail: MeshWeaver.Hosting.Orleans.MessageHubGrain[0]
[ACTIVATE] Grain Ops/Logs/memex-cloud-1790253235199752259-f8f895-ghgjt: activation faulted for Ops/Logs/memex-cloud-1790253235199752259-f8f895-ghgjt
System.TimeoutException: Node resolution for 'Ops/Logs/memex-cloud-1790253235199752259-f8f895-ghgjt' neither answered nor terminated within 30s. The activation source is SILENT — that is a stalled READ, and it says nothing about the node. An absent node does NOT reach here: it completes the source at once and faults with "No MeshNode resolvable for address …" instead. Look for the query fan-in's Initial-gate warning that lands BEFORE this line — it names the provider that did not deliver an Initial, which is the fault to fix.
Opened automatically from Admin/_LogIncident/e8bc602f83d6724a. Recurrences are folded into this issue rather than opening new ones.
It also stands for the whole log site 851f5c97b963af1c: other fingerprints of this site fold in here as comments rather than opening tickets of their own.
Activity
rbuergi commentedon Sep 27, 2026
Provider verification on 2026-09-27: the fresh activation failure is confirmed; the exact stalled edge is still under investigation.
Ops/Status/memex-cloudsampled at 12:47:34Z reports 5/5 ready on ci.9443, core311108bf16a70eee39a6618b36456cf82c58a85f. The incident pod555d9cd496-qntshstarted at 06:18:14Z and is still on that image. The incident's own samples show 124 activation faults at 11:02:33–40Z.A governed, narrower log read,
Ops/Actions/verify-5800-cloud-20260927-gate, selected this pod and the exact diagnostic phrasethe query is stalled on its all-providers Initialover the preceding 180 minutes, queried at 12:48:35Z. It completed with 9 read, 9 landed, 0 failed, untruncated. Every landed entry was inspected:The exact warning nodes are indexed by
namespace:Ops/Logs scope:descendants content.actionPath:Ops/Actions/verify-5800-cloud-20260927-gate(9 results over the Ops partition). These warnings establish multiple affected consumers/providers. They do not establish which shared resource or notification edge stalled, nor do these nine warning lines identify the exact Ops/Logs activation requests from the issue samples.The broader action
…-providerreturned 200 lines and explicitly reports truncation; its landing phase was still running at the last read. It is not a complete census. One entry read returned Unavailable, not Not found, and no absence or deletion is inferred from that result.#1186 remains open with its successful memex-specific six-hour verification recorded separately. No timeout, concurrency cap, or retry count is being increased.
rbuergi commentedon Sep 27, 2026
Further governed production reads, 2026-09-27 14:05 UTC. The stalled edge remains unresolved.
Ops/Actions/verify-5800-cloud-20260927-slow-dbqueried the incident pod555d9cd496-qntshat 14:02:43Z over the preceding 210 minutes, selecting CrossSchema SLOW timings of at least 10,000 ms, transient DB faults, and IoPool drain/teardown diagnostics. It finished Done, 0 read / 0 landed / 0 failed, untruncated. This is a zero for those emitted diagnostics, not proof that no work stalled.The separate broader read
…-db-edgeselected CrossSchema and related database/pool diagnostics on the same pod/window. It returned 100 lines, truncated, and finished Failed with 99 landed and 1 landing failure at 14:04:25Z. This is not a complete log census. Two inspected timing entries positively establish an active timing logger on that pod:Ops/Logs/memex-cloud-1790517154600768653-9cd496-qntsh: 13:52:34.600Z, NodeType query across 245 schemas, total 2,339 ms, first row 2,170 ms, 383 rows.Ops/Logs/memex-cloud-1790517179805023870-9cd496-qntsh: 13:52:59.805Z, Thread query across 245 schemas, total 1,559 ms, first row 1,463 ms, 500/500 rows.The source explains the measurement's limit:
PostgreSqlCrossSchemaQueryProviderstarts this timer around reader enumeration, after prerequisite table/schema probes and SQL construction.PostgreSqlPartitionedMeshQueryalso queues its work through the filesystem I/O pool before invoking that path. The activation's authoritative path resolver usesIMeshQueryCore, whose provider subscription is itself queued through the Query pool before the fan-in's Initial timer is armed. A missing timing line therefore cannot rule out queue delay, prerequisite reads, work that never terminated, or a notification defect. These are possible edges, not diagnosed causes; neither cap nor timeout is being increased.The broad action's failed landing is another #5717 witness: the read of the Hosting/LogEntry NodeType definition failed or did not answer within 10 seconds, so one log-entry node was not created. That recording failure must not be presented as missing source logs or as an activation failure on the cloud incident pod.
rbuergi commentedon Sep 27, 2026
Further runtime evidence, 2026-09-27 15:19Z. The provider stall remains unresolved; no timeout/cap/retry change has been made.
The incident image is CI.9443 (core
311108bf16a70eee39a6618b36456cf82c58a85f). Its successful portal-image job 108560088528 in CD run 36297895487 checked out Pluginsc86892669f44a453e5161ae22a5266eda0851f1eat 05:41:24Z. GitHub compare confirms the per-schema coalescing fixc0ce343808e6d5ebed2e1cb246e34a742aab45e1is an ancestor (behind_by=0, merge_base=c0ce3438). The known one-full-read-per-notification defect therefore already had its fix in this image; repeating that change is not a demonstrated repair for this recurrence.A read-only script also inspected the CURRENT Cloud provider instance, rather than an arbitrary loaded assembly:
rbuergi/Script/verify-5800-pg-runtime-20260927, successful activityrbuergi/_Activity/84f14dd49a554ac089bf3898a719d389. On56dbd958f4-wqnslat 15:19:47Z it found exactly one active PostgreSqlPartitionedMeshQuery. Its actual assembly MVID isdbefb3f6-caa1-436d-9052-7a96da07fb58; the scoped Query delegate's IL contains one call to CoalesceWhileRunning.The same script enumerated EXISTING pools via IoPoolRegistry.Snapshot (not Get, which could create a phantom): FileSystem cap 256, Query cap 256, pg-read:Postgres cap 16 and pg:Postgres cap 1 all reported zero in-flight and waiting. This is an idle reading on the CURRENT replica, not the incident pod and not evidence about 11:02Z. It neither attributes nor rules out historical admission delay. No queries, data, grants, configuration, or pools were created/changed by the script.
The exact boundary between admission, prerequisite database reads and Initial delivery still needs a contemporaneous witness or deterministic reproduction. The issue remains open.
rbuergi commentedon Sep 27, 2026
A further private reproduction attempt now covers the production log ownership pattern, 2026-09-27. Reading an existing governed log entry confirmed MainNode is its own path, not the Ops/Logs parent. The fixture therefore checks both independent main-node hubs and a shared-main control.
The test links the shipped LogEntryContent unchanged and uses a real private PostgreSQL store, three Orleans silos, 124 typed log entries with 4KB multiline messages, and the existing 15-second Initial bound. It exercises direct persistence and ordinary create/update landing separately. Against Plugins b382116249d143c590260902a5117aa814f97b2e and owned Core b21d138, Release/CIRun/warnings-as-errors builds with 0 warnings and 0 errors. Native xUnit reports 4 executed, 4 passed, 0 skipped/failed.
The private PostgreSQL container was deleted after the run. No production service provider, store, cache, grants, or credentials were used or changed. Receipts are /tmp/mwp-1186-independent-main.trx and /tmp/mwp-1186-independent-main-test.log; source is the private diagnostic TypedLogEntryActivationBurstTests.
This rules out typed payload size, normal write landing, and independent three-silo owning reads alone as a reproduction on current code. It does not reproduce the full production catalog, dynamic LogEntry layout configuration, retained subscriptions, or concurrent production traffic, and does not prove which historical change mattered. The provider Initial stall remains unresolved; no raised bound, retry, or recovery mechanism is being presented as a fix.
Further configuration and catalog controls completed at 20:15Z: the fixture now links the shipped layout and test-area sources unchanged and registers the full production LogEntry configuration (WithContentType, AddDefaultLayoutAreas, LogEntry views and Tests area). Four ownership/write cases passed with owning reads in 1045–1611ms.
A fifth case provisions 244 additional empty private partitions through the real PostgreSQL partition provider before normal landing and independent-hub activation. Provisioning acknowledgements settled in 4921ms; 124 writes landed in 111ms; all 124 owning reads settled in 1003ms. Final native receipt: 5 executed, 5 passed, zero failures/skips, private PostgreSQL container deleted. Build was Release/CIRun/warnings-as-errors with zero warnings/errors. Receipts: /tmp/mwp-1186-production-config.trx and /tmp/mwp-1186-partition-catalog.trx.
This adds the actual statically linked LogEntry hub configuration and larger empty partition registry to the negative reproduction. It does not model dynamic compilation of the full production NodeType catalog or background retained-query traffic. The investigation remains open.
rbuergi commentedon Sep 27, 2026
The private reproduction now covers the dynamic LogEntry NodeType path as well as the previous static-config controls. It embeds all four shipped LogEntry source/test files, seeds them as real Code nodes, requests a forced compile through the normal private node stream, then lands and reads 124 independent main-node instances over a real PostgreSQL database and three Orleans silos.
Final native xUnit receipt
/tmp/mwp-1186-dynamic-private-namespace.trx: 1 executed / 1 passed / 0 failed / 0 skipped (11.117 seconds including the private cluster). Its captured output records:DYNAMIC_LOG_COMPILE elapsed_ms=995 status=Ok error=DYNAMIC_LOG_LANDING nodes=124 elapsed_ms=213DYNAMIC_LOG_OWNING_READS nodes=124 elapsed_ms=1531Normal landing uses four concurrent writes. The compile has its own 45-second diagnostic bound; the writes and owning reads retain the production 15-second bound. Every returned content payload is checked through ContentAs, including its typed message value. The test project built with Release, CIRun=true and warnings-as-errors: zero warnings, zero errors, 6.54 seconds, using the owned Core checkout explicitly.
Fixture qualification: the earlier static diagnostic links these classes into the test host. The first dynamic fixture collided with that host's identical extension method (
CS0121); this was a fixture error, not a product repro. The runtime copy is therefore wrapped in a private namespace, preserving every shipped declaration and method body, and its configuration names the same content/layout/Tests methods explicitly. An assertion proves all four resources were loaded. No production node, DI service, access grant, module or configuration was changed. This remains local diagnostic work, not a product repair or a pushed branch.This rules out dynamic compilation plus a 124-instance burst alone on this private fixture. It does not reproduce the live catalog's hundreds of dynamic types, thousands of retained sync hubs, or the incident's enormous queued-work surge. The newly read exact-pod evidence at #5799 (comment) remains the stronger next target: 61,272–245,955 queued pool work items during the activation-failure window. No startup gate is opened to make these reads pass and no timeout/cap/retry is increased. The root-cause issue stays open.
rbuergi commentedon Sep 27, 2026
The private real-PostgreSQL / three-silo dynamic-log diagnostic now includes 5,000 live, independently addressed, dynamically typed owning-hub stream connections before the 124-node fresh burst. Setup uses normal owning-hub writes in small batches. Replay connections stay subscribed throughout the burst; completion/error counters verify these are retained live readers, not merely cached values from Take(1). The single runtime-compiled type still preserves all four shipped LogEntry source declarations/method bodies inside the private namespace required by the static diagnostic fixture.
First new condition: strict Release/CIRun/warnings-as-errors build, 0 warnings / 0 errors (5.53s); native test 1 executed / 1 passed / 0 failed or skipped, 70.253s total. Compile 1,083ms; warming 5,000 connections 57,223ms; at that instant pending pool work 0, managed heap 9,404,778,512 bytes. Fresh 124 normal writes 177ms, all 124 owning reads 1,189ms; all 5,000 retained streams remained live (0 completions, 0 errors). Receipt /tmp/mwp-1186-retained5000.trx.
A newly instrumented variant classifies memory AFTER every measured latency/liveness assertion, with the same connections still held. It is not a flake retry and collection cannot help the burst pass. Strict build 0 warnings / 0 errors (5.55s); 1 executed / 1 passed / 0 failed or skipped, 76.271s total. Warming 58,599ms, pending pool work 0, managed heap 9,248,473,296 bytes; 124 writes 164ms; 124 reads 1,158ms; retained reader completions/errors 0/0. Post-burst collection changed 9,480,159,024 to 8,587,227,736 bytes and took 7,304ms, while all 5,000 connections stayed held. Receipt /tmp/mwp-1186-retained5000-heap.trx.
This rules out the retained typed-hub count plus one dynamic type ALONE as sufficient to reproduce the 15-second Initial failure on this fixture, even with a large reachable heap. It does not reproduce the live hundreds-of-types catalog or the 61,272–245,955 queued-work surge. The large reachable heap is now independently reproducible, but its dominators are not attributed: private heap-graph inspection must distinguish actual hub/serializer retention from fixture logging and other roots before proposing a change. No capacity, timeout, retry, or watchdog was increased; no production data, service provider, grant, profiler, or operator was touched. Both root-cause issues remain open.
rbuergi commentedon Sep 27, 2026
The private 5,000-retained-typed-hub diagnostic still does not reproduce the activation timeout. A new EventPipe capture completed with one executed/passed test and no skips (92.218s); the fresh 124 writes took 159ms and owning reads 1,197ms, with all 5,000 subscriptions alive. Post-assertion full collection measured 8,740,074,840 managed bytes.
The EventPipe heap graph is not a complete heap census: its report contains exactly 10,000,000 objects / 561,908,177 bytes. Inspection of the installed dotnet-gcdump 10.0.731102 reader confirms a hard stop at
m_graph.NodeCount >= 10000000. Capture exit 0 and a nonempty file did not establish graph completeness. Its type totals cannot attribute the remaining heap or establish the production root cause. No capture cap, buffer, retry, or timeout has been raised.The full private native dump is preserved separately (10,549,531,992 bytes, private directory/file permissions). Both installed native readers crashed before producing statistics. Inspection identifies a signed-range defect: the reader casts
(segment.Size - offset)tointbeforeMath.Min(buffer.Length, ...). The dump has a 6,298,271,744-byte segment. A deterministic arithmetic repro inside that segment gives remaining=4,150,788,096, old read length=-144,179,200, corrected bounded read length=32. A private copy of the tool now computes the minimum in unsigned 64-bit arithmetic and narrows only the bounded result; its helper builds Release with warnings as errors, 0 warnings/errors. Analysis of the original full dump with that private correction is pending. The installed tool and dump remain intact; no production data/process/permissions were touched.These are instrument findings, not a product fix or evidence to close the activation/OOM issues. Full coverage and retaining roots still need to be established.
rbuergi commentedon Sep 30, 2026
Ownership note. This occurrence (memex-cloud
555d9cd496-qntsh, 11:02Z on 09-27) sits inside the ThreadPool/heap surge recorded on #5799 (poolPending up to 245,955; heap 2 → 10 GiB). That surge's producer is being worked under #5555 / #1186 (#5925 is open). So this issue waits on that root, not on the query providers. The newer memex stalls on #5315/#5344/#5478 do not carry that signature: #5315 (comment).rbuergi commentedon Oct 7, 2026
2026-10-07: the newest memex stall burst happens while the process spends most of its wall time in GC pauses. The provider is not the stalled edge.
All reads are governed
Logsactions on memex.systemorph.com. Nothing was mutated.The burst
Ops/Actions/verify-1186-memex-20261007-activationcovers{namespace="memex"}for the 24 h ending 2026-10-07 16:07Z. It read 432 lines, untruncated, 0 landing failures. The search could list only 200 of the landed entries. Of those 200:QueryProviderStalledException.PostgreSqlPartitionedMeshQuery.MessageHubGrainwarnings.145 of the 200 come from one pod,
memex-portal-deployment-65f6466887-5cx77. They fall between 09:52 and 09:56Z: 3, 6, 15, 105 and 16 per minute. A typical query ispath:Hosting/Queues/operator/agents scope:children nodeType:Hosting/Jobassystem-security. That query is anchored to one partition, so the per-schemaPostgreSqlMeshQuerydelegate serves it, not a 245-schema fan-out. I saw noactivation faultedline in the 200 I listed. That is not established for the other 232.What the process was doing:
[LIVENESS]on the same podSource:
Ops/Actions/verify-5800-memex-20261007-5cx77-hb, 09:30–10:05Z, 188 lines, untruncated.The process runs
serverGC=True, so these pauses stop the world. From 09:31 to 10:03Z the process accumulated 1,089 s of GC pause in 1,950 s of wall time (56 %). In the 09:50–09:58 band it is 65–93 % per tick. Gen2 collections went from 0–1 per tick to 2–3 per tick (about one every 4 s), while the heap stayed flat at about 11.3–11.6 GiB. That is a near-full live heap being collected repeatedly, not allocation growth. The stall burst at 09:52–09:56 sits inside that band. A query whose Initial needs about 1–2 s of execution cannot finish within 15 s when roughly 12 of every 13 seconds are spent stopped.What this excludes
PostgreSqlPartitionedMeshQueryorPostgreSqlMeshQuerywithholds an Initial in this burst. The provider times out with everything else on a process that is barely running managed code. The 09-27 ownership note on this issue already pointed this way for memex-cloud (poolPending ~245k, heap 2→10 GiB), and this burst on memex fits the same mechanism.Not established
…-5cx77-heapclimb, 200 of more than 200 listed) shows 115ObjectPolymorphicConverterwarnings, 44CreateNodelines, 12IMeshCatalogwarnings and Orleans health-monitor/watchdog warnings. None of those is a demonstrated allocator.So this issue's remaining evidence reduces to the memory-retention root (#5555). I made no code change. The issue stays open, because closing it needs a post-fix window with no GC-thrash band and no stalls.
rbuergi commentedon Oct 7, 2026
Related: #1186, #5555. Cross-linked, not merged. Both this issue and #1186 now reduce to GC thrash from the retention on #5555, but no fix PR is attributed to either stall, and #1186's chain (from 2026-08-10, including the
SubscribeRequestshape) is wider than this burst. Revisit as a merge once a fix on #5555 is verified to end the stall band.rbuergi commentedon Oct 8, 2026
Folded in as duplicates: #6164 (FleetCoordinator, gkkdv 10-05 16:36) and #6136 (Npgsql connector-open timeouts, cgd5r and gkkdv 10-05). In every occurrence GC was 95–98 % of wall time and heap was 11.1–11.6 GiB. This issue is now the tracking issue for stalls on GC-thrash pods, and it closes on a post-roll window carrying core #6261 + Plugins#3109. Per-stall table: #5315 (comment)
rbuergi commentedon Oct 10, 2026
Post-roll reading, 2026-10-10 ~05:20Z. All three portals serve core
479c9b72d5(/api/version), which carries #6261 (merged 2026-10-07 17:06Z). Plugins#3109 merged 2026-10-07 17:57Z, before any image in the windows below was built.Governed
Logsreads on the control instance (Ops/Actions/logs-*-20261010-*):warm-up complete)Thread Pool execution stalledThread Pool execution stalled for NN(≥ 10 s)DISPOSAL DEADLOCK DETECTED(MessageHub[7316]/[7317])MessageHub[73xx]or[DISPOSE-*[LIVENESS]tick with heap ≥ 6 GiB on the CURRENT ReplicaSetd9cd97, since 22:34Z; its[HEAPSTEP]lines peak at 5.06 GiB)477bf6, since ~23:00Z;[HEAPSTEP]peak 3.83 GiB)For comparison, the stall band this issue tracked ran at heap 10.2–11.6 GiB with GC at 86–98 % of wall time. The heaviest tick on a #6261 image in either window is memex
cfb58787b-b6kljon 10-09 14:35Z: heap 8.09 GiB, butgcPause +0.19 sper 10 s tick (~2 %) andpoolPending=0. The heap is high there, but the GC is not thrashing.Positive controls. (1) The pool-stall instrument fires in the window: 18 sub-3 s lines, so a ≥ 10 s stall would have printed. (2)
[LIVENESS]/[HEAPSTEP]are ingested from the current pods. (3) The same heap regex returns 255 ≥ 6 GiB ticks from the pre-#62616548d4pods (booted ~10-07 13:15Z, before the merge) up to 10-09 17:58Z. That control separates the images.Not established. Loki keeps under ~4.5 days. A probe for any line from the 10-05 pod
6cfcf78898-kss8qreturned 0, so the 10-05 episode itself cannot be re-read as an in-window control for theDISPOSAL DEADLOCKregex. That regex is the literal text atMessageHub.cs:3084/3116/3145. The windows are 12 h and 24 h of ordinary traffic plus frequent rolls. They are not a reproduction of the 10-05 surge.Closing. This became the tracking issue for stalls on GC-thrash pods, to close on a post-roll window carrying #6261 + Plugins#3109. Both windows above carry them. No
[LIVENESS]tick on a current ReplicaSet reaches 6 GiB, and no pool stall reaches 3 s. Reopen if a 30 s node-resolution silence recurs on a pod that is not GC-bound.sev:M sweep, Claude Opus 5.5 (agent session).