Repository navigation
Triage reconcile cannot record results on log incidents when storage query providers stall past the fan-in timeout #5478
Description
What is failing
The log-incident control plane's Triage reconcile cannot write updates onto the log-incident node it is managing. The write is routed through the mesh query fan-in, and that query never gets a snapshot: neither StorageAdapterMeshQueryProvider nor PostgreSqlPartitionedMeshQuery emits an Initial within the 15s fan-in bound, so path resolution for the incident node fails (DeliveryFailureException) and the control plane wraps it as IncidentWriteFailedException.
Probable cause
High confidence this is a downstream symptom of the storage-backed query providers stalling, not a defect in the control plane itself. The inner exception explicitly reports the merged Initial gating on every provider and the query being reported as unavailable/retryable — and the mesh shows a large volume of Query provider(s) [StorageAdapterMeshQueryProvider, PostgreSqlPartitionedMeshQuery] did not emit an Initial log entries from the same pod family (see Ops/Logs, 2026-09-21), indicating a recurring storage/query-layer stall or back-pressure. The control plane behaves as designed (retryable, re-attempted on next trigger), so the fix belongs in the stalled provider, per the exception's own guidance.
Impact
Single occurrence, single pod, memex namespace, at 2026-09-23T05:37:58Z. The concrete effect is that triage results and comments are not recorded on the affected incident until the next trigger succeeds — an observability gap on one incident, not user-facing data loss. If the underlying provider stall is systemic (the Ops/Logs volume suggests it recurs), expect this symptom to reappear on other incidents during stall windows.
Where to look
- Control plane write path:
MeshWeaver.Observability.LogIncidentControlPlane/ the Triage reconcile's incident-write logic (category on the incident). - Underlying stall:
MeshWeaver.Hosting.Persistence.Query.StorageAdapterMeshQueryProviderandMeshWeaver.Hosting.PostgreSql.PostgreSqlPartitionedMeshQuery— investigate why theInitialemission exceeds the 15s fan-in bound under load. - Related existing incident (same component, similar symptom):
Admin/_LogIncident/3b6b24e5f282f36b— "LogIncidentControlPlane: DeliveryFailureException — Incident {hex}: could not record the failure". If a ticket covering the provider stall already exists, this may be closable as a duplicate of it.
Evidence
| Fingerprint | 2633dcb2ad88b952 |
| Category | MeshWeaver.Observability.LogIncidentControlPlane |
| Severity | Error |
| Exception | MeshWeaver.Observability.IncidentWriteFailedException |
| Namespace | memex |
| Pods | memex-portal-deployment-5c99d79997-xs87x |
| Occurrences | 1 |
| First seen | 2026-09-23 05:37:58Z |
| Last seen | 2026-09-23 05:37:58Z |
| Routing | not determined — no configured route matches the category MeshWeaver.Observability.LogIncidentControlPlane. This repository is the configured fallback, not a finding about who owns the fault; the category names the LOGGER, which may not be the subject. |
Recent log lines
2026-09-23 05:37:58Z memex-portal-deployment-5c99d79997-xs87x fail: MeshWeaver.Observability.LogIncidentControlPlane[0]
Incident 16e8401077c15363: the Triage reconcile could not write Admin/_LogIncident/16e8401077c15363, so nothing was recorded on the incident. It is re-attempted the next time its trigger fires — the next collection emission or the next comment-debt tick.
MeshWeaver.Observability.IncidentWriteFailedException: The incident at 'Admin/_LogIncident/16e8401077c15363' could not be written.
---> MeshWeaver.Messaging.DeliveryFailureException: Path resolution for 'Admin/_LogIncident/16e8401077c15363' failed: Query provider(s) [MeshWeaver.Hosting.Persistence.Query.StorageAdapterMeshQueryProvider, MeshWeaver.Hosting.PostgreSql.PostgreSqlPartitionedMeshQuery] did not emit an Initial within the query fan-in's 15s bound for query 'path:"Admin/_LogIncident/16e8401077c15363"|"Admin/_LogIncident"|"Admin"' (user 'system-security'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.
--- End of inner exception stack trace ---
Re-addressed by the current identity function: this incident inherited a30c09f918c68632 (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.
Opened automatically from Admin/_LogIncident/2633dcb2ad88b952. Recurrences are folded into this issue rather than opening new ones.
Activity
rbuergi commentedon Sep 23, 2026
Downstream of the provider stall tracked on #5315 and #5344. The control plane is behaving as designed. The incident write resolves its path through the query fan-in, and that query was
path:"Admin/_LogIncident/…"|…. The path has a_segment, so bothStorageAdapterMeshQueryProviderandPostgreSqlPartitionedMeshQueryserve it, and both are named in this sample.Root-cause analysis and the fix in review are on #5315 (comment):
Both stop the per-schema providers from queueing one full Postgres read per change notification. That queue sits on the process-wide
pg-read:Postgresgate (cap 16), which every Initial also waits on.Nothing to change in
LogIncidentControlPlane: the failure is retryable, and it is re-attempted on the next trigger.Leaving this open. It closes together with #5315 on the same post-roll reading: whether
IncidentWriteFailedException/QueryProviderStalledExceptionrecurs after a roll carrying both halves.rbuergi commentedon Sep 24, 2026
Update: all three fixes are merged.
e45b95d8722d0fe843c63fa4bd36f5They are not live yet. They reach memex and memex-cloud with the next release, seal and roll that carries them. The post-roll reading named above decides whether this closes.
rbuergi commentedon Sep 24, 2026
Post-roll sweep 2026-09-24 ~06:50Z: the fix is not live on either portal yet
What is running.
/api/versionreturns9a61fed964.Ops/Status/memex-cloud(06:42:00Z) shows 4/4 replicas on3.0.0-ci.9263. CD run 9263 built core9a61fed964with Plugins41bcfe968f./api/versionreturns39f49494eb.Ops/Status/memexshows the serving pods on3.0.0-ci.9218, which is core39f49494ebwith Pluginse9d9283abe. A third pod, onci.9260, is unready, so the roll isconverged: false.This issue's fix: core #5615 (
e45b95d872) + Plugins#2337 (2d0fe843c6). It is not an ancestor of either running build (git merge-base --is-ancestor, exit 1).Where it is.
3.0.0-ci.9287(CD run35956222018) is corebf5ac85526with Pluginsc4795a8d5b, and it contains this fix. The registry has sealed it. memex-cloud's self-updater logged[SelfUpdate] handing 3.0.0-ci.9287 to the control lane (https://memex.systemorph…)at 06:47Z. So the roll now waits on the control instance, which is an owner step and was not done here.Still open. It closes on a post-roll reading once a
ci.9287-or-later image serves the namespace this fingerprint fires in.systemorph commentedon Sep 28, 2026
Still occurring: 1 more since the last update (2 total).
memex-portal-deployment-5c99d79997-xs87x,memex-portal-deployment-85469d6f4b-z9tfvRecent log lines
Re-addressed by the current identity function: this incident inherited
a30c09f918c68632(Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.systemorph commentedon Sep 29, 2026
Still occurring: 1 more since the last update (3 total).
memex-portal-deployment-5c99d79997-xs87x,memex-portal-deployment-85469d6f4b-z9tfv,memex-portal-deployment-5dc5995596-d9qj7Recent log lines
Re-addressed by the current identity function: this incident inherited
a30c09f918c68632(Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.rbuergi commentedon Sep 30, 2026
I read this issue's latest memex recurrences against the pod's LIVENESS heartbeats: #5315 (comment). At the stall minute, no heartbeat recorded ≥ 1,000 queued ThreadPool items. So this recurrence is not the pod-wide surge behind #5800/#5799. The 15 s Initial is being missed for another reason. The pg-read gate wait and Postgres lock waits at those minutes are the next readings, and I have not taken them. No bound was changed.
systemorph commentedon Oct 4, 2026
Still occurring: 1 more since the last update (4 total).
memex-portal-deployment-5c99d79997-xs87x,memex-portal-deployment-85469d6f4b-z9tfv,memex-portal-deployment-5dc5995596-d9qj7,memex-portal-deployment-67c67b456d-l7dp7Recent log lines
Re-addressed by the current identity function: this incident inherited
a30c09f918c68632(Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.systemorph commentedon Oct 5, 2026
Folded in — a DIFFERENT fingerprint from the same log site.
This issue tracks the log site
8ba036a09364b28a(MeshWeaver.Observability.LogIncidentControlPlane), and these occurrences belong to it. They carry their own fingerprint — the masking split one site across several — so before this they would have opened a SEPARATE ticket. One log site fanned out into 46 of those, which is why they fold here instead. Nothing is discarded: the distinguishing evidence is below, and splitting this back out is a person's call.f1e8bef1a37617218ba036a09364b28aMeshWeaver.Observability.LogIncidentControlPlaneMeshWeaver.Observability.IncidentWriteFailedExceptionThe incident at '{value}' could not be written. ---> System.TimeoutException: Update aborted: no initial state arrived for '{value}' within {n}s. Likely causes — ({n}) RLS silently rejected the prior CreateNode, ({n}) the path is misspelled / points at a namespace no NodeType claims, ({n}) the node was deleted between create and update, or ({n}) the per-node hub activated but its MeshDataSource didn't load the node from persistence. NO change item at all reached the mirror inside the bound — no snapshot ever hydrated. This does NOT establish that the owner never answered: a fresh subscribe is ACKNOWLEDGED BEFORE hydration, so an owner that acked and then produced no frame (a cold data source, a per-node hub still activating, an initial read that yielded nothing) reads identically here. Both the owner's {path} AND its initial read are still in scope. ---> MeshWeaver.Mesh.MeshNodeStreamHandle+BaseStateTimeoutException: NO change item at all reached the mirror inside the bound — no snapshot ever hydrated. This does NOT establish that the owner never answered: a fresh subscribe is ACKNOWLEDGED BEFORE hydration, so an owner that acked and then produced no frame (a cold data source, a per-node hub still activating, an initial read that yielded nothing) reads identically here. Both the owner's {path} AND its initial read are still in scope. --- End of inner exception stack trace --- --- End of inner exception stack trace ---memexmemex-portal-deployment-6cfcf78898-gkkdvAdmin/_LogIncident/f1e8bef1a3761721What triage wrote about this fingerprint
What is failing
The incident control plane's Comment reconcile (
MeshWeaver.Observability.LogIncidentControlPlane) could not write to an existing incident node: its update toAdmin/_LogIncident/ccec30127fa4da9a(the routing back-pressure incident, already filed as #5703) aborted withIncidentWriteFailedExceptionbecause no initial state ever arrived for the target node within the 30 s base-state bound — the subscribe was acked but no frame hydrated. The intended comment was therefore not recorded on the incident; the control plane re-attempts on its next trigger (next collection emission or comment-debt tick).Probable cause
The target node's owner-side stream acked the fresh subscribe but never produced an initial frame — a per-node hub that was still activating, wedged, or serving a cold initial read while the mesh was under heavy stress. Confidence: moderate — the shape of the failure is unambiguous, but the inner
BaseStateTimeoutExceptiondeliberately cannot distinguish "owner never answered" from "owner acked and produced nothing", and one sample can't separate them.The evidence:
Admin/_LogIncident/ccec30127fa4da9aexists, is Active, and had been written successfully (version 17) at 16:26:17 — only 27 s before this failure at 16:26:44. That largely rules out the message's candidate causes (1) RLS-rejected create and (2) misspelled path / unclaimed namespace, and nothing suggests (3) deletion. What remains is (4): the target's per-node hub /MeshDataSourcenot delivering an initial read in time.LocalSiloHealthMonitorthread-pool stall report ~16:20, and multiple routing back-pressure Criticals (including the very incident being commented on). The control plane's write timed out while the mesh it writes to was saturated — the observability pipeline is most likely to fail exactly when it is most needed.memex-portal-deployment-6cfcf78898-gkkdv) is a different pod from the target incident's samples (…-848f664fd5-jgkbf), consistent with a cross-pod write through a routing layer that was reporting back-pressure at the time.Impact
One occurrence, one pod, one missed comment. No user-visible data is lost — the reconcile is explicitly re-attempted on the next trigger, and the target incident was already filed and tracked (#5703). The defect class matters more than this instance, though: incident commentary silently fails to attach during mesh saturation, so during an outage the incident trail can lag or drop the record of what the control plane observed. Judged sev:M: a secondary path (comment recording) with a built-in retry; re-rule upward if the fingerprint recurs across pods or the re-attempt never lands.
Where to look
MeshWeaver.Observability.LogIncidentControlPlane— the Comment reconcile path that wraps the update in the 30 s base-state bound and logsIncidentWriteFailedExceptionon timeout. It already retries on its own trigger; the question is whether the retry is bounded/alerted or can lag indefinitely while the target hub stays cold.MeshWeaver.Mesh.MeshNodeStreamHandle+BaseStateTimeoutException— per its own message, a fresh subscribe is acked before hydration, so the reconcile's timeout cannot tell a dead owner from a slow one. If the control plane needs stronger guarantees during stress, the fix is on this distinction (e.g. an explicit owner-side probe) rather than a longer bound.OrleansMessageRejectionException~15:59, thread-pool stall ~16:20, routing back-pressure Criticals incl. Slow dispatch leg to Manufacturing import activity holds a routing slot and backs up same-stream legs past the 64-dispatch threshold #5703) — if those share one root cause, this ticket is collateral of it and can be closed as such.Not a duplicate: no prior incident carries this fingerprint. Correlated with #5703 (the incident being written to) and the fleet-stress window, but the failing component here is the observability control plane's own write path.
Still occurring: 1 more since the last update (1 total).
memex-portal-deployment-6cfcf78898-gkkdvRecent log lines
systemorph commentedon Oct 5, 2026
Still occurring: 1 more since the last update (2 total).
memex-portal-deployment-6cfcf78898-gkkdv,memex-portal-deployment-f86c77f48-twzk9Recent log lines
systemorph commentedon Oct 5, 2026
Still occurring: 1 more since the last update (2 total).
memex-portal-deployment-6cfcf78898-gkkdv,memex-portal-deployment-f86c77f48-twzk9Recent log lines
systemorph commentedon Oct 5, 2026
Folded in — a DIFFERENT fingerprint from the same log site.
This issue tracks the log site
8ba036a09364b28a(MeshWeaver.Observability.LogIncidentControlPlane), and these occurrences belong to it. They carry their own fingerprint — the masking split one site across several — so before this they would have opened a SEPARATE ticket. One log site fanned out into 46 of those, which is why they fold here instead. Nothing is discarded: the distinguishing evidence is below, and splitting this back out is a person's call.da6e457ed94838068ba036a09364b28aMeshWeaver.Observability.LogIncidentControlPlaneMeshWeaver.Observability.IncidentWriteFailedExceptionThe incident at '{value}' could not be written. ---> System.TimeoutException: Update aborted: no initial state arrived for '{value}' within {n}s. Likely causes — ({n}) RLS silently rejected the prior CreateNode, ({n}) the path is misspelled / points at a namespace no NodeType claims, ({n}) the node was deleted between create and update, or ({n}) the per-node hub activated but its MeshDataSource didn't load the node from persistence. ---> System.TimeoutException: No response received in hub {path} within {n}:{n}:{n} for request SubscribeRequest (id=p1{n}a_ISIZk2yjLhZPCubpg) → target {id} This hub: {id}=Started Queue(buffer={n},deferred={n},openGates={n},drainsInFlight={n},draining=False,handledWhileWaiting={n}). This hub is idle AT THE MOMENT IT GAVE UP — an instantaneous sample, which is why handledWhileWaiting above is printed beside it: that is the interval fact, and a hub that handled many messages was not idle throughout however empty its queue is now. Cause is UNKNOWN between: the target never received the request (routing), the target received it and is wedged (a per-node hub that stops answering — MeshWeaver#{n}), or the target answered and the reply was lost. Queue state alone cannot distinguish them; the trail below can, and so can the target's own RunLevel and queue. Trail: AWAITING SubscribeRequest→{path}@{path}(+{n}ms) → POSTED target={path}@{path}(+{n}ms) → RECEIVED runLevel=Started@{path}(+{n}ms) → ENQUEUED@{path}(+{n}ms) → QUEUED queue=main depth={n}@{path}(+{n}ms) → RECEIVED runLevel=Started@{path}(+{n}ms) → ENQUEUED@{path}(+{n}ms) → QUEUED queue=main depth={n}@{path}(+{n}ms) → ROUTED onTarget=False state=Forwarded@{path}(+{n}ms) → ROUTED onTarget=False state=Forwarded@{path}(+{n}ms) ⇒ the delivery reached a hub but no handler was ever entered — it is still being routed, or it was accepted and never executed. --- End of inner exception stack trace --- --- End of inner exception stack trace ---memexmemex-portal-deployment-f86c77f48-twzk9Admin/_LogIncident/da6e457ed9483806What triage wrote about this fingerprint
What is failing
The incident control plane's File step — recording on a log-incident node that it has been filed — could not update
Admin/_LogIncident/9a01f3ca3d46c5b7(the NodeType-census fault, filed as #6183). The update aborted because no initial state arrived within the 30 s base-state bound, and underneath that theSubscribeRequestto the target node reached a hub, was routed onward (ROUTED onTarget=False state=Forwarded, twice), but no handler was ever entered before the 60 s hub response bound expired. The delivery trail's own verdict: the request is still being routed, or it was accepted and never executed.Probable cause
The request was accepted by routing and forwarded toward the target's per-node hub but never executed there — confidence: moderate. The trail unambiguously rules out "the target never received the request": it was received, enqueued, and queued at two hubs before being forwarded off-target with no handler ever entered. What a single sample cannot separate is (a) cross-pod routing that never completed the forward from (b) the target's per-node hub accepting the request and wedging — the exception message itself references MeshWeaver#2896 for the latter class. The surrounding evidence favours general mesh stress over a specific wiring defect: this fired at 23:36 inside a ~23:00–23:37 window of instability on the same namespace — PostgreSQL connection resets (the target incident's own census fault at 23:30, #6183), repeated
LogIncidentControlPlanewrite failures against this same target from 23:31–23:36 (sibling fingerprint, #5478), and messaging/silo timeouts elsewhere in the fleet.Impact
One occurrence, one pod (
memex-portal-deployment-f86c77f48-twzk9), self-healing: the control plane re-attempts the write on the next change to the incident collection, and the target incident now shows status Filed with issue #6183 — nothing was lost. The defect class still matters: incident-node writes fail exactly when the mesh is degraded, so the incident trail lags during the window it is most needed. Judged sev:M — a secondary (observability bookkeeping) path with a built-in retry; re-rule if this fingerprint recurs outside stress windows or across pods.Where to look
MeshWeaver.Observability.LogIncidentControlPlane— the File write phase whose update wraps the base-state bound and logsIncidentWriteFailedExceptionon timeout.ROUTED onTarget=False state=Forwarded— the forwarding/routing layer that accepted theSubscribeRequestwithout executing it on the target. This fingerprint is distinct from sibling Triage reconcile cannot record results on log incidents when storage query providers stall past the fan-in timeout #5478 (owner acknowledged the subscribe but produced no initial frame): here no handler was entered at all, so the fault sits in delivery/routing or a wedged target hub, not in initial-read hydration.Admin/_LogIncident: DynamicTypePreWarmer latches its NodeType census as faulted on a single transient PostgreSQL connection reset — no retry until restart #6183 (the target node's own PostgreSQL census fault) and Triage reconcile cannot record results on log incidents when storage query providers stall past the fan-in timeout #5478 (sibling write failures against the same target) — if one root cause explains the window, this can be closed as collateral of it.Not a duplicate of #5478: same log site and component, but the inner failure has a different shape (forwarded-and-never-handled vs. acked-with-no-frame), so the masking rule currently keeps them as two fingerprints.
Still occurring: 1 more since the last update (1 total).
memex-portal-deployment-f86c77f48-twzk9Recent log lines
systemorph commentedon Oct 6, 2026
Still occurring: 7 more since the last update (9 total).
memex-portal-deployment-6cfcf78898-gkkdv,memex-portal-deployment-f86c77f48-twzk9Recent log lines
rbuergi commentedon Oct 8, 2026
Split for this issue's occurrences:
IncidentWriteFailedException, not a provider stall.[CrossSchema] SLOWfan-out reads over 159 schemas took 12–14 s to first row.SubscribeRequestwith no handler run.The healthy-pod half is tracked on #5315 and needs the #6260 IoPool census on a deployed image to split Postgres time from slot wait. Table: #5315 (comment)
rbuergi commentedon Oct 9, 2026
Consolidated into #5315. The split of this issue's occurrences is already in #5315's table (
issuecomment-6053628304):[CrossSchema] SLOWfan-outs over 159 schemas took 12–14 s to first row.IncidentWriteFailedwith a forwardedSubscribeRequestand no handler run. This is not a stall.The consumer behaves as designed: the incident "still carries its request and is picked up again on the next change". That leaves no defect specific to this issue.
Readings on the current memex image. memex runs
3.0.0-ci.10310(core8b791643) on all 3 replicas, booted 11:15–11:44Z today (Ops/Status/memex, 14:32Z). That image carries the #6260 IoPool census, which #5315 needs to split Postgres time from slot wait. A governedLogsread (Ops/Actions/sevm-20261009-memex-stall) covers 11:12–14:37Z withdid not emit an Initial|Failed to track activity|could not be claimed|UNANCHORED|TokenValidation. It returned 3 lines, none of them a stall: they are theInstanceSyncConfigGRACE line, once per replica, which is #5508 and fixed in Systemorph/MeshWeaver.Plugins#3230. The positive control is the 3DynamicTypePreWarmer: warm-up completelines in the same window (Ops/Actions/sevm-20261009-memex-prewarm), so the window is live. A 3.5 h zero does not by itself close a stall that recurs every few days. That is why this folds into #5315 instead of closing as fixed.Closing as a duplicate of #5315.
systemorph commentedon Oct 9, 2026
Left closed — this ticket was closed as a DUPLICATE, which is a decision that the fault is tracked on another issue. Reopening it would discard that decision, so the occurrences below are recorded here instead — GitHub renders the cross-reference on the issue this was closed onto, where the analysis lives.
Still occurring: 1 more since the last update (10 total).
memex-portal-deployment-6cfcf78898-gkkdv,memex-portal-deployment-f86c77f48-twzk9,memex-portal-deployment-5f6b9fc9b-hrdhqRecent log lines
systemorph commentedon Oct 9, 2026
Left closed — this ticket was closed as a DUPLICATE, which is a decision that the fault is tracked on another issue. Reopening it would discard that decision, so the occurrences below are recorded here instead — GitHub renders the cross-reference on the issue this was closed onto, where the analysis lives.
Still occurring: 7 more since the last update (16 total).
memex-portal-deployment-6cfcf78898-gkkdv,memex-portal-deployment-f86c77f48-twzk9,memex-portal-deployment-5f6b9fc9b-hrdhq,memex-portal-deployment-845898d964-ptszc,memex-portal-deployment-845898d964-f6wm5,memex-portal-deployment-845898d964-mwzl7Recent log lines
systemorph commentedon Oct 9, 2026
Folded in — a DIFFERENT fingerprint from the same log site.
This issue tracks the log site
8ba036a09364b28a(MeshWeaver.Observability.LogIncidentControlPlane), and these occurrences belong to it. They carry their own fingerprint — the masking split one site across several — so before this they would have opened a SEPARATE ticket. One log site fanned out into 46 of those, which is why they fold here instead. Nothing is discarded: the distinguishing evidence is below, and splitting this back out is a person's call.03f6cd1b9ce376b18ba036a09364b28aMeshWeaver.Observability.LogIncidentControlPlaneMeshWeaver.Observability.IncidentWriteFailedExceptionThe incident at '{value}' could not be written. ---> MeshWeaver.Messaging.DeliveryFailureException: Delivery to '{value}' failed: Grain placement operation timed out for grain {path} --- End of inner exception stack trace ---memexmemex-portal-deployment-5f6b9fc9b-hrdhq,memex-portal-deployment-845898d964-mwzl7Admin/_LogIncident/03f6cd1b9ce376b1What triage wrote about this fingerprint
What is failing
The log-incident control plane's write path (
MeshWeaver.Observability.LogIncidentControlPlane) cannot record results on incident nodes when the delivery to the target node's per-node messagehub grain fails: the update is wrapped inIncidentWriteFailedExceptionand nothing is written onto the incident. The affected writes target incident nodes inAdmin/_LogIncidentthat had just been created by the same burst the control plane was trying to triage — so the observability pipeline is failing to attach its own bookkeeping exactly on the incidents produced by the stress itself. Per the control plane's own message, the request is carried and re-attempted on the next change to the incident collection.Probable cause
Delivery failed because the Orleans grain placement for the target incident node's
messagehub/<path>grain timed out. Confidence: high on the mechanism — the exception chain states it directly (DeliveryFailureException: Delivery to '<incident path>' failed: Grain placement operation timed out). Confidence: moderate on the trigger being cluster saturation rather than a control-plane defect — the failure burst coincides with a fleet-wide stress window in which sibling incidents forSystem.TimeoutException("response did not arrive on time"),SiloUnavailableException, routing back-pressure, and the identicalDeliveryFailureExceptionat other log sites (LayoutAreaHost,RoutingGrain,Messaging) were all opened for the same minute. A grain placement timing out across pods points at a saturated/stalled cluster, not a bad path or missing node — the target nodes exist (they are incidents in the same collection that later received writes).Impact
Incident bookkeeping lags during mesh stress: triage results and comments silently fail to attach to the very incidents the stress is generating. Writes carry a built-in retry and no data is permanently lost, and the path is not user-visible. This is a real defect in the pipeline's robustness under load (the observability layer degrades first when the mesh degrades), but a secondary one.
Where to look
MeshWeaver.Observability.LogIncidentControlPlane— the write/reconcile path that logsIncidentWriteFailedException; check whether the retry-on-next-collection-change is bounded or alerted, or can lag indefinitely while the target hub stays unplaceable.MeshWeaver.Messagingdelivery layer wrappingDeliveryFailureException, and the Orleans grain placement timeout for per-nodemessagehubgrains.Admin/_LogIncident/f1e8bef1a3761721(filed as Triage reconcile cannot record results on log incidents when storage query providers stall past the fan-in timeout #5478) — same control plane and same symptom (incident write failed during stress), but a different inner mechanism (base-state hydration timeout instead of grain placement timeout). Same-window incidents8a6d880f930298f9(LayoutAreaHost, identical inner cause),d5afc72704385409(RoutingGraindirected delivery failure), andf7d8f9982b20cce5(Messagingaddress timeout) share the stress window; if those share one root cause, this ticket is collateral of it and can be closed as such.Left closed — this ticket was closed as a DUPLICATE, which is a decision that the fault is tracked on another issue. Reopening it would discard that decision, so the occurrences below are recorded here instead — GitHub renders the cross-reference on the issue this was closed onto, where the analysis lives.
Still occurring: 3 more since the last update (3 total).
memex-portal-deployment-5f6b9fc9b-hrdhq,memex-portal-deployment-845898d964-mwzl7Recent log lines
systemorph commentedon Oct 9, 2026
Folded in — a DIFFERENT fingerprint from the same log site.
This issue tracks the log site
8ba036a09364b28a(MeshWeaver.Observability.LogIncidentControlPlane), and these occurrences belong to it. They carry their own fingerprint — the masking split one site across several — so before this they would have opened a SEPARATE ticket. One log site fanned out into 46 of those, which is why they fold here instead. Nothing is discarded: the distinguishing evidence is below, and splitting this back out is a person's call.03f6cd1b9ce376b18ba036a09364b28aMeshWeaver.Observability.LogIncidentControlPlaneMeshWeaver.Observability.IncidentWriteFailedExceptionThe incident at '{value}' could not be written. ---> MeshWeaver.Messaging.DeliveryFailureException: Delivery to '{value}' failed: Grain placement operation timed out for grain {path} --- End of inner exception stack trace ---memexmemex-portal-deployment-5f6b9fc9b-hrdhq,memex-portal-deployment-845898d964-mwzl7Admin/_LogIncident/03f6cd1b9ce376b1What triage wrote about this fingerprint
What is failing
The log-incident control plane's write path (
MeshWeaver.Observability.LogIncidentControlPlane) cannot record results on incident nodes when the delivery to the target node's per-node messagehub grain fails: the update is wrapped inIncidentWriteFailedExceptionand nothing is written onto the incident. The affected writes target incident nodes inAdmin/_LogIncidentthat had just been created by the same burst the control plane was trying to triage — so the observability pipeline is failing to attach its own bookkeeping exactly on the incidents produced by the stress itself. Per the control plane's own message, the request is carried and re-attempted on the next change to the incident collection.Probable cause
Delivery failed because the Orleans grain placement for the target incident node's
messagehub/<path>grain timed out. Confidence: high on the mechanism — the exception chain states it directly (DeliveryFailureException: Delivery to '<incident path>' failed: Grain placement operation timed out). Confidence: moderate on the trigger being cluster saturation rather than a control-plane defect — the failure burst coincides with a fleet-wide stress window in which sibling incidents forSystem.TimeoutException("response did not arrive on time"),SiloUnavailableException, routing back-pressure, and the identicalDeliveryFailureExceptionat other log sites (LayoutAreaHost,RoutingGrain,Messaging) were all opened for the same minute. A grain placement timing out across pods points at a saturated/stalled cluster, not a bad path or missing node — the target nodes exist (they are incidents in the same collection that later received writes).Impact
Incident bookkeeping lags during mesh stress: triage results and comments silently fail to attach to the very incidents the stress is generating. Writes carry a built-in retry and no data is permanently lost, and the path is not user-visible. This is a real defect in the pipeline's robustness under load (the observability layer degrades first when the mesh degrades), but a secondary one.
Where to look
MeshWeaver.Observability.LogIncidentControlPlane— the write/reconcile path that logsIncidentWriteFailedException; check whether the retry-on-next-collection-change is bounded or alerted, or can lag indefinitely while the target hub stays unplaceable.MeshWeaver.Messagingdelivery layer wrappingDeliveryFailureException, and the Orleans grain placement timeout for per-nodemessagehubgrains.Admin/_LogIncident/f1e8bef1a3761721(filed as Triage reconcile cannot record results on log incidents when storage query providers stall past the fan-in timeout #5478) — same control plane and same symptom (incident write failed during stress), but a different inner mechanism (base-state hydration timeout instead of grain placement timeout). Same-window incidents8a6d880f930298f9(LayoutAreaHost, identical inner cause),d5afc72704385409(RoutingGraindirected delivery failure), andf7d8f9982b20cce5(Messagingaddress timeout) share the stress window; if those share one root cause, this ticket is collateral of it and can be closed as such.Left closed — this ticket was closed as a DUPLICATE, which is a decision that the fault is tracked on another issue. Reopening it would discard that decision, so the occurrences below are recorded here instead — GitHub renders the cross-reference on the issue this was closed onto, where the analysis lives.
Still occurring: 3 more since the last update (3 total).
memex-portal-deployment-5f6b9fc9b-hrdhq,memex-portal-deployment-845898d964-mwzl7Recent log lines
systemorph commentedon Oct 9, 2026
Folded in — a DIFFERENT fingerprint from the same log site.
This issue tracks the log site
8ba036a09364b28a(MeshWeaver.Observability.LogIncidentControlPlane), and these occurrences belong to it. They carry their own fingerprint — the masking split one site across several — so before this they would have opened a SEPARATE ticket. One log site fanned out into 46 of those, which is why they fold here instead. Nothing is discarded: the distinguishing evidence is below, and splitting this back out is a person's call.ea76324d9ceb36338ba036a09364b28aMeshWeaver.Observability.LogIncidentControlPlaneMeshWeaver.Observability.IncidentWriteFailedExceptionThe incident at '{value}' could not be written. ---> System.InvalidOperationException: Update aborted: this hub'{value}'s state, so no patch was built and none was posted. The write did NOT land; re-issue it. (The mirror was disposed or reclaimed mid-write — see Workspace.AcquireRemoteStreamUnchecked.) --- End of inner exception stack trace ---memexmemex-portal-deployment-845898d964-ptszcAdmin/_LogIncident/ea76324d9ceb3633What triage wrote about this fingerprint
What is failing
The log-incident control plane's triage write path (
MeshWeaver.Observability.LogIncidentControlPlane) failed to record its triage result onto two freshly-opened incident nodes inAdmin/_LogIncident: each update aborted before a patch was built, because the target hub's mirror stream ended without ever carrying the node's state — the mirror was disposed or reclaimed mid-write (Workspace.AcquireRemoteStreamUnchecked). Nothing was recorded at the time; the control plane re-attempts on the next change to the incident collection. Both writes demonstrably landed on retry — both target incidents carry their triage drafts and were filed (#5707, #5729) — so no data was lost.Probable cause
A mirror/hub disposal race on the per-node write path, triggered by cluster churn rather than an independent defect. Confidence: high on the mechanism (the abort message names it), moderate on churn being the trigger — the burst sits inside the same 20:39–20:44Z rollout-churn window that produced Orleans memory-stream dequeue timeouts and, on this very control plane and these very same target incidents, sibling failures with two different inner causes: grain placement timeouts (
Admin/_LogIncident/03f6cd1b9ce376b1) and base-state hydration timeouts (Admin/_LogIncident/f1e8bef1a3761721). Three distinct write-path failure modes striking the same nodes in the same minutes points at the hubs being torn down and reactivated under the churn, not at a defect in the triage logic itself. The framework documents this mirror/disposal write-path family in The Phantom Base After Owner Disposal — worth checking whether this abort belongs to that family's known teardown windows.Impact
Minimal for this instance: two writes delayed by one retry cycle on a single pod, self-recovered, no lost incident data, no user-visible path affected. The class concern — incident bookkeeping is the first thing to fail when the mesh is degraded — is real but already tracked.
Where to look
MeshWeaver.Observability.LogIncidentControlPlane— the triage write path that wraps the update and logsIncidentWriteFailedException; the retry on next collection emission worked here, so the open question is only whether it is bounded/alerted.MeshNodeStreamHandle.UpdateRemoteandWorkspace.AcquireRemoteStreamUnchecked(named in the abort message) — why a live write found a mirror already disposed/reclaimed, and whether the write path should re-acquire rather than abort.Duplicate / relation
Not an exact duplicate, but strongly related: sibling incidents on the same control-plane log site, same failure window, and the same target incident nodes were both filed into #5478 —
Admin/_LogIncident/f1e8bef1a3761721(base-state hydration timeout, filed as #5478) andAdmin/_LogIncident/03f6cd1b9ce376b1(grain placement timeout, also recorded against #5478). This incident is the third inner variant (mirror disposed mid-write) of that same condition. Recommend appending it to #5478 or closing it as collateral of the 2026-10-09 churn window rather than tracking separately — a human can decide in seconds.Left closed — this ticket was closed as a DUPLICATE, which is a decision that the fault is tracked on another issue. Reopening it would discard that decision, so the occurrences below are recorded here instead — GitHub renders the cross-reference on the issue this was closed onto, where the analysis lives.
Still occurring: 2 more since the last update (2 total).
memex-portal-deployment-845898d964-ptszcRecent log lines
systemorph commentedon Oct 10, 2026
Left closed — this ticket was closed as a DUPLICATE, which is a decision that the fault is tracked on another issue. Reopening it would discard that decision, so the occurrences below are recorded here instead — GitHub renders the cross-reference on the issue this was closed onto, where the analysis lives.
Still occurring: 14 more since the last update (30 total).
memex-portal-deployment-6cfcf78898-gkkdv,memex-portal-deployment-f86c77f48-twzk9,memex-portal-deployment-5f6b9fc9b-hrdhq,memex-portal-deployment-845898d964-ptszc,memex-portal-deployment-845898d964-f6wm5,memex-portal-deployment-845898d964-mwzl7Recent log lines
systemorph commentedon Oct 11, 2026
Left closed — this ticket was closed as a DUPLICATE, which is a decision that the fault is tracked on another issue. Reopening it would discard that decision, so the occurrences below are recorded here instead — GitHub renders the cross-reference on the issue this was closed onto, where the analysis lives.
Still occurring: 4 more since the last update (34 total).
memex-portal-deployment-6cfcf78898-gkkdv,memex-portal-deployment-f86c77f48-twzk9,memex-portal-deployment-5f6b9fc9b-hrdhq,memex-portal-deployment-845898d964-ptszc,memex-portal-deployment-845898d964-f6wm5,memex-portal-deployment-845898d964-mwzl7,memex-portal-deployment-6749fbfc59-hdlgzRecent log lines