Incident comment write stalls 30s then aborts when the incident hub cannot hydrate from persistence #4962

Activity

1. systemorph

   added

   bugSomething isn't workingSomething isn't working

   on Sep 20, 2026
2. rbuergi

   added

   area:meshCore subsystem: the mesh - nodes, partitions, addressing, activationCore subsystem: the mesh - nodes, partitions, addressing, activation

   feature:log-incidentFeature: log incidentFeature: log incident

   sev:LLow - cosmetic, rare edge case, or developer-onlyLow - cosmetic, rare edge case, or developer-only

   on Sep 20, 2026
3. rbuergi commented on Sep 22, 2026

   

   rbuergi

   on Sep 22, 2026

   Last edited by rbuergi

   Contributor

   More actions

   This ticket is a FOLD of at least three different terminals, so its 1074 cannot verify any fix — and its biggest share is not a mesh defect

   Grouped as part of the area:mesh timeout cluster. Read every retained sample's inner exception rather than the outer IncidentWriteFailedException, and this one fingerprint covers three distinct roots:

   terminal in the samples	what it means	whose defect
   TimeoutException: … no initial state arrived … within 30s → BaseStateTimeoutException — "NO change item at all reached the mirror"	nothing arrived inside the bound	unresolved: the owner's activation, its routing, or its initial read — the message itself says all three stay in scope
   MeshNodeStreamException: Host is shutting down, cannot route to 'Admin/_LogIncident/…'	the write was issued during host shutdown	the caller. The platform answered correctly
   TimeoutException: The operation has timed out. nested under the 30 s (the 2026-08-30 Npgsql window)	store unavailable	storage/cluster, transient

   Counting the samples that are quoted in the body: the Host is shutting down terminal appears 6 times of 10 (782ef3ac96819798 ×1, e4a97855ab595beb ×2, dce323795bb2ee6f ×3). That is the largest share of the visible population, and it is not a defect in the mesh at all — a write refused at shutdown is the routing layer answering truthfully. What is wrong is that LogIncidentControlPlane keeps issuing reconcile writes while the host is stopping and then reports the refusal at Error, which mints an incident on the way down. That is a classification fix, and it lives in the caller.

   Consequence for verification, and it is the reason this matters: a fix for any one of these three cannot be judged against this ticket's occurrence count, because the other two keep climbing. The honest form is per-terminal — the terminal you fixed must be absent from samples taken after the fix reached the replica — and a fix that changes the message also changes the fingerprint, which freezes this counter for a reason that is not the fix working.

   Where the work is

   The log site is MeshWeaver.Observability.LogIncidentControlPlane. src/MeshWeaver.Observability does not exist in core (ls src/MeshWeaver.Observability → no such directory; no LogIncidentControlPlane anywhere under src/ or memex/), so the Observability plugin and both halves of the actionable work — not writing during shutdown, and not reporting a shutdown refusal as a failure — are in MeshWeaver.Plugins.

   Core's half is the terminal vocabulary itself, and one of the three is now traced to a root: see #1186 / PR #5165 for the case where a store fault inside a query provider arrived downstream as a complete answer and was reported as a node that does not exist.

   Relationship to #5087 — a SPLIT, and they should stay separate

   Both tickets carry the footer "this incident inherited a30c09f918c68632". Same ancestor fingerprint, same log site (Incident …: Comment could not write Admin/_LogIncident/…), same outer exception — the current identity function separated them. That is a split, not two independent faults.

   But they are split on the terminal, and #5087's (InvalidOperationException: this hub's mirror ended without ever carrying the node's state — the mirror COMPLETED without state, from Workspace.AcquireRemoteStreamUnchecked) is distinct from all three above. So closing one into the other would bury a fourth root. Keep both, cross-referenced by terminal rather than by title. Recorded that way on #5087 too.

   Disposition

   Leave open, re-scoped: this is a reporting/classification ticket whose largest visible share is a shutdown-time write, and the work is in MeshWeaver.Plugins. It should be SPLIT per terminal rather than fixed as one thing.

   sev:L stands — the writes are retained and retried, so the cost is delayed annotations.

   What I did NOT establish

   • I did not count the full 1074 by terminal — only the 10 retained samples quoted in the body. The 6-of-10 shutdown share is a sample proportion, not the population's.
   • I did not verify against MeshWeaver.Plugins' main that LogIncidentControlPlane still issues writes during shutdown; that the terminal reaches it is established from the log line, the caller's behaviour is not.
   • The body's own note — "that retry model depends on a LATER change to the incident collection; for a quiet incident that may never come. Worth verifying that assumption in the code" — I did not verify. It is a Plugins read.
   • Nothing here attributes any individual occurrence to any root.

   Grouping method committed as Doc/Architecture/OneLogSiteManyTerminals (PR #5170).
4. rbuergi commented on Sep 22, 2026

   

   rbuergi

   on Sep 22, 2026

   Last edited by rbuergi

   Contributor

   More actions

   Follow-up: core already gives the caller everything needed to classify the shutdown share, so that half is a Plugins defect by measurement, not by elimination

   My previous comment said the Host is shutting down share — 6 of the 10 retained samples — is a caller defect. Here is why that is a measured statement rather than "it is not in core".

   The sentence is produced by the routing layer and is already classified transient

   src/MeshWeaver.Connection.Orleans/OrleansRoutingService.cs:428 and :606 are the only producers of Host is shutting down, cannot route to {address}. And src/MeshWeaver.Messaging.Contract/ShutdownNack.cs is a whole type devoted to making these refusals machine-readable. It states the contract in its own words:

   🚨 THE WORDING IS CONTRACT, not prose. The mesh classifies delivery failures by their MESSAGE TEXT as well as by ErrorType, and this sentence must be matched as TRANSIENT by MeshNodeStreamCache.IsTransientOwnerFailure, OrleansRoutingService.ClassifyRoutedFailure and AreaErrorClassifier.IsTransientHubFailure — so the caller RE-PROBES and lands on the fresh activation instead of taking a corpse's answer as final.

   and it distinguishes the two speakers explicitly:

   "No node found at 'x'", "No route to 'x'" and "Mesh/Host is shutting down, cannot route to x" all promise nothing, because none of them speaks for an address that reactivates … the routing layer never makes this address the subject of "is shutting down" — see IsAnsweredByOwner.

   So core provides three separate things a caller can use, all of them already wired:

   what the caller needs	what core provides
   is this transient?	ErrorType.ShuttingDown via OrleansRoutingService.ClassifyRoutedFailure
   who refused — the owner, or the routing layer?	ShutdownNack.IsAnsweredByOwner, derived from ShutdownNack.Banner rather than from an enumeration of sentences (the lesson of #3017)
   will this address come back?	ShutdownNack.ReactivationPromise — present on an owner-side refusal, deliberately absent from the host-shutdown routing refusal

   That third row is the operationally important one for this ticket: a host shutting down promises nothing about reactivation, so the correct caller behaviour is stop issuing, not retry — and a LogIncidentControlPlane reconcile that keeps writing into a stopping host and then reports each refusal at Error is minting incidents on the way down with a vocabulary available to tell it not to.

   What that settles and what it does not

   Settles: there is no core gap behind the shutdown share. The classification exists, is documented as contract, and is consumed by three named classifiers. The fix is for the caller to consult it — which is in MeshWeaver.Plugins, where LogIncidentControlPlane lives (src/MeshWeaver.Observability is absent from core; grep -rn LogIncidentControlPlane src memex finds nothing).

   Does not settle: I did not read LogIncidentControlPlane, so I have not seen what it does with the classification today — only that the classification is there to be read. And this covers one of the three terminals folded into this ticket. The BaseStateTimeoutException share and the Npgsql share are untouched by it, and the first of those is the one whose own message says all three of activation, routing and the initial read stay in scope.

   Nothing here changes the disposition: leave open, re-scoped, split per terminal.
5. rbuergi commented on Sep 22, 2026

   

   rbuergi

   on Sep 22, 2026

   Last edited by rbuergi

   Contributor

   More actions

   Correction to my own comments: the occurrence count here measures the LOG LINE, not an unrepaired failure

   Flagged by a session working area:ai, and it applies to this log site squarely — sharper than the caveat I wrote. main now carries the doc for the general pattern: Doc/Architecture/AFaultDoesNotStateTheVerdict — "a fault logged at Error claimed the outcome a later branch decides, so once a recovery path was added every benign blip filed a red line asserting held rollouts."

   Why it applies here, from this ticket's own text

   The log line is emitted at the refusal, and it says so in the same sentence:

   Incident X: Comment could not write Admin/_LogIncident/X, so nothing was recorded on the incident. It still carries its request and is picked up again on the next change to the incident collection.

   So the Error is written at a point where the outcome is not yet decided — the retry is a later branch. And this ticket's own Impact section records that the retry worked in the reported case: "the request was retained, the target incident node was written successfully again ~90 seconds later, and the control plane created this incident's own node seconds after that — mesh writes were healthy again immediately."

   So an occurrence is a refused ATTEMPT that was probably then retried and probably then landed. It is not a lost annotation, and the count is not a count of damage.

   What that changes, and what it does not

   Changes: any reading of the form "N occurrences, so this is still broken" is unsound at this log site, in both directions. A fixed root can keep this counter climbing for ever if anything still produces one transient refusal, and a recurrence check that reopens on the counter is reading the log statement's position in the code, not the defect. My earlier comment said the count cannot verify a fix; the stronger and more useful statement is that it cannot establish breakage either.

   Does not change: the grouping, or the disposition. Reading the retained samples' inner exceptions is unaffected — that tells you which terminals this fingerprint folds, which is a fact about the code paths reached, not about outcomes. And it does not soften the classification finding; it strengthens it. An Error line written before the retry that repairs it is precisely a fault claiming a verdict a later branch decides, which is the defect AFaultDoesNotStateTheVerdict names. The right report here is one that either waits for the retry's outcome or states plainly that it is reporting an attempt.

   What I did NOT establish

   • I did not read LogIncidentControlPlane (MeshWeaver.Plugins — absent from core), so I have not seen where its LogError sits relative to the retry, only that the message it emits asserts an outcome the retry can change. The area:ai finding is about [ThreadExec], a different site with the same shape; I am applying the pattern, not a measurement of this one.
   • I did not establish what fraction of these occurrences were repaired by a retry. One is documented in this ticket (the ~90-second one); the rest are unmeasured, which is the whole point — nobody can tell from the count.
   • The proportion I quoted earlier (6 of 10 retained samples carrying the Host is shutting down terminal) is a sample proportion of the retained ring and is unaffected by this: it counts which code path was reached, not whether the write eventually landed.
6. rbuergi

   added a commit that references this issue on Sep 22, 2026

   docs: a count whose log statement precedes the recovery branch establ…

   ...

   968dd28
7. rbuergi commented on Sep 24, 2026

   

   rbuergi

   on Sep 24, 2026

   Contributor

   More actions

   Two of the three terminals now have a merged fix. Both are awaiting roll.

   Per terminal, following the split recorded above:

   terminal	status
   MeshNodeStreamException: Host is shutting down, cannot route to … (6 of 10 retained samples)	Fixed in the caller: Systemorph/MeshWeaver.Plugins#2340 (merged). LogIncidentControlPlane.Run starts no transition once hub.IsLeaving() is true (hub shutting down, or ApplicationStopping). The request stays on the node for the replica that replaces this one. Any recoverable failure after the host began leaving (a reconcile, or a failure before the claim) records nothing and reports at Information. A stranded transition stays Error. Test ALeavingHostStartsNoIncidentWriteTest has two negative controls. LogIncidentUnwritableIncidentTest still shows Error on a host that is staying.
   BaseStateTimeoutException, "NO change item at all reached the mirror", 30 s	Core half: #5623 (merged 2026-09-23 22:30Z, 9dba8f8744). A grain on a stopping silo answered within about 1 ms, and its own router refused the answer, so the subscriber waited out 30 s. It now refuses and hands off with MigrateOnIdle. The retained samples fit that: 2026-09-17 14:25:47Z on d77fd7c88-x7nzm comes one minute after that pod's own shutdown refusals, and 2026-09-20 07:25:00Z on 78c786fcf6-bs4fn comes 30 s after 65f8c68995-z27tq began refusing. Both are roll windows.
   Npgsql TimeoutException (2026-08-30)	store unavailable, transient. Untouched.

   Deployed state, measured now (/api/version):

   • memex.meshweaver.cloud: 9a61fed964
   • memex.systemorph.com: 39f49494eb

   Both predate 9dba8f8744, and no image carries Plugins#2340 yet, so every share is awaiting roll.

   How to verify after the roll. Per the earlier comment, this counter cannot verify any fix. The check is per terminal: no Host is shutting down terminal at this log site, and no roll-window BaseStateTimeout, in samples from a replica that carries both fixes.

   Not established: I matched the BaseStateTimeout samples to roll windows by timestamp only. I did not read the owner silo's stop time for them. I also did not verify the body's note that a quiet incident may never be retried; that is unchanged by this work.
8. rbuergi commented on Sep 24, 2026

   

   rbuergi

   on Sep 24, 2026

   Contributor

   More actions

   Post-roll sweep 2026-09-24 ~06:50Z: the fix is not live on either portal yet

   What is running.

   • memex-cloud: /api/version returns 9a61fed964. Ops/Status/memex-cloud (06:42:00Z) shows 4/4 replicas on 3.0.0-ci.9263. CD run 9263 built core 9a61fed964 with Plugins 41bcfe968f.
   • memex: /api/version returns 39f49494eb. Ops/Status/memex shows the serving pods on 3.0.0-ci.9218, which is core 39f49494eb with Plugins e9d9283abe. A third pod, on ci.9260, is unready, so the roll is converged: false.

   This issue's fix: Plugins#2340 (68906340af). It is not an ancestor of either running build (git merge-base --is-ancestor, exit 1).

   Where it is. 3.0.0-ci.9287 (CD run 35956222018) is core bf5ac85526 with Plugins c4795a8d5b, and it contains this fix. The registry has sealed it. memex-cloud's self-updater logged [SelfUpdate] handing 3.0.0-ci.9287 to the control lane (https://memex.systemorph…) at 06:47Z. So the roll now waits on the control instance, which is an owner step and was not done here.

   Still open. It closes on a post-roll reading once a ci.9287-or-later image serves the namespace this fingerprint fires in.
9. rbuergi commented on Sep 24, 2026

   

   rbuergi

   on Sep 24, 2026

   Contributor

   More actions

   Post-roll reading on memex-cloud: quiet there, but this fingerprint's home is memex, which still runs pre-fix code

   Image. memex-cloud (memex.meshweaver.cloud) serves 3.0.0-ci.9291. /api/version returns core 140c5fb7ae, and Ops/Status/memex-cloud (sampled 14:49Z) shows all 4 replicas on that tag, converged: true. The Plugins half is b3601fecbe: the portal-ai build job of CD run 35970157348 (job 107542528900) logs HEAD is now at b3601fe. The first pods of this ReplicaSet (memex-portal-deployment-589ff8f895) started at 08:41:05Z and 08:42:17Z. More pods of the same ReplicaSet, on the same image, joined later (gqsfn at 14:06Z, vw5jj at 14:10Z, and others). The control instance memex.systemorph.com still serves core 39f49494eb.

   Window and method. I ran Logs InstanceActions on memex.meshweaver.cloud between 14:43Z and 14:54Z, with sinceMinutes 370/375, filtered to this ReplicaSet with | pod=~"memex-portal-deployment-589ff8f895-.+". That is ≥ 6 h of this image only, from 08:41Z to about 14:50Z. Coverage control: Ops/logs-memexcloud-20260924-close6h-c-bootcoverage (warm-up complete) lands 9 lines from 9 pods of this ReplicaSet. The earliest are ghgjt, vn49b and tp564 at about 08:51Z, so the window reaches back to the first boots.

   Fix: Plugins#2340 (68906340af). It is in memex-cloud's image. It is not in memex's, which runs core 39f49494eb + Plugins e9d9283abe (merge-base --is-ancestor exit 1).

   reading (memex-cloud, fixed image)	query	lines
   fingerprint	|~ "(?i)IncidentWriteFailedException|could not write Admin/_LogIncident" (Ops/logs-memexcloud-20260924-close6h-f-incidentwrite)	0
   positive control	|~ "(?i)LogIncidentControlPlane" (…-c-logincident): 9

   Every recorded occurrence was on memex, so the memex-cloud zero is supportive only and does not verify the fix. Stays open until memex runs an image containing the fix and a ≥ 6 h reading there is clean.
10. rbuergi commented on Oct 8, 2026

    

    rbuergi

    on Oct 8, 2026

    Contributor

    More actions

    Not closable yet. The BaseStateTimeout terminal (the per-node hub's mirror never got an initial snapshot) recurred 9 times on 2026-10-05 16:26–23:37Z on fixed images (pods 6cfcf78898-gkkdv, f86c77f48-twzk9), under the new fingerprint f1e8bef1a3761721 folded into #5478. A Host is shutting down write also hit the stranded path on 2026-09-27 06:27Z (15e59ffbf98b9794), where the request was already consumed. The identity change moved the counter to other fingerprints, so this one going quiet verifies neither fix. Related, not a duplicate: #5087 (closed, fixed by #5640).

    _{sev:L sweep, Claude Opus 5.5 (agent session).}
11. rbuergi commented on Oct 11, 2026

    

    rbuergi

    on Oct 11, 2026

    Contributor

    More actions

    Still occurring as of today. Stays open; this is now the only open issue for this terminal.

    Where the count went. The incident identity changed, so this fingerprint (c94d1352613d70de) went quiet while the same line kept firing under f1e8bef1a3761721, which files into #5478. #5478 is closed. So the recurrences below are being commented onto a closed issue, and this one is the open record.

    Reading — Admin/_LogIncident/f1e8bef1a3761721, memex.systemorph.com, 2026-10-11 ~11:20Z. 34 occurrences, first 2026-10-05 16:26Z, last 2026-10-11 03:57:20Z, namespace memex, 7 pods. Every sample is the BaseStateTimeoutException terminal ("NO change item at all reached the mirror inside the bound", 30 s). The retained samples fall in two clusters:

    when	pods	what else the samples show
    2026-10-09 20:43:05Z → 20:44:05Z	4 (5f6b9fc9b-hrdhq, 845898d964-f6wm5, -ptszc, -mwzl7)	two ReplicaSets at once, which suggests a roll in progress
    2026-10-11 03:55:20Z → 03:57:20Z	1 (6749fbfc59-hdlgz)	the same sample carries node kernel connect failed (Error: connect: no ack within 30s) from that pod

    Admin/_LogIncident/254804a0aad8b746 ([ROUTE] Directed delivery to pod hub … failed — surfacing Failed DeliveryFailure, TimeoutException) was last written at 04:14:53Z, the same second as this incident's node, so the watcher folded both in one pass. I did not open it to compare its sample times with the second cluster.

    What that points at, and how far. The pods named are the WRITERS (the line is logged by the caller). In the second cluster the writer itself was impaired: it could not reach its own node kernel in the same seconds. In the first, four writers on two ReplicaSets failed within a minute. Neither looks like one incident node failing to hydrate from a healthy store; both look like a process-level stall or a roll. Which pod OWNED the incident hubs in either cluster is not in these samples. The Npgsql and Host is shutting down terminals from the earlier split do not appear in these samples. That is a reading of ten retained samples, not of the 34 occurrences.

    Not established:

    • Whether hdlgz was stalled at 03:55Z, and by what (GC, thread pool, or a wedged turn). I could not read its log: the log store refused my four Logs actions today with 429 too many outstanding requests (ProvisionPlan log-ingestion run drops user identity on write-back (0 of 47 entries landed) and its Loki query phase also times out / hits 429 #5717; A LogStore pool: range reads against the log store run one at a time per process #6481 addresses that).
    • Whether the 10-09 cluster's owners were on the leaving ReplicaSet. fix(orleans): a grain on a stopping silo refuses and hands off instead of answering into the void (#5256) #5623 was meant to make a stopping silo refuse and hand off; if these owners were leaving pods, that path still has a hole.
    • The body's note that a quiet incident may never be retried.

    Next reading, once the log store answers: {namespace="memex",pod="memex-portal-deployment-6749fbfc59-hdlgz"} for 2026-10-11 03:50Z → 04:00Z, looking for Platform stalled, thread pool execution stalled and the pod-hub delivery timeouts.

    _{Fleet issue wave, Claude Opus 5.5 (agent session).}
12. rbuergi commented on Oct 11, 2026

    

    rbuergi

    on Oct 11, 2026

    Contributor

    More actions

    The "next reading" is taken: no incident hub failed to hydrate. Both clusters are a wedged silo (#6432), and the Error line is fixed in Systemorph/MeshWeaver.Plugins#3383

    The log store answered today. Two governed Logs actions on the control instance, both read-only: Ops/Actions/w21-4962-memex-hdlgz-0355-b and Ops/Actions/w21-4962-memex-ptszc-1009-2043.

    Cluster 2 — 2026-10-11 03:55Z, writer 6749fbfc59-hdlgz. The subscribe never left the writer's pod.

    • 03:57:19.256Z, [STALE-CALLBACK] cache/WeXQmrBhRESqawZSwzPp6Q: the SubscribeRequest for Admin/_LogIncident/31f12aedfc4bbbc8 has the trail POSTED → RECEIVED runLevel=Started → ENQUEUED → QUEUED queue=main depth=182, all on the pod's own cache/… hub, and nothing after it. No ROUTED, so the turn that would forward it had not started more than 30 s later.
    • 03:56:44Z, same hub: another SubscribeRequest in the same state at depth=153, 53 s old. The queue grew and did not drain. pool threads=24 pendingWork=0, so this is not pool starvation.
    • 03:56:50.769Z: [UpdateRemote] ERROR … type=BaseStateTimeoutException, then the IncidentWriteFailedException line.
    • This is the boot that ends in the 04:14Z eviction listed on Silo 5j9gx marked Dead by peers 26s after its last heartbeat and terminated itself — recurrence of the false-positive membership kill (#3053) #6432. The same pod logged Response did not arrive on time in 00:00:30 … messagehub/Provider/_Activity/import-manifest … executing for 00:00:25 at 03:55:01Z and the CreateNode lifecycle stalls at the validators stage with no verdict — a validator never answers within its window during the platform-startup boot-activity create #6391 create stalls at 03:53–03:58Z.

    Cluster 1 — 2026-10-09 20:43Z, writer 845898d964-ptszc. The owner was on the writer's own pod, and it was not the store.

    • [UpdateQueue] FAILED path=Admin/_LogIncident/90565fcee7617b58 at 20:41:07Z, 20:42:07Z, 20:43:09Z and 20:43:49Z.
    • 20:43:54Z, logged by ptszc itself: Dropping UnsubscribeRequest for stream … on hub Admin/_LogIncident/90565fcee7617b58: no synchronization hub found … never-created sync hub, six of them. So the incident's hub was hosted on ptszc, and the subscribes it was sent had created no stream by the time their unsubscribes arrived.
    • 5f6b9fc9b-hrdhq, the other writer in this cluster, was booting at 20:36Z (CreateNode lifecycle stalls at the validators stage with no verdict — a validator never answers within its window during the platform-startup boot-activity create #6391's first sample) and evicted at 20:44:12Z (Healthy Orleans silo falsely voted dead by peers and terminated itself during cluster roll-churn #6395).

    What that settles. For the terminal this issue still has open (BaseStateTimeoutException, "NO change item at all reached the mirror"):

    suspect in the abort message	reading
    the node is absent	No. All six incidents named in the retained samples exist and carry later versions (13 to 140) on memex.systemorph.com.
    the store read failed	No store fault in either pod's lines for these paths.
    the owner hub never hydrated	Not what happened. In cluster 2 the request never reached any owner. In cluster 1 the owner's hub never created the stream.

    Both are a silo that accepts work and dispatches none of it. #6432 has the dump: a convoy on the container's shared-instance lock, with MeshNodeStreamHandle.UpdateRemote and the grain activation's configuration resolve among the waiters. That is the root cause of these 34 occurrences, and it is tracked and being worked there (#6455 merged, #6490 open). Nothing in the write path can produce a NACK while the thread that would post it is parked on that lock, so there is no bound to change here.

    What was this issue's own defect, and is now fixed in Systemorph/MeshWeaver.Plugins#3383. The control plane reported a write it re-attempts by design at Error. Each attempt became an incident and a recurrence comment about a write that then landed. A recoverable write failure is now a Warning. The Error is an incident still unwritable after five minutes, once per outage, also raised by the comment-debt clock for an incident nothing re-attempts. That last part answers the body's open note about a quiet incident. A stranded transition stays Error. Tests: ARecoverableIncidentWriteFailureIsNotATicketTest; with the two lines put back to LogError, 6 of 8 cases fail.

    Why the recurrences landed on closed #5478. By design, not a bug: one log site folds into one ticket (siteFold 8ba036a09364b28a), that ticket was #5478, and a ticket closed as a duplicate is left closed and commented (LogIncidentFiler.CloseReasonThatOutranksTime). #5478 was closed as a duplicate of #5315 for a different terminal of the same site (the provider stall). The Plugins change removes the source: these attempts no longer file at all.

    Not established.

    • Which silo ptszc's incident hub was waiting on at 20:41–20:43Z. It came unstuck 23 s before hrdhq was evicted; that is a correlation.
    • What the cache/… hub's running turn on hdlgz was blocked in. The dump on Silo 5j9gx marked Dead by peers 26s after its last heartbeat and terminated itself — recurrence of the false-positive membership kill (#3053) #6432 is from another pod.
    • The 2026-10-05 occurrences (gkkdv, twzk9) were not re-read; Triage reconcile cannot record results on log incidents when storage query providers stall past the fan-in timeout #5478 records them as GC-bound and as a forwarded subscribe with no handler run.

    Disposition. Stays open until #3383 is on memex: the check is that no could not write Admin/_LogIncident line appears at fail: there outside a stranded transition. The wedge itself closes on #6432.

    _{Fleet issue wave, Claude Opus 5.5 (agent session). Read-only on the deployment.}
13. rbuergi commented on Oct 11, 2026

    

    rbuergi

    on Oct 11, 2026

    Contributor

    More actions

    Still owed (2026-10-11 19:55Z): Plugins#3383 is not in the running image (3.1.10565 carries Plugins 95439f8). It is in 3.1.10570 (Plugins 4b87135), which is in CD now. You can check it after that roll: a recoverable incident write failure should log at Warning and file nothing here.
14. rbuergi commented on Oct 13, 2026

    

    rbuergi

    on Oct 13, 2026

    Contributor

    More actions

    Closing: the fix is merged and live, and the incident has not moved in more than three weeks.

    • Fix. MeshWeaver.Plugins#3383 merged 2026-10-11 19:22Z. A recoverable incident-write failure logs at Warning and files nothing. It is in every memex image since 3.1.10570, and memex now runs 3.1.10716 (core ec7763883b).
    • Incident Admin/_LogIncident/c94d1352613d70de, the identity this issue was filed from (IncidentWriteFailedException), read on memex.systemorph.com 2026-10-13 ~15:12Z: lastModified is 2026-09-20 07:27Z. No occurrence has been recorded since, across roughly 30 boots in the last 24 h alone.

    Not separately measured: a Logs read of LogIncidentControlPlane over 24 h (Ops/Actions/wmlb-4962-memex-incidentwrite-24h) was cut at 200 lines. That logger is busy with ordinary work, so I did not use the read as a zero. The incident record is the instrument here.
15. rbuergi

    closed this as completedon Oct 13, 2026