LogIncidentControlPlane loses the first occurrence of every new incident — delivery races node registration #2167

What is failing

MeshWeaver.Observability.LogIncidentControlPlane records each detected log fingerprint by sending a message to Admin/_LogIncident/{fingerprint}. On the first occurrence of a fingerprint the send is dispatched before the incident node exists as a routable target, and the mesh router rejects the delivery with DeliveryFailureException: No node found at 'Admin/_LogIncident/{fingerprint}'. Closest ancestor is 'Admin'. The control plane then tries to record that failure — also on a not-yet-existing fingerprint — and the cycle repeats a few times until the retry budget runs out. The original incident is dropped.

Probable cause — high confidence

Race between send and node materialization on first-seen fingerprints. Evidence:

• The exception message is unambiguous about where the router looked and what it found (remainder='_LogIncident/{fingerprint}'), so the routing layer is not misconfigured — the target simply isn't there yet.
• Both fingerprints that appear in the samples (90a8113061a020a2, 6e643370636ed34a) are absent from the current Admin/_LogIncident/* listing, i.e. they were never successfully recorded. Meanwhile 30+ other incident nodes at the same parent path exist and are being updated normally — so the collection itself is fine, only the create-on-first-write path is broken.
• The bursts are tight (4 lines in 6s on one pod; 3 lines in <1s on another) — the shape of a fixed retry loop hitting the same missing target, not a slow-drip infrastructure problem.
• Two pods, two different fingerprints, two days apart — this is the code path, not a single-pod incident.

Impact

Low volume, meaningful correctness hole. 7 occurrences over ~2 days across 2 pods (memex-portal-deployment-*). Not an outage, but every genuinely new fingerprint in production is currently lost by the observability system — only recurring incidents that happen to survive the first-write race end up on the incident board. This silently biases triage toward already-known problems.

Where to look

• MeshWeaver.Observability.LogIncidentControlPlane — the recording path that posts to Admin/_LogIncident/{fingerprint}. It needs to either (a) ensure the node exists (idempotent create) before sending, or (b) use a send mode that materializes the target on demand.
• The self-referential guard: when the record-failure path itself throws DeliveryFailureException against Admin/_LogIncident/*, the control plane should short-circuit rather than re-enter its own recording loop — otherwise a single unroutable fingerprint amplifies into a burst of identical failures (visible in the samples).
• Related but distinct existing incident: Admin/_LogIncident/83ea6fdcf6fa198a (ContentRoute: DeliveryFailureException) — same exception type, different call site; keep separate.

Evidence

2026-08-22 13:47:37Z memex-portal-deployment-d7dd9796b-66g44 fail: MeshWeaver.Observability.LogIncidentControlPlane[0]
      Incident 90a8113061a020a2: could not record the failure
      MeshWeaver.Messaging.DeliveryFailureException: No node found at 'Admin/_LogIncident/90a8113061a020a2'. Closest ancestor is 'Admin' (remainder='_LogIncident/90a8113061a020a2').
2026-08-22 13:47:38Z memex-portal-deployment-d7dd9796b-66g44 fail: MeshWeaver.Observability.LogIncidentControlPlane[0]
      Incident 90a8113061a020a2: could not record the failure
      MeshWeaver.Messaging.DeliveryFailureException: No node found at 'Admin/_LogIncident/90a8113061a020a2'. Closest ancestor is 'Admin' (remainder='_LogIncident/90a8113061a020a2').
2026-08-22 13:47:38Z memex-portal-deployment-d7dd9796b-66g44 fail: MeshWeaver.Observability.LogIncidentControlPlane[0]
      Incident 90a8113061a020a2: could not record the failure
      MeshWeaver.Messaging.DeliveryFailureException: No node found at 'Admin/_LogIncident/90a8113061a020a2'. Closest ancestor is 'Admin' (remainder='_LogIncident/90a8113061a020a2').
2026-08-22 13:47:43Z memex-portal-deployment-d7dd9796b-66g44 fail: MeshWeaver.Observability.LogIncidentControlPlane[0]
      Incident 90a8113061a020a2: could not record the failure
      MeshWeaver.Messaging.DeliveryFailureException: No node found at 'Admin/_LogIncident/90a8113061a020a2'. Closest ancestor is 'Admin' (remainder='_LogIncident/90a8113061a020a2').
2026-08-24 17:58:49Z memex-portal-deployment-58c7bb4b9d-97pjf fail: MeshWeaver.Observability.LogIncidentControlPlane[0]
      Incident 6e643370636ed34a: could not record the failure
      MeshWeaver.Messaging.DeliveryFailureException: No node found at 'Admin/_LogIncident/6e643370636ed34a'. Closest ancestor is 'Admin' (remainder='_LogIncident/6e643370636ed34a').
2026-08-24 17:58:50Z memex-portal-deployment-58c7bb4b9d-97pjf fail: MeshWeaver.Observability.LogIncidentControlPlane[0]
      Incident 6e643370636ed34a: could not record the failure
      MeshWeaver.Messaging.DeliveryFailureException: No node found at 'Admin/_LogIncident/6e643370636ed34a'. Closest ancestor is 'Admin' (remainder='_LogIncident/6e643370636ed34a').
2026-08-24 17:58:50Z memex-portal-deployment-58c7bb4b9d-97pjf fail: MeshWeaver.Observability.LogIncidentControlPlane[0]
      Incident 6e643370636ed34a: could not record the failure
      MeshWeaver.Messaging.DeliveryFailureException: No node found at 'Admin/_LogIncident/6e643370636ed34a'. Closest ancestor is 'Admin' (remainder='_LogIncident/6e643370636ed34a').

_{Opened automatically from Admin/_LogIncident/3b6b24e5f282f36b. Recurrences are folded into this issue rather than opening new ones.}