A live query's early→live handoff dropped a change notification permanently by rbuergi · Pull Request #2295 · Systemorph/MeshWeaver

rbuergi · 2026-08-25T22:08:17Z

A live query's early→live handoff dropped a change notification — permanently

Every live children/descendants query opened two subscriptions on the adapter's change feed: an
"early" one that buffered into a backlog while !initialDone, and a "live" one attached inside the
initial-results callback, after which the early one was disposed.

IsolatedChangeFeed snapshots its observer list before delivering — it must, because
subscribers attach and detach mid-publish. So a write published in the instant before the live
subscription was attached is delivered against a snapshot holding only the early observer, and
by the time that delivery ran the callback had already set initialDone = true. The early
handler's if (!initialDone) threw the notification away; the live observer never saw it because it
was not in the snapshot; and nothing re-triggers the re-query. The row stayed invisible to that
subscription for its whole life — while a point read, and any fresh subscription, returned it
immediately.

A permanent miss dressed as a timeout, which is why no budget could ever have fixed it.

Measured, not inferred

This started as an investigation into whether cancelled agent rounds go unbilled — the writer
hypothesis #2244 pre-registered when its reader-side fix did not stop the WaitForUsage reds.
The writer is exonerated. Instrumenting the wait to read the node back at failure time, and
reproducing locally (MeshWeaver.AI.Test 4-wide with DOTNET_PROCESSOR_COUNT=4, one dotnet test
at a time, ~1 failure in 8 runs):

[USAGE-DIAG] leg1 emissions during the wait: 1
[USAGE-DIAG]   leg1 21:37:06.565 n=0 []
[USAGE-DIAG] listing(_Usage children) count=1   ← a FRESH listing, 3 ms later
[USAGE-DIAG]   listed …/_Usage/usage_model content={"inputTokens":137,"outputTokens":89}
[USAGE-DIAG]   cell e4533be4: status=Completed model=usage-model in=137 out=89 total=226
[USAGE-DIAG] POINT READ OK: …/_Usage/usage_model {"inputTokens":137,"outputTokens":89}

The satellite is written, with the right counts, keyed by the right model — on the Completed and
the Cancelled path. The test's own live subscription emitted exactly one snapshot, the empty
Initial, then nothing for the full 20 s.

Subscribing the same shared upstream under System (which makes WrapWithPerUserRls
short-circuit to the raw, unfiltered stream) exonerated the per-user RLS filter too — raw saw
exactly what filtered saw. Instrumenting the provider then named the culprit outright:

[QDIAG] SUBSCRIBE base=…/_Usage scope=Children
[QDIAG] INITIAL   base=…/_Usage items=0 backlog=0
[QDIAG] NOTIFY seen base=…/_Usage path=…/_Usage/usage_model match=True   ← EARLY observer, after the handoff

…and, for that base path, zero LIVE seen and zero REQUERY for the rest of the run.

The fix

One subscription for the query's whole lifetime. The backlog-vs-live routing decision is made
inside the same critical section that publishes the buffer, so a notification lands in exactly one
of the two and can never fall between them. The same shape existed in all four providers —
StorageAdapterMeshQueryProvider, PostgreSqlMeshQuery, PostgreSqlPartitionedMeshQuery,
CosmosMeshQuery — and is corrected in all four. It also removes the double-capture window the old
comments accepted, and puts Cosmos's disposal order right as a side effect.

No budget was widened, no retry or watchdog added.

The test

LiveQueryHandoffDropTest pins the window deterministically: it holds the query's scope walk open,
commits the write, takes the notification's fan-out snapshot while only the pre-Initial subscriber
exists, releases the walk, asserts the Initial is genuinely empty, and only then delivers.

Fails on the pre-fix code (verified by reverting the provider: 10 s, the assertion's message).
Passes on the fix in 43 ms.

MeshWeaver.Hosting.Test is 226/226 green.

Scope note

Related but not fixed here, and deliberately kept separate: a round whose provider never emitted
usage before the cancel/error records zero tokens, silently (a Debug-level no-op, no tokenizer
or estimate anywhere in the tree). That is real unbilled spend and it is #595, which I reopened —
its last comment said "Keeping this issue open, scoped to that estimate" and it was then
auto-closed as COMPLETED by a PR that shipped only the two surgical items.

Also noticed while reading the logs, not changed here: MeshQuery.InitialStallProbeDelay is 20 s
and ThreadTokenUsageTest.SettleBudgetMs is 20 000 — so in CI run 32894972304 the one warning
that would have named a stalled query provider fired ~30 ms after TEST END. A stall probe armed at
exactly the consumer's budget can never explain that consumer's timeout.

Root-causes the WaitForUsage flake family closed prematurely as #1040, #1812 and #2001 — each of
those diagnosed a different reader shape and none reached the notification drop underneath. Reported
on #1384 (the flake-population umbrella), which stays open.

…permanently Every live children/descendants query opened TWO subscriptions on the adapter's change feed: an "early" one buffering into a backlog while !initialDone, and a "live" one attached inside the initial-results callback, after which the early one was disposed. IsolatedChangeFeed snapshots its observer list BEFORE delivering — it must, because subscribers attach and detach mid-publish. So a write published in the instant before the live subscription was attached is delivered against a snapshot holding ONLY the early observer, and by the time that delivery ran the callback had already set initialDone = true. The early handler's `if (!initialDone)` threw the notification away; the live observer never saw it because it was not in the snapshot; and nothing re-triggers the re-query. The row stayed invisible to that subscription for its whole life — while a point read, and any fresh subscription, returned it immediately. Measured, not inferred. Instrumenting both observers reproduced it in MeshWeaver.AI.Test (4-wide, DOTNET_PROCESSOR_COUNT=4, ~1 in 8 runs): the notification arrives with match=True on the early observer AFTER "INITIAL items=0 backlog=0", and for that base path there is no live delivery and no re-query for the rest of the run. That is the long-running ThreadTokenUsageTest / ModelSubstitutionTest WaitForUsage red (#1040, #1812, #2001) — the token-usage satellite WAS written, with the right counts, on the Completed AND the Cancelled path; the reader's listing simply never learned of it. In the portal the same shape makes any live children listing (the chat token chip, a notification bell, a folder view) miss a node created in that window until something else changes under the same base path. One subscription for the query's whole lifetime now: the backlog-vs-live routing decision is made inside the same critical section that publishes the buffer, so a notification lands in exactly one of the two and can never fall between them. The shape existed in all four providers (StorageAdapterMeshQueryProvider, PostgreSqlMeshQuery, PostgreSqlPartitionedMeshQuery, CosmosMeshQuery) and is corrected in all four; it also removes the double-capture window the old comments accepted, and puts Cosmos's disposal order right. No budget widened, no retry, no watchdog. LiveQueryHandoffDropTest pins the window deterministically — it fails on the pre-fix provider and passes on the fix in 43 ms. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Copilot

Pull request overview

Fixes a race in live query subscriptions where a change notification could be dropped during the Initial→live handoff, causing a live children/descendants query to permanently miss a newly-created node until the subscription was restarted.

Changes:

Refactors live-query change-feed handling to use a single subscription for the query lifetime, switching from backlog buffering to the live requery buffer atomically.
Applies the same single-subscription handoff fix across in-memory adapter queries, PostgreSQL queries (both single and partitioned), and Cosmos queries.
Adds a deterministic regression test that pins the “snapshot-before-deliver” window and verifies the live requery still happens.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
test/MeshWeaver.Hosting.Test/LiveQueryHandoffDropTest.cs	New regression test that deterministically reproduces the dropped-notification handoff window.
src/MeshWeaver.Hosting/Persistence/Query/StorageAdapterMeshQueryProvider.cs	Uses one lifetime subscription and an atomic backlog→liveBuffer handoff to prevent permanent missed notifications.
src/MeshWeaver.Hosting.PostgreSql/PostgreSqlPartitionedMeshQuery.cs	Applies the same single-subscription backlog/live routing fix to partitioned PostgreSQL queries.
src/MeshWeaver.Hosting.PostgreSql/PostgreSqlMeshQuery.cs	Applies the same single-subscription backlog/live routing fix to PostgreSQL queries.
src/MeshWeaver.Hosting.Cosmos/CosmosMeshQuery.cs	Applies the same single-subscription backlog/live routing fix to Cosmos queries.
src/MeshWeaver.Documentation/Data/WhatsNew/2026-08-26-live-lists-stop-missing-a-just-created-node.md	Adds release-note entry describing the user-visible effect (live lists no longer silently miss a just-created node).

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

…iew) The gate that pins the Initial against the notification's fan-out snapshot ignored its wait's return value. A timed-out gate lets the scope walk continue on its own, which un-pins the two — the test would then pass having never held the window it exists to hold. It now records the timeout and the test asserts on it, so the run cannot silently verify nothing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

rbuergi · 2026-08-25T22:37:05Z

Controlled before/after on the reproducing loop

The deterministic LiveQueryHandoffDropTest is the proof (it fails pre-fix, passes post-fix). This
is the corroboration: the same 5-class filter, DOTNET_PROCESSOR_COUNT=4, one dotnet test at
a time, pre- and post-fix on the same machine.

build	iterations	`WaitForUsage`-family failures
pre-fix (first runs, machine under load ~40–100)	~25	3 (reproduced at iteration 3, 8 and 14 across three runs)
pre-fix control (same load as the post-fix runs)	70	1 — `CompletedRound_StampsTokensOnResponseCell_AndAccumulatesOnThread`
post-fix run 1	65	0
post-fix run 2	70	0

0 in 135 post-fix, 4 in ~95 pre-fix. Note the pre-fix control's hit is a Completed-round test —
more evidence the flake was never cancel-specific.

One unrelated failure, attributed rather than ignored

Post-fix run 1 also saw one
AgentToolCancellationTest.GetVersions_ParkedOnAStalledStore_UnwindsWhenTheRoundIsCancelled
(Expected 2 to be greater than 2 … cancelling must dispose the version read). It did not recur
in the further 70 post-fix iterations, and it is a static-counter timing assertion (VersionStore. Disposals must have advanced by the instant the OperationCanceledException is observed) with no
path to the change-feed handoff this PR touches. Recording it here rather than leaving it unsaid:
1-in-135, independent of this change as far as the evidence goes.

Note on the base

This branch is cut from ed4588222, where AiContentPackDriftTest.EveryFile_StillHashesToItsPinned ReconciliationPoint is red on main (content/ai/Skill/group.md changed in 123f6734c without
re-pinning AiContentPackSync.json). #2288 fixes it; once that lands I will merge main in rather
than investigate a red this diff cannot reach.

…f-drops-a-notification

github-actions · 2026-08-25T23:27:34Z