Abandoned cycles written before #2803 carry status=SUCCESS, so status-filtered analysis under-counts them silently

[erikdarlingdata]

Abandoned cycles written before #2803 carry status = SUCCESS, so any analysis filtering on status reports those days clean

An analysis trap rather than a live defect — the write path is already correct. Filing it because it silently voided the first pass of an investigation today and will do the same to the next one.

The shape

EnumeratedCollectorDriver classifies a run as abandoned ? AbandonedStatus : "SUCCESS", with AbandonedStatus = "ABANDONED". That distinction arrived in #2803 (26403150, "Give a wall-clock-budget-abandoned cycle its own collection_log status"). Rows written before it carry the abandonment's own error_message — "wall-clock budget (120s) reached; cycle abandoned" — together with rows_collected: 0 and status: SUCCESS.

So a query shaped like WHERE status <> 'SUCCESS', or an MCP read filtered the same way, reports those windows as 100% clean. Observed concretely: six such rows on one server, and the omission hid roughly a third of that server's abandonment events from a first scan. They were only recovered by filtering on error_message instead.

Why it is worth a guard rather than a note

The rows are historical and cannot be rewritten — collection_log is an append-only hypertable and the abandonment already happened. So the durable fix is not to the data but to the readers, and the risk is that each new analysis rediscovers this the hard way. Two options worth weighing:

• Derive "abandoned" from the conjunction that is true across the whole series — rows_collected = 0 together with the wall-clock message — rather than from status alone, wherever an abandonment count is computed. That works on both eras.
• Or bound the claim explicitly: have any surface that counts abandonments state the cutoff, so a reader knows a pre-Give a wall-clock-budget-abandoned cycle its own collection_log status #2803 window is under-counted rather than clean.

The first is preferable if anything in the read path already counts abandonments, because a filter that is wrong for one era of a retention window is wrong silently and in the reassuring direction.

The general shape, which is the reusable part

This is the same defect class as an empty column read as evidence of a disabled feature: a filter that is correct against current writes and silently wrong against older ones, failing in the direction that looks healthy. A count of zero from a status filter is indistinguishable from a genuinely quiet period, and nothing in the response marks the boundary.

Provenance

Found while root-causing the 120 s abandonment tail. The status-classification code path and #2803's commit were both read on current dev; the six affected rows and the one-third undercount were observed in collected data.

[erikdarlingdata]

Closed by PR #2930. The fix is two-sided, and the issue above understated the defect: a pre-#2803 abandoned row was not merely uncounted as abandoned, it was positively counted as a success — and Success is a displayed column sitting beside Abandoned. Fixing only the abandonment count would have made a pre-era window render Success + Abandoned > Runs on screen.

The most consequential correction: the wall-clock message exists in two forms, not one. query_store uses a 600 s budget (PerDatabaseWallClockBudget = 10 minutes), not 120 s, so a literal match on the (120s) sentence quoted above would have missed every query_store abandonment. The landed predicate is a LIKE on the stable part, derived from EnumeratedCollectorDriver.WholeCycleBudgetNoteFormat rather than from the example. A mutation replacing it with equality on the 120 s sentence counts 2 of 3 — that is the receipt.

Five reads were misclassified and are fixed: DarlingDataReader.CollectionHealthSql, DarlingFleetReader.FleetCollectionHealthSql, ViewerDataService's health and fleet reads, and LocalDataService.CollectionHealthSql — all five feed CollectorHealthClassifier's 0.5% WARNING band.

Two era-fragile reads were examined and deliberately left alone, which is worth recording: DarlingSelfAlertEvaluator's last_success/recent_success are server-scoped across ~40 collectors with no collector filter and last_success is a MAX any newer success supersedes, so one abandoning collector cannot make a server read fresh — changing it would move live alert firing on historical data for no reachable gain. And last_note/note_count are gated status = 'SUCCESS', which is exactly why the message was recoverable at all; suppressing it would remove the evidence.