Repository navigation
has_collector_errors drives the fleet card's health band and its semantics cannot be determined from the surface — four wrong inferences by two independent agents, and it reads false while errors sit inside the card's own window #3098
Description
What the field does
Every card get_fleet_overview returns carries has_collector_errors, and it participates in the card's band — the Healthy / Warning / Critical value a user reads first and the only one most consumers look at. A card can be banded on it while every other severity field says Healthy.
The problem
No consumer can tell what it means from the output, and the four readings that the output supports are each falsified by a different observation. These are all from live stores over one day:
It is not "a collector is currently failing." Four cards on a SQL Server store read has_collector_errors: true while every collector on each of them reported status: HEALTHY with a current last_success, failed_collector_count: 0, and healthy_collector_count: 39.
It is not scoped to the card's own window. Each card carries window_start/window_end exactly one hour apart. The store's collection_log dates the errors behind those four cards to a spread of roughly eleven hours on the previous day — between 15 and 26 hours before the card that reported them.
It is not the collector row's error count either. A card on a different store read has_collector_errors: false while the same response's collector row for that server carried errors: 2 over the seven-day window the row aggregates.
And it can read false while errors sit inside the card's own stated window. This is the sharpest one. A rollup generated at 14:28:27Z with window_start: 13:28:27Z returned has_collector_errors: false on all 43 cards, while the store holds four ERROR rows at 14:01:57, 14:01:57, 14:01:59 and 14:02:07 — inside that window, on three of those same servers.
Two fields on one card disagree about the same subject
On the PostgreSQL-target store, the single non-Healthy card currently reads:
band: Warning
collector_severity: Warning
failed_collector_count: 1
has_collector_errors: false
The band is correct and comes from failed_collector_count. has_collector_errors says the opposite about the same server in the same payload. Whatever the two fields are measuring, a reader cannot reconcile them, and one of them is redundant or wrong.
Why this is worth a code change rather than a doc note
Two independent agents made four wrong inferences about this field in one day, each time from the output plus the surrounding response, and each time the wrong reading survived until a store query contradicted it:
- read as "current failing state" — falsified by cards flagging
truewith all collectors healthy; - read as "a one-hour window" — falsified by 15-to-26-hour-old errors setting it;
- probed as a schema column on the same surface (
information_schema.columns), returning a clean and meaningless zero because the neighbouringlast_error_atis a computed projection, not stored; - and used, on that basis, to reason about when a cluster of errors occurred — a conclusion that was wrong and had to be retracted.
That last one is the practical cost. The field invites a temporal reading it does not support, and the response carries no *_at or window value beside it to correct the impression.
Suggested direction, not a prescription
Any one of these would resolve it; the first is cheapest:
- Give it a timestamp companion.
collector_errors_sinceor alast_collector_error_aton the card would make the field self-describing, the same waylast_error_at/last_denied_at/denied_since_last_successmade the collector rows self-describing. - Make it agree with the window the card already publishes. The card states
window_start/window_end; scoping the flag to that span would make it derivable and checkable. - Remove it if
failed_collector_countandcollector_severityalready carry the signal. The disagreement above suggests they do.
Whichever is chosen, the acceptance criterion should be that a reader can determine the field's population from the response alone — because the current failure mode is not that the value is wrong, but that nobody can say what it is the value of.
Notes
- Observed on
3.6.0-nightly.20260906, schema rung 112, across three stores. - The collector-row surface is a good model for the fix:
last_error_at,last_denied_atanddenied_since_last_successlanded recently and made "read and found nothing" distinguishable from "could not read", which is the same class of ambiguity one level down.
Activity
erikdarlingdata commentedon Sep 6, 2026
The field's population is determinable from the source, and it is not errors
DarlingFleetReaderbuilds it like this:has_collector_errorsis derived from the freshness classification and nothing else.FlagsFortakes a freshness band; its sibling in the same class is namedFromFreshness. No error count, nocollection_logread, no window.The confirmation is in the same file: when the flag is true, the card's human-readable reason string appends
"collection stale"— not "collector errors."That single fact explains all four falsified readings
Each observation in the issue above stops being a puzzle:
truewhile every collector isHEALTHYwithfailed_collector_count: 0window_start/window_endlastCollectionagainstnow, which has nothing to do with the published windowerrorscountfalsewhileERRORrows sit inside the card's own windowThe two-fields-disagree case resolves the same way:
failed_collector_countreports failure andhas_collector_errorsreports staleness. They are not in conflict, they are answering different questions under names that suggest one.Which changes the fix
The suggestions above assume the field means what its name says and needs qualifying - a timestamp companion, or scoping to the published window. If it is a freshness flag, the defect is the name, and the cheapest correct fix is to make the name match the computation (the reason string already does: "collection stale").
That also reframes the acceptance criterion. The original - a reader can determine the field's population from the response alone - is still right, but the current failure is not under-specification. It is a name that misdescribes a correctly-computed value, which is why four inferences from the output were wrong: every one of them was a reasonable reading of the name.
Worth noting the shape, since this product's audience is people reading exactly these surfaces: a field whose name implies a population it does not have is worse than an unnamed one, because the name supplies a false answer to "what is this the value of" and nothing in the payload contradicts it.
Not verified
Whether renaming is viable for consumers already reading
has_collector_errors, and whether any consumer depends on the current name. That is a compatibility question this comment does not answer.Fixes #3098 (#3106)
erikdarlingdata commentedon Sep 6, 2026
Fixed by #3106, merged to
devat751f02f6fd5101e34db0b3740a449a9f09753553.has_collector_errorsis nowcollection_stale— matching the reason string the card already emitted, which is the strongest evidence of intent available: the code was telling the truth in prose while the field name contradicted it.A rename rather than a companion field or an alias. The old JSON name had exactly one repo-wide hit — its own declaration — and no reader in the dashboard, MCP instructions, READMEs,
llms.txtordocs/. Verified absent afterwards with positive controls, so the zeros are real reads rather than failed fetches. Lite keepsHasCollectorErrors, correctly: its ladder is a separate type fed from an actual error count, so one identifier was naming two different populations across the SKUs and now each matches its own.No companion timestamp was added, and that is the right call — the card already carries
last_collection, which is the flag's exact input. What was missing was a name connecting them, not a value.The strongest argument against keeping an alias did not come from the wrong-guess count. One reader, prompted to grep its own artifacts, found three standing wrong characterisations of this field — and at the third it had written "I am recording it as unknown, do not date from it rather than proposing a third theory" and then proposed a third theory in the next sentence. A name that keeps generating plausible theories keeps getting them even from a reader who has already decided to stop. An alias would have left that name published.
Breaking for an MCP client keying on the old field: it reads absent-and-falsy, which is the same failure mode removing the field would have had, so the rename is strictly better. The value is recoverable from
status, or fromlast_collectionagainstgenerated_at.Closing by hand: this repo merges to
devwhile the default branch ismain, so a closing keyword never fires.erikdarlingdata commentedon Sep 6, 2026
Merged is not deployed — the fleet card still publishes the old name, and reading it will mislead you
Recorded because a reader doing exactly what this issue's authors did will get a wrong answer, and there is nothing in the payload to prevent it.
State as of closing, measured rather than assumed: a live store's
get_fleet_overviewreturnshas_collector_errorson all 42 cards andcollection_stalezero times. Positive control on that read: 42 cards parsed, collection-related keys exactlyawaiting_first_collection, band, collector_severity, deadlock_collector_band, failed_collector_count, has_collector_errors, healthy_collector_count, is_online, last_collection, status.deve2b3ab64d(squash of #3106 at751f02f6f)3.6.0-nightly.20260906, built fromd54ad00a5— which predates the mergeWhy this needs saying rather than being obvious
A reader who queries the card and then reasons — which is precisely how this issue was filed — has two available readings and both are wrong:
dev.Nothing in the payload distinguishes them: there is no version stamp beside the field, and
generated_atdescribes the rollup rather than the build.And that reproduces this issue's own failure mode. The complaint here was a name whose population could not be determined from the surface. The remediation is real but unobservable at the surface where it was reported, so a state check there returns "not fixed" — correctly, for that box, and misleadingly about the change.
Practical consequence: verify this fix against
devby content, not against a running store. Agit grepfor both spellings across the tree returns exactly the six Lite files (which keepHasCollectorErrorsdeliberately — Lite's flag is genuinely fed from an error count) and zero underDarling/,PerformanceMonitor.Common/ordocs/.Note that an ancestry probe is the wrong instrument for confirming it landed: this repository squash-merges, so
git merge-base --is-ancestor 751f02f6f devreturns false and reads as "never merged." Compare content, or check the squash commit on the base branch.When the next nightly is installed, the card's field name changes. Anything keyed to the old name — including monitoring baselines and line counts — goes stale at the install boundary rather than at merge, and a discontinuity there is the deployment, not a finding.