The fleet card reads a server dark for over 48 hours as Offline, not 'Awaiting first collection' (#3935)

[erikdarlingdata]

Closes #3935.

Why

The fleet card reads each server's newest collection from DarlingFleetReader.FleetLastCollectionSql. That read looks back 48 hours. A server dark for longer fell out of it and reached ClassifyFreshness as a null, and a null there means never collected. So the card said "Awaiting first collection" (Warning), and the Needs-attention reason said the same. The WPF viewer reads the same timestamp with no window and calls the server Offline. Only the viewer was right: the server collected once, then stopped.

The diagnosis in the issue holds. FleetReadsAreBoundedLivePostgresTests seeds a server dark for five days, and its card read IsOnline = null and awaiting first collection.

Erik's ruling (option 2): keep the 48-hour window. Tell "never collected" apart from "nothing in the window" using the registry row or the collection history, cheaply, and match the viewer's banding.

What changes

The last-collection read reports every enabled server, with its registration. FleetLastCollectionSql is still #3947's per-server LIMIT 1 probe, still bound to 48 hours on the partition column. Two things change:

• It is a LEFT JOIN LATERAL ... ON TRUE, so a server with nothing in the window is reported with a null instead of being dropped.
• It carries s.created_date from the registry. That column is the server's first successful connect: the upsert writes it on insert and never again.

There is no extra round trip and no second look at collection_log.

ClassifyWindowedFreshness decides what an empty window means. It is the shared ladder plus one arm:

• A collection inside the window bands exactly as before.
• Empty window, server registered before the window: Offline.
• Empty window, server registered inside the window: awaiting first collection. This one is exact, because the worker collects only after connecting. On DARLING01, all 17 registry rows (enabled or not) have their first retained collection after their created_date.
• No registration (only a hand-inserted row has none): the old reading.

Only a positive report can turn into Offline. A server that is missing from the read's result altogether gets the row's default: two nulls, which classify as never collected. That can only happen if the server is disabled between the two reads. The null fallback was written after actively-collecting servers showed as Offline on a production store (a missing row became a year-0001 timestamp), and a missing row still can't produce Offline.

The window stays, for the price the issue measured. The bind and the classifier share one derivation, LastCollectionWindowStart(now), so the window the read searched and the window the card reasons about cannot drift apart.

Web card. The status line on an Offline card showed a bare time of day (last collect 01:09 AM for a collection 20 hours old). It now shows an age (last collect 20h ago), or no recent collection when the window holds nothing. It used to render last collect — in that case. I checked this by rendering the shipped fleet.js under a DOM shim, against HEAD and this branch.

MCP payload. No shape change and no new fields. status / band / is_online / awaiting_first_collection / reason now read Offline for such a server. last_collection stays the newest collection inside the window, so it is null for a server dark longer than two days. status tells that null apart from the never-collected one, and the field's doc says so. The tool description needed no change: every field is still read for the call.

Behavior change, stated. A server dark for more than 48 hours now counts in offline_count rather than warning_count, and ranks as Offline in worst_servers.

Where the card and the viewer can still differ, deliberately. The viewer says "Awaiting first collection" whenever it finds no collection_log row at all. That covers two cases:

• A server dark longer than the 60-day collection_log retention. The viewer and list_servers side of that is filed as A server dark past collection_log's 60-day retention reads 'Awaiting first collection' in the WPF viewer and list_servers #3967.
• A server that connected more than two days ago and has written no row since. This is rare: every collector run writes a row whatever its outcome.

The card reads both as Offline. In both, the service reached the server and nothing has come back for two days.

Lite: no change needed. Lite's last-collection read has no window (hot table, then the archive view), and Lite has no fleet roll-up.

Not changed: the viewer, list_servers, FleetServersSql, the 7-day collection-health scan, and no migration.

Test plan

• DARLING01 (read-only, -U admin, default_transaction_read_only), 19 collection_log chunks, 16 compressed. Before and after were run alternately in fresh sessions, two rounds each. The plans are the same (Nested Loop becomes Nested Loop Left Join), with the same buffers:

  FleetLastCollectionSql	planning cold	planning warm	execution	exec buffers
  before (The fleet overview reads each server's newest sample instead of every retained row, and the web viewer asks for it once per refresh (#3895) #3947, CROSS JOIN)	24.7 / 29.2 ms (1,363 buf)	0.83–1.12 ms (50 buf)	0.20–0.67 ms	29
  after (LEFT JOIN + created_date)	27.3 / 28.7 ms (1,366 buf)	0.94–1.06 ms, one 3.6 ms (50 buf)	0.21–0.64 ms	29
  option 1, no window (for reference)	89.2 / 101.5 ms (8,937 buf)	3.1–3.3 ms (273 buf)	0.77–1.22 ms	—

  Cold-planning figures vary across sessions (an earlier round measured 9.8–13.1 ms for the windowed read). Each comparison above comes from back-to-back sessions.

• New FleetCardTellsDarkFromNeverCollectedTests (7 tests):

  • the ladder over every registration × collection combination;
  • the boundary at the bound instant;
  • an unreported server keeps the no-claim reading;
  • the dark card reads Offline on band, status, is_online, reason and roll-up counts;
  • a newly registered server still reads awaiting;
  • collection_stale keeps its population (has_collector_errors drives the fleet card's health band and its semantics cannot be determined from the surface — four wrong inferences by two independent agents, and it reads false while errors sit inside the card's own window #3098);
  • the read's shape: LEFT JOIN, created_date, the window kept, one collection_log reference, the shared bind.

• FleetReadsAreBoundedLivePostgresTests (rig, TimescaleDB):

  • sentinels registered the way the service registers them;
  • the probe returns the oracle's rows plus explicit empty-window rows for the dark and never-collected servers;
  • through GetFleetOverviewAsync, the five-days-dark card reads Offline and the ten-minutes-registered one reads awaiting, each matching the word the WPF viewer's unbounded freshness gives;
  • the windowed read plans only the window's chunks where the same statement unwindowed plans every one.

• Updated pins, deliberately: FleetLastCollectionSql_IsWindowed_NotAFullHistoryScan (CROSS to LEFT) and TheMcpCard_DerivesTheFlagFromFreshnessAndNothingElse (the classifier call).

• Red-watch, test project rebuilt each time. Four mutations of the reader, each caught:

  • BuildCard back on the shared ladder: 3 failures (pure card, source pin, live seam);
  • the probe back to CROSS JOIN: 4 failures;
  • the window dropped (option 1): 6 failures;
  • the window widened to 60 days: 6 failures, including the plan pin ("planned 9 of the unwindowed probe's 9 chunks").

• Full Darling.Tests with DARLING_TEST_PG on the rig: 12,603 total, 0 failed, 22 skipped. Every skip is gated on an environment this run did not set: the managed runtime or upgrade fixtures, a live SQL Server, symlink privilege, or the CI worker sizing. The first full run went red on two meta pins this change tripped, and both are fixed in the change. RepoFileAdoptionTests failed because the new test read the reader LF-normalised for single-line anchors, and it now uses the raw reader. TsqlConventionGuardTests.TheMemberScan_ReadsEveryDeclarationWhole failed because the member walker stops at a property pattern's braces, so the classifier is now written without one. Neither pin file is edited.

• Build: 0 Warning(s).

🤖 Generated with Claude Code

https://claude.ai/code/session_018Z4T51J4E6L8fjriPHpsif