Lite: stop the Job History header calling every Agent-less server "Stopped"

[erikdarlingdata]

Lite-only. The presentation-side twin of #1719, using the same two gates.

The problem

The Job History header drew a red Agent: Stopped for any row whose agent_running was false. That is a claim the collected data does not support in two ordinary cases:

• A server where Agent has never run — a container built without it, Express, Azure SQL DB — is not "stopped". It does not use Agent. There is nothing to report.
• A reading hours or days old says nothing about the service now. The backing read had no freshness filter at all, so a days-stale row rendered exactly like a current one.

Both rendered identically to a genuine outage. The result is the familiar one: the single state an operator must not learn to ignore was the state they saw constantly.

The fix

Judge only on a fresh reading, and only where Agent has been observed running at least once in the retained agent_status history — the evidence rule #1719 established for the Darling alert, reused rather than reinvented so the header and the alert cannot disagree about the same row.

State	Header	Colour
Fresh, running	Agent: Running · Next run: …	neutral
Fresh, stopped, seen running before	Agent: Stopped · Next run: …	red
Fresh, stopped, never seen running	Agent: never observed running	neutral
Stale or never collected	Agent: unknown (reading 3h old)	neutral

The real alarm is untouched — fresh + seen-running + stopped is still red. A currently-running Agent is its own proof of capability, so it short-circuits the history probe exactly as the evaluator does.

The all-servers roll-up gets the same treatment: never-used and stale servers are reported separately (2/3 running, 1 stopped, 4 not using Agent) instead of being folded into the stopped count, which is what made the fleet line overstate the number of problems.

Notes on two judgment calls

Wording. never observed running, deliberately not "not installed" / "not present". We can see that Agent has not been observed running; we cannot see whether it is installed. There is a test pinning that the header does not make the stronger claim.

Stale window is 30 minutes — matching Darling's StaleWindow and through it the Dashboard's threshold. Explicitly not the shared ServerHealthThresholds.StaleThreshold: that is two minutes, derived from the fastest collector's one-minute cadence, and agent_status collects every five (CollectorScheduleDefaults). Reusing it would have rendered a perfectly healthy Agent as stale most of the time — a pin asserts the window stays longer than the collector cadence.

The gate lives in Lite rather than PerformanceMonitor.Common to keep this Lite-only and off files other branches are editing. If Darling ever wants the same tri-state header, AgentHeaderStatus is the thing to lift.

Implementation

• Lite/Services/AgentHeaderStatus.cs (new) — the decision as a pure function, so it is testable without standing up WPF. The code-behind now only maps state to brush.
• LocalDataService.JobHistory.cs — collection_time and a derived ever_running added to the projection. ever_running is a window aggregate over the partition the query already scans, not a second round trip; MAX over a 0/1 CASE rather than BOOL_OR so it does not depend on which aggregates DuckDB exposes as window functions.
• JobHistoryTab.xaml.cs — renders whatever the classifier decides.

One thing this surfaced: my first draft had a clock-skew branch for a negative age, which is unreachable — a future timestamp is nearer than the stale window, so it classifies Fresh and never reaches the age renderer. Removed rather than left as untestable code, and the test now pins the actual behaviour: a skewed clock counts as fresh, so a genuinely stopped Agent still alerts. Refusing to judge there would silently suppress a true alarm on every server whose clock runs fast, which is the expensive direction to be wrong in.

Test plan

• Lite.Tests full suite: 1515 passed, 0 failed.
• Lite builds with 0 warnings.
• Never-seen → neutral, and does not claim Agent is absent.
• Stale → unknown (both directions: a days-old Running is no more current than a days-old Stopped); never-collected → unknown; stale-window boundary is exclusive; window is longer than the collector cadence.
• Seen + fresh + stopped → red still stands, and running-now is its own evidence.
• Roll-up: excludes never-observed from the stopped count, reports stale separately, still alerts on a real stopped Agent, neutral when nothing is judgeable, blank when empty.
• Real-DuckDB round-trips on the actual SQL: ever_running true when an earlier row ran though the newest is stopped, false when no retained row ever ran, does not leak across servers, and collection_time is projected and drives staleness.

Rides the nightly if it lands in time but does not gate it — cosmetic honesty.

🤖 Generated with Claude Code

[erikdarlingdata]

Closing as superseded by #1725, which shipped the same two fixes to dev while this was in review.

I checked their implementation against mine rather than assuming, and theirs is complete and equivalent:

• Same 30-minute stale window, arrived at for the same reason (mirror the headless service so both surfaces refuse to judge on the same age of data).
• IsAgentProblem = !IsStale && !AgentRunning && EverSeenRunning is exactly the rule my classifier called Stopped.
• StatusDisplay covers all three honest states — unknown (stale), a real Stopped, and not present.
• Fleet roll-up counts only servers the question applies to.

Merging this would have replaced working shipped code with a parallel implementation of the same thing, which is strictly worse than closing it. Auto-merge disarmed before closing — it was armed, and had the CHANGELOG conflict resolved in the other direction it could have reverted #1725.

One genuinely non-redundant piece is being carried forward separately: every test in #1725 runs against constructed model rows, so nothing proves the SQL. The ever_running derivation is a window aggregate that has to look across the whole retained partition while the surrounding query collapses to the newest row per server — the classic failure is silently returning the newest row's own value, or leaking across servers, and both would pass a model-level test. Following up with a test-only PR that round-trips it against real DuckDB, plus a pin that the stale window stays longer than the collector's 5-minute cadence (so nobody later "unifies" it with the shared ServerHealthThresholds.StaleThreshold, which is 2 minutes and would render a healthy Agent stale most of the time).

No production code in that follow-up, so it cannot conflict with #1725.