[BUG] A clean service stop logs 7 ERRORs from a disposed data source — 7 of the day's 9 ERROR lines

[erikdarlingdata]

What happens

Every clean, operator-initiated Stop-Service logs a burst of ERRORs from work that is still in flight after the collection loop has already reported it stopped. From prod-sql-use2-monitor-01 tonight:

18:52:12.940 [INFO ] [DarlingWebHostService] Stopping web dashboard
18:52:12.972 [INFO ] [DarlingMcpHostService] Stopping MCP server
18:52:13.379 [INFO ] [DarlingWorker] PerformanceMonitor Darling collection loop stopped
18:52:14.160 [ERROR] [DarlingWorker] [PgBaselineProvider] Failed to compute baselines for io_latency:
                     57P01: terminating connection due to administrator command
18:52:14.162 [ERROR] [DarlingWorker] [PgBaselineProvider] Failed to compute baselines for batch_requests:
                     Cannot access a disposed object.
18:52:14.162 [ERROR] [DarlingWorker] [PgBaselineProvider] Failed to compute baselines for session_count: ...
18:52:14.162 [ERROR] [DarlingWorker] [PgBaselineProvider] Failed to compute baselines for query_duration: ...
18:52:14.163 [ERROR] [DarlingWorker] [PgBaselineProvider] Failed to compute baselines for memory: ...
18:52:14.163 [ERROR] [DarlingWorker] [PgAnomalyDetector] Object stats anomaly detection failed:
                     Cannot access a disposed object.
18:52:14.164 [ERROR] [DarlingWorker] [PgFindingStore] FilterMutedFindingsAsync failed:
                     Cannot access a disposed object.

Seven ERRORs in 800 milliseconds, all after "collection loop stopped". Reproducible: the earlier stop today (12:54, a different build) produced the same shape from the same two components.

Why it is worth fixing rather than tolerating

The whole day's log on that box carries 9 ERROR lines. Seven of them are these. The other two are genuine (NpgsqlException: Exception while reading from stream during a drill-down and a baseline). So an operator or an alert rule looking at ERROR volume sees a 4.5x-inflated count whose dominant cause is "someone restarted the service", and the two real ones are the needles. That is the same failure mode as #2294 — a stop being reported as a fault — one layer up.

It is also, mechanically, unfinished work rather than pure noise: the analysis pass is still computing baselines and filtering findings when its data source is disposed underneath it, so whatever those five baselines would have written is lost. Losing it during shutdown is fine; not knowing whether it was lost or failed is not.

Mechanism

Two distinct causes in the same burst, and they want different handling:

• 57P01: terminating connection due to administrator command — the store's own postmaster is going down (the managed Postgres stops with the service), so an in-flight query is killed server-side. This is an expected consequence of a stop, not a fault.
• Cannot access a disposed object — the NpgsqlDataSource is disposed while the analysis pass is still running. The pass is started but the host does not wait for it (or cancel it) before disposing, so the shutdown is racing its own work.

The second is the one with a real fix: the analysis pass should observe the stopping token and complete or abandon before the data source goes away. Note the pass takes a serverId and cannot gate itself, which is the same seam #2213's review flagged — the gating belongs at the call site.

Suggested shape

1. Cancel-and-await the analysis pass in the host's StopAsync before disposal, so the disposed-object path becomes unreachable rather than merely quiet.
2. Classify the residue as shutdown, not error: with the stopping token signalled, 57P01, ObjectDisposedException and OperationCanceledException are the expected outcomes of a stop, and belong at Information ("baselines abandoned at shutdown") — one line, not seven. Keep them at ERROR when the token is not signalled, because then they mean something real.
3. Do not simply lower the log level unconditionally — that hides the case where a data source is disposed while the service is meant to be running, which is a genuine bug this text would otherwise be the only evidence of.

The distinction to preserve throughout: unfinished because we asked it to stop versus unfinished because something broke. Today the log cannot tell those apart, which is exactly why the seven lines are indistinguishable from the two that matter.

Found while dogfooding on the use2 box (#2150 flip deploy); unrelated to that change — the same burst is in the 12:54 stop on the previous build.

[erikdarlingdata]

Taking this one. Shape per the issue: plumb a CancellationToken through DarlingAnalysisService and its three components (today the Analysis project has no token anywhere, so the pass is not merely un-awaited — it is uncancellable), convert shutdown residue (ObjectDisposedException / 57P01 / OperationCanceledException with the stopping token signalled) into a single Information line at the worker instead of seven ERRORs, and keep the ERROR arm for the same exceptions when the token is NOT signalled — that case is a real bug and this text would be its only evidence. PR to follow.

[erikdarlingdata]

Fixed by #2307, squash-merged to dev as 696eb76, following the issue's three-point shape exactly: the pass now observes the host's stopping token (carried on AnalysisContext — the whole Analysis project had no CancellationToken before this, so the pass was not merely un-awaited but uncancellable), a stopping sweep holds it open for a bounded 5s unwind grace before the data source is disposed, and residue from a race it still loses (ObjectDisposedException / 57P01-03 / OCE, with the token SIGNALLED) collapses to one Information line that states the loss honestly. The same exceptions with the token NOT signalled keep their ERROR, and a command timeout is never relabelled shutdown, so the #2294 signal survives the coincidence. A category pin counts every ERROR-logging catch on the pass and goes red if a new one arrives unclassified.

Merge note for the record: the claude review ran twice and both passes are posted and clean (the one finding — the baseline-data gate's bare catch — was fixed in 58b6f96), but the review GUARD failed both attempts on an unguarded lookup 404, filed as #2309. Merged on the strength of the posted reviews.

Verification on the box comes with the next nightly: the before-state reproduced on 2 of 2 stops with an analysis pass in flight, so a clean stop on the new build should log exactly one Information line where the seven ERRORs were.