Darling tier-1 field-incident response: file log + Event Log source, store conf headroom, honest bootstrap status, PG18 docs - #1552
Merged
Conversation
…onest bootstrap status From the 24-server report (sequential first sweep crawling under store pressure looked like servers being permanently dropped, with no log anywhere to say otherwise): - DarlingFileLoggerProvider (new): rolling file log under %ProgramData%\PerformanceMonitorDarling\logs, buffered, daily-rotated, 14-day retention, never-throwing - the PRIMARY diagnostic surface. Every existing ILogger call site lands there unchanged. - Event Log: source pinned to "PerformanceMonitor Darling", best-effort registration at startup, and the README install steps gain the one elevated New-EventLog line - the provider was always wired but the NT SERVICE account can't register a source, so a by-the-book install has silently dropped every event since day one. - Managed store v4 conf block (append-and-heal like v2/v3): max_connections = 200, max_wal_size = 4GB. PG defaults (100/1GB) are toy-sized; the fleet bootstrap burst forced back-to-back spread checkpoints and backend-spawn churn (Windows error 487). - "Awaiting first collection": never-collected is a distinct freshness band (amber card + fleet Warning + grey sidebar dot) instead of the red Offline overlay - "queued, not reached yet" is the truth during bootstrap. MCP get_server_status gains the additive AwaitingFirstCollection value. - PostgreSQL 18 doc sweep: the bundled runtime has been PG 18.4 + TimescaleDB 2.28.1 since the runtime bump; README/sample/comments said 17. Tests: Darling suite 2151 passed / 0 failed (Release); new pins for the v4 block, the NeverCollected band (cards, fleet, sidebar), and updated freshness pins. Tier 2 (bounded-parallel sweep + cadence jitter) follows separately. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This was referenced Jul 17, 2026
pull Bot
pushed a commit
to ehtick/PerformanceMonitor
that referenced
this pull request
Jul 22, 2026
…itter (erikdarlingdata#1553) Tier 2 of the 24-server field-incident response (tier 1 was erikdarlingdata#1552). The service swept the fleet strictly sequentially -- one foreach over every server, every step awaited inline -- so under store distress one server's 3+ minute first-connect work left the alphabetical tail unreached for tens of minutes (indistinguishable from a permanent drop), and every 15s cadence boundary re-herded all servers at once. Fire-and-track sweep (D2/D3): the loop now LAUNCHES each server's collection body (new ProcessServerSweepAsync, tracked per-server in InFlightSweep) without awaiting it, falling straight through to the fleet-level reload/purge/disk-check steps, so one slow or hung server can no longer head-of-line-block the fleet. Concurrency is bounded to N=4 (MaxConcurrentServerSweeps, hardcoded) by a shared SemaphoreSlim acquired inside each body; collectors stay sequential within a server (Lite's RemoteCollectorService shape). Deliberately not await-all-per-sweep -- that recreates the incident in miniature. Cadence jitter: CadencePhaseOffset ((uint)serverId % periodSeconds; the serverId is already an FNV-1a hash) gives a deterministic, restart-stable per-server phase applied only at initial cadence stamps -- each collector's first post-connect due time, a reload's newly-enabled collectors, and the first scheduled analysis (fixed 2.5-minute window, not the full interval). The steady-state advance is never jittered. Per-server exception containment (D3, Design Goal 4): the extracted body has a catch-all (the gap the sweep loop previously had no cover for), so one server's throw is logged and isolated and a faulted fire-and-track task never surfaces unobserved. Concurrency-supporting invariants: per-server NextDue becomes a ConcurrentDictionary (D1) since a reload's recompute can now touch it while a body reads it (indexed only by static catalog keys -- a drop-in); disabling a server marks its state Retired (checked as the first statement inside each body after it acquires the gate, plus a connect-path re-check before any durable side-effect) so an in-flight/queued body for a removed server no-ops (never connects, never runs XE DDL, never re-writes self-alert edge state after Forget); and shutdown drains in-flight bodies (up to 15s, inside the host's 30s ShutdownTimeout) before the command loop, with the gate deliberately never disposed. Hang observability (D2b): because the incident was hangs, not throws, the skip branch logs Debug each sweep and one Warning per episode once a body crosses 60s continuous (queue time counts), with an Info on eventual completion; all skip-log state is outer-thread-owned. Contracts unchanged: collection_log write semantics, MCP surfaces, and alert dedup/edge semantics preserved; the per-server CollectionGate behavior is unchanged. New DarlingSweepSchedulingTests pins the jitter offset (deterministic, unsigned-modulo including negative FNV ids, bounded [0, period), not-all-identical across ids, non-positive period -> zero) and the N=4 cap. Full Darling.Tests (2157 passed / 134 gated-live skipped) and Lite.Tests (1352 passed) green. Darling service only -- no store schema change, no Lite/Dashboard/viewer changes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
MisterZeus
pushed a commit
to MisterZeus/PerformanceMonitor
that referenced
this pull request
Aug 24, 2026
…ered differently On dev, a registered-but-never-collected server draws a grey "Unknown" dot in the sidebar and an amber "Awaiting first collection" card one panel over. Both are stamped from the SAME ClassifyFreshness call. Verified before touching anything, and the premise holds exactly as erikdarlingdata#2473 describes it: DarlingServer.ApplyFreshness sets two flags where ServerSummaryItem.ApplyFreshness sets three, DarlingServer has no AwaitingFirstCollection to set, and MainWindow.xaml carries DataTriggers for Online/Offline/Warning only -- so the fifth state fell through to ForegroundMutedBrush and painted grey without failing anything. The dot is the thing a reader points at first, and it was silently giving the pre-erikdarlingdata#2429 answer. WHAT THE DOT NOW SAYS, AND WHY Amber, matching the card, with the card's word. The card already chose amber for this state and wrote down why -- a queued server is not a dead one, which is the 24-server field incident erikdarlingdata#1552 answered. Grey is arguably honest ("nothing is known yet") and it was the smaller change, but it is a SECOND answer to a question the card has already answered on the same screen, and having two answers is the entire defect. The sidebar's job here is to render the card's decision, not to hold an opinion about it. Amber now covers two states on the dot (a stale collection and a never-collected server), which is exactly what the card already does -- the card disambiguates them with a word and a dot has no room for one, so the dot gains a tooltip instead. Its first line is the shared headline, word for word what the card says. WHERE THE LADDER LIVES Not in the viewer. Collapsing the two viewer surfaces onto a viewer-local rules class would have produced a pin that could not span the files holding the other copies, and erikdarlingdata#2470's whole lesson is that a pin scoped shorter than the duplication is worth nothing. Looking for those files turned up two more: - DarlingFleetReader.StatusLabel -- the web/MCP fleet card's word. Agrees with the card today; a fourth independent derivation of the same five states. - DarlingMcpDataTools.FreshnessStatus -- list_servers, written as if statements, in another file, against its OWN copies of the 2-minute and 15-minute thresholds. That is the erikdarlingdata#2470 evasion shape twice over, plus a real drift bug: ServerHealthThresholds could move and list_servers would go on answering with the old numbers. Both live in the headless service, which cannot reference WPF, so the only assembly all four can render from is PerformanceMonitor.Common -- where ServerHealthClassifier and the thresholds already are, and where ClassifyBand already takes this exact flag triple. The type is ServerCollectionStatus, not ServerCardStatus, and the rename is load-bearing rather than cosmetic. Lite has a ServerCardStatus of its own answering a DIFFERENT question -- its word comes from a live connection check, this one from how old the newest collection is -- and erikdarlingdata#2457 turned down folding freshness into Lite's word precisely to keep the two axes apart. Common already names that distinction (ServerConnectionStatus is the connection answer); this is the collection one. Sharing a name would have invited the conflation both issues were about, and Lite/Models/ServerConnection.cs imports both namespaces, so it would also not have compiled. WHAT THE COLLAPSE ACTUALLY IS Classify(isOnline, hasCollectorErrors, awaitingFirstCollection) is the ladder. Word(), McpToken() and Headline() are three renderings of its result -- not three ladders, which is the correction erikdarlingdata#2470 had to make once already: a second switch is a second ladder even when it returns the right type. McpToken differs from Word in exactly one arm because "AwaitingFirstCollection" shipped as a published MCP status value and clients key on it; a status value is a consumer API, so the vocabularies stay apart and sit next to each other where the difference reads as a decision. FlagsFor(ServerFreshness) is the other half, and it is the half that fixes the bug rather than tidying it. Both ApplyFreshness implementations classified freshness and then wrote the flags out by hand; the sidebar's wrote two of three and dropped the awaiting marker on the floor. Nothing about a block of assignments makes a missing one visible. They now arrive as one value. Behaviour is unchanged everywhere except the sidebar dot. The MCP list_servers tokens were checked arm by arm against the thresholds they used to duplicate (2 min stale, 15 min offline -- identical), and StatusLabel's output is byte-identical. Verified on macOS: Common, the Darling viewer, the Darling service, Lite, Lite.Tests and Darling.Tests all build. The ladder itself was run against the real build in a throwaway net10.0 console -- every freshness band through FlagsFor and Classify agrees with FromFreshness, the erikdarlingdata#2429 flag pairs still resolve the way the card always resolved them, and all four list_servers tokens match the pre-change thresholds. The WPF suites cannot run on macOS; CI is the arbiter for those. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Tier 1 of the 24-server field-incident response (full diagnosis in the incident thread): the sequential first sweep crawling under store pressure was indistinguishable from servers being permanently dropped, because (a) there was no service log anywhere on a by-the-book install, and (b) the viewer rendered "hasn't been reached yet" as red Offline.
Changes
DarlingFileLoggerProvider(new) — rolling file log under%ProgramData%\PerformanceMonitorDarling\logs\darling-service_yyyyMMdd.log; buffered (5s flush), daily rotation, 14-day retention, never-throwing; anILoggerProvider, so every existing log site lands there with zero call-site changes. PRIMARY diagnostic surface.PerformanceMonitor Darling(matches service name + docs), best-effortCreateEventSourceat startup, and the README install steps gain the one elevatedNew-EventLogline. The provider was always wired, but the recommendedNT SERVICEaccount cannot register a source → every event silently dropped since day one (confirmed in the field report:SourceExistsreturned false).max_connections = 200,max_wal_size = 4GB. PG defaults (100 / 1GB) forced back-to-back spread checkpoints under the fleet-bootstrap write burst while Windows backend-spawn churn (error 487) surfaced as transient store write failures. Existing clusters heal on next service-owned start.ServerFreshness.NeverCollectedband: amber card status + fleet Needs-Attention Warning ("Awaiting first collection") + grey sidebar dot, never the red Offline overlay. MCPget_server_statusgains the additiveAwaitingFirstCollectionstatus value (existing values unchanged — contract note).darling.sample.json/workflow comments said 17; BYO guidance now "16 or newer, validated against 18".Contracts
get_server_status: additive status value only.Testing
NeverCollectedclassification/apply/status pins (cards, fleet band + reason, sidebar dot), and the updated no-collection pins.Tier 2 (bounded-parallel sweep + cadence jitter) follows as its own PR.
🤖 Generated with Claude Code