Repository navigation
A stock PostgreSQL target without pg_wait_sampling has no wait history at all and its empty chart reads as a broken product: the collector grows a service-side pg_stat_activity sampler arm as the floor, the three wait tiers become one connect-time decision, and every wait read discloses its instrument (#3604) - #3645
Conversation
…ulation at all, so its empty wait chart reads as a broken product: pg_wait_sampling grows a service-side pg_stat_activity sampler arm as the floor under stock targets, the three wait tiers are one connect-time decision, and every wait read discloses which instrument fed it (#3604) The pg_wait_sampling collector forks on a new connect-time fact, CollectorTargetInfo.HasPgWaitSamplingExtension: with the extension it reads the 10 ms profile as before; without it, it runs one multi-statement command of thirty one-second pg_stat_activity snapshots (each behind pg_stat_clear_snapshot, because the view is cached per transaction) and accumulates the tallies across cycles in its own collector_state, so the rows it writes are cumulative and the existing newest-minus-oldest read differences them unchanged. Which arm ran is recorded in state; get_pg_wait_sampling, get_pg_wait_stats and the Viewer panel disclose it as instrument: engine_cumulative | extension_sampled | service_sampled, with the floor caveat on the service tier. Aurora is gated off pg_wait_sampling (it cannot preload the module; the hourly EXTENSION_MISSING there was a permanent gap dressed as a precondition), and CoveredInsteadBy now points an Aurora caller at pg_wait_stats. The collector moves from hourly to five minutes and is detached from the sequential sweep body beside query_store and plan_correction, because a deliberate 30 s window inline would delay every other collector on the target; the window is 30 s and not the cycle because SweepPressureClassifier sums single-run costs against a 60 s body budget. No migration: same table, same columns, same collector name. The sampler needs nothing beyond pg_monitor.
| /// <summary> | ||
| /// The sentence the empty arm appends about the instrument (#3604): the service tier's floor caveat when | ||
| /// that is what is sampling, nothing when the extension is (its idle answer needs no qualification), and a | ||
| /// plain "not yet recorded" when no cycle has stated an arm. | ||
| /// </summary> |
There was a problem hiding this comment.
This new <summary> is inserted directly below the pre-existing doc comment for BuildWaitSamplingJson ("The response body, split out so the WIRE SHAPE can be asserted...", a few lines up), without moving that comment down. The net effect: the old summary is now misattached to DescribeInstrumentForEmpty (which it doesn't describe), and BuildWaitSamplingJson (below, line ~156) is left with no doc comment at all. Worth moving the pre-existing summary block down to sit directly above BuildWaitSamplingJson.
There was a problem hiding this comment.
Fixed in 2ee0… (second commit on this PR): DescribeInstrumentForEmpty and its own summary now sit ABOVE the pre-existing BuildWaitSamplingJson doc block, so that block is back on the member it describes. DocCommentHygieneTests.NoMemberCarriesTwoStackedSummaryBlocks executed green locally after the move (it was the CI failure on head 1).
| sb.Append(snapshot).Append(';'); | ||
| for (var i = 1; i < SamplerSnapshotsPerCycle; i++) | ||
| { | ||
| sb.Append("\nSELECT pg_sleep(").Append(SamplerPeriodMs / 1000.0).Append(");") |
There was a problem hiding this comment.
sb.Append(SamplerPeriodMs / 1000.0) formats the double with StringBuilder.Append(double), which is culture-sensitive (uses CultureInfo.CurrentCulture) — unlike ParseTally/SerializeTally a few dozen lines below, which explicitly use CultureInfo.InvariantCulture for exactly this reason. Right now it's harmless because 1000 / 1000.0 == 1.0 formats as "1" (no fractional part) in every culture, so pg_sleep(1) is always valid SQL. But if SamplerPeriodMs is ever tuned to a non-whole-second value (e.g. 500ms → 0.5), a Darling service host running under a culture that uses , as the decimal separator (common in much of Europe) would emit pg_sleep(0,5) — invalid syntax / wrong overload — and silently break this collector on that host. Worth using .ToString(CultureInfo.InvariantCulture) here to match the file's own convention and remove the latent trap.
There was a problem hiding this comment.
Fixed (third commit): (SamplerPeriodMs / 1000.0).ToString(CultureInfo.InvariantCulture), with a comment naming the trap so the next period tweak does not re-learn it. Good catch — the file already held itself to that rule in ParseTally/SerializeTally and this was the one number that slipped.
There was a problem hiding this comment.
LGTM — reviewed the pg_wait_sampling three-tier instrument change (Aurora engine_cumulative / extension_sampled / service_sampled): connect-time probe wiring, AppliesTo/CoveredInsteadBy gating, engine-capability sweep coverage, schedule-cadence justification, detach/gate wiring in DarlingWorker, the cumulative-tally sampler read and its cap/reset/parse round-trip, MCP tool and Viewer disclosure of the instrument, and Lite/Darling doc + test parity (README table, runbook, Lite.Tests mirrors). No correctness, security, or parity-drift issues found. Left two non-blocking inline notes: a misattached XML doc comment and a latent (currently inert) culture-formatting trap in the pg_sleep() literal.
…L collectors read'; pg_wait_sampling is now the one documented exception (the module cannot be preloaded there), so the kind-axis split, the pre-flight probe line and the compose measure census each name it rather than assume it away; the empty-arm helper moves above the JSON builder's doc block it had split (#3604)
…as every other number this file puts into SQL or state is: harmless at a whole-second period, a comma-decimal host's pg_sleep(0,5) the day it is not (#3645 review)
| } | ||
|
|
||
| /* The arm that ran, recorded every cycle rather than once: a target that gains or loses the extension | ||
| changes arm at its next connect, and the reads must follow on the next cycle, not the next restart. */ |
There was a problem hiding this comment.
Correctness: the sampler's cumulative tally is never cleared when the target switches to the extension arm, so it goes stale and can be resumed later.
When HasPgWaitSamplingExtension is true, ReadAsync returns here without touching TallyStateKey. Since only keys present in context.PendingState get written back to collector_state (DarlingCollectorRunner only upserts PendingState, see the if (context.PendingState.Count > 0) persistence block), the old sampler_tally row is left sitting in the store untouched, potentially for a very long time (the extension can stay installed for months).
If the extension is later dropped (or the probe starts failing) and the target reconnects onto the sampler arm again, ReadSamplerAsync calls ParseTally(context.State[TallyStateKey]) and resumes accumulating from that stale, pre-extension tally rather than starting fresh:
var tally = ParseTally(context.State.TryGetValue(TallyStateKey, out var carried) ? carried : null);In the common case the stale (small, old) tally is less than the extension arm's own cumulative counts, so DarlingPgWaitSamplingReader's (n.sample_count < o.sample_count) AS counter_reset heuristic happens to catch it and the discontinuity reads as an honest reset. But if the extension's own profile was itself recently reset (pg_wait_sampling_reset_profile() or a Postgres restart) shortly before the target falls back to the sampler arm, the resumed stale tally can be larger than the last extension-arm row, so the merge silently produces a value that looks like continued monotonic growth — no counter_reset flag — while actually splicing together two unrelated eras of data.
Given how carefully this PR reasons about staleness for the instrument token ("a target that gains or loses the extension changes arm at its next connect, and the reads must follow on the next cycle, not the next restart" / the TheExtensionArmRecordsItsInstrument test), the same care doesn't seem to have been applied to the tally. Worth clearing TallyStateKey (e.g. context.PendingState[TallyStateKey] = string.Empty) whenever the extension arm runs, so a later reversion to the sampler always starts from an empty tally instead of resuming an arbitrarily old one.
There was a problem hiding this comment.
Real catch — fixed in the fourth commit. The extension arm now writes PendingState[TallyStateKey] = string.Empty every cycle it runs (an empty tally is what ParseTally reads as start-over), with the reasoning you gave — including the just-reset-profile case where a stale-but-larger tally would read as monotonic growth with no reset flag — in the comment beside it. Pinned two ways in Lite.Tests/PgWaitSamplerArmTests: the extension arm's PendingState carries the empty tally, and a full sampler→extension→sampler reversion starts the fallback series at zero (27/27 executed green locally).
|
Reviewed the diff (25 files, PostgreSQL wait-sampling collector's new service-side sampler arm + instrument disclosure). This is entirely shared-library/Darling C# — no T-SQL to check, and Lite never runs PostgreSQL collectors in its app (only in Overall this is unusually well-reasoned and well-tested code (93/93 Darling.Tests + 26/26 Lite.Tests per the PR description, plus a live-rig test). I found one real correctness gap, left as an inline comment on
Everything else I checked came back clean:
|
There was a problem hiding this comment.
- Stale cumulative tally on arm reversion (
PerformanceMonitor.Collectors/PgWaitSamplingCollector.cs,ReadAsync/ReadSamplerAsync). The extension arm never touchesTallyStateKey, so the sampler arm'ssampler_tallystate row is left untouched (potentially for months) while the extension is in use. If the target later falls back to the sampler arm,ReadSamplerAsyncresumes accumulating from that stale tally instead of starting fresh. Usually this surfaces as an honestcounter_reset(stale tally < extension's cumulative count), but if the extension's profile was itself recently reset just before the fallback, the stale-but-larger tally can look like continued monotonic growth with no reset flag — silently splicing two unrelated eras of data together. See inline comment for detail and a suggested fix (clearTallyStateKeywhenever the extension arm runs).
… host persists only PendingState keys, so a tally left alone would sit in collector_state for as long as the extension stayed installed and be resumed as a months-old baseline the day the target fell back to the sampler arm — splicing two eras into one series when the extension's own profile had just been reset (#3645 review)
Remediated in 4th commit: the extension arm clears TallyStateKey every cycle; reversion path pinned by test (sampler→extension→sampler starts at zero). Replied inline.
| + "returned_samples is what the rows returned add up to." | ||
| + (serviceTier | ||
| ? " backend_count on this tier is the distinct backends seen in the LAST window, not " | ||
| + "since the profile started; a series' counter also resets when the service restarts, " |
There was a problem hiding this comment.
Correctness: this claim about the service_sampled tier appears to be false.
The note says: "a series' counter also resets when the service restarts, which the counter_reset flag reports the same way it reports an extension reset."
But sampler_tally (the state this claim is about) is persisted to the collector_state Postgres table via StateKeys / PendingState, and reloaded from there at the start of every cycle (DarlingCollectorRunner.GetCollectorStateAsync → context.State, then SaveCollectorStateAsync after). A Darling service restart does not clear that row — the next cycle after restart reloads the same cumulative tally and keeps adding to it. So newest.sample_count will never be smaller than oldest.sample_count purely because the Darling service restarted, and counter_reset will not fire for that reason. The only things that actually reset this tally are: a key getting evicted by the 500-key cap and later reappearing, or the extension arm running a cycle (which explicitly zeroes TallyStateKey).
This isn't just a doc nit — it's a factual claim shipped in an MCP tool's response text that an operator (or an LLM client) will use to interpret a discontinuity in the data. Believing "restarting the service resets this series" is actively misleading in both directions: it invites blaming an innocent restart for a series drop that has some other cause, and it means nobody should expect a service restart to actually reset a stale/suspicious baseline.
The same incorrect claim is echoed in the PR description ("a service restart resets the service tier's series") and isn't covered by any test (no test in PgWaitInstrumentDisclosureTests.cs or PgWaitSamplerArmTests.cs exercises restart behavior). Worth either fixing the text (drop the restart claim, or correctly scope it to "restarts of this collector's key cap eviction" / whatever was actually intended) or, if a restart-reset behavior was actually intended, wiring that up in the runner.
There was a problem hiding this comment.
Correct, and thank you — that sentence was a leftover from an earlier draft of the design that kept the tally in process memory; once it moved to StateKeys/collector_state (so it would survive restarts, deliberately) the note did not follow. Fifth commit: the note now says the tally SURVIVES a service restart and names what does start a series over — 500-key cap eviction, or the extension→sampler reversion the previous round made clear — with counter_reset reporting either. Pinned in PgWaitInstrumentDisclosureTests.ServiceTier_IsNamedAndCarriesTheFloorCaveat (contains the survives sentence, does not contain the reset-on-restart one). PR body corrected in the same two places.
There was a problem hiding this comment.
Reviewed the diff (dev...HEAD, 25 files, +1575/-54). This is a well-designed, thoroughly-documented change — the two-arm collector design, the connect-time tier decision, cadence/detach reasoning, and Lite/Darling parity are all handled carefully, and I didn't find SQL injection, secret-handling, or missing-index-DMV issues.
One substantive finding (left inline):
- Incorrect factual claim shipped in the MCP tool's response text.
DarlingMcpPgWaitSamplingTools.cs:238tells callers ofget_pg_wait_samplingthat a series' counter "resets when the service restarts, which the counter_reset flag reports." Butsampler_tallyis persisted to thecollector_statePostgres table (viaStateKeys/PendingState) and reloaded on every cycle, so a Darling service restart does not clear it — the tally keeps accumulating across restarts, andcounter_resetwill not fire because of one. This same incorrect claim appears in the PR description and isn't covered by any test. Since this text is read by operators and LLM clients to interpret discontinuities in the data, it should either be corrected to describe what actually resets the tally (key-cap eviction, or falling back from the extension arm) or the restart-reset behavior should actually be implemented if that was the intent.
Everything else (tier selection/gating, cadence and detach wiring, the batch SQL and result-set walking, tally accumulation/cap/parity tests, Aurora CoveredInsteadBy mirroring, Viewer/MCP disclosure, runbook updates) checked out.
…ts; the tally is persisted collector state reloaded every cycle, so it survives one — the note now names what actually starts a series over (cap eviction, arm reversion) and a pin keeps the false claim from returning (#3645 review)
Remediated in 5th commit: note corrected (tally survives restart; cap eviction / arm reversion are the resets), pinned by test; PR body corrected. Replied inline.
There was a problem hiding this comment.
LGTM — reviewed the sampler-arm collector logic (PgWaitSamplingCollector.cs), the connect-time probe (DarlingServerConnector.cs), the instrument disclosure on both MCP tools, the reader's delta/reset handling (DarlingPgWaitSamplingReader.cs), the engine-capability sweep additions (CollectorEngineCapability.cs / CollectorTargetInfo.cs), the detach-from-sequential-body wiring in DarlingWorker.cs, and the Viewer panel. No correctness, security, or Lite/Darling parity issues found — Lite never dispatches PostgreSQL collectors at all (DuckDbSchemaGenerator.StoredCollectors filters to SqlServer only), so this PostgreSQL-only change has no Lite counterpart to drift from. The SQL built in SamplerSnapshotSql/SamplerBatchSql only interpolates compile-time constants (no injection surface), the two review-fixup commits already landed (InvariantCulture on pg_sleep, tally clear on the extension arm's cycle) are present and tested, and OPTION(RECOMPILE)/T-SQL style rules don't apply since this PR touches no T-SQL.
…, and get_pg_wait_stats' instrument_note naming aurora_stat_system_waits() is accounted for as PROSE in the Aurora-surface allow-list (#3604)
|
Reviewed the diff against CONTRIBUTING.md (PostgreSQL-store conventions: schema qualification N/A — no migration rung in this PR; naive-UTC timestamp handling checked in Parity: Correctness spot checks that held up:
No substantive correctness, parity, or security issues found. A couple of very minor, non-blocking style notes below. |
|
Correction to my last comment: on closer look I don't have concrete style nits worth flagging — retracting that line. Nothing further to add. |
There was a problem hiding this comment.
LGTM — reviewed the collector fork (extension vs. service-sampler arm), connect-time probe, Aurora gating, detach/cadence wiring, MCP/Viewer instrument disclosure, and Lite/Darling parity (pg_wait_sampling is confirmed absent from Lite's stored-collector roster, so the shared-code changes are inert there and no parity gap exists). No correctness, security, or parity issues found; test coverage is thorough (unit + live-Postgres-gated).
…spliced once Fifty-nine PRs merged to dev today across the coordinator's lanes and the wave-2 worker's; each lane returned its entry to a buffer instead of touching this file, so that fifty-plus PRs did not each rebase the same twenty lines. This is the one splice. Every entry is one line (the archiver's compact() and the pins read them that way); riders fold into their parent's entry (#3599 under #3590, #3619 under #3611, #3623 under #3616, #3640 under #3633; #3617 test-only and #3661 re-cut as #3666 carry none); #3657's entry is in because it MERGED to dev - the twin to main is what is still pending. [Unreleased] gains a `### Added` above `### Fixed` (Keep-a-Changelog order) for the six new capabilities: per-user theme colours (#3606 / #3577 arm B), routed alert families (#3668 / #3598), the PostgreSQL logging audit tool (#3643 / #3607), the service-side wait sampler (#3645 / #3604), and the log-event classifier with its temp-file / autovacuum parser families (#3646 / #3601, #3664 / #3602 #3603). The other forty-eight are honesty fixes to existing surfaces and append to `### Fixed` after the wave-1 bullets, in PR-number order. Thirty-six reference definitions added for the issues the new entries cite and the index did not yet define; the [Unreleased] group is one ascending run again, which moves [#3557] into its slot (the one deleted line). Nothing under ## [3.8.0] or older is touched; the archive script was not run. One editorial touch: the #3585 entry ended in a dangling "Darling" and now reads "Darling only." (the tool exists only in the Darling MCP host). tools/changelog/changelog_archive.py verify: all PASS (1420 bold entries, floor 1,329; 1,358 distinct refs resolve; CRLF throughout; 356,539 bytes under the 750 KiB ceiling). ChangelogIndexAndArchiveTests: 5/5 pass via a net10.0 harness.
…spliced once (#3672) Fifty-nine PRs merged to dev today across the coordinator's lanes and the wave-2 worker's; each lane returned its entry to a buffer instead of touching this file, so that fifty-plus PRs did not each rebase the same twenty lines. This is the one splice. Every entry is one line (the archiver's compact() and the pins read them that way); riders fold into their parent's entry (#3599 under #3590, #3619 under #3611, #3623 under #3616, #3640 under #3633; #3617 test-only and #3661 re-cut as #3666 carry none); #3657's entry is in because it MERGED to dev - the twin to main is what is still pending. [Unreleased] gains a `### Added` above `### Fixed` (Keep-a-Changelog order) for the six new capabilities: per-user theme colours (#3606 / #3577 arm B), routed alert families (#3668 / #3598), the PostgreSQL logging audit tool (#3643 / #3607), the service-side wait sampler (#3645 / #3604), and the log-event classifier with its temp-file / autovacuum parser families (#3646 / #3601, #3664 / #3602 #3603). The other forty-eight are honesty fixes to existing surfaces and append to `### Fixed` after the wave-1 bullets, in PR-number order. Thirty-six reference definitions added for the issues the new entries cite and the index did not yet define; the [Unreleased] group is one ascending run again, which moves [#3557] into its slot (the one deleted line). Nothing under ## [3.8.0] or older is touched; the archive script was not run. One editorial touch: the #3585 entry ended in a dangling "Darling" and now reads "Darling only." (the tool exists only in the Darling MCP host). tools/changelog/changelog_archive.py verify: all PASS (1420 bold entries, floor 1,329; 1,358 distinct refs resolve; CRLF throughout; 356,539 bytes under the 750 KiB ceiling). ChangelogIndexAndArchiveTests: 5/5 pass via a net10.0 harness.
Wait accumulation on PostgreSQL had two paths and a hole. Aurora targets get engine-cumulative waits from
aurora_stat_system_waits()(pg_wait_stats, Aurora-gated). Stock targets get a real sampled profile IF thepg_wait_samplingextension is loaded. A stock target WITHOUT the extension — most first installs, and every shop where a preload library needs a change ticket — got nothing at all except an hourlyEXTENSION_MISSINGskip. For the SQL Server DBA whose whole performance worldview isdm_os_wait_stats, the flagship dimension was absent on exactly the targets most likely to be their first, and an empty wait chart reads as "the product is broken".Mechanism
PgWaitSamplingCollectornow has two arms behind one definition, forked on a new connect-time fact,CollectorTargetInfo.HasPgWaitSamplingExtension(probed frompg_extensionin the connected database byDarlingServerConnector, beside the Aurora probe, and carried onConnectionProbeResultso--test-connectioncomputes its collector count against the same shape the gate reads).pg_wait_sampling_profileread, unchanged.pg_stat_activitysnapshot, thenpg_sleep(1); pg_stat_clear_snapshot();snapshot, repeated toSamplerSnapshotsPerCycle= 30. Each snapshot is its own statement behind a clear becausepg_stat_activityis cached per transaction on first access and the batch is one implicit transaction; a single statement that slept and re-read in a loop would return thirty copies of one instant.ReadAsyncwalks result sets by shape (four columns = snapshot, one = sleep/clear), tallies by(event_type, event, query_id), folds the window into the tally carried in throughcontext.State, and emits the cumulative tally as rows in the extension arm's exact column shape withprofile_period_ms = 1000— soDarlingPgWaitSamplingReader's newest-minus-oldest differencing and itscounter_resetarm work unchanged, andestimated_wait_ms(samples × period) is right on both arms. The tally is capped at 500 keys (the extension arm'sLIMIT), least-sampled dropped.collector_state (server_id, 'pg_wait_sampling', 'instrument')as aPgWaitInstrumenttoken. The sampler's exclusions: its own backend, the service's other connections byapplication_name(pinned equal to the connector's literals), and the same three idle wait TYPES the extension arm excludes, spliced from the same set. A running backend with no wait isCPU/Running, as on the extension arm.query_idis selected only on PG14+.Tier selection is one decision at target attach, in the issue's order: Aurora native >
pg_wait_samplingpresent > service sampler.pg_wait_sampling.AppliesTois now!IsAurora(wastrue): Aurora cannot preload the module, so the hourlyEXTENSION_MISSINGit recorded there was a permanent gap wearing a fixable precondition's clothes, and the sampler arm running besidepg_wait_statswould be two answers to one question on the one engine with real counters.CollectorEngineCapability.CoveredInsteadBygainspg_wait_sampling → pg_wait_stats, the mirror of the pointer that already ran the other way, so an Aurora caller ofget_pg_wait_samplingis sent to the instrument that answers instead of told to install something they cannot. The new fact is swept as FIXABLE inTargetsWithEngineKind(the existing IL-decoding guardEveryFactAPostgresGateReads_IsVariedBySweepOrFixedByKinddemanded it, and passes). A new Lite test asserts over every swept PostgreSQL shape that exactly one ofpg_wait_stats/pg_wait_samplingapplies.Reads disclose the instrument.
get_pg_wait_samplingcarriesinstrument(extension_sampled|service_sampled|unknownfor a pre-#3604 store),instrument_recorded_at, andinstrument_note— on the service tier that isPgWaitInstrument.ServiceSampledCaveat, the one copy of the floor caveat, and the empty arm states it too.get_pg_wait_statscarriesinstrument: engine_cumulative. Both descriptions and a newDarlingMcpInstructionssection name the three tiers and the order. The Viewer's sampled-waits panel note names the arm.Why this shape and not a third collector
The catalog's one-collector-one-table rule is load-bearing:
PgSchemaGeneratorTestspinsAll.Count == Distinct(TargetTable).Count, and DDL generation, retention and the table census all rest on it. APgWaitSamplerCollectorwriting intopg_wait_samplingwould break that; its own table is a rung, and the ladder is spoken for. A target-aware definition is the established shape (cpu_utilization's Azure-vs-ring-buffer fork,pg_statement_stats' Aurora-vs-vanilla one). Provenance therefore cannot ridecollection_log.collector_name— the arm records itself in the collector's own state instead, which is what the reads consult.Cadence and detach. The collector moves from hourly to five minutes and is detached from the sequential sweep body (the third member after
query_storeandplan_correction, through the generic per-(server, collector) gate). Both halves of the hourly argument dissolved: it recordsEXTENSION_MISSINGon no target now, and on the sampler arm the cadence IS the duty cycle. Awaited inline, a deliberate 30 s run would delay every other collector on a stock target by half a minute every five — the #2700 starvation with a known cause.SweepBodyDetachPolicyTestspins that nothing detached sits on the one-minute tier, which is why the cadence is five and not one.Why 30 s at 1 Hz, and why that is a floor.
SweepPressureClassifiersums every collector's single-run cost against a 60,000 ms body budget and calls itBODY_OVERRUNpast that, with no detached exclusion — a four-minute sampling run would make every stock target read as saturated. Thirty seconds is the densest honest window that leaves that surface truthful; it is still 30× the resolutionpg_blockingandpg_lock_statsget from one snapshot a minute. One second rather than five because each doubling of the period halves the odds of seeing a wait shorter than it, while the cost does not move: measured on the rig, one snapshot executes in 0.065 ms (1.5 ms planning) at 12 backends, from a connection the pool holds open regardless. The honest limit is in every read that serves the tier: a wait shorter than a second is seen with probability roughly its length over a second; nothing between samples or between windows (a 10% duty cycle) is seen; shares are trustworthy for steady fractions of the server's time and approximate for rare short events. The tally is persisted collector state, reloaded every cycle, so it SURVIVES a service restart; a series starts over only on cap eviction (500 keys, least-sampled dropped) or on a reversion from the extension arm, and the reader's existingcounter_resetarm reports either.Measured on the rig
timescale/timescaledb:2.28.1-pg18as both store and stock target (nopg_wait_sampling), with an ACCESS EXCLUSIVE lock held and a blockedSELECTplus agenerate_seriesCPU burner running, one realDarlingCollectorRunner.RunAsynccycle: wall 29,184 ms, sql 29,130 ms (29,000 of it deliberate sleep), store 13 ms, 3 series —Lock/relationseen in 30/30 snapshots withbackend_count1,CPU/Running29/30, TimescaleDB's own workers asExtension/Extension;profile_period_ms = 1000on every row;collector_state.instrument = service_sampled; the tally round-tripped;BuildWaitSamplingJsonreturnedinstrument: service_sampledwith the floor caveat. The holder (idle in transaction,Client) and noActivity/Timeoutrow appeared — the shared exclusions hold on this arm.Least privilege
Nothing beyond
pg_monitor, which every PostgreSQL collector here already needs:pg_stat_activity's wait columns for other users' backends come frompg_read_all_stats, which it carries;pg_stat_clear_snapshot()andpg_sleep()are callable by any role. The first-target runbook says so where the grants are listed, and its cadence table and troubleshooting rows are updated.RequiredPgExtensionsstill declares the module with a narrowed meaning (the README's permissions paragraph is pinned to it, and a module dropped mid-connection on the extension arm still classifies asEXTENSION_MISSING).What it does NOT do
pg_wait_stats; adding a floor beside a ceiling would be noise.pg_plan_capture_readiness: that collector's facets are plan-capture readiness by contract ("never a claim that a plan was captured"), and its tool deliberately reduces to no verdict. The instrument disclosure lives on the wait reads themselves and in the collector's own state;--test-connectionshows the arm through the gate.StateKeys/collector_stateinstead (the default_trace_events re-reads the entire rollover set every cycle - validated 5.0x fix: read the current file, fall back on rollover #1962 mechanism), which is what makes it survive a service restart. The extension arm clears it every cycle it runs, so a later fallback to the sampler starts at zero rather than resuming a stale era (review catch, pinned).backend_counton the service tier is the WINDOW's distinct pids, not cumulative (pids recycle); the note says so.Tests
Lite.Tests/PgWaitSamplerArmTests.cs(13, executed here, green via the mactest pattern): exactly-one-collector over every swept PostgreSQL shape; arm follows the fact; Aurora permanent gap namespg_wait_stats; batch statement counts and clear-before-read order; window fits the body budget and command timeout; snapshot exclusions and CPU labelling;query_idby version; accumulation from a fixture result-set sequence with window-distinct backends; cumulative carry across cycles with every carried key re-emitted; empty window; tally cap; tolerant tally parse; extension arm records its instrument;StateKeys; vocabulary.Darling/Darling.Tests/PgWaitInstrumentDisclosureTests.cs(11, executed here): service/extension/unknown disclosure in the JSON; empty-arm sentence; the state SQL reads the collector's own row; Aurora read disclosesengine_cumulative; both descriptions and the instructions name the tiers; excluded application names match the connector; connect probe SQL andConnectionProbeResultplumbing; detach predicate.Darling/Darling.Tests/PgWaitSamplerLiveTests.cs(1, executed here against the rig,DARLING_TEST_PG-gated like its siblings): the run described above.SweepBodyDetachPolicyTests(third member, documented),CollectorStateContractTests(second state-declaring collector),PgWaitSamplingCollectorDefinitionTests/PgWaitExclusionParityTestscontexts take the extension arm.EngineCapabilityMissTests(incl. the IL gate-fact sweep),PgSchemaGeneratorTests,ReadmeDerivedCountPinTests,PgExtensionDependencyContractTests,CollectorStateContractTests,DarlingMcpPgPercentDenominatorTests— 93/93 Darling.Tests in the harness, 26/26 Lite.Tests.-c Release -p:EnableWindowsTargeting=truewith 0 warnings. WPF-only paths (the Viewer note) are compile-verified; CI is their first execution.Closes #3604