Skip to content

Darling: the store gains a collector run time table, and the shared run-time rules (part of #4938) - #5007

Merged
erikdarlingdata merged 21 commits into
devfrom
feature/4938-collector-run-time
Oct 3, 2026
Merged

erikdarlingdata merged 21 commits into
devfrom
feature/4938-collector-run-time

Conversation

@erikdarlingdata

@erikdarlingdata erikdarlingdata commented Oct 3, 2026 •

Copy link
Copy Markdown
Owner

Part of #4938.

Why

A collector that runs once a day, such as the index object statistics read, starts whenever the Darling service happened to start. Its load can land in the busiest hour of the monitored server. #4938 gives such a collector an optional run time: a quiet hour the operator picks.

This PR lays the groundwork only. It adds the table that stores the time, and the rules every part of the product will share when it reads, writes or applies one. The worker, the viewer, the CLI, MCP and Lite then cannot each invent their own.

The run time lives in a table of its own, not a column on the schedules table. The viewer's schedule Save deletes a scope's rows in config.config_collector_schedules and inserts them again with a fixed column list (version 3.9.0 ships the same Save). The connect check refuses only an older store, so every released viewer connects to a version 160 store. A column on the schedules table would be set back to NULL by each of their Saves, and a server's -1 would become NULL, so that server would take the fleet's run time. Released viewers cannot be changed, so the run time sits where their Save never writes.

What changes

  • Store version 160 (V160) creates config.config_collector_run_times. One row per collector, fleet-wide when server_id is NULL: collector_name, and run_at_minute, a smallint of minutes after midnight on the monitored server's own clock, 0 to 1439. A -1 means "no fixed time on this server". It stops a fleet-wide time, the way an empty array stops a fleet-wide database scope, so a CHECK allows -1 on a server row only. The same two partial unique indexes as the schedules table keep one fleet row and one server row per collector. No row means no run time, so every install that never sets one runs exactly as before. Every statement is IF NOT EXISTS or a drop and create, so a second run changes nothing. It sits on top of the install id's final shape (V159) and does not touch that table or the schedules table. The plan's migration 158 ships as 160.
  • A write to the table bumps the reload counter, with the same statement-level trigger the schedules table has, so a running service reloads the run times.
  • Server removal. Nothing deletes a removed server's schedule rows, and the table has no foreign key: removing a server deletes its definition only (ViewerDataService.MonitoredServers.cs line 98 and DarlingMcpServerAdminTools.cs line 390). The run-time table matches that: no foreign key, no cleanup.
  • Grants. No grant or role list names the schedules table. Provisioning grants the whole config schema (GRANT ... ON ALL TABLES IN SCHEMA config, and default privileges for new tables), so the new table has the same read and write rights with no change.
  • The shared run-time rules (CollectorRunTime, next to CollectorCadence). A run time is a 24-hour HH:MM time. It applies only to a collector whose interval is a whole number of days (an on-load collector counts as daily). The slot is taken on the monitored server's own clock. Each server gets a fixed offset of up to one hour, so a fleet does not start together. A run can start up to one hour after its slot. The messages every editor will show are shared too. CadencePhaseOffset moved from the Darling worker into CollectorCadence so these rules can use it. The worker forwards to it, with the values unchanged.
  • The service reads the table and resolves it. The schedule read is back to its original columns. A second read takes the run times, and each is carried onto the schedule row for the same scope and collector. A run time with no schedule row gets an entry that sets nothing else (its enabled state is now nullable there, so it cannot switch a collector on or off). ResolveSchedule layers the run time like the other columns: the server row, then the fleet row, then none. It treats -1 as none. It drops a run time whose effective interval is not a whole number of days. The service logs one warning for each such run time each time the schedules load, not each sweep.
  • A store at version 159 has no run-time table. The read answers "no run times", with no error and no warning (one debug line at most), and the rest of the load goes on.
  • A stored value the CHECK would refuse (a store whose CHECK was dropped by hand) resolves to none, and the load now warns once, naming the collector, the scope and the value. Before, it was dropped without a word.
  • The viewer's store version check asks for the table by name. A store the service has not yet migrated to 160 is reported as behind. The viewer's schedule Save is unchanged.
  • A schedule defaults comment said the PostgreSQL index bloat collector reads every page of an index. It now says the collector estimates from the catalog (Index bloat estimate arm (pg_stats-based): write the estimator #2561 measured, with the pg_read_all_data grant as its precondition #3234).

Not in this PR

Nothing schedules by the run time yet. The worker's due-time logic, the viewer's editor, the CLI, MCP and Lite follow in the next PR. With no run time set (the only state after this PR), every schedule resolves exactly as before.

Test plan

  • Build: Darling.Tests (with the service and the viewer), 0 warnings, 0 errors.
  • Without a store: 353 tests, 0 failed, 16 live tests skipped. These are the run-time, rung, store version, migration ladder, viewer schema, probe, repo-file adoption, doc-comment, comment-filter, collector coverage, managed roles, schedule SQL, sweep scheduling and provider classes.
  • Against a throwaway PostgreSQL cluster with TimescaleDB, the same classes: 323 tests, 0 failed, 0 skipped. The 8 run-time live tests are in it.
  • The viewer's schedule Save, run live with its real SQL, leaves every run-time row as it was: for a fleet scope, for a server scope, for a Save that leaves a scope empty, and for the reset of every server's rows. The schedule rows change as the Save says. At the previous head the old pin showed the opposite: the Save set the column to NULL.
  • The rung's SQL run twice on one store that holds rows: no error, the same columns, indexes, constraint, trigger and rows. (The old test migrated a store already stamped 160, so the runner skipped the rung and its SQL never ran twice.)
  • The table's columns, its CHECK (-2, 1440 and a fleet -1 refused; 0, 1439 and a server -1 taken), both unique indexes and the reload trigger.
  • A store at 159 migrates to 160 with an empty table, its schedule row and the schedules table's columns unchanged, and the install id row and table shape unchanged.
  • A store without the table loads with no run times, no warning and no error, and a second load works.
  • A value outside the CHECK's range logs one warning naming the collector, the scope and the value, live and without a store.
  • With the rung body emptied and the new warning switched off, five pins fail (the rung SQL pin, the 159-to-160 upgrade, the two-run test, the warning without a store, the warning live). Restored, they pass.
  • The full suite was not run, and neither was every class that reads a changed file.

Notes

  • Nothing is left from the five items. The follow-up writes run times to the new table (the viewer's editor, the CLI and MCP), and has to decide what the viewer's reset of a schedule does with a run time. Today no schedule reset or Save touches the table.
  • A run time for a removed server stays in the table, as its schedule rows do.

CHANGELOG

No entry in this PR; the release owner adds it under [Unreleased]. Proposed entry:

  • Darling's store can hold a run time for each collector schedule ([Set time on Collector Schedules #4938]) - The store gains a collector run-time table (store version 160). It holds minutes after midnight on the monitored server's own clock, with -1 meaning no fixed time on a server, and it is empty until a run time is set, so nothing changes yet. The worker, viewer, CLI, MCP tools and Lite use it in a follow-up.

…4938, package 1)

Adds CollectorRunTime to PerformanceMonitor.Collectors: the pure rules for
running a once-a-day collector at a chosen time on the monitored server's
wall clock. Nothing calls it yet except tests, so no behavior changes.

What it holds:
- SlotUtc: the UTC instant of a server's slot on a local date.
- NextDue: the next due time from now, the last run (nullable), the run-time
  minute, the interval and the server id.
- Spread (a fixed per-server offset under 60 minutes, from
  CadencePhaseOffset(serverId, 3600)) and Grace (60 minutes), both constants.
- AllowsRunAt: only a positive multiple of 1440 minutes.
- TryParse / Format for "HH:MM".
- MaxStampAhead: how far ahead a run-time stamp can sit (interval + 60 + 60
  minutes), for the clock-step check, so a stamp on a 25-hour autumn day is
  not read as a clock that stepped back.
- InvalidRunAtMessage and IntervalRefusalMessage: the two error texts, one
  copy for every editor.
- LocalIsUtc: the conversion to pass while a server's clock is not known.

CadencePhaseOffset moves from DarlingWorker to CollectorCadence (same code,
public). Its values are pinned first, as 24 literal (id, period) cases
including period 3600, which the existing pins did not reach, and they pass
unchanged on both sides of the move. DarlingWorker keeps a one-line
forwarding member, so the seed jitter, the cold-start spread, the
on-connect analysis stamp and their existing tests keep one spelling.

Design calls:
- The rules never know a time zone. They take a local-to-UTC delegate (an
  app passes ServerClock.ToUtc; LocalIsUtc while the clock is unknown).
- Only local-to-UTC is available, so the date of "now" is found by probing
  the few candidate dates around the UTC date (an offset is within 14 hours)
  and taking the latest slot at or before the instant. The same probe dates a
  last run. No UTC-to-local conversion is needed.
- A day is identified by its slot, not by the calendar date of a run. A slot
  that spills past local midnight (a late run time plus the spread) keeps the
  date it was computed for, so the run after it is one day later, not two.
  A run before the day's slot (an on-load run in the small hours) counts for
  the day before, so that slot is still owed.
- Each day's slot is computed from that date's local time (run time plus the
  spread, converted), never as the last slot plus 1440 minutes, so it stays
  at the same local time across a daylight-saving change.
- With a last run, the slots are every interval/1440 days from that run's
  date, so a two-day collector keeps its two-day grid. The latest slot at or
  before now is due when now is inside its grace (the answer is now, and the
  caller adds the seed jitter). Past the grace that day is skipped and the
  answer is the next grid slot. A slot still ahead is returned as is. With no
  last run the slots are every day. After a run, pass the run's instant (or
  the slot it ran for) as the last run.
- The grace is closed at both ends: a start at the slot, or at the slot plus
  60 minutes, is inside.
- A last run at the bottom of the calendar (DateTime.MinValue) is read as
  never run, so a sentinel cannot throw in the date arithmetic.
- TryParse takes exactly HH:MM (two digits, ASCII, 00:00 to 23:59) and
  ignores surrounding spaces; "24:00", "2:5", "2:00" and "abc" are refused.
- NextDue throws for an interval AllowsRunAt refuses or a run-time minute
  outside 0 to 1439; callers validate with AllowsRunAt first.

Tests (Darling.Tests, which references both projects, so the daylight-saving
cases use the real ServerClock.ToUtc): 74 in CollectorRunTimeTests, covering
before the slot, after the grace, inside the grace, a missed day, a
two-day interval, a spring-forward and a fall-back day in US Eastern, every
day of a year at the same local time, a fixed offset across the UTC date, an
unknown clock, the spread, the clamp bound, AllowsRunAt, parse and format
round trips and refusals, and a sweep over six clocks against a brute-force
check of every candidate date. The tests were run against stubs that return
wrong answers first: 42 of 74 failed.
…nd resolve (#4938, package 2a)

Migration 158 adds config.config_collector_schedules.run_at_minute, a nullable smallint of minutes
after midnight on the server's clock, CHECK -1 to 1439. NULL falls through per column as the
databases column does; -1 means no fixed time on this server and stops a fleet-wide time. The CHECK
is inline on the ADD COLUMN, so the statement stays idempotent and adds no validation pass. Nothing
writes the column yet.

The version bump brings its obligations: StorageVersion 158, the viewer probe's sentinel with its
MapProbedSchemaVersion arm and GetBoolean(133), the previous top rung's tests handed off, the
version-simulating live test drops the new column, and the pinned service SELECT now ends with it.

ScheduleOverride and EffectiveSchedule carry the run time, and the service's store read fills it.
ResolveSchedule: the server row's value wins (-1 resolves to none), else the fleet row's, else none;
a value outside the CHECK's range counts as not set. A value where the effective interval is not a
whole number of days (CollectorRunTime.AllowsRunAt on the recurring interval, so an on-load collector
counts as daily) resolves to none.

The warning for a dropped run time is raised in StoreConfigProvider.LoadViewAsync, after the servers
and the schedules are read, not in ResolveSchedule: that runs every sweep and stays pure and silent,
while a load happens once per config_version bump, which is once per schedule load. It is judged per
enabled server, not per row, because a per-server frequency override changes the interval (a
fleet-wide time can be fine on one server and refused on another), and a collector the server's
engine never runs is skipped. One warning per dropped run time names the collector, the server, which
row set the time, and the shared refusal text.

Also fixes the stale pg_index_bloat comment: since #3234 it estimates from pg_stats and the catalog
and reads no index pages.
…and word the probe comment without the banned term
…uster id is rebound (#4961, #4967)

Tests for the rule as a truth table, the live rebind and replacement facts, the new migration rung and its probe sentinel, and a reader on a store still below the rung.
…, #4967)

pg_upgrade makes a new cluster identifier and keeps the OIDs of the database and of the install id table. Migration 159 adds table_oid to the install id row. The service decides whether a row is this store's by the two OIDs once the row has its table OID, rebinds a changed cluster identifier without changing the id, and names which binding differed when it makes a new id. A row with no table OID follows the old rule once and has it filled in.
… cannot be saved is not used, a create is a move (#4961, #4967)
…e it could not save; a new id is created by a move from a temp file (#4961, #4967)
…plica only, with the Microsoft Learn link, at each stop and in both READMEs (#4961, #4967)
…-run-time

# Conflicts:
#	Darling/Darling.Tests/CaggGroupIndexDropLiveTests.cs
#	Darling/Darling.Tests/MuteRuleServerIdRungTests.cs
#	Darling/PerformanceMonitor.Darling.Storage/PgMigrations.cs
#	Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.cs
…o feature/4938-collector-run-time

# Conflicts:
#	Darling/Darling.Tests/CaggGroupIndexDropLiveTests.cs
#	Darling/PerformanceMonitor.Darling.Storage/PgMigrations.cs
#	Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.cs
…V158 and V159 (#4938)

The run-time column's migration is version 160, appended after the install id's two rungs, and
StorageVersion is 160. The viewer probe's sentinel for the column is the last arm (ordinal 135),
and the version map has a newest-first arm for it that returns 160.

The install id table OID rung (V159) is no longer the top of the ladder, so its tests hand the top
off the way the earlier rungs' did: the rung is registered in a dense ladder below the top, and a
store that stops at its sentinel maps to it. The run-time rung tests and their live half follow the
new number: the map's all-true and one-behind cases, the arm's return value, and the upgrade test
that starts from V159 and rewinds the version stamp from 160.

Comments that named the column's version now say V160.
…'s final shape, #4986)

This branch was cut on an earlier copy of the install id work. Dev now holds its final shape
(V159 adds table_oid and server_major to config.config_install_id), so every conflict in that
work resolves to dev's version, and the collector run time stays on top as V160.

Conflicted files and how each resolved:
- PgMigrations.cs: V159 is dev's (table_oid and server_major in one statement); V160 and its
  list entry are kept after it.
- StorageVersion.cs: 160.
- ViewerDataService.cs: this branch's V160 probe sentinel, its map parameter and its arm are
  kept on top of dev's V159 sentinel; the V159 arm's comment says it is no longer the top rung.
- CaggGroupIndexDropLiveTests.cs: dev's V159 removal (both columns) plus this branch's V160
  removal above it.
- InstallIdTableOidRungTests.cs: dev's version, with the V159 rung's tests handed off from
  "top rung" to "dense ladder, newer rung above".
- StoreInstallId.cs, Lite/Services/InstallIdStore.cs, StoreInstallIdBindingRuleTests.cs,
  StoreInstallIdLiveTests.cs, Darling/README.md: dev's version (the early copy of the install
  id work, nothing of this branch's own).
…stall id row and table shape (#4938)

A live test starts from a store at the install id's final shape (V159: table_oid and server_major),
puts an install id row in it, runs the migrations, and checks that the store reaches 160, the run
time column exists, the row reads the same, and the install id table keeps exactly its seven columns
and types. A fresh store is also checked to migrate to 160 with that shape.

With the V160 entry removed from the migration list, both 159-to-160 live tests fail.
…e Save from any viewer cannot clear it (#4938)

The viewer's schedule Save deletes a scope's rows in config_collector_schedules and inserts them again with a fixed column list, and every released viewer connects to a store that is newer than it is. A run_at_minute column on that table would be set back to NULL by each Save.

V160 now creates config.config_collector_run_times (server_id NULL = fleet-wide, collector_name, run_at_minute smallint NOT NULL, a CHECK of 0 to 1439 or -1 on a server row only, the same two partial unique indexes as the schedules table, and V17's reload trigger). No row means no run time. The column is gone from V160. Like the schedules table it has no foreign key and no cleanup when a server is removed: removing a server deletes its definition only.

The service reads the table and merges each run time onto the schedule row for the same scope and collector; a run time with no schedule row gets an entry that sets nothing else (Enabled is now nullable on that entry). A store at V159, which has no table, reads as no run times with at most one debug line. A stored value outside the CHECK's range is now warned about once per load, naming the collector, the scope and the value.

The viewer's probe for V160 asks for the table by name. The rung's SQL is now run twice on one store in a test, because the runner skips a rung a store is already stamped at.
@erikdarlingdata erikdarlingdata changed the title Darling: the collector schedules gain an optional run time column, and the shared run-time rules (part of #4938) Darling: the store gains a collector run time table, and the shared run-time rules (part of #4938) Oct 3, 2026
…n the serial-loop deadline (#4938)

The schedule read split into ReadScheduleRowsAsync and ReadRunTimesAsync, each building one command on
ServiceCommandDeadlines.SerialLoopSeconds. ReadScheduleOverridesAsync only layers one onto the other and builds
none, so the census listed a member with no command in it.

The census now lists the two new members in place of the old one and counts eleven serial-loop commands in ten
members. The comments that state those counts follow: eleven commands on the serial loop, eight of them also run
once at startup.
@erikdarlingdata
erikdarlingdata marked this pull request as ready for review October 3, 2026 08:49
@erikdarlingdata
erikdarlingdata merged commit 9d6c042 into dev Oct 3, 2026
21 of 22 checks passed
@erikdarlingdata
erikdarlingdata deleted the feature/4938-collector-run-time branch October 3, 2026 08:51
erikdarlingdata added a commit that referenced this pull request Oct 3, 2026
… every surface, and a schedule Reset clears it (#4938) (#5018)

A daily collector can have a run time: a fixed time of day such as 02:00 on the monitored server's clock, so a heavy collector runs in a quiet hour instead of whenever the service started. The table config.config_collector_run_times (store version 160, #5007) and the detached daily runs with their fleet-wide cap (#4999) were already on dev; this moves every surface onto that table.

Darling: the viewer's Collector Schedules window has a Run at column and saves run times and schedule rows in one transaction; --set-collector-run-at sets, stops or clears a run time fleet-wide or per server, and the command executor writes the same table with the catalog's canonical collector name; get_collection_health and the web Collection Health tab show each collector's run time and next run. The worker starts a collector with a run time at its slot (the run time plus a per-server spread under 60 minutes) and hands it off until 60 minutes after; a day it cannot hand off in that hour is skipped, not replayed. A slot counts as served only when the run takes its per-(server, collector) slot, and a run on load seeds the slot only when it succeeded. A store whose MCP role cannot read the table (42501) answers with no run times and one warning, and provision-roles.sql grants that read. A PostgreSQL server whose TimeZone does not resolve logs one warning per server and uses UTC.

Lite: the schedule file takes an optional run_at, the schedule editor sets and checks it, and the sweep runs such a collector once a day in the same hour-long window. At start-up Lite reads each daily collector's last run and the server's clock, so a restart no longer re-runs a collector that ran today.

Reset: Reset to Defaults, a server's Use default schedule and Apply Default to All Servers also delete the run times of the scope they reset, in the same transaction as the schedule rows. A viewer released before this change resets the schedule rows only. On a store below V160 the run-time half is skipped.

The docs say what the run time sets, that a run held by the fleet's cap can start after the hour, and that a run can start nearly two hours after run_at. A same-tick race between a run-time edit and a daily run's start is filed as #5033.

Closes #4938
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant