Repository navigation
Darling: the store gains a collector run time table, and the shared run-time rules (part of #4938) - #5007
Merged
Conversation
…4938, package 1) Adds CollectorRunTime to PerformanceMonitor.Collectors: the pure rules for running a once-a-day collector at a chosen time on the monitored server's wall clock. Nothing calls it yet except tests, so no behavior changes. What it holds: - SlotUtc: the UTC instant of a server's slot on a local date. - NextDue: the next due time from now, the last run (nullable), the run-time minute, the interval and the server id. - Spread (a fixed per-server offset under 60 minutes, from CadencePhaseOffset(serverId, 3600)) and Grace (60 minutes), both constants. - AllowsRunAt: only a positive multiple of 1440 minutes. - TryParse / Format for "HH:MM". - MaxStampAhead: how far ahead a run-time stamp can sit (interval + 60 + 60 minutes), for the clock-step check, so a stamp on a 25-hour autumn day is not read as a clock that stepped back. - InvalidRunAtMessage and IntervalRefusalMessage: the two error texts, one copy for every editor. - LocalIsUtc: the conversion to pass while a server's clock is not known. CadencePhaseOffset moves from DarlingWorker to CollectorCadence (same code, public). Its values are pinned first, as 24 literal (id, period) cases including period 3600, which the existing pins did not reach, and they pass unchanged on both sides of the move. DarlingWorker keeps a one-line forwarding member, so the seed jitter, the cold-start spread, the on-connect analysis stamp and their existing tests keep one spelling. Design calls: - The rules never know a time zone. They take a local-to-UTC delegate (an app passes ServerClock.ToUtc; LocalIsUtc while the clock is unknown). - Only local-to-UTC is available, so the date of "now" is found by probing the few candidate dates around the UTC date (an offset is within 14 hours) and taking the latest slot at or before the instant. The same probe dates a last run. No UTC-to-local conversion is needed. - A day is identified by its slot, not by the calendar date of a run. A slot that spills past local midnight (a late run time plus the spread) keeps the date it was computed for, so the run after it is one day later, not two. A run before the day's slot (an on-load run in the small hours) counts for the day before, so that slot is still owed. - Each day's slot is computed from that date's local time (run time plus the spread, converted), never as the last slot plus 1440 minutes, so it stays at the same local time across a daylight-saving change. - With a last run, the slots are every interval/1440 days from that run's date, so a two-day collector keeps its two-day grid. The latest slot at or before now is due when now is inside its grace (the answer is now, and the caller adds the seed jitter). Past the grace that day is skipped and the answer is the next grid slot. A slot still ahead is returned as is. With no last run the slots are every day. After a run, pass the run's instant (or the slot it ran for) as the last run. - The grace is closed at both ends: a start at the slot, or at the slot plus 60 minutes, is inside. - A last run at the bottom of the calendar (DateTime.MinValue) is read as never run, so a sentinel cannot throw in the date arithmetic. - TryParse takes exactly HH:MM (two digits, ASCII, 00:00 to 23:59) and ignores surrounding spaces; "24:00", "2:5", "2:00" and "abc" are refused. - NextDue throws for an interval AllowsRunAt refuses or a run-time minute outside 0 to 1439; callers validate with AllowsRunAt first. Tests (Darling.Tests, which references both projects, so the daylight-saving cases use the real ServerClock.ToUtc): 74 in CollectorRunTimeTests, covering before the slot, after the grace, inside the grace, a missed day, a two-day interval, a spring-forward and a fall-back day in US Eastern, every day of a year at the same local time, a fixed offset across the UTC date, an unknown clock, the spread, the clamp bound, AllowsRunAt, parse and format round trips and refusals, and a sweep over six clocks against a brute-force check of every candidate date. The tests were run against stubs that return wrong answers first: 42 of 74 failed.
…nd resolve (#4938, package 2a) Migration 158 adds config.config_collector_schedules.run_at_minute, a nullable smallint of minutes after midnight on the server's clock, CHECK -1 to 1439. NULL falls through per column as the databases column does; -1 means no fixed time on this server and stops a fleet-wide time. The CHECK is inline on the ADD COLUMN, so the statement stays idempotent and adds no validation pass. Nothing writes the column yet. The version bump brings its obligations: StorageVersion 158, the viewer probe's sentinel with its MapProbedSchemaVersion arm and GetBoolean(133), the previous top rung's tests handed off, the version-simulating live test drops the new column, and the pinned service SELECT now ends with it. ScheduleOverride and EffectiveSchedule carry the run time, and the service's store read fills it. ResolveSchedule: the server row's value wins (-1 resolves to none), else the fleet row's, else none; a value outside the CHECK's range counts as not set. A value where the effective interval is not a whole number of days (CollectorRunTime.AllowsRunAt on the recurring interval, so an on-load collector counts as daily) resolves to none. The warning for a dropped run time is raised in StoreConfigProvider.LoadViewAsync, after the servers and the schedules are read, not in ResolveSchedule: that runs every sweep and stays pure and silent, while a load happens once per config_version bump, which is once per schedule load. It is judged per enabled server, not per row, because a per-server frequency override changes the interval (a fleet-wide time can be fine on one server and refused on another), and a collector the server's engine never runs is skipped. One warning per dropped run time names the collector, the server, which row set the time, and the shared refusal text. Also fixes the stale pg_index_bloat comment: since #3234 it estimates from pg_stats and the catalog and reads no index pages.
…and word the probe comment without the banned term
…, #4967) pg_upgrade makes a new cluster identifier and keeps the OIDs of the database and of the install id table. Migration 159 adds table_oid to the install id row. The service decides whether a row is this store's by the two OIDs once the row has its table OID, rebinds a changed cluster identifier without changing the id, and names which binding differed when it makes a new id. A row with no table OID follows the old rule once and has it filled in.
…-run-time # Conflicts: # Darling/Darling.Tests/CaggGroupIndexDropLiveTests.cs # Darling/Darling.Tests/MuteRuleServerIdRungTests.cs # Darling/PerformanceMonitor.Darling.Storage/PgMigrations.cs # Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.cs
…o feature/4938-collector-run-time # Conflicts: # Darling/Darling.Tests/CaggGroupIndexDropLiveTests.cs # Darling/PerformanceMonitor.Darling.Storage/PgMigrations.cs # Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.cs
…V158 and V159 (#4938) The run-time column's migration is version 160, appended after the install id's two rungs, and StorageVersion is 160. The viewer probe's sentinel for the column is the last arm (ordinal 135), and the version map has a newest-first arm for it that returns 160. The install id table OID rung (V159) is no longer the top of the ladder, so its tests hand the top off the way the earlier rungs' did: the rung is registered in a dense ladder below the top, and a store that stops at its sentinel maps to it. The run-time rung tests and their live half follow the new number: the map's all-true and one-behind cases, the arm's return value, and the upgrade test that starts from V159 and rewinds the version stamp from 160. Comments that named the column's version now say V160.
…'s final shape, #4986) This branch was cut on an earlier copy of the install id work. Dev now holds its final shape (V159 adds table_oid and server_major to config.config_install_id), so every conflict in that work resolves to dev's version, and the collector run time stays on top as V160. Conflicted files and how each resolved: - PgMigrations.cs: V159 is dev's (table_oid and server_major in one statement); V160 and its list entry are kept after it. - StorageVersion.cs: 160. - ViewerDataService.cs: this branch's V160 probe sentinel, its map parameter and its arm are kept on top of dev's V159 sentinel; the V159 arm's comment says it is no longer the top rung. - CaggGroupIndexDropLiveTests.cs: dev's V159 removal (both columns) plus this branch's V160 removal above it. - InstallIdTableOidRungTests.cs: dev's version, with the V159 rung's tests handed off from "top rung" to "dense ladder, newer rung above". - StoreInstallId.cs, Lite/Services/InstallIdStore.cs, StoreInstallIdBindingRuleTests.cs, StoreInstallIdLiveTests.cs, Darling/README.md: dev's version (the early copy of the install id work, nothing of this branch's own).
…stall id row and table shape (#4938) A live test starts from a store at the install id's final shape (V159: table_oid and server_major), puts an install id row in it, runs the migrations, and checks that the store reaches 160, the run time column exists, the row reads the same, and the install id table keeps exactly its seven columns and types. A fresh store is also checked to migrate to 160 with that shape. With the V160 entry removed from the migration list, both 159-to-160 live tests fail.
…e Save from any viewer cannot clear it (#4938) The viewer's schedule Save deletes a scope's rows in config_collector_schedules and inserts them again with a fixed column list, and every released viewer connects to a store that is newer than it is. A run_at_minute column on that table would be set back to NULL by each Save. V160 now creates config.config_collector_run_times (server_id NULL = fleet-wide, collector_name, run_at_minute smallint NOT NULL, a CHECK of 0 to 1439 or -1 on a server row only, the same two partial unique indexes as the schedules table, and V17's reload trigger). No row means no run time. The column is gone from V160. Like the schedules table it has no foreign key and no cleanup when a server is removed: removing a server deletes its definition only. The service reads the table and merges each run time onto the schedule row for the same scope and collector; a run time with no schedule row gets an entry that sets nothing else (Enabled is now nullable on that entry). A store at V159, which has no table, reads as no run times with at most one debug line. A stored value outside the CHECK's range is now warned about once per load, naming the collector, the scope and the value. The viewer's probe for V160 asks for the table by name. The rung's SQL is now run twice on one store in a test, because the runner skips a rung a store is already stamped at.
…n the serial-loop deadline (#4938) The schedule read split into ReadScheduleRowsAsync and ReadRunTimesAsync, each building one command on ServiceCommandDeadlines.SerialLoopSeconds. ReadScheduleOverridesAsync only layers one onto the other and builds none, so the census listed a member with no command in it. The census now lists the two new members in place of the old one and counts eleven serial-loop commands in ten members. The comments that state those counts follow: eleven commands on the serial loop, eight of them also run once at startup.
erikdarlingdata
marked this pull request as ready for review
October 3, 2026 08:49
erikdarlingdata
added a commit
that referenced
this pull request
Oct 3, 2026
… every surface, and a schedule Reset clears it (#4938) (#5018) A daily collector can have a run time: a fixed time of day such as 02:00 on the monitored server's clock, so a heavy collector runs in a quiet hour instead of whenever the service started. The table config.config_collector_run_times (store version 160, #5007) and the detached daily runs with their fleet-wide cap (#4999) were already on dev; this moves every surface onto that table. Darling: the viewer's Collector Schedules window has a Run at column and saves run times and schedule rows in one transaction; --set-collector-run-at sets, stops or clears a run time fleet-wide or per server, and the command executor writes the same table with the catalog's canonical collector name; get_collection_health and the web Collection Health tab show each collector's run time and next run. The worker starts a collector with a run time at its slot (the run time plus a per-server spread under 60 minutes) and hands it off until 60 minutes after; a day it cannot hand off in that hour is skipped, not replayed. A slot counts as served only when the run takes its per-(server, collector) slot, and a run on load seeds the slot only when it succeeded. A store whose MCP role cannot read the table (42501) answers with no run times and one warning, and provision-roles.sql grants that read. A PostgreSQL server whose TimeZone does not resolve logs one warning per server and uses UTC. Lite: the schedule file takes an optional run_at, the schedule editor sets and checks it, and the sweep runs such a collector once a day in the same hour-long window. At start-up Lite reads each daily collector's last run and the server's clock, so a restart no longer re-runs a collector that ran today. Reset: Reset to Defaults, a server's Use default schedule and Apply Default to All Servers also delete the run times of the scope they reset, in the same transaction as the schedule rows. A viewer released before this change resets the schedule rows only. On a store below V160 the run-time half is skipped. The docs say what the run time sets, that a run held by the fleet's cap can start after the hour, and that a run can start nearly two hours after run_at. A same-tick race between a run-time edit and a daily run's start is filed as #5033. Closes #4938
This was referenced Oct 3, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #4938.
Why
A collector that runs once a day, such as the index object statistics read, starts whenever the Darling service happened to start. Its load can land in the busiest hour of the monitored server. #4938 gives such a collector an optional run time: a quiet hour the operator picks.
This PR lays the groundwork only. It adds the table that stores the time, and the rules every part of the product will share when it reads, writes or applies one. The worker, the viewer, the CLI, MCP and Lite then cannot each invent their own.
The run time lives in a table of its own, not a column on the schedules table. The viewer's schedule Save deletes a scope's rows in
config.config_collector_schedulesand inserts them again with a fixed column list (version 3.9.0 ships the same Save). The connect check refuses only an older store, so every released viewer connects to a version 160 store. A column on the schedules table would be set back to NULL by each of their Saves, and a server's -1 would become NULL, so that server would take the fleet's run time. Released viewers cannot be changed, so the run time sits where their Save never writes.What changes
config.config_collector_run_times. One row per collector, fleet-wide whenserver_idis NULL:collector_name, andrun_at_minute, a smallint of minutes after midnight on the monitored server's own clock, 0 to 1439. A -1 means "no fixed time on this server". It stops a fleet-wide time, the way an empty array stops a fleet-wide database scope, so a CHECK allows -1 on a server row only. The same two partial unique indexes as the schedules table keep one fleet row and one server row per collector. No row means no run time, so every install that never sets one runs exactly as before. Every statement isIF NOT EXISTSor a drop and create, so a second run changes nothing. It sits on top of the install id's final shape (V159) and does not touch that table or the schedules table. The plan's migration 158 ships as 160.ViewerDataService.MonitoredServers.csline 98 andDarlingMcpServerAdminTools.csline 390). The run-time table matches that: no foreign key, no cleanup.configschema (GRANT ... ON ALL TABLES IN SCHEMA config, and default privileges for new tables), so the new table has the same read and write rights with no change.CollectorRunTime, next toCollectorCadence). A run time is a 24-hour HH:MM time. It applies only to a collector whose interval is a whole number of days (an on-load collector counts as daily). The slot is taken on the monitored server's own clock. Each server gets a fixed offset of up to one hour, so a fleet does not start together. A run can start up to one hour after its slot. The messages every editor will show are shared too.CadencePhaseOffsetmoved from the Darling worker intoCollectorCadenceso these rules can use it. The worker forwards to it, with the values unchanged.ResolveSchedulelayers the run time like the other columns: the server row, then the fleet row, then none. It treats -1 as none. It drops a run time whose effective interval is not a whole number of days. The service logs one warning for each such run time each time the schedules load, not each sweep.Not in this PR
Nothing schedules by the run time yet. The worker's due-time logic, the viewer's editor, the CLI, MCP and Lite follow in the next PR. With no run time set (the only state after this PR), every schedule resolves exactly as before.
Test plan
Darling.Tests(with the service and the viewer), 0 warnings, 0 errors.Notes
CHANGELOG
No entry in this PR; the release owner adds it under [Unreleased]. Proposed entry: