Skip to content

DARLING: the store keeps an hourly history of its own pg_stat_statements deltas (V163, part of #5097) - #5203

Merged
erikdarlingdata merged 6 commits into
devfrom
feat/5097-store-statement-history
Oct 5, 2026
Merged

erikdarlingdata merged 6 commits into
devfrom
feat/5097-store-statement-history

Conversation

@erikdarlingdata

@erikdarlingdata erikdarlingdata commented Oct 5, 2026 •

Copy link
Copy Markdown
Owner

What changes for an operator

Once an hour the Darling service records, for the store's own statements, the change in calls, time, rows and blocks since the last snapshot. The 100 statements that spent the most time are kept 90 days in collect.store_statement_history; collect.store_statement_captures says how many were active, how many the extension evicted, and whether the hour was a fresh baseline. Nothing reads it yet; the reader tool is the next PR.

Design in short

  • Migration V163: three plain tables (not collectors, not in CollectorCatalog.All); the baseline sits in config so no purge ages it out.
  • The hourly store self-metrics tick runs StoreStatementHistory.SnapshotAsync in its own catch (Warning once, then Debug), before the collector-cost flush.
  • Deltas are computed in SQL against the stored baseline, so a service restart, store restart or nightly upgrade loses nothing. The epoch is stats_reset: unchanged means subtract; a reset after the previous capture credits the current counters whole; any other change records a new baseline and no history. A counter that fell inside one epoch credits the current value and flags a lower bound.
  • Reads only through config.store_statement_stats(); if it is unusable, one precondition capture and no history.
  • Viewer schema probe gains the V163 arm; ladder pins updated.

Test plan

  • Rung shape, epoch unit cases, SQL pins, command-timeout and worker-source pins.
  • Live (scratch store, planted reader functions): first capture, same-epoch delta with idle and new statements across a fresh connection, entry restart, reset inside interval, rebaseline, top-100 cap with statements_seen and hidden count, retention keeping the baseline, precondition capture.
  • Live, added after review: an installed but unloaded extension writes a precondition capture; a reader that raises rolls the whole pass back (no history, no capture, baseline unchanged); a same-epoch pass with non-NULL stamps on both sides, where the history row and its capture agree on interval_seconds.
  • Source pins, added after review: the snapshot runs under its own linked slice and an expired slice does not skip the flushes; the ready check requires the loaded setting and the TEMP privilege; the top-100 order breaks ties on role name.
  • Ladder guards: CaggGroupIndexDrop, plain-table census, alert-read exemptions.
  • No MCP tool and no tool description changes, so the tools/list budget does not move. The reader tool comes in a later PR.
  • CHANGELOG.md is not edited here; the entry below is spliced at merge.

CHANGELOG

SECTION: Added
ENTRY: - The Darling store keeps an hourly history of its own pg_stat_statements deltas ([#5203]) - migration V163 adds collect.store_statement_history: once an hour the service records, for the store's own statements, the change in calls, time, rows and blocks since the last snapshot. The 100 statements that spent the most time each hour are kept for 90 days, and the history survives service and store restarts and statistics resets.
REF: [#5203]: #5203

Part of #5097

Adds three plain tables and StoreStatementHistory.SnapshotAsync, which the
hourly store self-metrics tick runs in its own catch. The baseline lives in
the store so a service or store restart, a stats reset or the nightly
upgrade does not lose the deltas. Keeps the top 100 statements by time
spent for 90 days. Part of #5097.
… exemptions (the scanner keys on its first message)
…e interval value, deterministic top-100, documented limits
@erikdarlingdata erikdarlingdata added the merge-train In a merge train: its CI ran on the train PR label Oct 5, 2026
@erikdarlingdata
erikdarlingdata marked this pull request as ready for review October 5, 2026 09:18
@erikdarlingdata
erikdarlingdata merged commit d9b4713 into dev Oct 5, 2026
10 of 13 checks passed
@erikdarlingdata
erikdarlingdata deleted the feat/5097-store-statement-history branch October 5, 2026 09:18
erikdarlingdata added a commit that referenced this pull request Oct 5, 2026
…nt history (part of #5097) (#5217)

A new Darling MCP tool, get_store_query_history, reads the store's own hourly statement history (migration V163, #5203). Without query_id it ranks the statements that spent the most time over the window (Σ delta_total_exec_ms), each with its calls, window mean, the hours present, the role, per-statement counts of first_seen, restarted and reset hours, and the mean in the newest quarter of its hours against the rest (left out for first_seen hours, and null under four hours, so a lifetime credited to one hour cannot fake a slowdown). With query_id it returns that statement's hourly series, oldest first, with calls, mean, total, rows, block counts, temp writes, the cumulative max_exec_ms and the three flags. Both modes carry the window's captures (count, rebaselined and precondition hours, dealloc), so a missing hour reads as not captured rather than as idle.

Statement text is joined live from config.store_statement_stats(), the same reader get_store_query_stats uses; text the extension has since evicted reads as such, and when the reader is unusable now the history is still answered with "text unavailable" and a text_note carrying the reason. queryid goes out as a string. The answer carries effective_start, window_truncated and truncation_note (#4966), derived from the history's own first ok or rebaselined capture, with wording about the store's history rather than a monitored server. A store older than V163, one with no captures yet, and one whose captures are all precondition rows each answer a distinct precondition.

It is registered on the MCP host and mirrored on the web's /api/read. The tools/list ceiling rises to the measured 188,119.

Tests: refusals for bad role, top, hours_back and query_id; live cases planting the three tables (series with a reset hour, ranked order and truncation, the role filter, the slowdown split and its four-hour floor, a first_seen hour that would otherwise fake a slowdown, an unusable reader, each precondition, the data-start keys and the failed-read path); the budget, head-pin and cross-app inventory pins.

Part of #5097
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

merge-train In a merge train: its CI ran on the train PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant