get_store_metrics answers with a bounded summary instead of every store object's daily series (#3903)

[erikdarlingdata]

Closes #3903.

Why

get_store_metrics returned every store object's latest row and its daily series on every call. The object count is fixed by the schema, not by the fleet: about 250 objects (72 hypertables, 25 continuous aggregates, 146 TimescaleDB jobs, the payload dimensions, named tables and catch-alls). So the default answer was ~1.9 MB on every production store in the issue, about 500k tokens, and even days_back=1 was 347 KB because objects[] alone was 172 KB.

The diagnosis is confirmed on DARLING01 (253 objects, 17 days of history). The default response is 1,275,633 bytes:

key	bytes	share
daily (250 series, 3,518 points)	1,095,762	86%
objects (250 rows, 146 of them jobs with ~20 null byte fields each)	172,348	13.5%
everything else	~7,300	0.6%

The SQL is not the cost: on DARLING01's 83,355 store_metrics rows the latest read takes 225 ms and the 30-day daily read 504 ms (EXPLAIN (ANALYZE, BUFFERS), both in-memory quicksorts). The bytes are. The reads are unchanged by this PR.

What changes

• Summary first, the default. Every response keeps the store-level blocks: store (with the whole-store daily growth and the per-server ingest rate), inventory, retention, job_history and checkpointer. The per-object series is replaced by three ranked lists. Each list is bounded by limit (default 10) and carries its match count, truncated and order:
  • objects: the largest byte-bearing objects.
  • fastest_growing: ranked by growth_bytes over the window.
  • background_jobs: jobs whose failure count grew in the window first, then by duration_vs_cadence_percent.
• Each row carries its change over the window in place of its series. A byte-bearing row carries growth_bytes; a job row carries runs_in_window and failures_in_window. Each is measured from delta_since (the first daily point in the window) to the last point, the subtraction a caller would make from the series. The value is null, never zero, when there is nothing to state: fewer than two points, a null at either end, or a cumulative counter that went backwards.
• object_kind lists every object of one kind, bounded by limit. A kind that is not one of the seven listed kinds is refused (invalid). The store, job_history and checkpointer rows stay blocks and are never rows.
• object_name drills in. An exact name (case-insensitive) returns that object's row and its daily series over days_back. Otherwise every object whose name contains the text is listed. Exact matching comes first, so a name like wait_stats still reaches the hypertable alone, not its aggregate and jobs. A filter that matches nothing answers empty.
• Row shapes split by kind. Byte-bearing rows carry sizes, aggregate state and TOAST facts; job rows carry run telemetry. One shape for both had put about 20 null fields on every job row, and jobs are more than half the rows.
• Web mirror and triage. /api/read/get_store_metrics advertises and binds the three new parameters. The store alerts' triage page gains a "Background jobs" section (object_kind=background_job, limit=25). The page's table renders a section's first array, and in the summary the jobs are now a nested list.
• The MCP surface costs ~100k tokens of context before the first call: 157 tools, 336 KB of tools/list, 86 KB of instructions #3898 budget. The tool description went from 9,765 to 7,185 characters even with the new contract added. The cut is prose that the response's own notes already state where the value is read: the job-history setting and visibility arms, and the TOAST and checkpointer detail. Every pinned claim stays. The server-instructions paragraph went from 3,113 to 1,404 characters. The four parameter descriptions total 388 characters.

Behavior changes (wire)

• The default response no longer carries daily. objects is now a bounded page, not every object. series holds one object's daily points when an exact object_name is passed, and is null otherwise. The summary's note says the per-object series is not in the summary and names the parameter that brings it back.
• Background-job rows no longer carry the byte fields, which were always null on them. Byte-bearing rows no longer carry the job fields, which were always null on them.
• days_back keeps its meaning and its maximum of 400. It is now also the window for the deltas.
• No saved custom view reads get_store_metrics on DARLING01 (0 of 3). A saved panel elsewhere that keyed on daily would need object_name.
• There is no Lite change. get_store_metrics is Darling-only by architecture: CrossAppMcpToolInventoryPinTests records it as such.

Test plan

• Before, on DARLING01 (the deployed dev build, via mcp_one): default 1,245.7 KB; days_back=1 346.6 KB; days_back=7 710.8 KB; days_back=400 1,245.7 KB.
• After, on DARLING01's own 83,355 store_metrics rows copied to the rig: default 19.1 KB, 65 times smaller. days_back=1 17.9 KB; days_back=7 18.3 KB; days_back=400 19.1 KB; every job (object_kind=background_job, limit=1000) 55.7 KB; one object 12.4 KB; a partial name 13.5 KB.
• After, on a production-shaped fixture (253 objects, 43 servers, 400 days): default 18.3 KB; days_back=400 45.1 KB (only the whole-store series grows with the window); one object over 400 days 118.7 KB.
• New StoreMetricsSummaryFirstTests (pure): listed kinds, window deltas, exact-first selection, list and growth order, per-view notes, the description's contract together with the code behind it, a source pin that no per-object series projection returns, the web catalog, and the triage section.
• New StoreMetricsSummaryFirstLivePostgresTests, on its own scratch store seeded to production's shape:
  • The default stays under the issue's 50 KB acceptance, with every block present, no daily, and the lists' contents checked against arithmetic oracles.
  • delta_since and growth_bytes agree with the drill-down's own series.
  • days_back=400 stays bounded.
  • Filters, refusals and empty work.
  • The web mirror and the triage page's own section definition bind the new parameters into the same tool.
• Pins updated deliberately, never loosened:
  • The parameter contract.
  • The query_plan_dim's TOAST file sits at 40 % utilisation on the second SQL Server store (~93 GB slack from the V54 conversion + cycling deletes) — publish dimension-table TOAST utilisation in the store self-metrics; reclaim is --recompress-plan-dim --vacuum-full at the maintainer's word #3783 TOAST trailing-field adjacency, now on the byte-row shape and the series point.
  • The checkpointer exclusion, now ListedKinds plus SelectObjects in place of two inequalities.
  • The description-phrase pin now reads the description itself instead of the whole source file, so a comment can no longer satisfy it.
• Full Darling.Tests with DARLING_TEST_PG on the rig, rebased on dev 4b6e50c9: 12,499 total, 0 failed, 21 skipped (the gates that need DARLING_TEST_PGRUNTIME or DARLING_TEST_SQL, and the symlink test without Developer Mode). An earlier run on the pre-rebase build failed only PgWaitSamplerLiveTests (31 samples against an upper bound of 30). That test passed in two other full runs and 3 of 3 times alone; it samples pg_stat_activity cluster-wide while the own-store classes run in parallel. Filed as PgWaitSamplerLiveTests can count another test's lock wait: 31 samples from a 30-snapshot cycle under full-suite load #3939.
• Full Lite.Tests on the pre-rebase build (nothing in Lite or a shared project changed; run because Lite tests read Darling sources): 5,124 total, 0 failed.
• Mutation checks, each restored:
  • Putting back the unbounded object list fails the source pin and both live tests.
  • Contains-only name matching fails the selection pin and both live tests.
  • Letting a counter reset read as a count fails the window-delta pin.

Follow-up filed

• The store-metrics latest read sorts all 400 days of collect.store_metrics on every call: 7.8 s at full retention #3934: the latest read this tool and the hourly TOAST-slack self-alert share sorts all 400 days of collect.store_metrics on every call. At full retention it takes 7.8 s on a CI-sized rig, against 15 ms with a (object_kind, object_name, metric_time DESC) index and a skip scan. That fix needs a migration rung, so it is out of this change.

🤖 Generated with Claude Code

https://claude.ai/code/session_018Z4T51J4E6L8fjriPHpsif