Skip to content

get_pg_io_trend's automatic subject choice reads each series' endpoints on TimescaleDB, not every row in order (#3961) - #4003

Merged
erikdarlingdata merged 1 commit into
devfrom
fix/3961-pg-io-trend-subject
Sep 23, 2026
Merged

erikdarlingdata merged 1 commit into
devfrom
fix/3961-pg-io-trend-subject

Conversation

@erikdarlingdata

Copy link
Copy Markdown
Owner

Closes #3961.

Why

get_pg_io_trend's automatic subject choice (DominantIoSubjectSql) sorts every pg_io_stats row of the
window before its LAG window function can rank the (backend_type, object_type, context) series: on
DARLING01 over 168 h that was measured at 3,767-5,791 ms (per Erik's issue comment, re-measured deciding
whether to fold this into #3897/#3968 - it was not, because the fix needs a ruling on trusting endpoints).
The trend it feeds answers in 269 ms on the same window.

What changes

  • New DarlingPgTrendReader.DominantIoSubjectEndpointsSql: ranks each series on `last(x, collection_time)
    • first(x, collection_time)(aHashAggregate, no sort) instead of differencing every row, and reports any_rewound - true when any series in the window shows a sign it went backwards (stats_resetmoved, a counter is NULL on some rows but not others, or a counter dipped below its first value or ended below its peak). Its HAVING/ORDER BY tail is byte-identical toDominantIoSubjectSql`'s (pinned by test) so the two
      can never rank the same sums differently.
  • GetDominantIoSubjectAsync takes a new fromEndpoints parameter: true tries the endpoints read first and
    only falls through to the exact LAG form when any_rewound says a reset happened in this window, or the
    endpoints read found no qualifying pair at all. first()/last() are TimescaleDB hyperfunctions, so a
    plain-PostgreSQL store (Store-object convergence never re-runs on a plain-PostgreSQL store: the hourly pass is gated on TimescaleDB, so the Ungated and Tuning steps heal only at restart #3913) always keeps the LAG form.
  • New DarlingPgTrendReader.IsTimescaleDbStoreAsync, gating fromEndpoints at the one call site in
    DarlingMcpPgTrendTools.GetPgIoTrend - the same pg_extension check TimescaleSupport.DetectAsync uses
    on a live connection, asked here of a data source. Cheap enough (one indexed catalog lookup) to ask fresh
    per call rather than memoize, unlike the window-scanning reads it gates.
  • Answers are unchanged: the endpoints read either matches the LAG form's choice, or a window that rewound
    is hand-checked by the LAG form itself.

Diagnosis, confirmed on my rig

Erik's own comment on the issue already measured this on DARLING01 (real production scale, 788,341 rows,
8 chunks / 7 compressed): LAG form 3,767-5,791 ms, endpoints form 304 ms, identical top-8 pairs, no
stats_reset in that window - and flagged the two open questions this PR resolves: the reset-detection
rule (a ruling, since the shipped comment on the LAG query argues deliberately about resets) and
first()/last() being TimescaleDB-only (routing needed).

On my own rig (60,480 synthetic pg_io_stats rows, 7 daily chunks, 6 of 7 compressed - the same shape,
smaller scale): LAG form 62.6 ms, endpoints form 52.9 ms, identical chosen pair. The relative win is smaller
here because the win scales with rows-per-chunk (sort cost is worse than linear) and my rig has about 13x
fewer rows per chunk than DARLING01 - direction and mechanism confirmed, magnitude needs DARLING01's scale
to reproduce exactly.

Lite: no equivalent. Lite has no PostgreSQL targets and no pg_io_stats table or "dominant subject"
auto-choice pattern of any kind (grepped Lite/ for pg_io_stats, Dominant, first(...collection_time),
last(...collection_time) - no matches). Nothing to port there.

Test plan

  • PgTrendReaderTests.TheEndpointChoice_RanksTheLagFormsSums_AndFlagsAWindowThatRewound - unit, pins the
    endpoints SQL's shape (HAVING/ORDER BY tail identical to the LAG form's, filters identical, every
    counter's rewind check present, no LAG( in it) and TimescaleExtensionPresentSql's text.
  • TrendBucketingLiveTests.PgIoSubjectChoice_FromEndpoints_IsTheLagFormsChoice_AndARewoundWindowIsHandedToIt -
    live, three pairs over five snapshots where trusting the endpoints blindly would pick the wrong
    subject (an autovacuum-worker reset drops its sum from 5,100 to 100, which would hand the default to
    a client-backend series that never rewound); proves fromEndpoints: true/false agree everywhere,
    any_rewound fires correctly (including a rewind that never moves stats_reset), half a subject still
    constrains the endpoint read, IsTimescaleDbStoreAsync reads true on a real TimescaleDB store, and an
    end-to-end call through GetPgIoTrend itself resolves the same subject. Proved to fail on the old
    (rewind-blind) shape by temporarily dropping the any_rewound check and re-running - it failed
    exactly as expected, then the sabotage was reverted.
  • Targeted: *PgIo*, *PgTrend*, *DarlingMcpPgTrendTools*, *McpPayloadContractCensusTests*,
    *DocCommentHygiene*, *LivePostgresCollectionHygieneTests*, *StorageCommandTimeoutTests* - all
    green (the timeout gate caught a real miss: my new IsTimescaleDbStoreAsync command needed an
    explicit CommandTimeout, now fixed).
  • Full suite: Darling.Tests, DARLING_TEST_PG set - 12,951 total, 0 failed, 29 skipped (unrelated live
    SQL Server E2E tests with no DARLING_TEST_SQL). One run surfaced a genuine, pre-existing flake: my
    new live test originally shared its time anchor and (backend_type, object_type, context) key with an
    existing pg_io_stats fact in the same file/server_id, and running concurrently with it in the full
    suite (not just in isolation) cross-contaminated both. Fixed by moving my test's window to a date none
    of this file's other facts touch. Re-ran the full suite after the fix: clean.
  • Lite unaffected; Lite.Tests not run (no Lite file touched, no shared project touched).

🤖 Generated with Claude Code

https://claude.ai/code/session_01GdmA4ND1wLSqA91ax1m4xv

…ts on TimescaleDB, not every row in order (#3961)

DominantIoSubjectSql sorts every pg_io_stats row in the window before its LAG window function can rank
them: 3.8-5.8 s over 168 h on DARLING01. The ranking only needs each series' summed increase, and for a
counter that never rewinds that sum is its last value minus its first, answered from a HashAggregate in
0.30 s on the same window with an identical top-8. DominantIoSubjectEndpointsSql reads the fast way on a
TimescaleDB store (first()/last() are TimescaleDB's) and flags any series that shows a sign of rewinding -
a moved stats_reset, a partly-NULL counter, or a counter below its first value or below its peak - so the
reader falls back to the exact LAG form for that window instead of trusting an endpoint reading that could
be wrong. A plain-PostgreSQL store keeps the LAG form outright, gated by IsTimescaleDbStoreAsync.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GdmA4ND1wLSqA91ax1m4xv
@erikdarlingdata
erikdarlingdata enabled auto-merge (squash) September 23, 2026 06:53
@erikdarlingdata
erikdarlingdata merged commit 57b6912 into dev Sep 23, 2026
15 of 16 checks passed
@erikdarlingdata
erikdarlingdata deleted the fix/3961-pg-io-trend-subject branch September 23, 2026 06:59
erikdarlingdata added a commit that referenced this pull request Sep 23, 2026
…ntries in their sections (#4080)

Adds 42 entries and 42 link refs (#3992, #3995, #3996, #3998, #4001, #4002, #4003, #4007, #4010, #4011, #4013, #4015, #4020, #4022, #4025, #4029, #4030, #4031, #4036, #4038, #4039, #4040, #4044, #4047, #4048, #4049, #4050, #4051, #4055, #4061, #4063, #4064, #4065, #4066, #4067, #4068, #4069, #4070, #4071, #4073, #4074, #4078). Each PR's entry was buffered, and this lands every entry whose PR was merged on origin/dev when it ran.

#3989 left 26 entries under bare 'Changed' and 'Fixed' lines above '### Added'. They move into '### Changed' and '### Fixed', below the new entries, and one blank line stays under [Unreleased].


Claude-Session: https://claude.ai/code/session_01Ua31ugERL5DmhFVRtf6keQ

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant