Skip to content

PostgreSQL analysis: the plan-regression pick sees the worst statements, and a standby, slot or database that left is no longer described or graded as current (#4759, #4760, #4761) - #4780

Merged
erikdarlingdata merged 6 commits into
devfrom
fix/4759-pg-facts-last-seen
Sep 29, 2026
Merged

erikdarlingdata merged 6 commits into
devfrom
fix/4759-pg-facts-last-seen

Conversation

@erikdarlingdata

@erikdarlingdata erikdarlingdata commented Sep 29, 2026 •

Copy link
Copy Markdown
Owner

Fixes #4759
Fixes #4760
Fixes #4761

Three commits, one per issue, in the order #4760, #4759, #4761.

Why

What changes

  • The plan-regression fact picks from the first 50 changed statements by query_id, not the worst #4760. PgTargetPlanFlipSql orders by PickRegression's tiers: rows with enough calls on both sides, a ratio, and the delta and concerning-ratio bars first; then rows with enough calls and a ratio; then the rest. Each tier is by ratio DESC NULLS LAST, then query_id. The means and the ratio are computed in SQL the way PlanFlipRow does (a mean over no calls and a ratio over a zero mean are NULL). The three scorer bars are passed as $5, $6, $7. LIMIT $4 and COUNT(*) OVER () are unchanged.
  • A standby that is gone keeps its last state, and outranks live standbys, for the rest of the window #4759. PgTargetReplicationLagSql returns each standby's own first and last row and standby_gone: the last row is older than twice the table's effective cadence in the window, (last capture - first capture) / (collections - 1) floored at 2 minutes, measured back from the window end ($3), never from the table's last capture and never from a clock. Live standbys rank first, then peak as before. The fact records standby_gone and minutes_since_last_seen; the span is the picked standby's own; the synchronous weighting skips a gone standby; the advice for a gone standby is in the past tense ("was N behind when last seen ... before the window ended").
  • A dropped replication slot keeps grading Critical for the rest of the window #4761. The slot read returns the same verdict per slot (slot_gone), and the wraparound read per database (database_gone). Slots and databases still reporting rank first. A gone slot or database is recorded on the fact and grades nothing. A gone slot's inactive hours run to its last row. The slot advice says the slot stopped being reported.
  • Decision for a window with a single collection. There is no cadence to measure, so nobody is called gone in it (status quo). The 2 minute floor applies once a cadence exists.
  • Wraparound advice for a gone database. The advice says the database stopped being reported and when it was last seen, the way the slot advice does, puts its age figures in the past tense, and recommends a VACUUM only once the database is confirmed to still exist, while a live database's text is unchanged.

Test plan

  • RED first, The plan-regression fact picks from the first 50 changed statements by query_id, not the worst #4760: with the old ordering, SixtyChangedStatements_... named id 49 instead of 59, and TheFiftyRowPage_... named the 4 ms comparable flip (id 25) instead of the statement that clears the bars (id 57). Both pass with the fix.
  • RED first, A standby that is gone keeps its last state, and outranks live standbys, for the rest of the window #4759: with only the standby_gone ranking removed, AStandbyThatStoppedAnHourBeforeTheWindowEnd_... picked the departed standby (expected replica-b, got replica-a). It passes with the fix.
  • A dropped replication slot keeps grading Critical for the rest of the window #4761: ADroppedSlot_WhoseRowsStoppedTwoHoursBeforeTheWindowEnd_DoesNotGradeCritical passes (live slot is the pick, the gone slot grades 0 with arm 0, and its inactive hours are 27.95, not 30). I did not run it against the old code; it asserts keys the old code never wrote, so it cannot pass there.
  • RED first, wraparound advice for a gone database: against the previous PgTargetAdvice.Vacuum.cs, all three cases of TheWraparoundAdvice_ForADatabaseThatStoppedBeingReported_SaysWhenItWasLastSeen_AndPutsItsFiguresInThePastTense (emergency, routine crossing, healthy) failed, 3 of 72 in PgTargetVacuumTests. All 72 pass with the change. TheWraparoundAdvice_ForALiveDatabase_IsWordForWordTheSameWhenTheGoneFlagIsZero pins the full headline, investigation and remediation of three live facts (captured from the old code, with the gone flag at 0); it passes both before and after.
  • Pure tests: AStandbyThatLeft_IsDescribedInThePastTense_AndTheSyncWeightingSkipsIt, plus text pins for the new ORDER BY and the three bar parameters.
  • Live classes on a throwaway PostgreSQL (PgTargetReplicationTests, PgTargetVacuumTests, PgTargetFactCollectorTests, PgTargetPlanTests): 153 passed, 0 failed.
  • Darling.Tests builds with 0 warnings, 0 errors (after merging origin/dev).
  • Full Darling.Tests suite once, without DARLING_TEST_PG: Total 17203, Failed 0, Skipped 1127 (the live tests), Not Run 1.
  • Full Darling.Tests suite once more after the wraparound wording change, without DARLING_TEST_PG: Total 17207, Errors 0, Failed 0, Skipped 1127, Not Run 1.
  • The full suite with DARLING_TEST_PG set was not run; CI's PostgreSQL job runs it.
  • No EXPLAIN (ANALYZE, BUFFERS) was taken for the new plan-change ORDER BY. The sort is over the flipped statements only (the same rows the old sort held).

Worth a second look:

  • An existing pin was updated deliberately: TheCollectorReads_AreInTheInventory_... now expects ORDER BY standby_gone, p.peak_replay_bytes DESC NULLS LAST (the old text is the tail of the new one) plus the gone expression.
  • The gone rule is written three times in SQL (standby, slot, wraparound reads) with the same expression. The wraparound read gained a small span CTE and a CROSS JOIN.
  • PgTargetReplicationTests.cs is stored with LF line endings in this checkout; the other touched files are CRLF. Git normalizes both.

CHANGELOG

SECTION: Fixed
ENTRY:

SECTION: Fixed
ENTRY:

SECTION: Fixed
ENTRY:

…ick chooses from, so the worst regression is reported (#4760)

The plan-change read kept the 50 lowest query ids, and PickRegression chose only
among those, so with more than 50 changed statements the worst regression could be
dropped, and 50 rows that all failed the call floors reported no regression while
dropped rows passed. The read now orders by PickRegression's two tiers (both call
floors, a ratio, the delta and concerning-ratio bars first; then both call floors
and a ratio; then the rest), each tier by ratio with NULLs last, then query id. The
means and the ratio are computed in SQL the way PlanFlipRow does, and the scorer's
three bars ride as parameters. LIMIT and COUNT(*) OVER () are unchanged.
…e live ones, and its advice says when it was last seen (#4759)

The standby read returned no time for a standby's last row and picked the standby
with the highest peak, so a standby that was removed, replaced or reconnected from a
new address stayed the fact's state for the rest of the window: the advice spoke of
it in the present tense, the synchronous weighting still applied, and the span was
the whole table's. The read now returns each standby's own first and last row and a
gone flag: the last row is older than twice the table's effective cadence in the
window ((last capture - first capture) / (collections - 1), floored at two minutes),
measured back from the window end. Live standbys rank first. A gone standby is
recorded on the fact, skips the synchronous weighting, and its advice is in the past
tense (was N behind when last seen); the span is the standby's own.
…se no longer grades from its last row for the rest of the window (#4761)

A dropped replication slot kept its last row in the window, so an inactive, growing
slot over 10 GiB kept grading Critical after the operator dropped it, and its
inactive hours ran on to the window end. The slot read and the wraparound read now
return the same last-seen verdict as the standby read: the latest row is older than
twice the table's effective cadence in the window, measured back from the window end.
Slots and databases still reporting rank first; a gone slot or database is recorded
on the fact, grades nothing, and a gone slot's inactive hours run to its last row.
The slot advice says the slot stopped being reported.
…pped being reported, and a live database's text (#4761)
…is worded as the state it was last seen in, and no VACUUM is recommended until it is confirmed to exist (#4761)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant