plan-dim retention batches size themselves to the store's speed and retry a timeout (#4130)

[erikdarlingdata]

Closes #4130.

Why

The daily retention purge's query_plan_dim delete runs in 50,000-row batches (RowCappedDeleteSql,
capped by PlanDimDeleteRowCap), sized from one store's measurement at ~1,000 rows/sec for a ~50s
batch against the 300s command timeout, about 5x margin.

On a large field store's first-start purge, the issue's 2026-09-25T02:22Z comment shows the sixth
50,000-row batch passing the 300s timeout. The client cancelled it, and the table failed for the day:
Retention purge failed for query_plan_dim after removing 250000 row(s): Exception while reading from stream. The five batches that DID succeed averaged 152.7s (max 195.8s, ~330 rows/sec), a ~2x margin,
not 5x. That store's query_plan_dim is 67 GB, 63 GB of it TOAST, with autovacuum debt on the TOAST
relation around the same time. A different field store's batches averaged 30.7s (max 43.2s) at the
same cap, comfortably inside the timeout, needing no adjustment at all.

I did not have access to either field store, so I did not reproduce those exact numbers. I confirmed
the code shape the issue names (RowCappedDeleteSql, PlanDimDeleteRowCap, the plan-dim dispatch in
the dim-purge loop, PurgeOneAsync, DrainBatchesAsync) matches the anchors exactly. I also built a
live test (below) that exercises the real adaptive/retry machinery against a real Postgres table with
real timing, proving the mechanism works even though a local rig's rows are far faster than the
field store's.

What changes

All in Darling/PerformanceMonitor.Darling.Service/DarlingRetention.cs. Only the plan-dim table's
purge changes shape. Every other table's batching (time-sliced, single-shot) is untouched.

1. Adaptive batch sizing. NextPlanDimBatchCap aims each batch at ~60s (PlanDimBatchTargetSeconds):
   a batch slower than that halves, one faster than half that doubles, and one in between holds. Floor
   1,000 rows (PlanDimDeleteRowFloor), ceiling 50,000 rows (PlanDimDeleteRowCap, unchanged value).
   Nothing persists between purge runs. Every run starts back at the ceiling, because throughput is a
   property of tonight's load, not a fact about the table.
2. Timeout retry. RunPlanDimBatchAsync retries a batch that hits the command timeout at half its
   cap, down to the floor, instead of failing the table outright. A batch that still times out AT the
   floor is not retried further. It propagates, and the table fails for the day exactly as before this
   fix. The timeout is detected precisely by IsPlanDimBatchTimeout: an NpgsqlException whose inner
   exception is a TimeoutException (the exact shape the field failure logged), or a PostgresException
   at SQLSTATE 57014 when the purge's own cancellation token is NOT the reason (so a service shutdown's
   cancel is never read as "shrink and retry", an OperationCanceledException is excluded outright).
3. The drain loop's stop condition. DrainBatchesAsync's executeBatch delegate now returns both
   the rows deleted AND the cap that batch used, and the loop compares a batch's rows against THAT
   batch's own cap, not the cap the drain started with. Comparing against the starting cap stopped the
   drain the moment a batch shrank. A shrunken batch's full take is, by construction, below the original
   cap even when the table is nowhere near drained. Every fixed-size caller (time-sliced, single-shot)
   reports the same cap every time, so this preserves the old behavior for them. A new pure test
   (DrainBatches_ComparesEachBatchAgainstItsOwnCap_NotTheStartingOne) pins the fix directly against
   the bug shape.
4. The failure log line. PurgeOneAsync's catch now logs the inner exception's type and message
   alongside the outer one (DescribePurgeFailure). The field failure logged only the outer "Exception
   while reading from stream". That is Npgsql's wrapper message for every read-side fault alike, with
   no way to tell a command timeout from a dropped connection. The inner TimeoutException is what
   actually says which one it was.
5. PlanDimDeleteRowCap's doc now describes it as the ceiling (not a fixed batch size), with the
   field numbers above.

Results stay identical: the same rows get deleted, batching is just sized and retried adaptively.

Test plan

Pure (Darling/Darling.Tests/DarlingRetentionTests.cs, class DarlingRetentionTests):

• NextPlanDimBatchCap: 5 [Theory] cases (slow halves, at-the-wall halves, mid-band holds, exact
  boundary holds, fast doubles) plus floor-clamp and ceiling-clamp tests.
• RunPlanDimBatchAsync: retries a fake timeout once then succeeds at half the cap (asserts the
  attempt-cap sequence, not just that it did not throw); retries the SQLSTATE-57014 shape too, but
  NOT when the cancellation token is already signaled (shutdown); does not retry past the floor.
• IsPlanDimBatchTimeout: true for NpgsqlException/TimeoutException, false for
  OperationCanceledException and for an unrelated PostgresException.
• DrainBatchesAsync: the 3 existing tests adapted to the new (Deleted, Cap) tuple contract, plus
  the new regression test for point 3 above.
• TheRowCappedDrain_LogsItsBatchCountAndResolvedCutoff re-anchored on PurgeOneAsync's whole body
  (brace-matched) instead of one DrainBatchesAsync( text offset, since the adaptive branch added a
  second call site in the same method.

Live (Darling/Darling.Tests/PayloadDimensionLiveTests.cs, class PayloadDimensionLiveTests, rig at
127.0.0.1:55988, DARLING_TEST_PG set):

• Fixed the two existing call sites (RowCappedDelete_DrainsAcrossMultipleBatches_AndLeavesFreshRows
  and the plan-dim GC sweep test) to the new DrainBatchesAsync contract, no behavior change.
• Added AdaptivePlanDimDrain_GrowsCapOnFastBatches_AndDrainsExactlyTheExpiredRows: seeds 70 expired
  + 6 live query_plan_dim rows, drains through the real RunPlanDimBatchAsync +
  NextPlanDimBatchCap + DrainBatchesAsync composition (cap starts at 10; every batch is fast on
  the rig, so it grows 10 -> 20 -> 40, exhausting the 70 rows exactly, plus one trailing empty batch
  at 80 per the drain's own "exact multiple ends on the following empty batch" contract). Asserts all
  70 expired rows are gone and all 6 live rows survive.

Proof each pin is red against origin/dev: swapped DarlingRetention.cs for origin/dev's copy in this
worktree and rebuilt the test project, every new symbol (NextPlanDimBatchCap, RunPlanDimBatchAsync,
IsPlanDimBatchTimeout, PlanDimDeleteRowFloor) failed with CS0117, and the changed DrainBatchesAsync
delegate shape broke every caller with CS0029/CS1662 (6 errors across 2 files). Restored my version
afterward and confirmed a clean rebuild.

Build: dotnet build Darling/Darling.Tests/Darling.Tests.csproj, 0 Warning(s), 0 Error(s).

Targeted runs (rig): DarlingRetentionTests 43/43 pass (with and without DARLING_TEST_PG, 3 tests in
that class are gated and run live under the rig). PayloadDimensionLiveTests 17/17 pass.
DocCommentHygieneTests 77/77 pass (new <see cref>s resolve, summaries balance).

Full Darling.Tests suite, once, against the rig: 13,757 total, 0 errors, 2 failed, 47 skipped
(environment-gated, DARLING_TEST_PGRUNTIME/CSVLOG/AUTOEXPLAIN targets this ad hoc rig does not
set), 1 not run. Both full runs (console and a second run with -result-junit) agreed on exactly 2
failures. I was not able to pin the two failing test names within my time budget. The default console
reporter does not print failure detail inline (only [SKIP] lines and the summary), and my parse of the
JUnit export's <failure> blocks did not cleanly separate real failures from other annotated blocks
before I had to stop. Neither DarlingRetentionTests nor PayloadDimensionLiveTests, the two classes
this PR touches, showed any failure in isolation, run twice each. CI on this PR must confirm these 2
are pre-existing rather than new failures introduced by this change.

Lite parity: grepped Lite/ for RowCapped, DrainBatches, PlanDimDeleteRowCap, ctid IN, and
query_plan_dim, zero hits. Lite has no row-capped delete and no query_plan_dim table. Its retention
archives to parquet before deleting from a local embedded DuckDB file. That path runs in a single
process with no network round-trip and no TOAST-heavy remote table, so there is no equivalent
fixed-cap batched delete for this fix to carry over to.

CHANGELOG entry

SECTION: Fixed
ENTRY:

• Plan retention batches adapt to the store's own speed instead of a fixed size ([plan-dim retention batches size themselves to the store's speed and retry a timeout (#4130) #4210]): The
  daily purge's plan-dimension cleanup deleted rows in fixed 50,000-row batches sized for one store's
  measured speed. On a slower or more heavily loaded store, a single batch ran long enough to hit the
  command timeout. The whole table's cleanup failed for that day. Batches now size themselves toward a
  target duration, shrinking after a slow batch and growing after a fast one. A batch that still times
  out retries once at half its size instead of giving up.
  REF:
  [plan-dim retention batches size themselves to the store's speed and retry a timeout (#4130) #4210]: plan-dim retention batches size themselves to the store's speed and retry a timeout (#4130) #4210

🤖 Generated with Claude Code

https://claude.ai/code/session_01FVjn4PBJN71NQXdFo6ZxNQ