Perfmon data is compressed per counter, so a one-counter trend reads only that counter's data (#5574)

[erikdarlingdata]

A one-counter perfmon trend over a long window now reads only that counter's compressed data. New perfmon chunks compress by server and counter. A background loop re-groups the chunks that were compressed by server alone. The hourly pass starts the loop. It works newest first, on the last 30 days, one chunk at a time, with a pause after each. A store whose TimescaleDB cannot do this keeps working exactly as before.

SECTION: Changed
ENTRY: - A long one-counter perfmon trend loads faster ([#5579]) - perfmon data is now compressed per counter as well as per server. Reading one counter (for example Batch Requests/sec) over 30 days opens only that counter's batches, not every counter's. Older compressed chunks are re-grouped in the background, back to 30 days or to the day before retention drops them, whichever is sooner. The loop takes one chunk at a time, newest first, and after each one it pauses as long as that chunk took. A read beside a re-group waits a few seconds at most. The web viewer's perfmon trend and the get_perfmon_trend MCP tool now reach 30 days instead of 7.

Details (part 1: the compression change)

The 30-day raise and the timing split are in this PR: see "The 30-day raise and the timing split".

What changes (file and symbol, all under Darling/):

• PerformanceMonitor.Darling.Storage/TimescaleSupport.cs
  • PerfmonStatsSegmentBy = "server_id, counter_name". CompressionSegmentByFor returns it for perfmon_stats. The enable statement, the "is it converged" read and the new-chunk grouping all go through that one lookup, as collection_log compression: segment by collector as well as by server #4951 did for collection_log.
  • The trend read filters server_id and counter_name by equality and nothing else. Its object_name test is a suffix match. So those are the only two columns. The orderby is not set, so it stays collection_time DESC.
  • The settings change reaches the existing store through the existing hourly "compression policies" pass (bounded lock, own transaction). Below TimescaleDB 2.14 with compressed chunks, the pass skips the change with one Information line. This is the same as for collection_log (SettingsChangeBlockedAsync). The old collection_log method now calls it.
  • RegroupPerfmonChunksAsync picks compressed perfmon chunks inside the 30-day reach. Their own grouping (from timescaledb_information.chunk_compression_settings) must differ from server_id,counter_name. It takes the newest first.
  • Each chunk runs decompress_chunk, a row count and compress_chunk in ONE transaction behind SET LOCAL lock_timeout = '3s'. A busy lock skips the chunk to the next pass.
  • No new chunk starts after 2 minutes. The first chunk always starts. Three failures in a row stop the pass.
  • The pass logs one Information line per chunk (name, rows, old and new grouping, ms). It logs one line per pass with the chunks left. It logs at Debug only when nothing is left.
  • It does nothing until the hypertable itself carries the new grouping. Otherwise it writes the old grouping again.
  • TryRunBoundedDdlAsync got an overload that takes a body, because the re-group counts rows between the two calls. The statement-list overload calls it and behaves the same.
• PerformanceMonitor.Darling.Service/DarlingWorker.cs gets a new last step, perfmon chunk re-group, in the store-object convergence list. It runs hourly only. The start-path delegate does nothing, so a restart is not held for minutes.
• Tests: PerfmonRegroupGateTests is new, with 5 tests. They cover the grouping and its spellings, the three block reasons and the 2.14.1 view floor. TimescaleSupportTests is updated deliberately. Every collector table keeps server_id. Only perfmon_stats also has counter_name.

TimescaleDB version finding (checked, not assumed):

• Settings can change with compressed chunks from 2.14.0.
• But timescaledb_information.chunk_compression_settings first appears in 2.14.1. The file sql/updates/2.14.0--2.14.1.sql creates it. The 2.14.0 views file does not have it.
• So the re-group is gated on the view existing, not on "2.14".
• On 2.14.0 the grouping changes for new chunks, and the old chunks age out.
• The existing CollectionLogSegmentByLiveTests.RequirePerChunkSettingsAsync has the same blind spot, because it checks >= 2.14. It is harmless there because it also checks the view.

Measured on a local PostgreSQL 18.6 with TimescaleDB 2.30.1 (synthetic)

The data had 3 servers and 260 counter/instance series (200 distinct counter names). Samples came every minute for 30.7 days. That is 34.5 million rows, with 1.12 M rows per daily chunk. 27 chunks were compressed with the OLD grouping first. Then they were converged through the new code path.

	old (server_id)	new (server_id, counter_name)
compression of the 27 chunks	4286 MB to 534 MB (8.0x)	4511 MB to 217 MB (20.8x) (hypertable_compression_stats). The sizes differ a little because the 4 newest chunks are not compressed at the old measurement.
compressed batches a one-counter read opens (1 server, 27 chunks)	10,125	54
trend read, 1 counter, 30 days: shared buffers	582,000 (402,669 hit + 179,472 read)	135,700
trend read: rows decompressed then thrown away	22.4 M	2.4 M
trend read: time (warm, 3 runs)	2.2 s / 3.7 s / 5.7 s after a restart (noisy laptop)	0.9 s / 0.7 s / 0.6 s
get_perfmon_stats (latest snapshot)	1.4 to 8 ms	3 to 9 ms. It reads the newest, uncompressed chunk. Compression grouping does not touch it.
every-counter snapshot from a COMPRESSED chunk (worst case for the new grouping, read only for a server silent for 3+ days)	0.7 to 1.1 ms, 2 batches	5.9 to 8.6 ms, about 260 batches

• Rows are identical before and after the re-group. The row count is 34,486,140 and the per-row hash sums are equal. An md5 over the per-counter (count, hash sum) of all 200 counters is also equal (d40a35fd...).
• Re-grouping one 1.12 M-row chunk took 12.9 s to 20.9 s on this laptop. That includes the row count. The average was about 13.8 s over 27 chunks. The last 25 took 347 s in one pass.
• Chunks are re-compressed at about 80,000 rows/s here. A chunk with 10x the rows takes minutes.
• Disk: the chunk is held uncompressed until the commit. That is about 166 MB with its indexes here (4286 MB / 27 chunks). The old and the new compressed copy sit next to it. All of it goes through the WAL.
• The cold-cache disk read of the issue (11.2 s) cannot be reproduced on this laptop, because the OS cache is warm. The buffer counts above are the reproducible part.
• The large-store timing is for the separate run. It uses a read-only SQL file kept with the notes: the trend and get_perfmon_stats EXPLAIN (ANALYZE, BUFFERS), plus a progress query. It takes -v server_id=<id>.

Round 2: readers are not blocked, and the 30-day reach converges in hours

This section replaces the RegroupPerfmonChunksAsync and "2 minutes per pass" bullets of part 1.

The numbers come from a local PostgreSQL 18.6 with TimescaleDB 2.30.1 (shared_buffers 1 GB). The test chunk had 13,127,040 rows (43 servers x 61 counters, 1-minute samples). The largest store's chunks hold about 14 M. The chunk was compressed by server_id first.

A trend-read loop ran beside each re-group. It read one counter over 30 days, using the section-1 SQL of the rehearsal file. An idle read takes 15 ms.

1. Readers during a re-group

way to re-group	total	lock held by the writer	what a reader saw
shipped: decompress_chunk, count, compress_chunk in ONE transaction	113 s (58.4 + 0.9 + 53.7)	EXCLUSIVE, then ACCESS EXCLUSIVE from the end of the decompress to the commit	reads passed during the decompress (about 0.9 s each), then one read waited 54.4 s, the whole compress half
compress_chunk(chunk, recompress => true)	85 s	EXCLUSIVE, then ACCESS EXCLUSIVE for the compress half	one read waited 34 s
decompress in one transaction, compress in a second (this PR)	93.5 s (49.0 + 44.4). A second run took 62.8 s (42.8 + 20.0).	EXCLUSIVE only, in both	no waiting lock request seen. Slowest read of 131: 2.0 s. Slowest of 118: 1.6 s (disk contention, not a lock).

Why:

• decompress_chunk keeps EXCLUSIVE (reads allowed) until its last step. There it takes ACCESS EXCLUSIVE to drop the old compressed chunk.
• compress_chunk keeps EXCLUSIVE. This is from TimescaleDB 2.30.1 tsl/src/compression/api.c, decompress_chunk_impl and compress_chunk_impl.
• A lock lasts to the end of its transaction. So one transaction carried the ACCESS EXCLUSIVE through the whole compress half. Two transactions release it at the first commit.
• Docs checked: recompress => true is documented as In-memory recompression is attempted first; it falls back to internal decompress/compress. See https://github.com/timescale/docs/blob/latest/api/compression/compress_chunk.md. The same sentence appears for convert_to_columnstore at https://www.tigerdata.com/docs/api/latest/hypercore/convert_to_columnstore.
• With changed settings, recompress => true fell back to the full rewrite in one statement (row 2).
• Segment-wise recompression is for partially compressed chunks with unchanged settings. So it cannot apply new settings. This is from the docs and was not run here.
• So there is no supported way to keep reads going through a settings-changing re-compress in one call. The change is the two transactions.

A long reader can hold the chunk (an open transaction that read it). Then the decompress ran 57 s and hit the 3 s lock timeout at its end. It rolled back, because it failed to get ACCESS EXCLUSIVE. A reader queued behind it waited at most 4.1 s (3 s plus noise).

That is the existing "busy lock skips the chunk" behaviour. It is bounded to a few seconds for readers. The lost work is the cost.

A crash between the two transactions

A crash leaves the chunk uncompressed, not broken. The compression policy compresses every uncompressed chunk older than a day with the hypertable's current grouping. That is the new one. The compress half is tried 3 times (3 s lock timeout each, 5 s apart, if_not_compressed => true). After that the chunk is left to the policy, with one Warning. A service stop between the two says so at Information.

2. Convergence in hours

PerfmonRegroupDrain is new (Darling/PerformanceMonitor.Darling.Storage/PerfmonRegroupDrain.cs). It is a background loop.

• It takes one chunk at a time on its own connection, newest first.
• After each chunk it pauses as long as the chunk took, and never under 15 s.
• A chunk with a busy lock is skipped for the rest of the run.
• Three failures in a row stop the run.
• It stops by itself when nothing is left (two catalog reads at Debug).
• It ends on the service's stopping token.
• The hourly pass (DarlingWorker.ConvergeStoreObjectsAsync) only calls StartIfIdle after the list and behind the TimescaleDB flag. No database work runs on its thread, so the pass is bounded again. The old list step is gone.
• The start path does not start it.
• The gates are the same. PerfmonRegroupBlockedReason is asked before every chunk.

Rate and projection:

• 13.1 M rows took 93.5 s (140 k rows/s). The one-transaction version took 113 s (116 k rows/s). This is synthetic data, which compresses far better than real counters (114 MB compressed for 1.8 GB).
• The first round's 80 k rows/s on higher-entropy rows is the safer figure.
• Projection for the large store: 30 chunks of 14 M rows each. That is 100 s per chunk at 140 k rows/s, and 175 s at 80 k rows/s.
• With a pause equal to the chunk, 30 x (100 + 100) s is 1.7 hours. And 30 x (175 + 175) s is 2.9 hours. The store is busy with the rewrite at most half the time.
• This is projected from the SQL halves run by hand at 13.1 M rows. The loop itself ran only on small chunks (live test below).

Test plan (round 2)

• Build Darling.Tests with 0 warnings (after merging origin/dev).
• PerfmonRegroupGateTests Total 8, Failed 0. There are 3 new tests. The loop starts without waiting and runs once. A failed run lets a new one start. A stop ends it quietly. There is also the wiring pin.
• PerfmonRegroupLiveTests Total 1, Failed 0 on a local PostgreSQL 18.6 / TimescaleDB 2.30.1. The gate keeps the loop idle while the hypertable has the old grouping. After the settings change, 5 in-reach chunks are re-grouped newest first (log order equals catalog order). The chunk 45 days back is untouched. The count and a hash per (server, counter) are identical. There are 4 pauses of 90 s for 5 chunks (none after the last). A second run is a no-op with no Information line.
• CollectionLogSegmentByLiveTests Total 8, Failed 0. The bounded-DDL overload change did not move it. StoreObjectConvergenceTests 14, StoreObjectHourlyStepSelectionTests 3, TimescaleSupportTests 79, RetentionReevaluationTests 11 and TimescaleAvailabilityReprobeTests 4: Failed 0.
• BoundedDdlSingleCopyTests.TheHourlyLockTimeout_IsSetInOneMethodOnly was FAILING on the previous head. It counts the helper's overloads, and the first round added a second one. The first round never ran it. Fixed deliberately: it now expects exactly 2 overloads. The statement-list one delegates to the body one. The single SET LOCAL lock_timeout is still pinned inside the helper.
• *Census* 477, *Ratchet* 15, *Hygiene* 90, *Scrub* 123, *Guard* 338, *DarlingStoreUpgrade* 269 and *SingleCopy* 4: Failed 0. The skips are live tests without a store.
• Not run: the full suite (CI decides).

Test plan (round 1)

• Build Darling.Tests with 0 warnings.
• PerfmonRegroupGateTests Total 5, Failed 0. TimescaleSupportTests Total 79, Failed 0 (16 live skipped, no DARLING_TEST_PG). StoreObjectConvergenceTests Total 7, Failed 0 (also on the merged rehearsal head).
• *Census* Total 477, Failed 0. *Hygiene* Total 90, Failed 0. *Guard* Total 338, Failed 0.
• The code path itself ran end to end on the local PostgreSQL. First the settings change. Then the re-group of 27 chunks newest first, with the budget honoured. A 20 s budget did 2 chunks, then 25 more in the next pass. The row checks are above.
• *Ratchet* and *Scrub* were run in round 2 (above).
• Not run: the full suite (CI decides).

Not done (handoff)

• The 30-day raise is done: see "The 30-day raise and the timing split".
• Live tests: all written and seen RED in Round 2b (see that section).
• RED proofs: done in Round 2b.
• Rehearsal branch: rebuilt in Round 2b, and merged forward again (see the sha there).
• The reader-wait numbers above come from psql with the same statements the product runs. They do not come from the product's loop at 13 M rows.
• *Ratchet* and *Scrub*: round 1 did not run them. Round 2 and Round 2b did.
• Parity check: the Darling Viewer reads the same table with the same filter (ViewerDataService.Perfmon.cs PerfmonTrendsSql: server_id = $1, counter_name IN (...)). So it benefits from the new grouping with no change. Its window cap was not checked. Lite stores perfmon in DuckDB (no TimescaleDB), so nothing changes there. Its chart window was not checked.
• Known behaviour of part 1, replaced by round 2: the pass ran on the serial sweep loop. Up to about 2 minutes plus one chunk of the hourly pass went to re-grouping while chunks remained. The pass's 5-minute budget bounded it. A chunk being re-grouped took an ACCESS EXCLUSIVE lock on that day's chunk until commit, so reads of that one day waited. Round 2 moved the work into the background loop and split the transaction, so this no longer holds.
• The commit carries attribution trailers. Leave them out of the squash message.

Round 2b

The crash between the two halves is safe. Every remaining behaviour now has a live test that was seen failing with its production line reverted. No production code changed in this round, so the rehearsal build from before still stands. Round 2b changed only PerfmonRegroupLiveTests.cs.

Rehearsal branch rehearsal/5574-on-5571:

• It merged origin/dev, then Query Store interval tables are partitioned by day #5573 (0cd56725b), then this branch.
• One conflict arose, in the convergence list of DarlingWorker.cs. It kept the Query Store interval step. It dropped the old hourly re-group step, which this branch had already replaced with the background loop.
• The Darling service builds with 0 warnings.
• Head at the build: dda26e6a8.
• Head after merging this branch's test commit: 8ee8668e3.
• Merged forward again: origin/dev (already contained), Query Store interval tables are partitioned by day #5573 at c5fca7d1b, and this branch at 8cec10e6d. No conflict this time. New head: ca0db631c018cd5084e8526829d98b249a066256. The Darling service builds with 0 warnings. Darling.Tests builds with 0 warnings. *QueryStoreInterval* Total 242, Failed 0. *PerfmonRegroup* Total 21, Failed 0. *Census* 477, *Ratchet* 15, *Hygiene* 90, *Scrub* 123 and *Guard* 338: Failed 0. The skips are live tests without a store.

The crash between the halves:

• CompressRegroupedChunkAsync already passes if_not_compressed => true. The code already leaves an uncompressed chunk to the compression policy. So no fix was needed.
• Two tests now show it. In the first, a chunk is left decompressed. The drain does not see it and finishes the other four with no warning. The policy's next run compresses it with the new grouping, with the same rows.
• In the second, the policy compresses the chunk before the compress half runs. The compress half reports done with no warning.
• On TimescaleDB 2.30.1, if_not_compressed already defaults to true. So the explicit argument only matters on releases where the default was false. The RED proof for that test sets it to false.

New live tests in PerfmonRegroupLiveTests (13 in the class now). Each makes its own scratch database.

Test	Production line reverted for the RED proof	What failed
fresh store compresses its first chunks with the new grouping	CompressionSegmentByFor returns server_id for perfmon_stats	the hypertable grouping assertion
crash between the halves	same revert	the hypertable grouping assertion
policy compressed the chunk first	if_not_compressed => false	compress half returned false
pause is at least the chunk's own time	pause set to the floor only	no pause was asked for (floor 0)
pause never under the floor (the existing test)	pause set to the chunk's time only	pauses were not 90 s
busy lock skips the chunk for the run	the skip on a busy lock removed	the drain did not end in 60 s
three failures in a row stop the run	the failure-limit check disabled	5 failures logged, not 3
compress retry (success on a later try, and all tries busy)	the loop cut to one attempt	returned false. 1 busy line, not 3
plan shape	the segmentby revert above	the drain re-grouped nothing (gate)
reader beside a 1.5 M-row re-group	decompress and compress in one transaction	a read waited 7.2 s of an 18.2 s re-group (4.9 s of 10.3 s in another run). With two transactions it waited 0.1 to 0.2 s.
service stop during the pause	pause waited on without the stopping token	the run did not end
service stop during the compress half	the "service is stopping" log line removed	line missing

The plan test compares the same chunk before and after the re-group. Before, only server_id is an index condition on the compressed chunk. After, counter_name is there too. Its own RED proof is the segmentby revert, which stops the drain before the chunk is touched. The test's "before" half shows the plan can tell the two groupings apart.

The reader test bound is 2.5 s. It proves nothing on a machine where a 1.5 M-row compress takes under 2.5 s. But it cannot fail there either.

RED proofs for the first round's test (AnUpgradedStore_IsReGroupedNewestFirst_...) each failed on their own assertion:

• ORDER BY ascending: the order.
• The reach filter removed: 6 chunks, not 5.
• The gate on the hypertable grouping removed: the run never ends.
• The "nothing to log" check always logging: the second run.
• The pause floor removed: pauses not 90 s.

The test file holds a chunk's lock through a small disposable class, because the live-cleanup ratchet refuses a finally teardown in a live test.

The 30-day ceiling raise is done in the last commit: see "The 30-day raise and the timing split".

Test plan (round 2b)

• Build Darling.Tests with 0 warnings.
• *PerfmonRegroup* Total 21, Failed 0 (PerfmonRegroupLiveTests Total 13, Failed 0 after the last edit). *CollectionLogSegmentByLiveTests* 8, *TimescaleSupportTests* 79, *BoundedDdlSingleCopyTests* 4 and *DarlingStoreUpgrade* 269 (13 skipped): Failed 0.
• *Census* 477, *Hygiene* 90, *Scrub* 123 and *Guard* 338: Failed 0. *Ratchet* Total 15 had Failed 1 on the first run (the lock finally blocks). It was fixed, then Failed 0.
• Not run: the full suite (CI decides), Lite.Tests (nothing in Lite changed).

Check round 1 fixes

Nine findings from the first correctness check are fixed on this branch (head b0a1a80). There is no migration.

• A busy lock pauses as long as the work it cost. Each chunk waits the larger of its own time and the 15 s floor. A lock lost at the end of a long decompress throws the whole rewrite away. The wait is now that time, not the floor. The 3 s lock timeout stays. A busy chunk that lost 10 s or more of work gets its own Information line with the time. The comment that said "a 3 s lock wait, not a rewrite" is corrected.
• No 300 s cap. The decompress and compress statements get a 30-minute cap, and the service stop token still cancels them. A chunk whose decompress hits the cap is remembered while the process lives. It is not tried again. The statement's own Warning is the one Warning. The compress half now retries only on a busy lock (55P03). Any other error ends the tries at once, and the chunk goes to the compression policy.
• One Warning when every compress try is busy. The three tries in one call no longer feed the "busy for 3 passes in a row" streak. The test counts every Warning line.
• A done chunk that is a candidate again is skipped. The drain keeps a set of the chunks it re-grouped. A repeat gets one Warning per process. It names the setting text the view returned.
• A compressed chunk with no per-chunk settings row counts as the old grouping. The read uses a LEFT JOIN now.
• Retention. The reach is the shorter of 30 days and the perfmon retention less one day. The retention is the fleet override in the schedule table, else the default. A chunk that is gone at decompress time (42P01) is skipped at Debug. It no longer counts as a failure.
• Run-end reasons. A blocked gate, a failed candidates read and "nothing left" are three different outcomes with separate log lines. A failed read logs one Warning per process, then Debug. The old "converged" Debug line no longer appears for a blocked gate or a failed read.
• Test de-flaking, nothing loosened. The reader test bounds the extra wait, still 2.5 s. That is the read during a re-group minus the same read without one, measured in the test. The pause bound tolerates 2 s and the stop test waits 10 s. Every test day comes from one timestamp captured once, and seed times are built in UTC.
• The gated and no-op tests assert the run-end value (Blocked, NothingLeft), not a log text.
• Startup wording. A comment in ConvergeStoreObjectsAsync said the first :30 tick starts the re-group. The first hourly pass, about 30 s after a start, starts it, and so does every hourly pass after. The comment is fixed. No test pinned the old wording.

Revert proofs: I reverted each production line, saw the named test fail, then restored it. All nine failed together on the reverted build.

• ABusyLock_PausesAsLongAsTheTimeItCost_AndALongOneSaysSo: the busy arm back to the floor.
• ADecompressThatHitsTheStatementCap_IsNotTriedAgainForTheLifeOfTheProcess: the remembered-chunk add removed.
• TheCompressHalf_IsNotRetriedAfterAnythingButABusyLock: retry on any outcome.
• WhenEveryCompressTryIsBusy_TheChunkIsLeftToThePolicy_WithOneWarning: streak fed again, two Warnings.
• AChunkThatStaysACandidateAfterItsReGroup_IsSkipped_WithOneWarningPerProcess: done-set check removed.
• AChunkWithNoPerChunkSettingsRow_IsACandidate: JOIN back.
• AChunkThatIsGoneAtDecompressTime_IsSkippedQuietly_NotFailed: the 42P01 arm removed.
• TheReach_IsClampedToTheRetentionLessADay: clamp removed.
• AFailedCandidatesRead_IsWarnedOncePerProcess_AndIsNotConvergence: Warning every time.

BoundedDdlBusyStreakTests.TheWiring_ClearsOnApplied_AndOnFailed_AndEscalatesOnBusy pins the number of BusyStreaks.Clear calls in the helper. It now expects 3, because the gone-relation arm clears a chunk's streak too. I changed it on purpose.

Tests ran on a local PostgreSQL 18 with TimescaleDB 2.30.1 (Total / Failed):

• PerfmonRegroup* 29 / 0
• CollectionLogSegmentByLive* 8 / 0
• TimescaleSupportTests 79 / 0
• BoundedDdl* 11 / 0
• DarlingStoreUpgrade* 269 / 0 (13 skipped)
• StoreObjectConvergence* 14 / 0
• Census 477 / 0, Ratchet 15 / 0, Hygiene 90 / 0, Scrub 123 / 0, Guard 338 / 0

The Darling tests and the service build with 0 warnings. I ran no full suite, because CI decides.

Check round 2 fixes

Four small defects and one weak assertion from the second check. The changes are in TimescaleSupport.cs, PerfmonRegroupDrain.cs and PerfmonRegroupLiveTests.cs. There is no migration, and the user-facing entry above does not change.

• A server cancel is logged as that. A raw 57014 comes from a statement_timeout on the role or database, or a cancel from another session. It still stops the drain from retrying the chunk until the service restarts, so a short server timeout does not cost a rewrite every hour. The Information line in the TimedOut arm of PerfmonRegroupDrain.cs now gives the real elapsed time. When the cancel came first, it says "cancelled by the server (a statement_timeout or a cancel) after N s". It then names the 1800 s cap it fell short of. It says "hit the statement cap after N s" only when the statement ran to the cap. The new test ADecompressTheServerCancelsBeforeTheCap_IsNotTriedAgain_AndTheLogSaysTheServerDidIt gives the drain a 2 s session statement_timeout beside a held lock. The existing cap test now also asserts the cap wording. RED: forcing the cap wording on every time-out fails the new test.
• A chunk dropped between the two halves is Gone. A relation error counts as gone only when the chunk catalog agrees. ResolveMissingRelationAsync in TimescaleSupport.cs re-reads timescaledb_information.chunks after a 42P01 at either half. A chunk that is not listed is Gone: a Debug line, no failure count, and no "the policy compresses it later" text. A chunk that is still listed, or a catalog that cannot be read, is a Failed chunk with a Warning that says so. CompressRegroupedChunkAsync now returns the chunk result (Regrouped, Gone or LeftUncompressed) instead of a bool, so the compress half can report Gone. The new test ARelationErrorIsGoneOnlyWhenTheChunkCatalogAgrees_AtEitherHalf covers the compress half on a missing chunk, and the helper on a listed chunk at both halves. RED: skipping the compress-half Gone return fails it. So does making the helper always answer Gone. We found no way to make a real decompress raise 42P01 on a chunk that still exists. The listed-chunk branch is therefore proven on the helper, not end to end.
• The gone-chunk test proves the run goes on. It stubs three gone chunks, and the failure limit is three in a row. It asserts the run ends AllSkipped with no Warning, and it finds a Debug "no longer exists" line for each chunk. RED: counting a gone chunk as a failure ends the run at the third with a Warning, and the test fails.
• The reach test no longer depends on when the class loaded. TheReach_IsClampedToTheRetentionLessADay reads the chunks inside its reach from the database now(). It reads them just before the run. It expects exactly those chunks re-grouped and the rest left alone. The comment on RunStartUtc says which tests rely on slack and which read the clock. This is a test-only fix, so no production line can be reverted.
• The reader baseline is the median of warm reads. The first, cold read is dropped. The median of five warm reads is the baseline, so one slow outlier cannot widen the 2.5 s bound. The bound is unchanged. This is test-only too.

Test plan (check round 2)

• PerfmonRegroup* classes: Total 31, Failed 0, on a local PostgreSQL 18 with TimescaleDB 2.30.1 in UTC.
• BoundedDdl*, TimescaleSupportTests and StoreObjectConvergence* together: Total 104, Failed 0.
• *Census*, *Ratchet*, *Hygiene*, *Scrub* and *Guard* in Darling.Tests: Total 965, Failed 0.
• Darling.Tests and the Darling service build with 0 warnings and 0 errors. The rehearsal branch builds too, after merging origin/dev and this branch (rehearsal head 9bd5474).
• Reverts, not committed: each of four production lines made its test fail. They were the cap wording, the compress-half Gone return, the helper answering Gone, and a gone chunk counted as a failure. All were restored.
• Lite.Tests not run, because no Lite file changed.

The 30-day raise and the timing split

Merge of origin/dev

#5570 (the time range picker) is in dev. The merge had no conflicts, so no file needed a hand resolution. WebReadReach.cs and McpHelpers.cs came in clean.

The raise

get_perfmon_trend reaches 720 hours (30 days) on the web and over MCP, the same as get_wait_trend. On a large store, the cold 30-day read of one counter went from 11.2 s to 1.3 s. That was after the data was re-grouped by counter (see the table above). That is inside the 10 s bar the other raised trends were held to.

• WebReadReach.cs: the row is Trend("perfmon_stats", RawTrendHours). The block comment above the raw-table rows no longer lists get_perfmon_trend as left at 168 and cites the measured read.
• DarlingMcpTrendTools.cs (GetPerfmonTrend): the validator takes WebReadReach.MaxHoursFor("get_perfmon_trend"), like get_wait_trend. The tool's guide tail now says "hours_back reaches 720 (30 days)". The number is not in the hours_back parameter text. EveryRaisedTool_StatesItsRowsNumberInTheGuideTail... requires it in the tail, because the parameter line is in every tools/list answer.
• The web catalog's max_hours follows the row. Catalog_EveryHoursParam_CarriesMaxHoursFromTheTable pins it, and now pins 720 for this read.
• Tests: WebReadReachTests lists get_perfmon_trend among the raised reads and pins MaxHoursFor at RawTrendHours. A new test, ThePerfmonReGroupDrainsReach_IsThePerfmonTrendsReachInDays, ties TimescaleSupport.PerfmonRegroupReachDays to WebReadReach.RawTrendHours / 24 and to the read's own reach, so the re-group always covers what the read reaches. WebReadReachLiveTests asks get_perfmon_trend at 720 (answered, no error) and 721 (refused with "exceeds maximum of 720 hours (30 days)"), as it does for the other raw trends.
• RED proofs: each reverted only the production line and was restored, never committed.
  • The row back to the default made 5 of the 18 WebReadReachTests fail.
  • The drain's reach set to 7 failed ThePerfmonReGroupDrainsReach_IsThePerfmonTrendsReachInDays.
  • The validator back to the 168-hour overload made EveryOptedInValidator_... and WebReadReachLiveTests fail.
• Lite: checked Lite/Mcp/McpPerfmonTools.cs (line 77). Its get_perfmon_trend uses the default 168-hour ValidateWindow, as does every other Lite MCP trend. Lite has no reach table and no raised trend, and its perfmon data is in DuckDB, which the re-group does not touch. Nothing to mirror.

The timing split

AReaderBesideTheReGroupOfALargeChunk_NeverWaitsMoreThanAFewSeconds and AServiceStopDuringTheCompressHalf_CancelsTheWait_AndSaysTheChunkIsLeftToThePolicy start a clock and judge a time against a limit. They moved, with every assertion and bar unchanged, into the new PerfmonRegroupTimingLiveTests with [Collection("timing")], so they can run on a quiet process. Both mint their own scratch database, so the class carries the #1776 own-store comment instead of live-postgres.

The helpers they share with PerfmonRegroupLiveTests are in the new static PerfmonRegroupLiveSupport (nothing was copied). No guard rejects the undefined collection name: Census, Ratchet, Hygiene, Scrub, Guard, DocComment and the Stage=Guard trait all pass. I ran the clock rule of the open timing-collection change over the nine test files this PR touches. The only matches are the two Stopwatch.StartNew calls in PerfmonRegroupTimingLiveTests. None of the files starts an STA thread.

A time-of-day defect fixed in the same file

AFreshStore_CompressesItsFirstChunksWithTheNewGrouping_AndTheDrainHasNothingToDo seeded the chunks 2, 3 and 4 days back and expected the policy to compress all of them. The perfmon policy compresses after 1 day plus 3 hours. Between 00:00 and 03:00 UTC the chunk 2 days back is not yet due, and the test failed (it did, at 00:37 UTC). It now seeds 3, 4 and 5 days back, which are due at any hour. Every assertion is unchanged.

Test plan (this part)

On a local PostgreSQL 18.6 with TimescaleDB 2.30.1, in UTC:

• Darling.Tests and the Darling service build with 0 warnings and 0 errors.
• *PerfmonRegroup*, *BoundedDdl*, *TimescaleSupportTests*, *WebReadReach* and *DarlingMcpTrendTools* together: Total 187, Failed 0.
• *McpToolGuide*, *McpToolsListBudget*, *WebRangeKeptHistory*, *WebServerPageRange*, *WebPerfmon*, *WebTimeRange*, *TrendPayloadBudget*, *TrendStaleStats*, *PerfmonWaitStatisticsArtifact*: Total 237, Failed 0.
• *Census*, *Ratchet*, *Hygiene*, *Scrub*, *Guard*, *DocComment*: Total 973, Failed 0. Stage=Guard trait: Total 1434, Failed 0.
• AReaderBesideTheReGroupOfALargeChunk_NeverWaitsMoreThanAFewSeconds failed once in its new class. That run was on a cold local server, beside WebReadReachLiveTests in parallel. The timing collection does not serialize until the timing-collection change merges. The test passed alone, beside that class again, and in the two full-class runs that include it. The bar is unchanged.
• Lite.Tests not run: no Lite file changed.
• Not run: the full suite (CI decides).

Fixes #5574

Measured on a copy of a large store

These are from #5574's Part A (before) and Part B (after) on a restored copy of a large SQL Server store. Details are in issuecomment-6067584601 and issuecomment-6069386286.

Read	Before	After
One counter (Batch Requests/sec) over 30 days, first read after a PostgreSQL restart	11,216 ms, 35,735 blocks read from disk	1,303 ms, 18,642 blocks read from disk
The same read again	1,165 ms	219 ms
Compressed batches read in each UNION ALL leg	4,494	57
Rows the filter threw away in each leg	about 10.4 M	about 0.5 M

• The re-group on that copy: 38 chunks and 341.5 M rows in 4,595 s, about 77 minutes. The median chunk took 69.6 s. No chunk was skipped, no Warning was logged, and no chunk was left uncompressed.
• During the re-group: the same 30-day read took 1,053 ms with a warm cache. No chunk rewrite was running at that moment.
• A small cost: the newest-values read (280 rows from the newest, uncompressed chunk) went from 1.3 ms to 9.7 ms. Its plan now uses a Merge Append where it used a Sort. It stays under 10 ms.
• The PostgreSQL restart empties PostgreSQL's cache only. The operating system's file cache was not emptied, in both runs.