Skip to content

The twenty continuous aggregates join the compression ladder: compress_after derived from each tier's refresh window, a once-a-day band off the hourly grid, and the backlog staged one aggregate per night, largest first (#3581) - #3611

Merged
erikdarlingdata merged 2 commits into
devfrom
fix/3581-lane
Sep 18, 2026

Conversation

@erikdarlingdata

Copy link
Copy Markdown
Owner

Closes nothing by keyword — #3581 is closed by hand at merge, per house convention. PARTIAL against #3581: the daily-tier retention question the issue's comment raises is deliberately untouched (see What this does not do).

The omission, with the numbers

Every raw hypertable in the store has compressed since the archival tier existed (CompressAfterDays, segmentby server_id, the hourly tick, the #3035 phase grid). No continuous aggregate ever did, and nothing in TimescaleSupport said so — not a rejected design, an omission. On the largest production store the twenty materializations were 235 GiB of a 415 GiB database, every one reading compression_enabled = false and zero policy_compression rows on any materialization. The four Query Store aggregates alone were 71.5 / 55.6 / 54.7 / 33.5 GiB against a raw query_store_stats of 25 GiB compressing at ~9x — because the hourly grain is only a ~4x row reduction, so an uncompressed rollup of a 9x-compressed source is larger than the source. The 90-day hourly tier was a month into its first fill; uncompressed, the store was on course for a 600–650 GB plateau in mid-November, and at the raw tier's ratio the same plateau is ~200 GB.

The rig reproduced the shape in miniature (PG 18.4 / TimescaleDB 2.28.1, six servers × 150 queries × 10-minute snapshots × 25 days): 1,056 MB of raw compressed to 32 MB while its five Query Store aggregates held 110 / 110 / 179 / 148 / 5 MB uncompressed. Once enabled they compressed at 10.1x to 18.6x.

What ships

TimescaleSupport.EnsureAggregateCompressionAsync, called from the worker's TimescaleDB block right after EnsureContinuousAggregatesAsync and before EnsureRetentionPoliciesAsync. An idempotent startup ensure, not a rung: ALTER MATERIALIZED VIEW collect.<view> SET (timescaledb.compress, timescaledb.compress_segmentby = 'server_id', timescaledb.compress_orderby = 'bucket DESC') gated on continuous_aggregates.compression_enabled, then add_compression_policy(...) with if_not_exists, one aggregate at a time inside its own try (the ALTER must precede the policy — the rig raises columnstore not enabled on continuous aggregate the other way round). A converge (SetAggregateCompressionPolicySql) moves an existing policy whose compress_after, cadence, fixed-schedule flag or band instant differs, because add_compression_policy returns -1 and changes nothing against a policy the store already has — the #1778/#1937 drift, pre-empted rather than filed later. The summary line names every aggregate and its window, the #1958 way.

The registry is derived, not hand-listed: AggregateCompressionTargets = HourlyAggregates ∪ DailyAggregates ∪ BaselineAggregates, in creation order. DailyAggregates is hoisted out of the ensure sweep's inline array for the reason HourlyAggregates was hoisted at #3012 — the creation sweep, the compression registry and the tests now read one list. The segmentby column and the orderby alias are recovered from each shipped CREATE (RefreshGroupingTermsFor, AggregateBucketColumnFor) and asserted per definition, so they are properties of the registry rather than assumptions.

compress_after, derived

AggregateCompressMarginSpan      = 1 day   (one raw chunk, ChunkIntervalDays)
HourlyAggregateCompressAfterSpan = HourlyRefreshStartSpan (1 day)  + margin = 2 days   (13 aggregates: the six hourlies + seven baselines)
DailyAggregateCompressAfterSpan  = DailyRefreshStartSpan  (3 days) + margin = 4 days   (7 aggregates: the hierarchical dailies)

Why a margin at all, measured: a refresh that reaches into a compressed materialization chunk does not fail on 2.28.1. It decompresses every compressed batch overlapping the buckets it re-materializes (1,000 rows per server_id segment — a six-row backdated write into one hourly bucket staged 6,006 rows), rewrites them into the chunk's heap side, and leaves the chunk PARTIAL (catalog status 9) until the next compression run recompresses it. Single-bucket refresh: 8 ms uncompressed → 29 ms compressed + a 195 ms recompression on the next policy pass. Forced eight-day refresh: 1,150 ms → 1,514 ms, leaving 180,000 heap rows for a 265 ms recompression. So an overlapping window would not break anything; it would make the newest chunk a permanent decompress-and-recompress churn, once an hour, forever.

Why one raw chunk rather than one bucket: the refresh window reaches back start_offset and aligns its start down to a bucket boundary, so disjointness needs compress_after ≥ start_offset + bucket — 1 day 1 hour hourly, 4 days daily. The daily tier's bucket is a day so the floor is a day there already; the hourly tier's is an hour, and an hour is the wrong unit for the gap — every other boundary in the store moves in days (chunks close at UTC midnight, raw eligibility flips at UTC midnight), and a sub-day margin puts the refresh's aligned start and the compression boundary within one scheduling jitter of each other. One raw chunk on every tier lands the daily tier on its own floor and puts a whole day between the two on the hourly tier: 2 and 4, the values the issue was opened for, reached as an expression. Pinned per tier with the tier's own bucket, and pinned against every retained aggregate's horizon (compress_after < drop_after, and 2 × compress_after < drop_after so the short interval-identity tiers still spend most of their life compressed at day-width chunks).

The band, and why not #3174's compression band

The hourly grid tiles the hour with no minute unassigned, but two stretches carry no start: the guard after the light band, and the heaviest refresh's window past its own start. The compression grid deliberately leaves the tail of that window unused for hourly policies ("the other 6 minutes of the window are past the refresh and are left on the table"), because recovering them for jobs that run every hour would size a band against a still-moving ceiling. A once-a-day job is a different trade.

AggregateCompressionBandMinute = HeaviestRefreshStartMinute + HeaviestRefreshWindowMinutes − 1 = :35 — the window's last minute, derived from geometry rather than from the ceiling constant (so re-taking the ceiling per #3182/#3188 moves no job). It is 1,200 s past the heaviest refresh's start against an 896 s recorded ceiling (the first clear minute, :30, would be 900 s — a four-second margin), past every light refresh's 226.8 s ceiling measured from its own start, on no hourly refresh's minute, on no raw compression minute, and the minute immediately before CompressionPhaseMinutes[0]. All of that is asserted against the shipped grid's output, not a copy of its rule. Clearance to the next refresh start is 25 minutes.

One aggregate per hour, from 01:35Z, in registry order (AggregateCompressionBandHourFor). An hour apart rather than "a few minutes apart" for the reason #3185 recorded: distinct starts are a lock guarantee, not an overlap guarantee — a run longer than the step overlaps its successor for the rest of its run, and these runs are minutes (a day's chunk of the largest aggregate is roughly the raw table's, which #3112 measured at 552 s). The hour also has only six minutes past every ceiling and on no grid start, so a twenty-member minute-stagger had nowhere to go. One per hour makes "no two aggregates decompress and recompress at once" hold by construction, and keeps the family inside the managed store's background-worker headroom (at most one of these jobs runs at a time; the + 2 over HypertableCount is not re-derived). Hour 1 is the hour after the one that carries the raw tier's midnight rewrite (#3112's midnight band, query_store_stats at :42 for 552 s); aggregate chunks become eligible at the same UTC midnight, so it is the earliest hour that both sees the new eligibility and is clear of that burst. Twenty-three hours are available; a twenty-fourth member is red rather than wrapped onto midnight.

Why not the hourly compression band: its width is HypertableCount / CompressionPhaseMaxPerMinute, and every minute it takes comes out of HeaviestRefreshWindowMinutes. Twenty more members would widen it from 24 minutes to 31 and shrink the window from 21 to 14 — 840 s against the 896 s ceiling, which TimescaleSupportTests holds red.

Lock geometry, measured on the rig with pg_locks mid-run: compress_chunk holds AccessShareLock on the materialization hypertable and escalates only on the chunk it rewrites (ShareLock → ExclusiveLock → AccessExclusiveLock at the swap, all three observed in one run). So a compression run cannot block the aggregate's own refresh, which writes the newest chunk — the one the margin keeps out of the eligible set. What it can hold up is a reader of exactly the chunk being swapped, for the swap, which is the cost the raw tier pays today. The daily refreshes (finish-to-start, drifting) and the nightly purge (_nextPurgeUtc, anchored to service start) run at instants no fixed band can avoid by construction; the purge is chunk drops and the refreshes are read-only against these chunks, so neither is a lock hazard, and the comment says so rather than claiming a schedule that dodges them.

The staging

A compression policy's first run compresses every chunk older than compress_after, one chunk per transaction, in one pass — on an existing store that is the whole aggregate minus its newest two or four days, and on the largest store that is the 235 GiB event. Every run after the first finds only what aged in since yesterday. So the staging is about first runs only, and it is carried by the schedule rather than by state: every policy is created at once, and initial_start is the next UTC midnight plus a night offset — nights assigned largest first by measured materialization size to aggregates with eligible chunks (0, 1, 2, …), and night zero to every aggregate with nothing eligible. A fresh store therefore gets all twenty policies at once with no special case; on a backlog store the plan is visible in timescaledb_information.jobs as consecutive next_start dates, each aggregate at its own hour; a restart remembers nothing and re-adds nothing. StageAggregateCompressionNights is pure and pinned (largest first; eligible-chunk count does not order — bytes do; ties break on registry order; a fresh store is all night zero).

The one thing this shape does not do is re-plan around a partial failure: a start that created ten of twenty policies and lost its connection stages the remaining ten from night zero on the next start, so up to two aggregates can share a night on that path, each still at its own hour. Stated in the comment and accepted over a stateful cursor.

On the rig, the "night 0" run for the largest aggregate compressed three 10-day chunks (137 MB → 10 MB) in 445 ms and left the newest chunk alone.

What the rig measured

measurement before after
viewer month-scale trend read (QueryStoreTrendRouting.BuildRollupTrendSql's rollup arm over query_store_stats_corrected_hourly, 30-day window, one server) 15,179 shared buffers, 11.2 ms, Index Scan Backward on (server_id, bucket) per chunk 3,522 shared buffers, 6.6 ms, Custom Scan (ColumnarScan) per compressed chunk with Index Cond: (server_id = 3) and the bucket range as a vectorized filter over batch min/max metadata; the uncompressed newest chunk keeps its index scan
query_store_stats_corrected_hourly materialization 110 MB 30 MB
steady-state policy refreshes (1-day window) with compressed old chunks present — 22 / 31 / 12 ms, touching nothing compressed
daily hierarchical refresh reading a compressed hourly region (3 days) — 578 ms, works, leaves the daily's chunk PARTIAL until its own policy recompresses
single-bucket refresh into a compressed chunk 8 ms 29 ms + 195 ms recompression on the next policy pass
forced 8-day refresh into a compressed chunk 1,150 ms 1,514 ms + 265 ms recompression

The segmentby server_id matches the read shape exactly: the compressed chunks are read by segment with the index condition on server_id alone.

Reader plan before (EXPLAIN ANALYZE, BUFFERS, rollup arm of the trend read)
Finalize GroupAggregate (actual rows=599.00 loops=1)
  Group Key: _materialized_hypertable_15.bucket
  Buffers: shared hit=15179
  ->  Sort (actual rows=599.00 loops=1)
        ->  Custom Scan (ChunkAppend) on _materialized_hypertable_15 (actual rows=599.00 loops=1)
              ->  Partial GroupAggregate  (chunk 35)  Buffers: shared hit=2801
                    ->  Index Scan Backward using _hyper_15_35_chunk__materialized_hypertable_15_server_id_bucket (actual rows=16500.00)
                          Index Cond: ((server_id = 3) AND (bucket >= ...) AND (bucket <= ...) AND (bucket < ...))
              ->  Partial GroupAggregate  (chunk 36)  Buffers: shared hit=6080
                    ->  Index Scan Backward using _hyper_15_36_chunk__materialized_hypertable_15_server_id_bucket (actual rows=36000.00)
              ->  Partial GroupAggregate  (chunk 37)  Buffers: shared hit=6080
                    ->  Index Scan Backward using _hyper_15_37_chunk__materialized_hypertable_15_server_id_bucket (actual rows=36000.00)
              ->  Partial GroupAggregate  (chunk 38)  Buffers: shared hit=218
                    ->  Index Scan Backward using _hyper_15_38_chunk__materialized_hypertable_15_server_id_bucket (actual rows=1350.00)
Execution Time: 11.242 ms
Reader plan after (three chunks compressed, newest uncompressed)
Finalize GroupAggregate (actual rows=600.00 loops=1)
  Group Key: _materialized_hypertable_15.bucket
  Buffers: shared hit=3522
  ->  Sort (actual rows=600.00 loops=1)
        ->  Custom Scan (ChunkAppend) on _materialized_hypertable_15 (actual rows=600.00 loops=1)
              ->  Partial GroupAggregate  (chunk 35, uncompressed)  Buffers: shared hit=2782
                    ->  Index Scan Backward using _hyper_15_35_chunk__materialized_hypertable_15_server_id_bucket (actual rows=16800.00)
              ->  Partial GroupAggregate  (chunk 36)  Buffers: shared hit=362
                    ->  Custom Scan (ColumnarScan) on _hyper_15_36_chunk (actual rows=36000.00)
                          Vectorized Filter: ((bucket <= ...) AND (bucket >= ...) AND (bucket < ...))
                          ->  Index Scan Backward using compress_hyper_33_77_chunk_server_id__ts_meta_v2_last_bucke_idx (actual rows=36.00)
                                Index Cond: (server_id = 3)
                                Filter: ((_ts_meta_min_1 <= ...) AND (_ts_meta_max_1 >= ...) AND (_ts_meta_min_1 < ...))
              ->  Partial GroupAggregate  (chunk 37)  Buffers: shared hit=364
                    ->  Custom Scan (ColumnarScan) on _hyper_15_37_chunk (actual rows=36000.00)  ... Index Cond: (server_id = 3)
              ->  Partial GroupAggregate  (chunk 38)  Buffers: shared hit=14
                    ->  Custom Scan (ColumnarScan) on _hyper_15_38_chunk (actual rows=1350.00)   ... Index Cond: (server_id = 3)
Execution Time: 6.616 ms

Two load-bearing edits outside the new section

ConvergeCompressionScheduleAsync skips the family. An aggregate's compression job reports the aggregate's user view as its hypertable (collect / <view> — measured, the same resolution ContinuousAggregateRefreshStateSql documents for refresh jobs), so it lands in the raw converge's unscoped proc_name LIKE '%compression%' read, where a cadence other than one hour is the staleness test (#1778). Without the exclusion the first start after this build would retune every one of these once-a-day jobs to the hourly tick and put all twenty on the hourly grid — the placement the ruling excluded. Excluded by name on the narrow read (no schema column) and by name-in-collect on the wide one, so a foreign hypertable elsewhere that happens to share an aggregate's name keeps #1778's converge exactly as before. The live test asserts the family stays on the daily cadence after the raw converge runs.

CompressionActivitySql resolves the materialization. timescaledb_information.chunks knows an aggregate only by its _materialized_hypertable_N, so #1778's eligible-chunk count keyed on the job's own name read zero forever for every aggregate job — the #3582 shape in another reader. It now LEFT JOINs continuous_aggregates to count on the materialization when the job is an aggregate's, at the job's own compress_after from its config rather than the raw constant, and emits that delay so the log line says "2d" / "4d" / "every 1 day" for these jobs instead of the raw tier's values (#1958's rule). CompressionActivity gains an optional trailing CompressAfter, so the five-argument shape every caller and test constructs is unchanged. The #3112 clearance watch deliberately still reads these jobs as foreign — its overrun text reasons from the raw chunk close, and misdescribing an aggregate run would be worse than not judging it; a watch over the daily band's own clearance is a follow-up, said so on AssignedPhaseMinute.

A production fact the issue and the rig disagree on, flagged rather than assumed

The issue's staging note describes the backlog as "~60 1-day chunks" per aggregate. On a fresh 2.28.1 store the materialization chunk interval is ten times the root raw hypertable's — measured: a 1-day raw table yields 10-day materialization chunks on every aggregate built over it, hierarchical ones included (a 7-day raw yields 70-day, a 1-hour raw 10-hour). A store whose raw tables were created at ChunkIntervalDays = 1 therefore has 10-day materialization chunks, not 1-day. The design here is correct at any width — a chunk compresses only when its whole range is past compress_after, so the separation argument does not depend on it — but the shape of the work does: at 10-day chunks each aggregate compresses one 10-day chunk every tenth night rather than one day's chunk nightly, a 10-day chunk of the 7-day interval tier lives seventeen days and is compressed for the last five, and the 90-day tiers hold up to 100 days. The ensure's comment states both cases. Verifying the width on the store is one read the coordinator can fire — SELECT ca.view_name, c.range_end - c.range_start, count(*) FROM timescaledb_information.chunks c JOIN timescaledb_information.continuous_aggregates ca ON c.hypertable_name = ca.materialization_hypertable_name WHERE ca.view_schema = 'collect' GROUP BY 1, 2 — and narrowing new materialization chunks to a day (set_chunk_time_interval on the materialization; existing chunks keep their width) is a separate decision this PR does not take, reported in the lane report.

Tests

TimescaleAggregateCompressionTests (new, in the live-postgres collection). Ungated: the derivation (margin = one raw chunk; each tier's compress_after = its offset + margin = 2 / 4 days; ≥ offset + bucket per tier; the literal renderer refuses non-day spans); the registry (20 = 6 + 7 + 7, once each, tier by source list, bucket alias and server_id group key recovered from every CREATE, the parse controlled with a differently-aliased CREATE and two refusals); every retained aggregate compressing inside its horizon, walked over RetentionPolicies (14 tiers, control asserted); the band (:35 as geometry, on no refresh minute, on no compression minute, the minute before the raw band, 1,200 s past the heaviest start > 896, past every light ceiling, 25 minutes of clearance, hours 1..20 distinct and inside the day, both throws); the statements (tier window, daily cadence, if_not_exists, the UTC anchor at hour/night, never the raw tier's values; the converge's bound job id, jsonb_set, fixed schedule, no scheduled; the state read's either-identity join and both delays; the activity read's materialization resolution); the staging on a synthetic store (largest first one per night, bytes not chunk count, night zero for no backlog, a fresh store all night zero, registry-order tie-break); the exclusion predicate over all twenty and none of the raw hypertables. Gated on DARLING_TEST_PG: the ensure against the fixture — every aggregate compression_enabled, exactly one compression job each at its tier's window on the daily cadence, fixed schedule, its hour and :35 read back from the catalog; a settled second pass adds and converges nothing and says so; the raw converge leaves all twenty on the daily cadence; the activity read sees them at their own delay — and a second live test that drifts one policy onto the raw tier's values and asserts the converge moves it and then settles.

Tests cannot execute on the authoring machine (net10.0-windows); CI is their first execution. Every ungated assertion was mirrored line-for-line in a throwaway console against the built assembly and holds; the two live flows were run against the rig on a fresh database through the exact call sequence the tests use (MigrateAsync → TryEnableAsync → ConvertToHypertablesAsync → ConvergeContinuousAggregateRefreshAsync → EnsureContinuousAggregatesAsync → the ensure, twice → the raw converge → the drift and re-converge), with every read-back matching.

Storage, Service, Viewer and Darling.Tests build at 0 warnings with -c Release -p:EnableWindowsTargeting=true.

What this does NOT do

Docs: Darling/README.md's TimescaleDB section gains the aggregate-compression bullet; docs/retention-hold-runbook.md gains a section naming the four job families an operator meets in timescaledb_information.jobs, including the new one and what its staged next_start dates mean. CHANGELOG.md deliberately not touched; the entry is in the lane report.

…s_after derived from each tier's refresh window, a once-a-day band off the hourly grid, and the backlog staged one aggregate per night, largest first (#3581)

Every raw hypertable has compressed since the archival tier existed; no continuous aggregate ever did, and nothing in source said so. On the largest production store the twenty materializations were 235 GiB of a 415 GiB database, all reading compression_enabled = false — the four Query Store aggregates alone 71.5/55.6/54.7/33.5 GiB against a 25 GiB raw table compressing at ~9x, because an uncompressed rollup of a 9x-compressed source is larger than the source.

TimescaleSupport.EnsureAggregateCompressionAsync is an idempotent startup ensure, not a rung: ALTER MATERIALIZED VIEW ... SET (timescaledb.compress, segmentby server_id, orderby bucket DESC) under a catalog check, then add_compression_policy once a day per aggregate, failure-isolated per aggregate, with a #1958-style summary naming every aggregate and its window.

compress_after is an expression: each tier's refresh start_offset plus one raw chunk (ChunkIntervalDays) — 2 days for the thirteen hourly-refreshed aggregates, 4 for the seven dailies. Measured on 2.28.1 why the two boundaries must stay apart: a refresh into a compressed region works but decompresses 1,000-row batches per segment and leaves the chunk PARTIAL for the next policy run (8 ms -> 29 ms + 195 ms recompress for one bucket; 1,150 ms -> 1,514 ms + 265 ms for a forced eight-day refresh).

The band is the last minute of the heaviest refresh's window (:35, 1,200 s past its start against an 896 s ceiling, the minute before the raw compression band opens), one aggregate per hour from 01:35Z in registry order — an hour apart rather than minutes because distinct starts are not non-overlap (#3185), and because joining #3174's compression band would widen it from 24 to 31 minutes and take the heaviest refresh's window under its ceiling.

The backlog is staged in the schedule itself: every policy is created at once with initial_start on the next UTC midnight plus its night, nights assigned largest first by measured materialization size to aggregates with eligible chunks and night zero to everything without — so a fresh store gets all twenty at once, the plan is visible as consecutive next_start dates, and a restart remembers nothing.

The raw compression converge now skips these jobs by name — their once-a-day cadence would otherwise read as stale against the hourly tick and the first start after this build would put all twenty on the hourly grid. #1778's activity read resolves an aggregate's chunks through its materialization at the job's own delay instead of reading zero forever. DailyAggregates is hoisted out of the ensure sweep so the creation sweep, the compression registry and the tests read one list.
@erikdarlingdata
erikdarlingdata enabled auto-merge (squash) September 18, 2026 16:43
@claude

claude Bot commented Sep 18, 2026

Copy link
Copy Markdown

Reviewed the diff (TimescaleSupport.cs, DarlingWorker.cs, the new TimescaleAggregateCompressionTests.cs, and the two docs). Scope notes and what I checked:

Lite/Darling parity: not applicable here — TimescaleDB continuous-aggregate compression only exists on Darling's PostgreSQL store (Lite has no TimescaleDB analog, per CONTRIBUTING.md's note that TimescaleDB support is runtime setup outside the shared migration ladder). No parity gap.

Correctness, traced through by hand:

  • compress_after derivation (HourlyRefreshStartSpan/DailyRefreshStartSpan + AggregateCompressMarginSpan) checks out against the actual constants in the file (1d/3d refresh offsets, 1d margin → 2d/4d), and the disjointness inequality (compress_after ≥ start_offset + bucket) holds for both tiers including the daily tier's equality case.
  • BaselineAggregates are correctly folded into the hourly tier (AddHourlyRefreshPolicySql/HourlyRefreshStartOffset), so they get the 2-day compress_after — matches how their refresh policy is actually attached in EnsureContinuousAggregatesAsync.
  • IsAggregateCompressionTarget's bare-name matching and the two exclusion paths in ConvergeCompressionScheduleAsync (schema-scoped on the phased read, unscoped on the narrow fallback) behave as documented, and the registries are asserted disjoint by test.
  • EnsureAggregateCompressionAsync's per-aggregate failure isolation, idempotency (catalog-gated ALTER, if_not_exists, converge-on-drift only), and the AggregateCompressionStateSql column ordering (0–10) all line up with how the reader consumes them.
  • DailyAggregates hoist is a clean extraction — same 7 entries, same order, single remaining reference — so the "one list" claim holds.
  • CompressionActivitySql's LEFT JOIN to resolve an aggregate's materialization and read compress_after from the job's own config (fixing the always-zero backlog count) is correct and backward-compatible (CompressionActivity.CompressAfter is optional/trailing).

Security: no externally-influenced input reaches any interpolated SQL — every interpolated value (view, intervals, hour/minute) comes from the internal registry or validated integer/enum-like inputs; parameterized values (job_id, compress_after text, hour, minute) go through $n placeholders with explicit casts.

Minor, non-blocking observation: SetAggregateCompressionPolicySql's converge always re-anchors initial_start to "tomorrow" (night zero) rather than preserving a staged night, but this is the same pattern the existing raw-tier SetCompressionSchedulePhaseSql already uses, and the docstring's reasoning (a freshly-staged policy can't differ from its wanted values before its first run, so it never enters the converge branch) holds under normal operation. Not a new risk class introduced by this PR.

No correctness, parity, or security issues found.

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM — traced the compress_after derivation, the daily band/staging math, the raw-converge exclusion, and the CompressionActivitySql materialization-resolution fix against the actual constants and callers in TimescaleSupport.cs; no correctness, Lite/Darling parity, or security issues found. Full notes in the comment above.

… snapshot the standing aggregates, drop only the ones they created, through LiveCleanupBatch (#1873 residue guard caught the twenty they left)
@erikdarlingdata

Copy link
Copy Markdown
Owner Author

Disposition of the review's one observation (the converge re-anchoring initial_start to night zero): acknowledged, no change. The converge branch is entered only when a job's compress_after, cadence, fixed-schedule flag, hour or minute differs from the shipped values, and a policy this build creates cannot differ before its first run — so a staged night is never re-anchored by it. The one path that does lose staging is the partial-failure one (a start that created some policies and lost its connection stages the remainder from night zero on the next start, so up to two aggregates can share a night, each at its own hour); that is stated on StageAggregateCompressionNights and accepted over a stateful cursor.

The second commit (004d357a) is test-only: the two live tests now snapshot the standing aggregates and drop the ones they created through LiveCleanupBatch, because the #1873 residue guard correctly caught the twenty they left on the fixture. Every test passed on the first run (9,986 total, 0 failed); the collection cleanup was the only red.

{
if (hour < 0 || hour >= 24 || minute < 0 || minute >= 60 || nightOffset < 0)
{
throw new ArgumentOutOfRangeException(nameof(nightOffset), "the band instant must be a valid hour and minute of the day and a non-negative night");

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking nit: this always names nightOffset in the ArgumentOutOfRangeException, even when hour or minute is the value that's actually out of range. Doesn't affect behavior (nothing inspects ParamName here), but worth nameof(hour)/nameof(minute) per-branch if this ever needs to be debugged from a thrown exception message.

@claude

claude Bot commented Sep 18, 2026

Copy link
Copy Markdown

Reviewed the diff (TimescaleSupport.cs, DarlingWorker.cs, Darling.Tests, Darling/README.md, docs/retention-hold-runbook.md).

Scope check: Darling/TimescaleDB-only change (per-CAGG columnar compression + policy). Confirmed Lite has no analogous rollup/materialization objects (its "daily summary" is computed at read time over raw views, not a persisted CAGG) and DuckDB has no comparable user-facing compression-policy concept — its Parquet export path already uses COMPRESSION ZSTD. No Lite/Darling parity gap; touching only Darling/* and the runbook doc is correct here.

Correctness walk:

  • AggregateCompressionTargets is derived from HourlyAggregates/DailyAggregates/BaselineAggregates in declaration order — verified the hoisted DailyAggregates list preserves the exact prior inline-array order (matters for the L2-before-L3 day-grain dependency).
  • StageAggregateCompressionNights: backlog aggregates get consecutive nights largest-first, empty ones all land on night 0 but at distinct hours via AggregateCompressionBandHourFor, so no real scheduling collision — matches the pinned test expectations.
  • SetAggregateCompressionPolicySql parameter order ($1 job id, $2 compress_after text, $3 hour, $4 minute) matches the AddWithValue call order in EnsureAggregateCompressionAsync.
  • IsAggregateCompressionTarget disjointness from CompressionPhaseOrder (raw hypertables) is asserted by test, so the raw converge's exclusion can't accidentally swallow a raw table.
  • The one edge case worth naming but not blocking: on ConvergeCompressionScheduleAsync's narrow (pre-fixed_schedule-column) fallback path, IsAggregateCompressionTarget(hypertable) is checked with no schema qualifier, so a foreign hypertable in another schema that happens to collide by bare name with one of the 20 new aggregate view names would no longer get its cadence retuned there, on an old-enough TimescaleDB to need that fallback. Same risk class the code already accepts elsewhere for CompressionPhaseOrder name collisions — not something I'd hold up the PR for.

Security/perf: no user input flows into the interpolated SQL (all view names/intervals come from the internal compile-time registry); the new per-start catalog reads are small (~20 rows) and failure-isolated per aggregate, consistent with the rest of the file's ensure pattern.

Left one non-blocking inline nit on AggregateCompressionInitialStartSql's exception ParamName.

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM — reviewed the CAGG compression ensure/converge logic, staging math, raw-converge exclusion, test coverage, and confirmed no Lite/Darling parity gap (Lite has no analogous materialized rollups). No substantive correctness, security, or performance issues found; one non-blocking nit left inline.

@erikdarlingdata
erikdarlingdata merged commit b6d60dd into dev Sep 18, 2026
8 checks passed
@erikdarlingdata
erikdarlingdata deleted the fix/3581-lane branch September 18, 2026 17:16
erikdarlingdata added a commit that referenced this pull request Sep 19, 2026
…spliced once

Fifty-nine PRs merged to dev today across the coordinator's lanes and the wave-2
worker's; each lane returned its entry to a buffer instead of touching this file,
so that fifty-plus PRs did not each rebase the same twenty lines. This is the one
splice. Every entry is one line (the archiver's compact() and the pins read them
that way); riders fold into their parent's entry (#3599 under #3590, #3619 under
#3611, #3623 under #3616, #3640 under #3633; #3617 test-only and #3661 re-cut as
#3666 carry none); #3657's entry is in because it MERGED to dev - the twin to main
is what is still pending.

[Unreleased] gains a `### Added` above `### Fixed` (Keep-a-Changelog order) for the
six new capabilities: per-user theme colours (#3606 / #3577 arm B), routed alert
families (#3668 / #3598), the PostgreSQL logging audit tool (#3643 / #3607), the
service-side wait sampler (#3645 / #3604), and the log-event classifier with its
temp-file / autovacuum parser families (#3646 / #3601, #3664 / #3602 #3603). The
other forty-eight are honesty fixes to existing surfaces and append to `### Fixed`
after the wave-1 bullets, in PR-number order.

Thirty-six reference definitions added for the issues the new entries cite and the
index did not yet define; the [Unreleased] group is one ascending run again, which
moves [#3557] into its slot (the one deleted line). Nothing under ## [3.8.0] or
older is touched; the archive script was not run. One editorial touch: the #3585
entry ended in a dangling "Darling" and now reads "Darling only." (the tool exists
only in the Darling MCP host).

tools/changelog/changelog_archive.py verify: all PASS (1420 bold entries, floor
1,329; 1,358 distinct refs resolve; CRLF throughout; 356,539 bytes under the
750 KiB ceiling). ChangelogIndexAndArchiveTests: 5/5 pass via a net10.0 harness.
erikdarlingdata added a commit that referenced this pull request Sep 19, 2026
…spliced once (#3672)

Fifty-nine PRs merged to dev today across the coordinator's lanes and the wave-2
worker's; each lane returned its entry to a buffer instead of touching this file,
so that fifty-plus PRs did not each rebase the same twenty lines. This is the one
splice. Every entry is one line (the archiver's compact() and the pins read them
that way); riders fold into their parent's entry (#3599 under #3590, #3619 under
#3611, #3623 under #3616, #3640 under #3633; #3617 test-only and #3661 re-cut as
#3666 carry none); #3657's entry is in because it MERGED to dev - the twin to main
is what is still pending.

[Unreleased] gains a `### Added` above `### Fixed` (Keep-a-Changelog order) for the
six new capabilities: per-user theme colours (#3606 / #3577 arm B), routed alert
families (#3668 / #3598), the PostgreSQL logging audit tool (#3643 / #3607), the
service-side wait sampler (#3645 / #3604), and the log-event classifier with its
temp-file / autovacuum parser families (#3646 / #3601, #3664 / #3602 #3603). The
other forty-eight are honesty fixes to existing surfaces and append to `### Fixed`
after the wave-1 bullets, in PR-number order.

Thirty-six reference definitions added for the issues the new entries cite and the
index did not yet define; the [Unreleased] group is one ascending run again, which
moves [#3557] into its slot (the one deleted line). Nothing under ## [3.8.0] or
older is touched; the archive script was not run. One editorial touch: the #3585
entry ended in a dangling "Darling" and now reads "Darling only." (the tool exists
only in the Darling MCP host).

tools/changelog/changelog_archive.py verify: all PASS (1420 bold entries, floor
1,329; 1,358 distinct refs resolve; CRLF throughout; 356,539 bytes under the
750 KiB ceiling). ChangelogIndexAndArchiveTests: 5/5 pass via a net10.0 harness.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant