Repository navigation
The twenty continuous aggregates join the compression ladder: compress_after derived from each tier's refresh window, a once-a-day band off the hourly grid, and the backlog staged one aggregate per night, largest first (#3581) - #3611
Conversation
…s_after derived from each tier's refresh window, a once-a-day band off the hourly grid, and the backlog staged one aggregate per night, largest first (#3581) Every raw hypertable has compressed since the archival tier existed; no continuous aggregate ever did, and nothing in source said so. On the largest production store the twenty materializations were 235 GiB of a 415 GiB database, all reading compression_enabled = false — the four Query Store aggregates alone 71.5/55.6/54.7/33.5 GiB against a 25 GiB raw table compressing at ~9x, because an uncompressed rollup of a 9x-compressed source is larger than the source. TimescaleSupport.EnsureAggregateCompressionAsync is an idempotent startup ensure, not a rung: ALTER MATERIALIZED VIEW ... SET (timescaledb.compress, segmentby server_id, orderby bucket DESC) under a catalog check, then add_compression_policy once a day per aggregate, failure-isolated per aggregate, with a #1958-style summary naming every aggregate and its window. compress_after is an expression: each tier's refresh start_offset plus one raw chunk (ChunkIntervalDays) — 2 days for the thirteen hourly-refreshed aggregates, 4 for the seven dailies. Measured on 2.28.1 why the two boundaries must stay apart: a refresh into a compressed region works but decompresses 1,000-row batches per segment and leaves the chunk PARTIAL for the next policy run (8 ms -> 29 ms + 195 ms recompress for one bucket; 1,150 ms -> 1,514 ms + 265 ms for a forced eight-day refresh). The band is the last minute of the heaviest refresh's window (:35, 1,200 s past its start against an 896 s ceiling, the minute before the raw compression band opens), one aggregate per hour from 01:35Z in registry order — an hour apart rather than minutes because distinct starts are not non-overlap (#3185), and because joining #3174's compression band would widen it from 24 to 31 minutes and take the heaviest refresh's window under its ceiling. The backlog is staged in the schedule itself: every policy is created at once with initial_start on the next UTC midnight plus its night, nights assigned largest first by measured materialization size to aggregates with eligible chunks and night zero to everything without — so a fresh store gets all twenty at once, the plan is visible as consecutive next_start dates, and a restart remembers nothing. The raw compression converge now skips these jobs by name — their once-a-day cadence would otherwise read as stale against the hourly tick and the first start after this build would put all twenty on the hourly grid. #1778's activity read resolves an aggregate's chunks through its materialization at the job's own delay instead of reading zero forever. DailyAggregates is hoisted out of the ensure sweep so the creation sweep, the compression registry and the tests read one list.
|
Reviewed the diff (TimescaleSupport.cs, DarlingWorker.cs, the new TimescaleAggregateCompressionTests.cs, and the two docs). Scope notes and what I checked: Lite/Darling parity: not applicable here — TimescaleDB continuous-aggregate compression only exists on Darling's PostgreSQL store (Lite has no TimescaleDB analog, per CONTRIBUTING.md's note that TimescaleDB support is runtime setup outside the shared migration ladder). No parity gap. Correctness, traced through by hand:
Security: no externally-influenced input reaches any interpolated SQL — every interpolated value ( Minor, non-blocking observation: No correctness, parity, or security issues found. |
There was a problem hiding this comment.
LGTM — traced the compress_after derivation, the daily band/staging math, the raw-converge exclusion, and the CompressionActivitySql materialization-resolution fix against the actual constants and callers in TimescaleSupport.cs; no correctness, Lite/Darling parity, or security issues found. Full notes in the comment above.
… snapshot the standing aggregates, drop only the ones they created, through LiveCleanupBatch (#1873 residue guard caught the twenty they left)
|
Disposition of the review's one observation (the converge re-anchoring The second commit ( |
| { | ||
| if (hour < 0 || hour >= 24 || minute < 0 || minute >= 60 || nightOffset < 0) | ||
| { | ||
| throw new ArgumentOutOfRangeException(nameof(nightOffset), "the band instant must be a valid hour and minute of the day and a non-negative night"); |
There was a problem hiding this comment.
Non-blocking nit: this always names nightOffset in the ArgumentOutOfRangeException, even when hour or minute is the value that's actually out of range. Doesn't affect behavior (nothing inspects ParamName here), but worth nameof(hour)/nameof(minute) per-branch if this ever needs to be debugged from a thrown exception message.
|
Reviewed the diff (TimescaleSupport.cs, DarlingWorker.cs, Darling.Tests, Darling/README.md, docs/retention-hold-runbook.md). Scope check: Darling/TimescaleDB-only change (per-CAGG columnar compression + policy). Confirmed Lite has no analogous rollup/materialization objects (its "daily summary" is computed at read time over raw views, not a persisted CAGG) and DuckDB has no comparable user-facing compression-policy concept — its Parquet export path already uses Correctness walk:
Security/perf: no user input flows into the interpolated SQL (all view names/intervals come from the internal compile-time registry); the new per-start catalog reads are small (~20 rows) and failure-isolated per aggregate, consistent with the rest of the file's ensure pattern. Left one non-blocking inline nit on |
There was a problem hiding this comment.
LGTM — reviewed the CAGG compression ensure/converge logic, staging math, raw-converge exclusion, test coverage, and confirmed no Lite/Darling parity gap (Lite has no analogous materialized rollups). No substantive correctness, security, or performance issues found; one non-blocking nit left inline.
…spliced once Fifty-nine PRs merged to dev today across the coordinator's lanes and the wave-2 worker's; each lane returned its entry to a buffer instead of touching this file, so that fifty-plus PRs did not each rebase the same twenty lines. This is the one splice. Every entry is one line (the archiver's compact() and the pins read them that way); riders fold into their parent's entry (#3599 under #3590, #3619 under #3611, #3623 under #3616, #3640 under #3633; #3617 test-only and #3661 re-cut as #3666 carry none); #3657's entry is in because it MERGED to dev - the twin to main is what is still pending. [Unreleased] gains a `### Added` above `### Fixed` (Keep-a-Changelog order) for the six new capabilities: per-user theme colours (#3606 / #3577 arm B), routed alert families (#3668 / #3598), the PostgreSQL logging audit tool (#3643 / #3607), the service-side wait sampler (#3645 / #3604), and the log-event classifier with its temp-file / autovacuum parser families (#3646 / #3601, #3664 / #3602 #3603). The other forty-eight are honesty fixes to existing surfaces and append to `### Fixed` after the wave-1 bullets, in PR-number order. Thirty-six reference definitions added for the issues the new entries cite and the index did not yet define; the [Unreleased] group is one ascending run again, which moves [#3557] into its slot (the one deleted line). Nothing under ## [3.8.0] or older is touched; the archive script was not run. One editorial touch: the #3585 entry ended in a dangling "Darling" and now reads "Darling only." (the tool exists only in the Darling MCP host). tools/changelog/changelog_archive.py verify: all PASS (1420 bold entries, floor 1,329; 1,358 distinct refs resolve; CRLF throughout; 356,539 bytes under the 750 KiB ceiling). ChangelogIndexAndArchiveTests: 5/5 pass via a net10.0 harness.
…spliced once (#3672) Fifty-nine PRs merged to dev today across the coordinator's lanes and the wave-2 worker's; each lane returned its entry to a buffer instead of touching this file, so that fifty-plus PRs did not each rebase the same twenty lines. This is the one splice. Every entry is one line (the archiver's compact() and the pins read them that way); riders fold into their parent's entry (#3599 under #3590, #3619 under #3611, #3623 under #3616, #3640 under #3633; #3617 test-only and #3661 re-cut as #3666 carry none); #3657's entry is in because it MERGED to dev - the twin to main is what is still pending. [Unreleased] gains a `### Added` above `### Fixed` (Keep-a-Changelog order) for the six new capabilities: per-user theme colours (#3606 / #3577 arm B), routed alert families (#3668 / #3598), the PostgreSQL logging audit tool (#3643 / #3607), the service-side wait sampler (#3645 / #3604), and the log-event classifier with its temp-file / autovacuum parser families (#3646 / #3601, #3664 / #3602 #3603). The other forty-eight are honesty fixes to existing surfaces and append to `### Fixed` after the wave-1 bullets, in PR-number order. Thirty-six reference definitions added for the issues the new entries cite and the index did not yet define; the [Unreleased] group is one ascending run again, which moves [#3557] into its slot (the one deleted line). Nothing under ## [3.8.0] or older is touched; the archive script was not run. One editorial touch: the #3585 entry ended in a dangling "Darling" and now reads "Darling only." (the tool exists only in the Darling MCP host). tools/changelog/changelog_archive.py verify: all PASS (1420 bold entries, floor 1,329; 1,358 distinct refs resolve; CRLF throughout; 356,539 bytes under the 750 KiB ceiling). ChangelogIndexAndArchiveTests: 5/5 pass via a net10.0 harness.
Closes nothing by keyword — #3581 is closed by hand at merge, per house convention. PARTIAL against #3581: the daily-tier retention question the issue's comment raises is deliberately untouched (see What this does not do).
The omission, with the numbers
Every raw hypertable in the store has compressed since the archival tier existed (
CompressAfterDays, segmentbyserver_id, the hourly tick, the #3035 phase grid). No continuous aggregate ever did, and nothing inTimescaleSupportsaid so — not a rejected design, an omission. On the largest production store the twenty materializations were 235 GiB of a 415 GiB database, every one readingcompression_enabled = falseand zeropolicy_compressionrows on any materialization. The four Query Store aggregates alone were 71.5 / 55.6 / 54.7 / 33.5 GiB against a rawquery_store_statsof 25 GiB compressing at ~9x — because the hourly grain is only a ~4x row reduction, so an uncompressed rollup of a 9x-compressed source is larger than the source. The 90-day hourly tier was a month into its first fill; uncompressed, the store was on course for a 600–650 GB plateau in mid-November, and at the raw tier's ratio the same plateau is ~200 GB.The rig reproduced the shape in miniature (PG 18.4 / TimescaleDB 2.28.1, six servers × 150 queries × 10-minute snapshots × 25 days): 1,056 MB of raw compressed to 32 MB while its five Query Store aggregates held 110 / 110 / 179 / 148 / 5 MB uncompressed. Once enabled they compressed at 10.1x to 18.6x.
What ships
TimescaleSupport.EnsureAggregateCompressionAsync, called from the worker's TimescaleDB block right afterEnsureContinuousAggregatesAsyncand beforeEnsureRetentionPoliciesAsync. An idempotent startup ensure, not a rung:ALTER MATERIALIZED VIEW collect.<view> SET (timescaledb.compress, timescaledb.compress_segmentby = 'server_id', timescaledb.compress_orderby = 'bucket DESC')gated oncontinuous_aggregates.compression_enabled, thenadd_compression_policy(...)withif_not_exists, one aggregate at a time inside its own try (the ALTER must precede the policy — the rig raisescolumnstore not enabled on continuous aggregatethe other way round). A converge (SetAggregateCompressionPolicySql) moves an existing policy whosecompress_after, cadence, fixed-schedule flag or band instant differs, becauseadd_compression_policyreturns -1 and changes nothing against a policy the store already has — the #1778/#1937 drift, pre-empted rather than filed later. The summary line names every aggregate and its window, the #1958 way.The registry is derived, not hand-listed:
AggregateCompressionTargets=HourlyAggregates∪DailyAggregates∪BaselineAggregates, in creation order.DailyAggregatesis hoisted out of the ensure sweep's inline array for the reasonHourlyAggregateswas hoisted at #3012 — the creation sweep, the compression registry and the tests now read one list. Thesegmentbycolumn and theorderbyalias are recovered from each shipped CREATE (RefreshGroupingTermsFor,AggregateBucketColumnFor) and asserted per definition, so they are properties of the registry rather than assumptions.compress_after, derivedWhy a margin at all, measured: a refresh that reaches into a compressed materialization chunk does not fail on 2.28.1. It decompresses every compressed batch overlapping the buckets it re-materializes (1,000 rows per
server_idsegment — a six-row backdated write into one hourly bucket staged 6,006 rows), rewrites them into the chunk's heap side, and leaves the chunk PARTIAL (catalog status 9) until the next compression run recompresses it. Single-bucket refresh: 8 ms uncompressed → 29 ms compressed + a 195 ms recompression on the next policy pass. Forced eight-day refresh: 1,150 ms → 1,514 ms, leaving 180,000 heap rows for a 265 ms recompression. So an overlapping window would not break anything; it would make the newest chunk a permanent decompress-and-recompress churn, once an hour, forever.Why one raw chunk rather than one bucket: the refresh window reaches back
start_offsetand aligns its start down to a bucket boundary, so disjointness needscompress_after ≥ start_offset + bucket— 1 day 1 hour hourly, 4 days daily. The daily tier's bucket is a day so the floor is a day there already; the hourly tier's is an hour, and an hour is the wrong unit for the gap — every other boundary in the store moves in days (chunks close at UTC midnight, raw eligibility flips at UTC midnight), and a sub-day margin puts the refresh's aligned start and the compression boundary within one scheduling jitter of each other. One raw chunk on every tier lands the daily tier on its own floor and puts a whole day between the two on the hourly tier: 2 and 4, the values the issue was opened for, reached as an expression. Pinned per tier with the tier's own bucket, and pinned against every retained aggregate's horizon (compress_after < drop_after, and2 × compress_after < drop_afterso the short interval-identity tiers still spend most of their life compressed at day-width chunks).The band, and why not #3174's compression band
The hourly grid tiles the hour with no minute unassigned, but two stretches carry no start: the guard after the light band, and the heaviest refresh's window past its own start. The compression grid deliberately leaves the tail of that window unused for hourly policies ("the other 6 minutes of the window are past the refresh and are left on the table"), because recovering them for jobs that run every hour would size a band against a still-moving ceiling. A once-a-day job is a different trade.
AggregateCompressionBandMinute = HeaviestRefreshStartMinute + HeaviestRefreshWindowMinutes − 1 = :35— the window's last minute, derived from geometry rather than from the ceiling constant (so re-taking the ceiling per #3182/#3188 moves no job). It is 1,200 s past the heaviest refresh's start against an 896 s recorded ceiling (the first clear minute, :30, would be 900 s — a four-second margin), past every light refresh's 226.8 s ceiling measured from its own start, on no hourly refresh's minute, on no raw compression minute, and the minute immediately beforeCompressionPhaseMinutes[0]. All of that is asserted against the shipped grid's output, not a copy of its rule. Clearance to the next refresh start is 25 minutes.One aggregate per hour, from 01:35Z, in registry order (
AggregateCompressionBandHourFor). An hour apart rather than "a few minutes apart" for the reason #3185 recorded: distinct starts are a lock guarantee, not an overlap guarantee — a run longer than the step overlaps its successor for the rest of its run, and these runs are minutes (a day's chunk of the largest aggregate is roughly the raw table's, which #3112 measured at 552 s). The hour also has only six minutes past every ceiling and on no grid start, so a twenty-member minute-stagger had nowhere to go. One per hour makes "no two aggregates decompress and recompress at once" hold by construction, and keeps the family inside the managed store's background-worker headroom (at most one of these jobs runs at a time; the+ 2overHypertableCountis not re-derived). Hour 1 is the hour after the one that carries the raw tier's midnight rewrite (#3112's midnight band,query_store_statsat :42 for 552 s); aggregate chunks become eligible at the same UTC midnight, so it is the earliest hour that both sees the new eligibility and is clear of that burst. Twenty-three hours are available; a twenty-fourth member is red rather than wrapped onto midnight.Why not the hourly compression band: its width is
HypertableCount / CompressionPhaseMaxPerMinute, and every minute it takes comes out ofHeaviestRefreshWindowMinutes. Twenty more members would widen it from 24 minutes to 31 and shrink the window from 21 to 14 — 840 s against the 896 s ceiling, whichTimescaleSupportTestsholds red.Lock geometry, measured on the rig with
pg_locksmid-run:compress_chunkholdsAccessShareLockon the materialization hypertable and escalates only on the chunk it rewrites (ShareLock→ExclusiveLock→AccessExclusiveLockat the swap, all three observed in one run). So a compression run cannot block the aggregate's own refresh, which writes the newest chunk — the one the margin keeps out of the eligible set. What it can hold up is a reader of exactly the chunk being swapped, for the swap, which is the cost the raw tier pays today. The daily refreshes (finish-to-start, drifting) and the nightly purge (_nextPurgeUtc, anchored to service start) run at instants no fixed band can avoid by construction; the purge is chunk drops and the refreshes are read-only against these chunks, so neither is a lock hazard, and the comment says so rather than claiming a schedule that dodges them.The staging
A compression policy's first run compresses every chunk older than
compress_after, one chunk per transaction, in one pass — on an existing store that is the whole aggregate minus its newest two or four days, and on the largest store that is the 235 GiB event. Every run after the first finds only what aged in since yesterday. So the staging is about first runs only, and it is carried by the schedule rather than by state: every policy is created at once, andinitial_startis the next UTC midnight plus a night offset — nights assigned largest first by measured materialization size to aggregates with eligible chunks (0, 1, 2, …), and night zero to every aggregate with nothing eligible. A fresh store therefore gets all twenty policies at once with no special case; on a backlog store the plan is visible intimescaledb_information.jobsas consecutivenext_startdates, each aggregate at its own hour; a restart remembers nothing and re-adds nothing.StageAggregateCompressionNightsis pure and pinned (largest first; eligible-chunk count does not order — bytes do; ties break on registry order; a fresh store is all night zero).The one thing this shape does not do is re-plan around a partial failure: a start that created ten of twenty policies and lost its connection stages the remaining ten from night zero on the next start, so up to two aggregates can share a night on that path, each still at its own hour. Stated in the comment and accepted over a stateful cursor.
On the rig, the "night 0" run for the largest aggregate compressed three 10-day chunks (137 MB → 10 MB) in 445 ms and left the newest chunk alone.
What the rig measured
QueryStoreTrendRouting.BuildRollupTrendSql's rollup arm overquery_store_stats_corrected_hourly, 30-day window, one server)Index Scan Backwardon(server_id, bucket)per chunkCustom Scan (ColumnarScan)per compressed chunk withIndex Cond: (server_id = 3)and the bucket range as a vectorized filter over batch min/max metadata; the uncompressed newest chunk keeps its index scanquery_store_stats_corrected_hourlymaterializationThe
segmentby server_idmatches the read shape exactly: the compressed chunks are read by segment with the index condition onserver_idalone.Reader plan before (EXPLAIN ANALYZE, BUFFERS, rollup arm of the trend read)
Reader plan after (three chunks compressed, newest uncompressed)
Two load-bearing edits outside the new section
ConvergeCompressionScheduleAsyncskips the family. An aggregate's compression job reports the aggregate's user view as its hypertable (collect/<view>— measured, the same resolutionContinuousAggregateRefreshStateSqldocuments for refresh jobs), so it lands in the raw converge's unscopedproc_name LIKE '%compression%'read, where a cadence other than one hour is the staleness test (#1778). Without the exclusion the first start after this build would retune every one of these once-a-day jobs to the hourly tick and put all twenty on the hourly grid — the placement the ruling excluded. Excluded by name on the narrow read (no schema column) and by name-in-collecton the wide one, so a foreign hypertable elsewhere that happens to share an aggregate's name keeps #1778's converge exactly as before. The live test asserts the family stays on the daily cadence after the raw converge runs.CompressionActivitySqlresolves the materialization.timescaledb_information.chunksknows an aggregate only by its_materialized_hypertable_N, so #1778's eligible-chunk count keyed on the job's own name read zero forever for every aggregate job — the #3582 shape in another reader. It nowLEFT JOINscontinuous_aggregatesto count on the materialization when the job is an aggregate's, at the job's owncompress_afterfrom its config rather than the raw constant, and emits that delay so the log line says "2d" / "4d" / "every 1 day" for these jobs instead of the raw tier's values (#1958's rule).CompressionActivitygains an optional trailingCompressAfter, so the five-argument shape every caller and test constructs is unchanged. The #3112 clearance watch deliberately still reads these jobs as foreign — its overrun text reasons from the raw chunk close, and misdescribing an aggregate run would be worse than not judging it; a watch over the daily band's own clearance is a follow-up, said so onAssignedPhaseMinute.A production fact the issue and the rig disagree on, flagged rather than assumed
The issue's staging note describes the backlog as "~60 1-day chunks" per aggregate. On a fresh 2.28.1 store the materialization chunk interval is ten times the root raw hypertable's — measured: a 1-day raw table yields 10-day materialization chunks on every aggregate built over it, hierarchical ones included (a 7-day raw yields 70-day, a 1-hour raw 10-hour). A store whose raw tables were created at
ChunkIntervalDays = 1therefore has 10-day materialization chunks, not 1-day. The design here is correct at any width — a chunk compresses only when its whole range is pastcompress_after, so the separation argument does not depend on it — but the shape of the work does: at 10-day chunks each aggregate compresses one 10-day chunk every tenth night rather than one day's chunk nightly, a 10-day chunk of the 7-day interval tier lives seventeen days and is compressed for the last five, and the 90-day tiers hold up to 100 days. The ensure's comment states both cases. Verifying the width on the store is one read the coordinator can fire —SELECT ca.view_name, c.range_end - c.range_start, count(*) FROM timescaledb_information.chunks c JOIN timescaledb_information.continuous_aggregates ca ON c.hypertable_name = ca.materialization_hypertable_name WHERE ca.view_schema = 'collect' GROUP BY 1, 2— and narrowing new materialization chunks to a day (set_chunk_time_intervalon the materialization; existing chunks keep their width) is a separate decision this PR does not take, reported in the lane report.Tests
TimescaleAggregateCompressionTests(new, in thelive-postgrescollection). Ungated: the derivation (margin = one raw chunk; each tier'scompress_after= its offset + margin = 2 / 4 days; ≥ offset + bucket per tier; the literal renderer refuses non-day spans); the registry (20 = 6 + 7 + 7, once each, tier by source list,bucketalias andserver_idgroup key recovered from every CREATE, the parse controlled with a differently-aliased CREATE and two refusals); every retained aggregate compressing inside its horizon, walked overRetentionPolicies(14 tiers, control asserted); the band (:35 as geometry, on no refresh minute, on no compression minute, the minute before the raw band, 1,200 s past the heaviest start > 896, past every light ceiling, 25 minutes of clearance, hours 1..20 distinct and inside the day, both throws); the statements (tier window, daily cadence,if_not_exists, the UTC anchor at hour/night, never the raw tier's values; the converge's bound job id,jsonb_set, fixed schedule, noscheduled; the state read's either-identity join and both delays; the activity read's materialization resolution); the staging on a synthetic store (largest first one per night, bytes not chunk count, night zero for no backlog, a fresh store all night zero, registry-order tie-break); the exclusion predicate over all twenty and none of the raw hypertables. Gated onDARLING_TEST_PG: the ensure against the fixture — every aggregatecompression_enabled, exactly one compression job each at its tier's window on the daily cadence, fixed schedule, its hour and :35 read back from the catalog; a settled second pass adds and converges nothing and says so; the raw converge leaves all twenty on the daily cadence; the activity read sees them at their own delay — and a second live test that drifts one policy onto the raw tier's values and asserts the converge moves it and then settles.Tests cannot execute on the authoring machine (net10.0-windows); CI is their first execution. Every ungated assertion was mirrored line-for-line in a throwaway console against the built assembly and holds; the two live flows were run against the rig on a fresh database through the exact call sequence the tests use (
MigrateAsync→TryEnableAsync→ConvertToHypertablesAsync→ConvergeContinuousAggregateRefreshAsync→EnsureContinuousAggregatesAsync→ the ensure, twice → the raw converge → the drift and re-converge), with every read-back matching.Storage, Service, Viewer and Darling.Tests build at 0 warnings with
-c Release -p:EnableWindowsTargeting=true.What this does NOT do
DarlingStoreMetricsReader/DarlingMcpStoreMetricsTools/StoreSelfMetricsare get_store_metrics' inventory is blind to 57% of the store it reports on: continuous aggregates and un-enumerated plain tables never appear, and the total is never reconciled #3582's lane and untouched; the new jobs appear there aspolicy_compression <view> [id]through the existing label.Docs:
Darling/README.md's TimescaleDB section gains the aggregate-compression bullet;docs/retention-hold-runbook.mdgains a section naming the four job families an operator meets intimescaledb_information.jobs, including the new one and what its stagednext_startdates mean.CHANGELOG.mddeliberately not touched; the entry is in the lane report.