Repository navigation
Derive the insert-autovacuum tuning from the catalog, not a hand list - #2406
Conversation
…#2405) The tuning reached 4 hypertables out of 51 on the dogfood store, and the untuned set included tables larger than the tuned ones -- perfmon_stats at 5.3 GB, spinlock_stats at 3.7 GB, wait_stats at 2.8 GB, index_object_stats at 1.9 GB. All are pure-insert collector fact tables, structurally identical to the four that were covered. The four were not chosen; they are whichever tables had been the subject of an EXPLAIN investigation. This file already recorded the mechanism, about a table it had missed once before: "it was simply missed when the PostgreSQL collectors landed, since this list is hand-maintained rather than derived from the catalog." A list of which tables are append-heavy is a list describing a property EVERY collector target has by construction, so it should not be a list at all. The sweep resolves hypertables lacking the option from timescaledb_information.hypertables and ALTERs those, per-table failure-isolated like the statements above it. Idempotence lives in the predicate, so a re-run selects nothing rather than re-ALTERing 51 tables. The four literals stay: a plain-PostgreSQL store has no TimescaleDB catalog to read, and they are the measured floor for that case. They now share one const with the sweep, because two spellings would tune a store two ways depending on which path reached a table first -- and the sweep's predicate matches on insert_scale_factor, so it would consider the other spelling already tuned and make the divergence permanent. Scoped to hypertables deliberately. Insert-driven autovacuum governs the visibility map on append-only data; a plain table whose churn is DELETEs needs autovacuum_vacuum_scale_factor instead (#2404, query_plan_dim). Sweeping every table in collect would silently apply the inert knob. Refs #2405 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Reviewed. This is Darling/PostgreSQL-only ( Went through the sweep logic in detail:
Nothing blocking. No missing-index-DMV suggestions per project convention (not applicable here anyway). Didn't flag the missing |
Conflict was #2404 adding the query_plan_dim dead-tuple ALTER into the same array this branch rewrote to share a const. Both are kept: the four INSERT literals go through InsertTuningOptions, and query_plan_dim's dead-tuple line stays a literal -- it deliberately does not share that const, because it is the other knob for the other failure. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
| public const string UntunedHypertablesSql = @" | ||
| SELECT format('%I.%I', h.hypertable_schema, h.hypertable_name) | ||
| FROM timescaledb_information.hypertables h | ||
| JOIN pg_class c ON c.relname = h.hypertable_name | ||
| JOIN pg_namespace n ON n.oid = c.relnamespace AND n.nspname = h.hypertable_schema | ||
| WHERE COALESCE(array_to_string(c.reloptions, ','), '') NOT LIKE '%insert_scale_factor%'"; |
There was a problem hiding this comment.
Minor consistency nit: resolving a hypertable's pg_class row via a manual relname/nspname join is more fragile than the idiom already established two files over — StoreSelfMetrics.cs's HypertableInsertSql resolves the same timescaledb_information.hypertables row with format('%I.%I', h.hypertable_schema, h.hypertable_name)::regclass. Reusing that cast here would be a single join instead of two:
SELECT format('%I.%I', h.hypertable_schema, h.hypertable_name)
FROM timescaledb_information.hypertables h
JOIN pg_class c ON c.oid = format('%I.%I', h.hypertable_schema, h.hypertable_name)::regclass
WHERE COALESCE(array_to_string(c.reloptions, ','), '') NOT LIKE '%insert_scale_factor%'Not a correctness bug as written (the pg_namespace join correctly disambiguates same-named relations across schemas), just an unnecessary reinvention of a pattern this codebase already has a name for.
| catch (Exception ex) when (ex is not OperationCanceledException) | ||
| { | ||
| /* No TimescaleDB (a plain-PostgreSQL store) or an unreadable catalog — the literal statements above | ||
| already ran, so this is a no-op rather than a failure. */ | ||
| logger?.LogDebug("Hypertable insert-tuning sweep skipped: {Message}", ex.Message); | ||
| return 0; | ||
| } |
There was a problem hiding this comment.
This catch conflates two very different failure modes at the same LogDebug level: "plain-PostgreSQL store, no timescaledb_information catalog" (expected, high-frequency, fine to bury at Debug) and "TimescaleDB store where the catalog read itself broke" — e.g. a permissions change on pg_namespace/pg_class, or a future edit to UntunedHypertablesSql that introduces a real SQL error. The second case is a genuine bug on a TimescaleDB store and would silently degrade to "untuned forever" with no operator-visible signal, unlike the per-table ALTER failures a few lines below which are surfaced at LogWarning. Worth at least logging ex (not just ex.Message) at Debug, or distinguishing "relation timescaledb_information.hypertables does not exist" from other exceptions and warning on the latter.
|
Reviewed the diff ( Summary: Solid change, low risk. Replaces the hand-maintained 4-table literal list with a catalog-derived sweep over all TimescaleDB hypertables, keeping the four literals as the plain-PostgreSQL floor and sharing one Parity: This is Darling/PostgreSQL-only functionality (TimescaleDB autovacuum tuning) with no Lite equivalent — Lite uses embedded DuckDB, so no counterpart change is needed there. Left two non-blocking inline notes:
|
Measured on the dogfood store:
with untuned tables larger than tuned ones:
Every one is a pure-insert collector fact table, structurally identical to the four that were covered. The four were not selected on merit — they are whichever tables had been the subject of an EXPLAIN investigation.
This file already recorded the mechanism, about a table it had missed once before:
A list of which tables are append-heavy describes a property every collector target has by construction. It should not be a list.
The sweep resolves hypertables lacking the option from
timescaledb_information.hypertablesand ALTERs those, per-table failure-isolated like the statements above it. Idempotence is in the predicate, so a re-run selects nothing rather than re-ALTERing 51 tables every service start.The four literals stay. A plain-PostgreSQL store has no TimescaleDB catalog to read, and those four are the measured floor for that case. They now share a single
constwith the sweep — two spellings would tune a store two different ways depending on which path reached a table first, and because the sweep's predicate matches oninsert_scale_factor, it would treat the other spelling as already-tuned and make the divergence permanent and invisible.Scoped to hypertables deliberately. Insert-driven autovacuum governs the visibility map on append-only data. A plain table whose churn is DELETEs needs
autovacuum_vacuum_scale_factorinstead — that is #2404 forquery_plan_dim. Sweeping everything incollectwould silently apply the inert knob to the tables that need the other one.The
10000threshold is what keeps this safe on the many small hypertables: the scale factor is not consulted until that many inserts accumulate.