Skip to content

Derive the insert-autovacuum tuning from the catalog, not a hand list - #2406

Merged
erikdarlingdata merged 2 commits into
devfrom
fix/2405-derive-hypertable-insert-tuning
Aug 21, 2026
Merged

erikdarlingdata merged 2 commits into
devfrom
fix/2405-derive-hypertable-insert-tuning

Conversation

@erikdarlingdata

Copy link
Copy Markdown
Owner

Measured on the dogfood store:

hypertables WITH the insert tuning:   4
hypertables WITHOUT it:              47

with untuned tables larger than tuned ones:

untuned size
perfmon_stats 5330 MB
spinlock_stats 3707 MB
wait_stats 2773 MB
index_object_stats 1907 MB
query_snapshots 1259 MB
file_io_stats 1227 MB

Every one is a pure-insert collector fact table, structurally identical to the four that were covered. The four were not selected on merit — they are whichever tables had been the subject of an EXPLAIN investigation.

This file already recorded the mechanism, about a table it had missed once before:

It was simply missed when the PostgreSQL collectors landed, since this list is hand-maintained rather than derived from the catalog.

A list of which tables are append-heavy describes a property every collector target has by construction. It should not be a list.

The sweep resolves hypertables lacking the option from timescaledb_information.hypertables and ALTERs those, per-table failure-isolated like the statements above it. Idempotence is in the predicate, so a re-run selects nothing rather than re-ALTERing 51 tables every service start.

The four literals stay. A plain-PostgreSQL store has no TimescaleDB catalog to read, and those four are the measured floor for that case. They now share a single const with the sweep — two spellings would tune a store two different ways depending on which path reached a table first, and because the sweep's predicate matches on insert_scale_factor, it would treat the other spelling as already-tuned and make the divergence permanent and invisible.

Scoped to hypertables deliberately. Insert-driven autovacuum governs the visibility map on append-only data. A plain table whose churn is DELETEs needs autovacuum_vacuum_scale_factor instead — that is #2404 for query_plan_dim. Sweeping everything in collect would silently apply the inert knob to the tables that need the other one.

The 10000 threshold is what keeps this safe on the many small hypertables: the scale factor is not consulted until that many inserts accumulate.

…#2405)

The tuning reached 4 hypertables out of 51 on the dogfood store, and the
untuned set included tables larger than the tuned ones -- perfmon_stats at
5.3 GB, spinlock_stats at 3.7 GB, wait_stats at 2.8 GB, index_object_stats
at 1.9 GB. All are pure-insert collector fact tables, structurally
identical to the four that were covered.

The four were not chosen; they are whichever tables had been the subject of
an EXPLAIN investigation. This file already recorded the mechanism, about a
table it had missed once before: "it was simply missed when the PostgreSQL
collectors landed, since this list is hand-maintained rather than derived
from the catalog." A list of which tables are append-heavy is a list
describing a property EVERY collector target has by construction, so it
should not be a list at all.

The sweep resolves hypertables lacking the option from
timescaledb_information.hypertables and ALTERs those, per-table
failure-isolated like the statements above it. Idempotence lives in the
predicate, so a re-run selects nothing rather than re-ALTERing 51 tables.

The four literals stay: a plain-PostgreSQL store has no TimescaleDB catalog
to read, and they are the measured floor for that case. They now share one
const with the sweep, because two spellings would tune a store two ways
depending on which path reached a table first -- and the sweep's predicate
matches on insert_scale_factor, so it would consider the other spelling
already tuned and make the divergence permanent.

Scoped to hypertables deliberately. Insert-driven autovacuum governs the
visibility map on append-only data; a plain table whose churn is DELETEs
needs autovacuum_vacuum_scale_factor instead (#2404, query_plan_dim).
Sweeping every table in collect would silently apply the inert knob.

Refs #2405

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@claude

claude Bot commented Aug 21, 2026

Copy link
Copy Markdown

Reviewed. This is Darling/PostgreSQL-only (PgTableTuning.cs, PgTableTuningTests.cs) — Lite has no TimescaleDB/PostgreSQL equivalent, so no parity gap here.

Went through the sweep logic in detail:

  • Injection safety: the target identifier is built server-side via format('%I.%I', h.hypertable_schema, h.hypertable_name), which properly escapes/quotes on the Postgres side before it's string-interpolated into the ALTER TABLE in C#. Source is the TimescaleDB catalog (timescaledb_information.hypertables joined to pg_class/pg_namespace), not user input — same trust model already used elsewhere in this file (Statements) and in StoreSelfMetrics.cs.
  • Idempotence: correctly lives in the predicate (NOT LIKE '%insert_scale_factor%' against reloptions), not a guard at the call site, so a re-run selects nothing once a store converges. COALESCE(..., '') correctly handles the common case of reloptions IS NULL on an untuned table.
  • pg_class/pg_namespace join: filtering on c.relname = h.hypertable_name before also constraining n.nspname = h.hypertable_schema looks like it could be ambiguous, but (relnamespace, relname) is unique in pg_class, so the combined join always resolves to exactly the intended row — no correctness issue.
  • Failure isolation: both the catalog SELECT (no TimescaleDB → catch → no-op, matches the "literal statements already covered a plain-PostgreSQL store" reasoning) and the per-table ALTER loop are isolated consistently with the existing ApplyAsync pattern.
  • Shared constant: InsertTuningOptions is genuinely shared between the four literals and the sweep's ALTER, and the test (TheLiteralStatementsAndTheDerivedSweep_ShareOneSpellingOfTheOptions) pins that they can't drift apart — good, since the sweep's own idempotence predicate keys off that exact spelling.
  • Ordering: sweep runs after the TimescaleDB block and reuses the existing tuning connection, so it never runs before hypertables exist.

Nothing blocking. No missing-index-DMV suggestions per project convention (not applicable here anyway). Didn't flag the missing AS on the new query's table aliases (FROM timescaledb_information.hypertables h, JOIN pg_class c) since StoreSelfMetrics.cs already establishes that same no-AS style for this exact view — CONTRIBUTING's AS-required rule is scoped to the T-SQL section, and this is embedded PostgreSQL SQL.

Conflict was #2404 adding the query_plan_dim dead-tuple ALTER into the same
array this branch rewrote to share a const. Both are kept: the four INSERT
literals go through InsertTuningOptions, and query_plan_dim's dead-tuple
line stays a literal -- it deliberately does not share that const, because
it is the other knob for the other failure.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Comment on lines +143 to +148
public const string UntunedHypertablesSql = @"
SELECT format('%I.%I', h.hypertable_schema, h.hypertable_name)
FROM timescaledb_information.hypertables h
JOIN pg_class c ON c.relname = h.hypertable_name
JOIN pg_namespace n ON n.oid = c.relnamespace AND n.nspname = h.hypertable_schema
WHERE COALESCE(array_to_string(c.reloptions, ','), '') NOT LIKE '%insert_scale_factor%'";

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor consistency nit: resolving a hypertable's pg_class row via a manual relname/nspname join is more fragile than the idiom already established two files over — StoreSelfMetrics.cs's HypertableInsertSql resolves the same timescaledb_information.hypertables row with format('%I.%I', h.hypertable_schema, h.hypertable_name)::regclass. Reusing that cast here would be a single join instead of two:

SELECT format('%I.%I', h.hypertable_schema, h.hypertable_name)
FROM timescaledb_information.hypertables h
JOIN pg_class c ON c.oid = format('%I.%I', h.hypertable_schema, h.hypertable_name)::regclass
WHERE COALESCE(array_to_string(c.reloptions, ','), '') NOT LIKE '%insert_scale_factor%'

Not a correctness bug as written (the pg_namespace join correctly disambiguates same-named relations across schemas), just an unnecessary reinvention of a pattern this codebase already has a name for.

Comment on lines +191 to +197
catch (Exception ex) when (ex is not OperationCanceledException)
{
/* No TimescaleDB (a plain-PostgreSQL store) or an unreadable catalog — the literal statements above
already ran, so this is a no-op rather than a failure. */
logger?.LogDebug("Hypertable insert-tuning sweep skipped: {Message}", ex.Message);
return 0;
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This catch conflates two very different failure modes at the same LogDebug level: "plain-PostgreSQL store, no timescaledb_information catalog" (expected, high-frequency, fine to bury at Debug) and "TimescaleDB store where the catalog read itself broke" — e.g. a permissions change on pg_namespace/pg_class, or a future edit to UntunedHypertablesSql that introduces a real SQL error. The second case is a genuine bug on a TimescaleDB store and would silently degrade to "untuned forever" with no operator-visible signal, unlike the per-table ALTER failures a few lines below which are surfaced at LogWarning. Worth at least logging ex (not just ex.Message) at Debug, or distinguishing "relation timescaledb_information.hypertables does not exist" from other exceptions and warning on the latter.

@claude

claude Bot commented Aug 21, 2026

Copy link
Copy Markdown

Reviewed the diff (Darling/PerformanceMonitor.Darling.Storage/PgTableTuning.cs + tests).

Summary: Solid change, low risk. Replaces the hand-maintained 4-table literal list with a catalog-derived sweep over all TimescaleDB hypertables, keeping the four literals as the plain-PostgreSQL floor and sharing one InsertTuningOptions const between both paths so the two can't drift into different spellings that the idempotence predicate would then treat as already-tuned. Per-table failure isolation matches the existing ApplyAsync pattern, the identifier used in the dynamic ALTER is built with Postgres's own format('%I.%I', ...) (not user input, no injection concern), and query_plan_dim correctly stays out of the sweep since it isn't a hypertable. Tests pin the SQL shape and the shared-spelling invariant.

Parity: This is Darling/PostgreSQL-only functionality (TimescaleDB autovacuum tuning) with no Lite equivalent — Lite uses embedded DuckDB, so no counterpart change is needed there.

Left two non-blocking inline notes:

  1. The untuned-hypertable query resolves pg_class via a manual relname/nspname join rather than the format('%I.%I', ...)::regclass idiom already used for the same lookup in StoreSelfMetrics.cs — functionally correct, just an avoidable reinvention.
  2. The catalog-read catch logs all exceptions (not just "no TimescaleDB") at LogDebug, which would silently swallow a genuine bug on a TimescaleDB store rather than surfacing it the way the per-table ALTER failures do at LogWarning.

@erikdarlingdata
erikdarlingdata merged commit 2f8e65e into dev Aug 21, 2026
6 checks passed
@erikdarlingdata
erikdarlingdata deleted the fix/2405-derive-hypertable-insert-tuning branch August 21, 2026 11:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant