The log said what the spill cost and what the vacuum cost, and the store kept the sentence: two parser families lift bytes and run figures into V130 columns of pg_log_events, and autovacuum health shows per-table run cost beside the catalog (#3602, #3603)

[erikdarlingdata]

The log said what the spill cost and what the vacuum cost; the store kept the sentence and threw away the number

#3646 landed the classified log-event pipeline (V129) and stored log_temp_files and log_autovacuum_min_duration lines as recognised-only events: the family was set, the prose was redacted and kept, and nothing was lifted. So family = temp_file returned "temporary file: path …, size 4294967296" as text, and family = autovacuum returned a fourteen-line report as one message — the figures a DBA needs to sum, rank or alert on were inside a string. This PR adds the two parser families the pipeline was built to receive, and the one rung that gives them columns: V130.

The mechanism

Two classes implementing IPgLogFamilyParser, registered in PgLogEventClassifier.DefaultParsers ahead of PgRecognisedFamilyParser exactly as the seam's doc prescribed, so they take their families' lines out of the generic arm. Their numbers travel on the SAME row as thirteen nullable columns (PgLogEventMetrics, passed through PgLogEvent.From(...) — still the one constructor path, still the one redaction site), not as sibling tables. PgRecognisedFamilyParser now recognises checkpoint alone; PgLogFamilies.Parsed is five and RecognisedOnly one. No stored row is relabelled: a V129-era temp_file or autovacuum row keeps its family and has null metrics.

• PgTempFileEventParser (Per-event temp spill attribution from log_temp_files — counters know a database spilled, not which execution spilled 4 GB at 03:07 #3602) — accepts on ^temporary file: (the recogniser Ingest target PostgreSQL logs as telemetry — errors, spills, autovacuum runs, and connection churn are invisible between counter samples #3601 carried), lifts size N → bytes. The path stays in message (it names a pgsql_tmp file, not a relation, and is redacted like any message); the STATEMENT: companion is fingerprinted and never stored — the pipeline's existing rule, unchanged. Two files from one statement (a multi-batch sort, a parallel hash) are two events with one fingerprint, which is the join back to pg_stat_statements' temp_blks_* and the per-execution grain the issue asked for.
• PgAutovacuumEventParser (Autovacuum per-run cost from log_autovacuum_min_duration — the health view says whether it ran, not what it cost #3603) — accepts on the header (automatic [aggressive ]vacuum[ to prevent wraparound] of table "db.schema.table" / automatic analyze of table "…"), splits the relation ONCE on the first dot (database → database_name, schema.table → relation_name), sets is_analyze, then matches each clause independently by keyword so a clause a version does not write leaves its columns null rather than failing the line. elapsed: N.NN s → duration_ms. All parsing is InvariantCulture (PostgreSQL prints through its own snprintf; there is no locale in the numbers), pinned under de-DE.

The line formats, per version, read from the source (not remembered)

clause	16 / 17 (REL_16_STABLE, REL_17_STABLE — identical)	18 (REL_18_STABLE)	lifted
header	automatic vacuum of table "d.s.t": index scans: N / automatic analyze of table "d.s.t"	same	relation, kind
pages	pages: N removed, N remain, N scanned (P% of total)	…, N eagerly scanned appended	removed, remain
tuples	tuples: N removed, N remain, N are dead but not yet removable	same	removed, remain
buffers	buffer usage: N hits, N misses, N dirtied	N hits, N reads, N dirtied — misses renamed reads	hits, misses/reads → buffer_misses, dirtied
WAL	WAL usage: N records, N full page images, N bytes — vacuum only	…, N buffers full appended, and written on analyze lines for the first time	records, bytes
system	system usage: CPU: user: N.NN s, system: N.NN s, elapsed: N.NN s (pg_rusage.c, unchanged)	same	elapsed → duration_ms
temp file	temporary file: path "%s", size %lu (fd.c, unchanged since 8.3)	same	bytes

PostgreSQL 15's one-line autoanalyze (automatic analyze of table "…" system usage: CPU: …) still yields relation and duration. Not lifted, deliberately: index scans, removable cutoff, relfrozenxid, frozen, visibility map, the per-index lines, I/O timings, the avg read/write rates (derived, and 0.000 on any sub-second run), full page images, buffers full, delay time.

The rung — V130 pg-log-event-metrics

ALTER TABLE collect.pg_log_events
    ADD COLUMN IF NOT EXISTS relation_name text,
    ADD COLUMN IF NOT EXISTS bytes bigint,
    ADD COLUMN IF NOT EXISTS duration_ms bigint,
    ADD COLUMN IF NOT EXISTS pages_removed bigint,
    ADD COLUMN IF NOT EXISTS pages_remaining bigint,
    ADD COLUMN IF NOT EXISTS tuples_removed bigint,
    ADD COLUMN IF NOT EXISTS tuples_remaining bigint,
    ADD COLUMN IF NOT EXISTS buffer_hits bigint,
    ADD COLUMN IF NOT EXISTS buffer_misses bigint,
    ADD COLUMN IF NOT EXISTS buffer_dirtied bigint,
    ADD COLUMN IF NOT EXISTS wal_records bigint,
    ADD COLUMN IF NOT EXISTS wal_bytes bigint,
    ADD COLUMN IF NOT EXISTS is_analyze boolean;

Nullable, no DEFAULT, no backfill (the V80/V81/V101/V114/V121/V127/V128 shape; catalog-only on a compressed hypertable, segmentby unchanged). Added in two places, the V101 rule: V129's CREATE TABLE text gains the same thirteen columns after raw_line_hash for the fresh population — PgSchemaGeneratorTests.EveryPostgresRung_IsIdenticalToTheGeneratedSchema requires that rung to BE the generator's output, and the generator emits the collector's current PayloadColumns — and V130's ALTER carries the store that built the seventeen-column table first. No index (the reads still enter through idx_pg_log_events_time; a (server_id, relation_name, collection_time) index is the same separate rung as V129's family index). StorageVersion 130; the Viewer probe gains sentinel 105 (information_schema.columns … 'pg_log_events' … 'wal_bytes') and the hasPgLogEventMetrics top arm; V129's top-rung claims hand off to PgLogEventMetricsRungTests.

Why columns and not the sibling tables #3601 planned

A pg_temp_files and a pg_autovacuum_runs table would each have needed its own identity column and its own dedupe of the same overlapping tail, its own retention, hypertable, compression policy and CI worker slot, and a join on raw_line_hash for every read that wanted the event beside its numbers. Thirteen nullable columns cost none of that: the row already has the identity, the dedupe, the fingerprint and the retention, and "everything at 03:07" and "what did the 03:07 spill cost" are one query. family says which columns can be non-null. What this trades away — per-family retention and per-family indexing — are the separate rungs V129's doc already reserves. Census unchanged: 28 PG collectors, 70 tables / 71 hypertables, 73/84 workers, 152 tools; the README pins did not move.

The reads

• get_pg_log_events publishes the numbers as nested objects present only where the line carried them: temp_file: { bytes, mb } on a spill, autovacuum: { relation_name, kind, duration_ms, pages_*, tuples_*, buffer_*, wal_*, read_mb, written_mb } on a run, null otherwise — a connection event does not carry thirteen nulls. Inside the run object a null is KEPT and means "this line had no such clause" (an analyze has no pages/tuples; 16/17 print no WAL on an analyze). The description says all of this, names the spill-storm rule as a custom-alert source later, not this tool, and stops saying the two families "gain structured tables in later work".
• get_pg_autovacuum_health gains recent_runs per table — aggregates over the newest 25 runs in the window (runs_counted, history_capped, vacuum_runs/analyze_runs, total/max/avg duration_ms, total_tuples_removed with runs_with_tuples because sums skip nulls, total_read_mb/total_written_mb, total_wal_bytes) and the newest 5 runs in the same per-run shape the events read uses (one spelling, AutovacuumRunShape, shared). Read through a new DarlingPgLogEventReader.GetAutovacuumRunsAsync (family = 'autovacuum' AND relation_name = ANY($4::text[]), ROW_NUMBER() … <= $5 per relation, same dedupe and chunk bound as the events read). Matched on (database, relation), falling back to relation alone for a V129-era row whose database the log did not name. tables_with_run_history and run_history_note say at the top how much of the page the cost half could speak for and — when none — the three reasons this read cannot tell apart. No new parameter; the history rides the tool's window and page.
• get_pg_logging_audit's two facets for log_temp_files / log_autovacuum_min_duration now say SHIPPED and name the read; the pin that required every consumer to say PLANNED asserts exactly those two flipped (load-bearing edit outside the named files — the pin's own doc said it was meant to go red when a consumer landed).

Evidence

• Rig (timescale/timescaledb:2.28.1-pg18, PostgreSQL 18.4, log_temp_files=0 log_autovacuum_min_duration=0 autovacuum_naptime=2s track_io_timing=on, torn down): a 50k-row table, a work_mem=64kB sort and a hash aggregate, an UPDATE/DELETE for dead tuples, 1/0, a unique violation, a lock wait past deadlock_timeout. The self-hosted collector's shipped SQL ran on the target's own log, the classifier ran on what came back, the rows went through the collector's COPY into a store migrated to 130, and both tools read them. Classified: autovacuum=54, checkpoint=3, connection=160, error=15, lock_wait=2, temp_file=3. 0 of 54 autovacuum rows missing duration or buffers. Spills: 11370496 (the sort), 892928 + 122880 (the hash agg's fileset pair, ONE fingerprint) — "temp_file": { "bytes": 11370496, "mb": 10.84 }. The public.orders vacuum: duration_ms 70, pages 0/1702, tuples 10001/40000, buffers 5610/0/4, wal 5475/542577, kind: vacuum; its autoanalyze duration_ms 60, buffers 1985/1/0, wal 8/2347 with pages/tuples null. get_pg_autovacuum_health on a seeded catalog row for the same table: recent_runs { runs_counted 2, vacuum_runs 1, analyze_runs 1, total_duration_ms 130, total_wal_bytes 544924, … }.
• Live gated tests on the same rig (DARLING_TEST_PG / DARLING_TEST_PG_LOG_TARGET): PgLogEventMetricsLivePostgresTests (information_schema agrees with PayloadColumns column-for-column after migration; six fixture events through COPY; both reads with the lifted values; per-relation cap; recent_runs on the table with runs and null on the one without), both PgLogEventsLivePostgresTests (extended: bytes/mb on the spill page, relation/pages/tuples and duration_ms: null on the autovacuum page, no bytes key on the lock-wait page), DarlingPgReadSqlParsesLiveTests (the new AutovacuumRunsSql PREPAREs on 18), DarlingPgAutovacuumVerdictLiveTests — 6/6.
• Executed here via the mactest pattern (net10.0-windows Darling.Tests-named harness, WindowsDesktop entry dropped): 737 tests across the touched pin classes — PgLogEventMetricsParserTests (16/17/18 vacuum + analyze fixtures with the real clause texts; wraparound variant; PG15 one-liner; de-DE culture; two spills with a two-line STATEMENT and the literal-never-stored sweep; every Ingest target PostgreSQL logs as telemetry — errors, spills, autovacuum runs, and connection churn are invisible between counter samples #3601 header shape accepted by the structured parser and refused by the recognised arm; the seam's one From; metrics members ↔ columns name-for-name), PgLogEventMetricsRungTests (top rung; thirteen ADD COLUMN, no DEFAULT/NOT NULL/UPDATE/INDEX/VIEW; each column's type rendered by the generator's own mapping and present in BOTH texts in the collector's order; probe ordinal 105 and arm order; census unchanged; runbook + README), the updated PgLogEventsPipelineTests/PgLogEventsRungTests (handoff), PgSchemaGeneratorTests, DeltaFamilyIntervalCompletionRungTests, DarlingMcpPgLoggingAuditToolsTests, DarlingPgAutovacuumReaderTests, CiClusterWorkerSizingTests, ConsumedTimestampFrameDisciplineTests, McpPageContractTests, McpToolTypeRegistrationTests, ServerPageTabsTests, ViewerPostgresTabsTests, MigrationLadderPins, MigrationDataMovingRungCensusPins, ReadmeDerivedCountPinTests, StoreCopyPhaseTests, StorageCommandTimeoutTests, LiveCleanupConversionRatchetTests, LivePostgresCollectionHygieneTests and the rest of the PostgreSQL's log is read by a classifier now, not by two regexes: errors, connections and lock waits land in pg_log_events off the same tail and RDS API the deadlock and plan readers use, redacted with the plan parser's own patterns (#3601) #3646 harness — the only failures the three D:\darling\… Windows-path assertions. Every touched project builds -c Release -p:EnableWindowsTargeting=true with 0 warnings.

What this does NOT do

• No spill-storm read or alert rule — the issue calls it a natural custom-alert source once data exists; the per-event grain is now there for it.
• No cost-limit / naptime recommendation from the run history — recent_runs is the evidence the health read's recommendations can consume next; nothing here judges it.
• No per-family retention or index — separate rungs, as V129's doc reserves.
• read_mb / written_mb assume the default 8 kB block_size; the block counts beside them are exact.
• avg read/write rate, full page images, buffers full, the per-index lines and the freeze/cutoff clauses stay in message and are not lifted.
• The three Ingest target PostgreSQL logs as telemetry — errors, spills, autovacuum runs, and connection churn are invisible between counter samples #3601 facets in get_pg_logging_audit (lock waits, connections, checkpoints) still say PLANNED although the pipeline ships them — the pipeline's own follow-up, pinned so the flip is deliberate.
• No Viewer panel columns for the new fields (the panel is served by the same table; the probe gate is in place for when it names them).
• CHANGELOG untouched (entry returned to the coordinator).

Closes #3602
Closes #3603