Skip to content

Job history and default trace watermark fallback reads the server's whole slice when nothing happened in 6 hours #5515

Closed

Description

@erikdarlingdata

Claude posting for Erik Darling.

Found by the #5508 audit. The numbers are from its step 1 (comment 6051350950): pg_stat_statements over 13.2 days on a store with 43 servers at schema V165. Line numbers are at dev 25f68ae.

What runs

The watermark probe first looks back 6 hours. When it finds no row, BuildServerWatermarkSql(bounded: false) (Darling/PerformanceMonitor.Darling.Service/DarlingCollectorRunner.cs:4745 and :4908) reads with only server_id in the filter. It reads MAX(run_datetime) from collect.job_history and MAX(event_time) from collect.default_trace_events.

Numbers

  • job_history: 14,728 calls at 5,426 blocks each, 79.9 M blocks.
  • default_trace_events: 15,215 calls at 3,758 blocks each, 57.2 M blocks.
  • 3,941 s of execution together.

Why it matters

This is the fallback, yet it ran about 30,000 times. A quiet server, with no job runs or trace events in 6 hours, pays it every cycle. job_history has idx_job_history_server_run (server_id, run_datetime DESC) and still reads 5,426 blocks a call. The likely cause is compressed chunks the MAX has to open. That is inferred, not observed.

Fix direction

Keep the watermark at write time, so the unbounded probe runs only when no watermark is stored. #4197 did this for the ring-buffer collectors. Code only.

Activity

  1. added
    in-progressActively being worked by a local session or its agents (PR open or in flight)
    on Oct 8, 2026
  2. erikdarlingdata commented on Oct 8, 2026

    @erikdarlingdata
    OwnerAuthor

    Fixed by #5532, merged to dev. It ships in the next release. Claude posting for Erik Darling.

  3. removed
    in-progressActively being worked by a local session or its agents (PR open or in flight)
    on Oct 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions