Skip to content

Field stores cannot tell us which query is slow: preload pg_stat_statements in the managed store and surface it by role #3899

Closed

Description

@erikdarlingdata

Summary

When a field user reports "the web viewer / MCP tools are slow", nothing in the product can say which query is slow. The managed store:

  • loads only timescaledb in shared_preload_libraries,
  • has log_min_duration_statement = -1,
  • and has no pg_stat_statements.

So the store-log sweep (#3021) has no slow-statement lines to classify, and get_store_metrics has no per-statement view.

Diagnosing the current slowness reports (#3895, #3896, #3897, #3898) needed the store owner's credential and a temporary per-role log_min_duration_statement = 0 to attribute time per tool. A field operator cannot do that, and should not have to.

What is already there

The bundled runtime ships the module. On DARLING01 (PostgreSQL 18.4, nightly 458), pg_stat_statements.dll and its control file are present, and pg_available_extensions lists pg_stat_statements 1.12, not installed.

Proposal

  1. Preload and create it in the managed store. Add pg_stat_statements to shared_preload_libraries in the managed conf, and CREATE EXTENSION IF NOT EXISTS pg_stat_statements in the darling database.
    • The preload change needs a PostgreSQL restart, so it should ride a path that already restarts the store (install/upgrade), with the extension created on the next start.
    • The product's statements are parameterized, so the normalized query text carries no collected data.
  2. Surface it. An MCP/web read (for example get_store_query_stats): top statements by total and mean time, calls, rows and shared blocks, split by role. The role split is the useful axis: viewer is the web viewer, mcp is MCP tools, the owner is collection and maintenance, so "the web viewer is slow" becomes a ranked list in one call. A reset stamp from pg_stat_statements_info belongs beside it, so a reader knows the window.
  3. Log slow reader statements by default. For example, ALTER ROLE viewer SET log_min_duration_statement = '5s' and the same for mcp, set during role provisioning, plus a slow_statement class in the store-log sweep so those lines are counted and retained with text. This is cheap, needs no restart, and catches the statements that are approaching the 15 s statement_timeout before they start failing.
  4. Document the BYO equivalent for stores the product does not manage.

Why now

#3895 and #3896 are both "invisible at our scale" defects that were only found by measuring. A reporting user with this in place could send us the answer instead of a symptom.

Activity

  1. erikdarlingdata commented on Sep 22, 2026

    @erikdarlingdata
    OwnerAuthor

    Confirmed on all three production stores (nightly 458, 2026-09-22): shared_preload_libraries = timescaledb only; pg_stat_statements 1.12 listed in pg_available_extensions, not installed. It blocked a real production attribution today — per-statement calls / mean / max / stddev for the #3893 rollup and the per-server health read could not be taken, so the #3893/#3901/#3902 attributions had to come from pg_stat_activity sampling (lower bounds only). With this issue shipped those would have been one query.

  2. erikdarlingdata commented on Sep 22, 2026

    @erikdarlingdata
    OwnerAuthor

    Implemented in PR #3904, and verified live on DARLING01 (dev build, 2026-09-22):

    Once it reaches the production stores, the #3893, #3901 and #3902 attributions become a single call (order_by: mean_time or max_time), with per-execution means and maxima instead of pg_stat_activity lower bounds.

  3. erikdarlingdata commented on Sep 22, 2026

    @erikdarlingdata
    OwnerAuthor

    Closed by the watcher: delivered in PR #3904, merged to dev.

  4. erikdarlingdata commented on Sep 22, 2026

    @erikdarlingdata
    OwnerAuthor

    Follow-up: #3915 fixes the post-merge review findings on #3904 (details there). Filed alongside it: #3910 (plaintext passwords in provisioning), #3913 (plain-PostgreSQL convergence cadence), #3914 (compose/BYO web and MCP connect as the owner).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions