Skip to content

Hash-grouped query readers blend INSERT...EXEC callers and label them with one arbitrary text #2012

Closed

Description

@erikdarlingdata

The mechanism (reproduced, not theoretical)

INSERT INTO #t EXEC <proc> statements produce the same query_hash across different callee procedures — the hash normalizes the callee away. Reproduced on SQL Server 2022:

query_hash statement host object
0x9C01BDF7F0F05FC4 INSERT INTO #items EXEC dbo.inner_v3 wrapper_v3
0x9C01BDF7F0F05FC4 INSERT INTO #items EXEC dbo.inner_v4 wrapper_v4

This bit during live production triage on 2026-08-01: hash-keyed text served a V3-core's text for stats that belonged to a V4-core call on two of three per-tenant attribution datapoints — the wrong-callee label survived until disambiguated by hand via OBJECT_DEFINITION.

Where it lands in the product

Per-row storage is correct: the collector captures each row's own statement text (offset SUBSTRING) and #1767's payload dims key by content digest, which explicitly rejected query_hash for this class of reason. The corruption is read-side, in every reader that groups by query_hash and picks one representative text:

  • DarlingDataReader.TopQueriesSql (get_top_queries_by_cpu): groups (database_name, query_hash), takes MAX(sql_handle) + a latest-text LATERAL — colliding callers' stats are summed together and the line is labeled with one arbitrary caller's text.
  • ViewerDataService.QueryStats.cs (four hash-grouped sites) and ViewerDataService.FinOps.Workload.cs — same shape.
  • Lite's twins of the above.
  • The plan-analysis MCP tools resolve per-row and are unaffected; query_store_stats composers group by (module_name, query_hash), which already carries the host-object dimension and is much less exposed.

Staged fix

Stage 1 (no schema, this issue): make the merged groups honest. The #1767 digest column is on every row, so the hash-grouped readers can carry COUNT(DISTINCT query_text_digest) AS distinct_texts almost for free. When a group has more than one distinct text, the MCP payload says so (distinct_texts + a caveat that the shown text is one representative — literal variants are the common benign cause, INSERT...EXEC callers the misattributing one) and the Viewer/Lite grids get the count as a column so a "1 of 4 texts" group stops masquerading as a single statement.

Stage 2 (follow-up, schema): capture the host-object identity at collection — sys.dm_exec_sql_text.objectid is already in the collector's OUTER APPLY, just not selected. With object_name on the row, readers group by (query_hash, host object) where an object exists: separates proc-hosted callees exactly, preserves the ad-hoc literal-collapse behavior (objectid NULL) unchanged. New column + migration + reader/CAGG updates, so it gets its own PR.

Validation path: the container repro above feeds a live end-to-end (collect from the repro database, read back through the deduped reader, assert distinct_texts = 2 for the colliding hash); the production monitor then confirms real fleet collisions surface the flag.

Activity

  1. added 6 commits that reference this issue on Aug 3, 2026
  2. erikdarlingdata commented on Aug 4, 2026

    @erikdarlingdata
    OwnerAuthor

    Completed in two stages, both merged to dev:

    Stage 1 (#2012 disclosure): hash-grouped top-queries reads carry distinct_texts + text_note so a blended group stops masquerading as one statement.

    Stage 2 (PR #2055): the collector captures each statement's hosting module (sys.dm_exec_sql_text.objectid → schema.object) as host_object_name (Darling V51 / Lite v52, appended last, no backfill). Every hash-grouped top-queries surface — both MCP get_top_queries_by_cpu tools (new host_object field), the REST route, and both apps' Top Queries grids — now groups by (database, query_hash, host object) with the representative-text lookup constrained to the group's own rows, so INSERT...EXEC callers sharing a hash report separately with their own text. Ad-hoc literal-collapse unchanged (NULL hosts group as one). Grids' Module column prefers the collection-time host over the #1568 sql_handle stitch.

    Validated in CI against dev Postgres: V51 ran green on both the fresh ladder and a ≥46 rewind, and the regrouped reader passed the distinct-texts live round-trip (blend counts 2/1/0 exactly as planted). Deliberately out of scope: the comparison and FinOps 80/20 surfaces keep hash-level grouping — splitting their keys would false-flag NEW/GONE across the upgrade boundary on every proc-hosted query.

  3. added a commit that references this issue on Aug 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions