Skip to content

Stage index_object_stats definition metadata instead of correlating per index (#3508) - #3510

Merged
erikdarlingdata merged 1 commit into
devfrom
feature/3508-index-gather-staging
Sep 17, 2026
Merged

erikdarlingdata merged 1 commit into
devfrom
feature/3508-index-gather-staging

Conversation

@erikdarlingdata

Copy link
Copy Markdown
Owner

Closes #3508. The body-cost half of #3477 (the width half is the per-collector DB scope; this is the per-database body).

Problem

index_object_stats staged the three stats DMVs into #temps (the #1135 fix) but built the per-index definition metadata in the final SELECT as five correlated subqueries against the live catalog, per index row: key_columns/included_columns (FOR XML over sys.index_columns/sys.columns), two FK EXISTS over sys.foreign_key_columns, and the compression TOP (1) over sys.partitions. ~5 catalog probes per index, scaling with index count — a 54K-index database spends ~28 s in this body.

Fix

Extend the existing staging to the definition catalog — the technique sp_IndexCleanup already uses. Stage sys.index_columns (+ column names, once, with a supporting clustered index), the FK-backed / FK-referenced key-column sets, and the per-index lowest partition compression into #temps; then LEFT JOIN them in the final SELECT with no correlated subqueries. Emitted strings and flags are byte-identical, so Stage-2 dedupe and FK-protection are unaffected — a plan-shape change only. The collector is shared, so Lite and Darling both get it.

Measured

Synthetic database, 56,001 indexes on empty tables (so this isolates the definition-metadata cost):

body time
before (correlated) ~4.1 s
after (staged) ~2.0 s

EXCEPT both ways returns 0 rows in either direction at both 12K and 56K indexes. Per-step instrumentation confirms the correlated per-index catalog probing is gone; the residual is the unchanged dm_db_index_operational_stats read plus the irreducible cost of building the delimited key/include strings. (Empty tables, so absolute times run below a real 54K-index instance whose operational-stats DMV is heavier; the ~2x and the parity are the transferable results.)

Tests

  • New guard DefinitionMetadata_IsStaged_NotCorrelatedPerIndex pins that the definition metadata is staged, not correlated (a revert drops the markers).
  • Full local suites green: Lite.Tests 3916/0, Darling.Tests 9627 total / 0 errors. The one Darling failure (NpgsqlRootCertificateValidationTests.ChainShape…) is the known machine-local TLS cert-chain test on a path this change never touches (it passes in CI on Linux).

🤖 Generated with Claude Code

https://claude.ai/code/session_01RzwoqcD62jKyp6tzsWpZLz

…er index (#3508)

The per-database body staged the three stats DMVs into #temps (the #1135 fix) but
built the per-index DEFINITION metadata in the final SELECT as five correlated
subqueries against the live catalog, evaluated per index row: key_columns and
included_columns (FOR XML over sys.index_columns/sys.columns), the two FK EXISTS
over sys.foreign_key_columns, and the compression TOP(1) over sys.partitions. On a
wide schema that is ~5 catalog probes per index, and the cost scales with index
count - the body-cost half of #3477 (a 54K-index database spends ~28 s here).

Extend the existing staging to the definition catalog, the technique sp_IndexCleanup
already uses: stage sys.index_columns (+ column names, once, with a supporting
clustered index), the FK-backed/FK-referenced key-column sets, and the per-index
lowest partition compression into #temps, then LEFT JOIN them in the final SELECT
with no correlated subqueries. The emitted key/include strings and every flag are
byte-identical, so the Stage-2 dedupe and FK-protection logic are unaffected - a
plan-shape change only. IndexObjectStatsCollector is shared, so Lite and Darling
both get it.

Measured on a synthetic 56,001-index database (empty tables, so this is the
definition-metadata cost in isolation): the body drops from ~4.1 s to ~2.0 s, and an
EXCEPT-both-ways parity check returns zero rows in either direction at 12K and 56K
indexes. A guard test pins that the definition metadata is staged, not correlated.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RzwoqcD62jKyp6tzsWpZLz
@erikdarlingdata
erikdarlingdata merged commit 8be9539 into dev Sep 17, 2026
7 of 8 checks passed
@erikdarlingdata
erikdarlingdata deleted the feature/3508-index-gather-staging branch September 17, 2026 14:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant