Repository navigation
[FEATURE] Query Store backfill needs an off switch (global service toggle, hot-applied) #2167
Description
Component
Both apps (Darling first — fleet-operational need)
Why now
Field event 2026-08-10: a freshly restored tenant catalog put the #2058 backfill into sustained drain against a cross-region 4-core production primary — 64MB slices every cycle, dominated by ASYNC_NETWORK_IO (server holding the query open while the remote client drains), stacked on budget-bound tick collection. The operator's correct call was "stop the backfill on this deployment until the tuning lands" — and there is NO way to do that today: the loop runs unconditionally for every server; the only related knob (capture_plans) is global and guts plan collection everywhere, which is not an acceptable trade.
Ask
config_service.query_store_backfill_enabled boolean NOT NULL DEFAULT TRUE (new migration rung), read into DarlingConfig via the live store-reload path so a flip applies next cycle without a restart, gating RunQueryStoreBackfillLoopAsync (skip iteration + log state transitions). Viewer settings toggle beside the capture-plans checkbox. Lite parity via its app settings.
Per-server granularity is the natural follow-up but the global switch is the emergency brake and ships first. Related: #2164 (budget knob + known-hash skip), #2165 (tick/backfill serialization).
Activity
Changelog: add the missing [#2167] footnote definition (review catch)
Merge pull request #2168 from erikdarlingdata/backfill-switch-2166
erikdarlingdata commentedon Aug 10, 2026
The Darling half is merged (#2168):
config_service.query_store_backfill_enabled(V58), live-read by the backfill loop, Viewer toggle beside plan capture. Deploying to the dogfood fleet deployment now. This issue stays OPEN for the Lite half — Lite's backfill shares the class but needs a global settings surface to hang the toggle on.Merge pull request #2176 from erikdarlingdata/lite-backfill-switch-2167
erikdarlingdata commentedon Aug 11, 2026
Complete in both apps now. Darling got the store-backed switch in #2168 (V58, live-reloaded, Viewer checkbox); Lite got the Settings checkbox in #2176 (
query_store_backfill_enabledin settings.json, read live so it takes effect without a restart). Ships in 3.5.0.The surfaces differ deliberately — a store column for the headless service whose operator may be nowhere near the box, a checkbox for the app with a settings window and one local user — and that reasoning is recorded in both places so the asymmetry doesn't read as an oversight later.
The Query Store backfill gets an off switch (erikdarlingdata#2167)
Lite gets the backfill off switch too (erikdarlingdata#2167 parity)