Darling: unbounded memory growth in the collector row accumulation — query_store OOM cascades to Postgres commit-limit failures and process crash #1556

Field report (24-server fleet, 16GB host, nightly 3.1.0-nightly.20260717 post-#1553): service memory grew 0 → ~13GB in under 9 minutes, OutOfMemoryException in query_store collection (caught, logged, growth continued), then system-wide commit-limit exhaustion — bundled Postgres threw could not reattach to shared memory (error code 1455) / database system is in recovery mode, RDP and SSM on the box degraded, and the service process died twice (exit 1067, no WER dump). Reproduced on two consecutive restarts.

Root cause (grounded)

1. Unbounded accumulation across databases: DarlingCollectorRunner.RunAsync creates ONE List<TRow> per collector run and every database's rows accumulate into it; storage happens only after the full database loop (DarlingCollectorRunner.cs:262-278). A server's entire per-database haul is held in memory at once.
2. The per-database catch retains the accumulation: the Failed to collect {Collector} from [{Database}] handler (:272-277 — the exact line the field log captured) swallows the OOM and CONTINUES the loop with the already-accumulated list fully retained.
3. The query_store read is unbounded and plan-heavy: no TOP/row cap ("no size guard" per its own comment), and CONVERT(nvarchar(max), qsp.query_plan) is selected on EVERY runtime-stats interval row — the same plan XML duplicated per interval (QueryStoreCollector.cs:347-353, :430-445). The store-derived watermark makes each restart after an outage re-read everything since the last successful collect.
4. Darling: bounded-parallel fire-and-track collection sweep + cadence jitter (#1553) #1553's 4-wide sweep multiplies the peak ×4 (four servers' hauls concurrently). Exacerbation, not cause — the reporter's hypothesis was exactly right.

Fix directions

• Flush-and-clear rows per database (or per N rows) instead of accumulating the whole run — bounds peak to one batch.
• Bound the query_store read (row cap per database per cycle, logged when hit) and stop duplicating plan XML per interval row (capture it once per plan per cycle — newest interval row only).
• OOM-specific handling in the item loop: clear the accumulation and ABORT the collector run rather than continuing.
• Cap watermark catch-up staleness so post-outage restarts do not re-read unbounded history.
• Viewer: surface a store-unreachable indicator instead of an empty screen (reporter ask 5).
• Investigate/document the no-WER-dump exit path (reporter ask 4; likely CLR fail-fast under commit exhaustion).

Immediate operator mitigation (no new build): "capturePlans": false in darling.json removes the dominant memory term (plan text becomes a NULL placeholder).

🤖 Generated with Claude Code