Repository navigation
Classify the 1s lock-timeout yield as YIELDED, not a collection error (#1805) - #1808
Merged
Merged
Conversation
…#1805) One collector deliberately sets SET LOCK_TIMEOUT 1000 (query_snapshots, both variants); its 1222 is the never-be-a-blocker promise working, but it surfaced as a generic collection error - and one collection error alone paints a CRITICAL day on the health calendar. - Shared YieldsOnLockTimeout flag on the collector definition, true only for query_snapshots; CollectorCatalog.YieldsOnLockTimeout(name) for the runners' catch sites (unknown names default false) - Both runners (Lite RemoteCollectorService, DarlingWorker.RunOneAsync) catch 1222 through the flag and log YIELDED with a message that names the re-frame: target lock contention, not a monitoring failure - Reader audit across both apps found every error/success predicate exact-match, so YIELDED is excluded from the Critical band, collector health, fleet banding, and the collection-stopped self-alert with zero reader changes; pins guard the predicate shape in both daily summaries - In-memory health is yield-neutral: no streak reset, no streak growth - Yields stay visible: Yields column on both Collection Health grids and a yields field on both get_collection_health MCP tools Verified: catalog pin watched red by flipping the override; full Lite.Tests 1598/1598 and Darling.Tests 3564+189skip green; all builds zero warnings. DailyHealthBandTests untouched. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1805.
What
Classification only, exactly per the issue's fix sketch — the collection behavior (the 1s
LOCK_TIMEOUTyield itself) was already correct and is untouched.The shared flag.
ICollectorSchemaInfo.YieldsOnLockTimeout(default false; true only onQuerySnapshotsCollector, the one collector that sets the guard — pinned, including a source check that the guard is really in both query variants). Evaluable by name viaCollectorCatalog.YieldsOnLockTimeout(name), same shape asAppliesTo— with the opposite unknown-name default (false): a 1222 from a collector without the guard is unexpected and stays an ERROR.Both runners classify. Lite (
RemoteCollectorService) and Darling (DarlingWorker.RunOneAsync) catchSqlException1222 through the flag and log a newYIELDEDcollection_log status instead ofERROR, with a message that says what it means (target lock contention, one snapshot sweep skipped, not a monitoring failure). Both catches are filtered (when), so every other 1222 and every other error path is byte-identical to before.The exclusion needed zero reader changes — audited, not assumed. Every reader of collection_log status in both apps matches statuses exactly: the daily band's collErrors input (
COUNT(*) FILTER (WHERE status = 'ERROR')in Lite'sLocalDataService.DailySummaryand Darling'sDailySummarySql), collector-health error counts (Lite CollectionHealth, Darling viewer per-server + fleet, both MCP tools), the collection-stopped self-alert'sIN ('SUCCESS','SKIPPED')recency (a yield correctly does not count as success there), last-error stamps (IN ('ERROR','PERMISSIONS')), and the fleet Healthy/Failing banding. New pins in BOTH test suites guard the predicate shape — exact-match= 'ERROR'— so a future broadening to a not-success form cannot silently re-admit yields into the Critical band. The Dashboard is out of scope: its collection is the SQL-side Agent pipeline and never runs this collector.In-memory health is yield-neutral.
RecordCollectorResult("YIELDED")neither resets a real error streak (a yield is not proof the collector works) nor grows one (the guard worked as designed). Behavioral tests cover both directions plus the success-clears control.The re-frame stays visible (the issue's "own log shape and counter"). Yields get a
Yieldscolumn on both Collection Health grids (Lite tab + Darling viewer, with a tooltip saying what it means) and ayieldsfield on bothget_collection_healthMCP tools — clustering is a lock-contention signal about the monitored server, now labeled as such. The log windows already render unknown statuses in the plain foreground, soYIELDEDrows read neutrally there too.Deliberately unchanged, per the issue: the 1s timeout, no retries, no extending the guard to other collectors, and
CollectorHealthClassifier— a collector yielding for hours goes honestly stale, with the Yields column beside the staleness explaining why.DailyHealthBandTests(both apps) are untouched and green: one real collection error alone is still a Critical day.Test plan
CollectorCatalogpins: query_snapshots is the only flagged collector; unknown names default false; the flagged collector's source really carries the guard in both variants — watched RED by flipping the override to false🤖 Generated with Claude Code