Repository navigation
The analysis said an Agent job was running long and could not say which: the RUNNING_JOBS fact now names the job furthest past its own history on both SKUs, and the card, the frozen finding text and the headline say which one (partial #3653) - #3693
Merged
Merged
Conversation
…ning-long job furthest past its own history (percent_of_average, then duration, then name) and carry it on Fact.ObjectName, and the job card's headline, first sentence and remediation say which job instead of "1 Agent job running long" (partial #3653, A9) The RUNNING_JOBS aggregate said how many Agent jobs overran and by how much and could not say which; the story, the advice card and the persisted finding all sent the operator to the Running Jobs view to learn the name. The PostgreSQL and DuckDB collectors gain one column - an ordered, FILTERed ARRAY_AGG's first element over the running-long rows only - and put it on the fact's ObjectName (never Metadata, which is doubles by contract). ComposeRunningJobs names the job when the slot is present and a job ran long, keeps its pre-existing text otherwise, and refuses a name beside a zero count. Counts and window maxima keep their existing per-row meaning. Pins: Darling SQL/emission text pin over both SKUs plus a DARLING_TEST_PG live fixture (three rows: FILTER, percent-before-duration, NULL when nothing long); Lite in-process DuckDB collector tests (named rows; the identical-rows tie-break; null when nothing long); advice with and without a name. No CHANGELOG edit.
Closed
39 tasks done
erikdarlingdata
enabled auto-merge (squash)
September 19, 2026 04:56
This was referenced Sep 19, 2026
erikdarlingdata
added a commit
that referenced
this pull request
Sep 19, 2026
…ame the job (Fixes #3704) (#3709) #3632's three maintenance edges (SCH_M / IO_WRITE_LATENCY_MS / WRITELOG -> RUNNING_JOBS, gated on the job having FIRED) are keyed on REGULAR symptom facts, and AnomalyIncidentReconciler folded an ANOMALY_* story only onto a regular story carrying its mapped family. An anomaly is exactly the sub-threshold case where the regular symptom did not fire, so four one-fact maintenance-family anomalies during one server's own 116-minute index-maintenance job paged separately and never named the job. The reconciler now adds RUNNING_JOBS as the LAST fold-target candidate for an anomaly whose resolved family is one of the three maintenance symptoms (MaintenanceFamilies, pinned to the graph's edge sources), so it folds onto the regular story that consumed the job. The fired-gate is structural: a story carries RUNNING_JOBS only when the fact scored above zero, and one long job is base 0.5 on the (1, 3) ramp, which both fires and roots a story - so there is no fired-but-storyless case to mint an incident for. Reconcile gains a (stories, facts) overload; both live SKUs pass the run's facts so the folded anomaly's frozen StoryText Investigation gets one sentence naming the job from the RUNNING_JOBS fact's ObjectName (#3693). The one-argument form stays for the frozen deprecated/Dashboard twin: it folds, it cannot name. Non-maintenance anomalies never gain the job as a target. RelationshipGraph.BuildMaintenanceEdges doc: the "does not name the job" sentence has been false since #3693; corrected. Pins in Lite.Tests/AnomalyIncidentReconcilerTests: the issue's four-anomaly fixture, the present-but-quiet job, thirteen non-maintenance keys, own-family-first, the one-argument form, the unnamed job, idempotence, the graph-parity pin, and an end-to-end run through the real scorer, engine, clusterer and stamper. Executed here: 38/38, red-first with the set emptied (6 red).
This was referenced Sep 19, 2026
erikdarlingdata
added a commit
that referenced
this pull request
Sep 19, 2026
… PRs across the #3653, #3542 and #3691 waves, #3704, #3710, #3712 and #3735, spliced once (#3751) Sixty-five PRs merged to dev since #3672 (the first splice) across two seats - the coordinator's #3653 brains-review residue lanes and the wave-2 worker's #3542 / #3691 PostgreSQL-target engine lanes - and by rule none touched this file; each returned its entry to a buffer. This is the one splice. Text-only; one file; nothing outside ## [Unreleased] and the definition block moves. [Unreleased] gains a ### Changed between ### Added and ### Fixed (Keep-a-Changelog order) for the nine entries whose subject is a designed behaviour that now behaves differently - routing (#3712 confidence chooses the channel), gating (#3692 tempdb persistence, #3734 blocking/LRQ gates, #3724 peak-and-mean), the wire shape (#3719 one error envelope), the review guard's drift arm (#3673), lineage metadata (#3723) and two grading changes (#3693, #3747). Twenty-two new capabilities append to ### Added (the sixteen PostgreSQL-target fact families and rungs, CONFIG_CHANGED, the recurrence label, the V134/V135 columns, the review traps file, the measurement census); thirty-two defects append to ### Fixed. Order within each section is ascending PR number, as #3672 did. Every entry is one line and verbatim from its buffer (trailing whitespace only). Four PR pairs fold into one entry as the buffers wrote them (#3738+#3746, #3683+#3686, #3681+#3685, #3726+#3729); #3695 carries three entries. #3673 has no issue and cites none, like the file's ninety-six other ref-less entries. Five definitions added ([#3691], [#3704], [#3710], [#3712], [#3735]) at the end of the [Unreleased] run, which stays one ascending run (3517..3735). No archive touched; the split was not run. tools/changelog/changelog_archive.py verify: all PASS (1,483 bold entries, floor 1,329; 1,363 distinct refs resolve; 1,372 definitions; CRLF throughout; 409,263 bytes under the ceiling). Client-hygiene grep over the file: zero hits.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Headline mechanism. The
RUNNING_JOBSfact now carries the NAME of the job that is running long. Both fact collectors (PostgreSQLPgFactCollector.Activity.cs, DuckDBDuckDbFactCollector.Activity.cs) gain one column beside the four existing aggregates — the first element of an ordered,FILTERedARRAY_AGG(job_name)over the running-long rows only — and put it onFact.ObjectName. TheRUNNING_JOBSadvice arm (FactAdvice.ComposeRunningJobs) names that job in the headline, the first sentence and the remediation. Partial #3653 (the mechanical sentence of the A9 structural-half item); recurrence-aware scoring and moving-maintenance-window detection are NOT here — they await the maintainer's Q3 ruling on the issue.The lie it stops telling
The analysis said "1 Agent job running well past normal — likely stuck, not busy" and could not say which.
RunningJobsSqlwasCOUNT(*),COUNT(CASE WHEN is_running_long …),MAX(percent_of_average),MAX(current_duration_seconds)— one row of doubles — andFact.MetadataisDictionary<string, double>by contract, so the name had nowhere to ride. Every surface downstream (the story, the frozen advice in the persisted finding'sStoryText, the MCPget_analysis_findingscard, the notification headline) inherited the blank and sent the operator to the Running Jobs view to learn the name of the thing the engine had just scored. Therunning_jobstable has carriedjob_nameon both stores all along.Why this shape
Fact. The chosen one is the story's SUBJECT: the job furthest past its OWN history (percent_of_average DESC NULLS LAST), because "running long" is defined against the job's own history, not against the wall clock.current_duration_seconds DESCbreaks a tie the way the collector's ownORDER BYdoes.job_nameis the final tie-break so two rows equal on both yield the same answer on every pass — the composed advice is frozen into the finding'sStoryText, and a nondeterministic pick would rewrite a finding's text between passes over an unchanged window. The Lite fixture pins exactly that case (three identical long rows → the ordinal-first name).FILTER (WHERE is_running_long): a job merely present in the window is not the subject. Without the FILTER, the longest steady job would be named — the fixtures plant a not-long row with the highest duration AND the highest percent to catch that regression.Fact.ObjectName, neverMetadata. Metadata is doubles-only (the PostgreSQL-target analysis engine: design and v1 build #3542 lane-7 lesson: every reader — scorer, advice, MCP, the finding stores' JSON — treats a metadata value as a number).ObjectNameis the string slot the object-scoped facts already use for "the name the source query selected". I read every consumer before choosing it:Fact.ObjectNameis NOT copied ontoAnalysisStory/AnalysisFindingand no renderer prints it generically or parses it asschema.table— it reaches the operator exactly one way, throughFactAdvice.Composeat analysis time, whose output is frozen intoStoryTextand read back by every card. So there is no drill-down lookup to confuse and no persisted-row shape change.ARRAY_AGG … FILTER … [1]rather than a subquery or a second scan. One round trip, one row, and the identical clause is valid on both engines (DuckDB lists are 1-based; FILTER and ordered aggregates are supported at the pinned 1.5.5), so the two collectors carry it verbatim. The repo already usesarray_agg(col ORDER BY …)[1]inPlanRegressionSql. Rejected:arg_max(DuckDB-only, and no secondary tie-break),any_value(unordered; the dialect pin confines it to the plan-regression query), aROW_NUMBERCTE (a second pass over the same rows for one string).max_percent_of_averageandmax_duration_secondsare window maxima over EVERY running row, long or not (pre-existing semantics, deliberately unchanged — see "What it does NOT do"); the name is chosen among the long rows only. The two coincide in the ordinary case and the wording does not depend on it.What it does NOT do
running_long_count(existing per-row semantics, out of this slice; noted for the coordinator).max_percent_of_average/max_duration_secondsto the long rows — a not-long job with a high-variance history can own those maxima today. Existing behaviour, reported rather than changed here.LinkedJobClause(the SCH_M / IO_WRITE_LATENCY_MS / WRITELOG symptom cards' "RUNNING_JOBS fired in the same window" sentence). That is those arms' change; its doc comment is corrected to say the job card now names the job and this clause still points at the view.FactScorer,RelationshipGraph,InferenceEngine, anyPgTarget*file,AnalysisModels.cs, or the frozendeprecated/Dashboardtwin.Blast radius
PgFactCollector.RunningJobsSqlis a public const insidePgFactCollector.AllSql, so every existing dialect census still walks it (noQUALIFY/NOW(/@/N',$1positional,FROM running_jobsresolves to a collector table,any_valueabsent). Column ordinals 0–3 are unchanged; the C# reads the new column at ordinal 4 and keeps NULL asnull. Lite's inline query is the same clause overv_running_jobs.RUNNING_JOBSmetadata keys are unchanged, so the scorer's(1, 3)ramp, the maintenance graph edges (BaseSeverity > 0) andLinkedJobClausesee exactly what they saw.Tests
New, all EXECUTED here (Lite in-process DuckDB 1.5.5 through a throwaway harness Compile-Including the classes; Darling through the
Darling.Tests-named mactest harness; the live fixture against a throwawaytimescale/timescaledb:2.28.1-pg18migrated to ladder 131 withDARLING_TEST_PGset), and each proven RED with the product change stashed:Darling.Tests/PgFactCollectorTests.RunningJobsSql_NamesTheJobFurthestPastItsOwnHistory_AmongLongRowsOnly_OnObjectName_BothSkus— pins the clause text (FILTER, the three-key ORDER BY,[1], the fifth column's position), the ordinal-4 NULL-preserving read andObjectName = worstLongJobNamein BOTH collectors' source, and that theRUNNING_JOBSmetadata block is still exactly the four figures.Darling.Tests/PgFactCollectorTests.EndToEnd_RunningJobs_NamesTheWorstLongJob_AgainstDevPostgres(gated) — three rows (400 % long at 7,200 s; 250 % long at 9,000 s; a not-long row at 900 % / 99,999 s): the 400 % job is named, value 2, counts/maxima keep per-row meaning; a second server with nothing long emits the fact with a NULL name. Cleans up throughLiveStoreCleanup.RunAsync. Ran green against the rig; CI'sDarling PostgreSQL testsruns it too.Lite.Tests/FactCollectorTests.CollectFacts_RunningJobs_NamesTheJobFurthestPastItsOwnHistory_AmongTheLongRowsOnly— the same three rows through the realDuckDbFactCollector; plus the existingCollectFacts_RunningJobs_ReturnsLongRunningCountnow asserts the identical-rows tie-break (Test Job 0). New seederTestDataSeeder.SeedRunningJobRowsAsync.Lite.Tests/FactCollectorMiseryTests.RunningJobs_NoneRunningLong_FactStillCreatedWithZeroValue— assertsObjectNameis null when nothing ran long.Lite.Tests/StoryConfidenceTests.RunningJobsAdvice_NamesTheJob_WhenTheFactCarriesOneand…_KeepsTheUnnamedSentence_WithoutAName_AndRefusesANameOnAZeroCount— the headline/investigation/remediation with one job, with several ("and 2 others"), without a name, and the refused name-beside-zero case.RebuildFactsgains an optionalworstJob.Executed locally: Lite harness 88/88 (the three classes); Darling harness 170/170 across
PgFactCollectorTestsand the census classes (StoreSqlClockDiscipline,FactCollectorCommandTimeout,AnalysisPassTokenThreading,DocCommentHygiene,RepoFileAdoption,CommentFilterAdoption,LiveCleanupConversionRatchet,LivePostgresCollectionHygiene,ConsumedTimestampFrameDiscipline,CreationTimeClockFrameDiscipline), the gated pair 2/2 against the rig. Builds-c Release -p:EnableWindowsTargeting=true:PerformanceMonitor.Analysis,PerformanceMonitor.Darling.Analysis,PerformanceMonitorLite,Darling.Tests,Dashboard.Testsat 0 warnings;Lite.Testscarries one pre-existing xUnit2000 warning inLogTailOverlapThresholdPinTests.cs:95(not touched here, present on dev).Which component(s) does this affect?
How was this tested?
See Tests above. No SQL Server target is involved — the change is store-side (PostgreSQL 18 via the migrated rig; DuckDB 1.5.5 in-process) and shared-library prose.
Checklist
dotnet build -c Debug)References #3653 (partial — do not close).