Skip to content

The analysis said an Agent job was running long and could not say which: the RUNNING_JOBS fact now names the job furthest past its own history on both SKUs, and the card, the frozen finding text and the headline say which one (partial #3653) - #3693

Merged
erikdarlingdata merged 1 commit into
devfrom
fix/3653-running-jobs-name
Sep 19, 2026

Conversation

@erikdarlingdata

Copy link
Copy Markdown
Owner

What does this PR do?

Headline mechanism. The RUNNING_JOBS fact now carries the NAME of the job that is running long. Both fact collectors (PostgreSQL PgFactCollector.Activity.cs, DuckDB DuckDbFactCollector.Activity.cs) gain one column beside the four existing aggregates — the first element of an ordered, FILTERed ARRAY_AGG(job_name) over the running-long rows only — and put it on Fact.ObjectName. The RUNNING_JOBS advice arm (FactAdvice.ComposeRunningJobs) names that job in the headline, the first sentence and the remediation. Partial #3653 (the mechanical sentence of the A9 structural-half item); recurrence-aware scoring and moving-maintenance-window detection are NOT here — they await the maintainer's Q3 ruling on the issue.

The lie it stops telling

The analysis said "1 Agent job running well past normal — likely stuck, not busy" and could not say which. RunningJobsSql was COUNT(*), COUNT(CASE WHEN is_running_long …), MAX(percent_of_average), MAX(current_duration_seconds) — one row of doubles — and Fact.Metadata is Dictionary<string, double> by contract, so the name had nowhere to ride. Every surface downstream (the story, the frozen advice in the persisted finding's StoryText, the MCP get_analysis_findings card, the notification headline) inherited the blank and sent the operator to the Running Jobs view to learn the name of the thing the engine had just scored. The running_jobs table has carried job_name on both stores all along.

Why this shape

  • One name, not a list. A window aggregate cannot name every long-running job without becoming a list, and a list has no honest home on a Fact. The chosen one is the story's SUBJECT: the job furthest past its OWN history (percent_of_average DESC NULLS LAST), because "running long" is defined against the job's own history, not against the wall clock. current_duration_seconds DESC breaks a tie the way the collector's own ORDER BY does. job_name is the final tie-break so two rows equal on both yield the same answer on every pass — the composed advice is frozen into the finding's StoryText, and a nondeterministic pick would rewrite a finding's text between passes over an unchanged window. The Lite fixture pins exactly that case (three identical long rows → the ordinal-first name).
  • Running-long rows only. FILTER (WHERE is_running_long): a job merely present in the window is not the subject. Without the FILTER, the longest steady job would be named — the fixtures plant a not-long row with the highest duration AND the highest percent to catch that regression.
  • Fact.ObjectName, never Metadata. Metadata is doubles-only (the PostgreSQL-target analysis engine: design and v1 build #3542 lane-7 lesson: every reader — scorer, advice, MCP, the finding stores' JSON — treats a metadata value as a number). ObjectName is the string slot the object-scoped facts already use for "the name the source query selected". I read every consumer before choosing it: Fact.ObjectName is NOT copied onto AnalysisStory/AnalysisFinding and no renderer prints it generically or parses it as schema.table — it reaches the operator exactly one way, through FactAdvice.Compose at analysis time, whose output is frozen into StoryText and read back by every card. So there is no drill-down lookup to confuse and no persisted-row shape change.
  • ARRAY_AGG … FILTER … [1] rather than a subquery or a second scan. One round trip, one row, and the identical clause is valid on both engines (DuckDB lists are 1-based; FILTER and ordered aggregates are supported at the pinned 1.5.5), so the two collectors carry it verbatim. The repo already uses array_agg(col ORDER BY …)[1] in PlanRegressionSql. Rejected: arg_max (DuckDB-only, and no secondary tie-break), any_value (unordered; the dialect pin confines it to the plan-regression query), a ROW_NUMBER CTE (a second pass over the same rows for one string).
  • The advice keeps every pre-existing sentence when the name is absent. A finding persisted before this PR, or a window where jobs ran but none ran long, reads exactly as before. The name is also refused beside a zero running-long count — the collectors cannot produce that pairing (the FILTERed aggregate is NULL when nothing ran long), and naming a job that did not run long would be the lie this arm exists to stop.
  • The overrun figures stay attributed to "the worst", not to the named job. max_percent_of_average and max_duration_seconds are window maxima over EVERY running row, long or not (pre-existing semantics, deliberately unchanged — see "What it does NOT do"); the name is chosen among the long rows only. The two coincide in the ordinary case and the wording does not depend on it.

What it does NOT do

  • Does not de-duplicate a job that ran long across several collection ticks in running_long_count (existing per-row semantics, out of this slice; noted for the coordinator).
  • Does not re-scope max_percent_of_average / max_duration_seconds to the long rows — a not-long job with a high-variance history can own those maxima today. Existing behaviour, reported rather than changed here.
  • Does not carry the name into LinkedJobClause (the SCH_M / IO_WRITE_LATENCY_MS / WRITELOG symptom cards' "RUNNING_JOBS fired in the same window" sentence). That is those arms' change; its doc comment is corrected to say the job card now names the job and this clause still points at the view.
  • Does not touch FactScorer, RelationshipGraph, InferenceEngine, any PgTarget* file, AnalysisModels.cs, or the frozen deprecated/Dashboard twin.
  • No CHANGELOG edit (the entry is returned to the coordinator).

Blast radius

PgFactCollector.RunningJobsSql is a public const inside PgFactCollector.AllSql, so every existing dialect census still walks it (no QUALIFY/NOW(/@/N', $1 positional, FROM running_jobs resolves to a collector table, any_value absent). Column ordinals 0–3 are unchanged; the C# reads the new column at ordinal 4 and keeps NULL as null. Lite's inline query is the same clause over v_running_jobs. RUNNING_JOBS metadata keys are unchanged, so the scorer's (1, 3) ramp, the maintenance graph edges (BaseSeverity > 0) and LinkedJobClause see exactly what they saw.

Tests

New, all EXECUTED here (Lite in-process DuckDB 1.5.5 through a throwaway harness Compile-Including the classes; Darling through the Darling.Tests-named mactest harness; the live fixture against a throwaway timescale/timescaledb:2.28.1-pg18 migrated to ladder 131 with DARLING_TEST_PG set), and each proven RED with the product change stashed:

  • Darling.Tests/PgFactCollectorTests.RunningJobsSql_NamesTheJobFurthestPastItsOwnHistory_AmongLongRowsOnly_OnObjectName_BothSkus — pins the clause text (FILTER, the three-key ORDER BY, [1], the fifth column's position), the ordinal-4 NULL-preserving read and ObjectName = worstLongJobName in BOTH collectors' source, and that the RUNNING_JOBS metadata block is still exactly the four figures.
  • Darling.Tests/PgFactCollectorTests.EndToEnd_RunningJobs_NamesTheWorstLongJob_AgainstDevPostgres (gated) — three rows (400 % long at 7,200 s; 250 % long at 9,000 s; a not-long row at 900 % / 99,999 s): the 400 % job is named, value 2, counts/maxima keep per-row meaning; a second server with nothing long emits the fact with a NULL name. Cleans up through LiveStoreCleanup.RunAsync. Ran green against the rig; CI's Darling PostgreSQL tests runs it too.
  • Lite.Tests/FactCollectorTests.CollectFacts_RunningJobs_NamesTheJobFurthestPastItsOwnHistory_AmongTheLongRowsOnly — the same three rows through the real DuckDbFactCollector; plus the existing CollectFacts_RunningJobs_ReturnsLongRunningCount now asserts the identical-rows tie-break (Test Job 0). New seeder TestDataSeeder.SeedRunningJobRowsAsync.
  • Lite.Tests/FactCollectorMiseryTests.RunningJobs_NoneRunningLong_FactStillCreatedWithZeroValue — asserts ObjectName is null when nothing ran long.
  • Lite.Tests/StoryConfidenceTests.RunningJobsAdvice_NamesTheJob_WhenTheFactCarriesOne and …_KeepsTheUnnamedSentence_WithoutAName_AndRefusesANameOnAZeroCount — the headline/investigation/remediation with one job, with several ("and 2 others"), without a name, and the refused name-beside-zero case. RebuildFacts gains an optional worstJob.

Executed locally: Lite harness 88/88 (the three classes); Darling harness 170/170 across PgFactCollectorTests and the census classes (StoreSqlClockDiscipline, FactCollectorCommandTimeout, AnalysisPassTokenThreading, DocCommentHygiene, RepoFileAdoption, CommentFilterAdoption, LiveCleanupConversionRatchet, LivePostgresCollectionHygiene, ConsumedTimestampFrameDiscipline, CreationTimeClockFrameDiscipline), the gated pair 2/2 against the rig. Builds -c Release -p:EnableWindowsTargeting=true: PerformanceMonitor.Analysis, PerformanceMonitor.Darling.Analysis, PerformanceMonitorLite, Darling.Tests, Dashboard.Tests at 0 warnings; Lite.Tests carries one pre-existing xUnit2000 warning in LogTailOverlapThresholdPinTests.cs:95 (not touched here, present on dev).

Which component(s) does this affect?

  • Lite
  • Darling
  • Lite Tests
  • Darling Tests
  • SQL collection scripts
  • Documentation
  • Full Dashboard (deprecated)
  • CLI Installer (deprecated)

How was this tested?

See Tests above. No SQL Server target is involved — the change is store-side (PostgreSQL 18 via the migrated rig; DuckDB 1.5.5 in-process) and shared-library prose.

Checklist

  • I have read the contributing guide
  • My code builds with zero warnings (dotnet build -c Debug)
  • I have tested my changes against at least one SQL Server version (n/a — store-side; PostgreSQL 18 + DuckDB 1.5.5 executed)
  • I have not introduced any hardcoded credentials or server names

References #3653 (partial — do not close).

…ning-long job furthest past its own history (percent_of_average, then duration, then name) and carry it on Fact.ObjectName, and the job card's headline, first sentence and remediation say which job instead of "1 Agent job running long" (partial #3653, A9)

The RUNNING_JOBS aggregate said how many Agent jobs overran and by how much and
could not say which; the story, the advice card and the persisted finding all
sent the operator to the Running Jobs view to learn the name. The PostgreSQL
and DuckDB collectors gain one column - an ordered, FILTERed ARRAY_AGG's first
element over the running-long rows only - and put it on the fact's ObjectName
(never Metadata, which is doubles by contract). ComposeRunningJobs names the
job when the slot is present and a job ran long, keeps its pre-existing text
otherwise, and refuses a name beside a zero count. Counts and window maxima
keep their existing per-row meaning.

Pins: Darling SQL/emission text pin over both SKUs plus a DARLING_TEST_PG live
fixture (three rows: FILTER, percent-before-duration, NULL when nothing long);
Lite in-process DuckDB collector tests (named rows; the identical-rows tie-break;
null when nothing long); advice with and without a name. No CHANGELOG edit.
@erikdarlingdata
erikdarlingdata enabled auto-merge (squash) September 19, 2026 04:56
@erikdarlingdata
erikdarlingdata merged commit 97de1b3 into dev Sep 19, 2026
7 of 8 checks passed
@erikdarlingdata
erikdarlingdata deleted the fix/3653-running-jobs-name branch September 19, 2026 04:56
erikdarlingdata added a commit that referenced this pull request Sep 19, 2026
…ame the job (Fixes #3704) (#3709)

#3632's three maintenance edges (SCH_M / IO_WRITE_LATENCY_MS / WRITELOG -> RUNNING_JOBS, gated on the
job having FIRED) are keyed on REGULAR symptom facts, and AnomalyIncidentReconciler folded an ANOMALY_*
story only onto a regular story carrying its mapped family. An anomaly is exactly the sub-threshold case
where the regular symptom did not fire, so four one-fact maintenance-family anomalies during one server's
own 116-minute index-maintenance job paged separately and never named the job.

The reconciler now adds RUNNING_JOBS as the LAST fold-target candidate for an anomaly whose resolved
family is one of the three maintenance symptoms (MaintenanceFamilies, pinned to the graph's edge sources),
so it folds onto the regular story that consumed the job. The fired-gate is structural: a story carries
RUNNING_JOBS only when the fact scored above zero, and one long job is base 0.5 on the (1, 3) ramp,
which both fires and roots a story - so there is no fired-but-storyless case to mint an incident for.
Reconcile gains a (stories, facts) overload; both live SKUs pass the run's facts so the folded anomaly's
frozen StoryText Investigation gets one sentence naming the job from the RUNNING_JOBS fact's ObjectName
(#3693). The one-argument form stays for the frozen deprecated/Dashboard twin: it folds, it cannot name.
Non-maintenance anomalies never gain the job as a target.

RelationshipGraph.BuildMaintenanceEdges doc: the "does not name the job" sentence has been false since
#3693; corrected. Pins in Lite.Tests/AnomalyIncidentReconcilerTests: the issue's four-anomaly fixture,
the present-but-quiet job, thirteen non-maintenance keys, own-family-first, the one-argument form, the
unnamed job, idempotence, the graph-parity pin, and an end-to-end run through the real scorer, engine,
clusterer and stamper. Executed here: 38/38, red-first with the set emptied (6 red).
erikdarlingdata added a commit that referenced this pull request Sep 19, 2026
… PRs across the #3653, #3542 and #3691 waves, #3704, #3710, #3712 and #3735, spliced once (#3751)

Sixty-five PRs merged to dev since #3672 (the first splice) across two seats -
the coordinator's #3653 brains-review residue lanes and the wave-2 worker's
#3542 / #3691 PostgreSQL-target engine lanes - and by rule none touched this
file; each returned its entry to a buffer. This is the one splice. Text-only;
one file; nothing outside ## [Unreleased] and the definition block moves.

[Unreleased] gains a ### Changed between ### Added and ### Fixed (Keep-a-Changelog
order) for the nine entries whose subject is a designed behaviour that now behaves
differently - routing (#3712 confidence chooses the channel), gating (#3692 tempdb
persistence, #3734 blocking/LRQ gates, #3724 peak-and-mean), the wire shape
(#3719 one error envelope), the review guard's drift arm (#3673), lineage
metadata (#3723) and two grading changes (#3693, #3747). Twenty-two new
capabilities append to ### Added (the sixteen PostgreSQL-target fact families and
rungs, CONFIG_CHANGED, the recurrence label, the V134/V135 columns, the review
traps file, the measurement census); thirty-two defects append to ### Fixed.
Order within each section is ascending PR number, as #3672 did.

Every entry is one line and verbatim from its buffer (trailing whitespace only).
Four PR pairs fold into one entry as the buffers wrote them (#3738+#3746,
#3683+#3686, #3681+#3685, #3726+#3729); #3695 carries three entries. #3673 has
no issue and cites none, like the file's ninety-six other ref-less entries. Five
definitions added ([#3691], [#3704], [#3710], [#3712], [#3735]) at the end of the
[Unreleased] run, which stays one ascending run (3517..3735). No archive touched;
the split was not run.

tools/changelog/changelog_archive.py verify: all PASS (1,483 bold entries, floor
1,329; 1,363 distinct refs resolve; 1,372 definitions; CRLF throughout; 409,263
bytes under the ceiling). Client-hygiene grep over the file: zero hits.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant