Lanes pick the test classes that name a changed symbol, and catalog-wide tests always run (#5776)

[erikdarlingdata]

Claude posting for Erik Darling.

Fixes #5776

Lane A

ci-select.py local-classes lists the test classes whose source names a symbol the change declares or edits. map-select takes the same answer (tight tokens only) as a new reason, "names a changed symbol". Commit 7018936.

What changes

• .github/scripts/ci-select.py
  • New subcommand local-classes --base origin/dev [--map FILE] [--wide] [--cap 25] [--root DIR] (A1). Standard library only.
  • The diff runs from the merge base of --base and HEAD to the working tree, so it works before a commit. It covers .cs, .sql, .js and .xaml, and untracked files count as all-added lines (A2).
  • Tight tokens (A3):
    • names a changed line declares: class, record, struct, enum, interface, const, method, property and field (.cs); function, const/let and class (.js); x:Name, x:Class and Binding (.xaml);
    • the hunk headers, read twice: once with git's built-in csharp driver, which names the member, and once with the default header, which names the column-0 type of a file-scoped namespace;
    • SQL objects: v_* anywhere, and CREATE/ALTER TABLE/VIEW/FUNCTION/PROCEDURE names in .sql.
  • A token taken only from test files is dropped. --wide adds every identifier of 6 or more characters on a changed line (A4).
  • Dropped tokens (A5): stop words (C# keywords, test-API words, 72 common words such as Server, Collector and Query) and names under 4 characters. A token named in more than --cap test files is dropped and printed as NOISY <token> <count>.
  • Output (A6):
    • one line per class (suite Class tokens), most specific first: a changed test file's own classes, then by the fewest test files naming the class's rarest token;
    • then RUN <suite>: -class "*.X" ... lines, 40 classes a line. Classes whose file quotes "DARLING_TEST_PG" are marked needs rig and get their own RUN line.
    • Over 300 classes it prints <n> classes: CI will run FULL; run the first 300 below and lists the first 300 in order.
    • With --map, a separate MAP block gives the map's coverage answer for the changed files, as a count per suite and per reason.
  • map_select() takes a new keyword named ({suite: {Class}}). Each named class gets setdefault(cls, "names a changed symbol"), after every other reason, so a class keeps its earlier reason and the selection only grows (A7). The map-select CLI computes named only when --base is given; CI passes --base "${BASE_SHA}". If git fails, named is None and the selection is what it was before.
  • The function stays pure: no file or git access; the CLI does the I/O. map_replay passes no named, so the replay is unchanged.
  • Not changed: build.yml and nightly.yml (their ci-select calls are byte-identical), and the map-keep and selection JSON formats. The only new thing is a reason string in why.
• Lite.Tests/CiSelectLocalClassesGuardTests.cs (new, Stage=Guard, Reads=Darling; A8): 5 cases on a planted, uncommitted change in a throwaway git repository:
  • a changed method selects the class that names it, and marks the rig class needs rig;
  • a stop word (Server) selects nothing;
  • --cap 1 drops a token named in 2 files and prints NOISY FrobnicateWidget 2;
  • 301 classes print the FULL line and list exactly 300;
  • map-select --base adds names a changed symbol for two classes, and without --base it does not.
• .claude/skills/lane-orders/SKILL.md (Build and test, A9): run python .github/scripts/ci-select.py local-classes --base origin/dev, then its -class lines plus *Census*, *Ratchet* and -trait Stage=Guard.

Measured (A10)

Each pull request's diff from its merge base with dev to its head, against today's test tree (3,000 test files, the same basis as the plan's prototype). The class counts include the classes of the pull request's own changed test files.

PR (head, merge base)	mode	tokens	NOISY	classes	section-1 red classes hit
#5714 (9a1d2f2, 5805c0e)	tight, cap 25	290	64	574	4 of 7
#5714	wide, cap 25	1975	981	2204	7 of 7
#5714	wide, cap 10	1975	1168	1164	7 of 7
#5741 (4eb7589, 7835169)	tight, cap 25	93	13	276	4 of 5
#5741	wide, cap 25	484	271	922	5 of 5
#5741	wide, cap 10	484	306	414	5 of 5

• Tight mode misses three Alerts: a Client Timeouts alert from attention events, with an always-on capture, for Lite and Darling (#5640) #5714 classes: DarlingWebEndpointsTests, WebPlainTextTests and XeTimeFilterSweepTests. They reflect over the catalog and name no changed symbol, so Lane B's R4 tags cover them. It also misses SkipCreditFloorTests on Darling records the collection slots it stepped over (#5735) #5741, which reads _fleetGateStats?.RecordSlot( and is found by --wide.
• Without the second, default-header pass, Darling records the collection slots it stepped over (#5735) #5741 tight also missed DarlingPeerDisclosureTests, which names the class DarlingMcpInstructions. That pass is why both headers are read.
• Alerts: a Client Timeouts alert from attention events, with an always-on capture, for Lite and Darling (#5640) #5714 tight goes over 300, so a lane gets the FULL line and the first 300 in order.
• Runtime: about 3 s on this tree (the git diff is 0.5 s; indexing the test files is 2.4 s).

Test plan

• Lite.Tests built: 0 warnings, 0 errors.
• CiSelectLocalClassesGuardTests: 5 of 5 pass.
• Red-plant 1: rename the planted method to FrobnicateWidgetRenamed. ChangedMethod_SelectsTheClassThatNamesIt fails (1 of 5). Restored.
• Red-plant 2: replace the map_select reason with pass. MapSelect_WithTheBase_AddsTheClassThatNamesAChangedSymbol fails (1 of 5). Restored.
• Lite: *CiSelect*, CrossAppGuardCiGateTests, DarlingReadsTraitGuardTests, TestMapShadowGuardTests, WatermarkPolicyTests (every Lite class that names ci-select), *Census*, *Ratchet*: 170 total, 0 failed.
• Lite -trait Stage=Guard: 507 total, 0 failed.
• Darling.Tests built (0 warnings). SlowClassCensusTests and GuardStageWorkflowTests (they read ci-select.py): 23 total, 0 failed. FleetIdentifierScrubTests (reads SKILL.md): 3 of 3 pass.
• python .github/scripts/ci-select.py --replay: 731 rows, 2035 failing classes checked, 0 misses.
• local-classes --map on a synthetic map: the MAP block prints lite: 1 of 1 classes (names a changed symbol 1).
• Not run: map-select with --base against a real nightly map in CI. Its first pull request run shows how many classes the new reason adds.

For the coordinator

• On a large pull request, the "names a changed symbol" reason adds hundreds of classes to map-select (574 for Alerts: a Client Timeouts alert from attention events, with an always-on capture, for Lite and Darling (#5640) #5714's diff). Those pull requests are FULL anyway (drift over 300). On a small pull request it adds only the classes that name its symbols.
• Matching runs per test file, so every class in a file that names a token is listed.

Lane B

Rule R4 in GuardStageScanner: catalog-wide classes always run. Pushed as cde71dcae + 1ff86809b, merged over Lane A's 701893612. Lane C (below) replaced its Stage=Census trait with Stage=Guard; this section describes the result after Lane C.

What changes

• B1. Rule R4 (Darling/Darling.Tests/GuardStageScanner.cs, linked into Lite.Tests). A test class "reaches a whole catalog" when its code (with comments and strings blanked) contains any of these:

  • McpServerToolTypeAttribute or WithToolsFromAssembly (the MCP tool catalog);
  • GetTypes() / GetExportedTypes() on an assembly that is not a test class's own (typeof(XTests).Assembly and GetExecutingAssembly() are excluded);
  • CollectorCatalog.All.

  Such a class carries [Trait("Stage", "Guard")], or is exempt (R3, it needs a store), or is on the scanner's slow list (see Lane C). R1, R2 and R3 are unchanged.

• B2. Trait: Stage=Guard (after Lane C). The R4 set sums to 545.1 s in Darling.Tests in the nightly test map. Three slow classes account for 501 s of that, and they stay off the guard stage. The rest sum to about 44 s in Darling.Tests and 38 s in Lite.Tests, so the existing trait is enough. build.yml, ci-select.py and the lane-orders line need no change for it.

• B3. Tags, applied by script: 87 Darling.Tests classes and 35 Lite.Tests classes carry Stage=Guard. Exemptions: 25 Darling.Tests R4 classes need a store (R3: live-postgres collection, a DARLING_TEST_ variable, or a store helper). They are in a new s_exemptCatalog with a pinned ceiling of 25. DarlingEmptyEnumerationInventoryTests was already in s_exempt. Lite has no R4 store class, so its exemption list stays empty.

• B4. Both GuardStageCensusTests gain three cases:

  • TheCatalogDetector_... (probe text: 6 planted reaches flagged; the test assembly's own types, GetExecutingAssembly, CollectorCatalog.Find/AllowsX, a single tool attribute, comments and literals not flagged);
  • a synthetic census case: TheCensus_FlagsAnUntaggedCatalogClass_AndHonoursOnlyARuleR4OnlySlowEntry (Darling) and TheCensus_FlagsAnUntaggedCatalogClassAndATaggedStoreClass (Lite);
  • EveryCatalogWideClass_CarriesTheStageGuardTrait_OrIsExempt... (real tree; floor 100 Darling / 35 Lite; the named classes must be R4 and tagged).

  The exemption ceilings are pinned in TheExemptionList_HasItsPinnedSize (Darling) and TheExemptionList_StaysEmpty (Lite).

B5. The red classes of plan section 1

Class	Suite	R4 tagged?
DarlingWebEndpointsTests	Darling	yes
WebPlainTextTests	Darling	yes
RuntimePreconditionReadWiringTests	Darling	yes
XeTimeFilterSweepTests	Lite	yes
DefaultTraceEventsCollectorDefinitionTests	Lite	yes
StatementCutSourceScanTests	Darling	no, Lane A's name rule
XeSessionPermissionLogLevelTests	Lite	no, name rule
CollectionSweepCommandTimeoutTests	Darling	no, name rule
FleetGateStatsTests	Darling	no, name rule
SkipCreditFloorTests	Darling	no, name rule
FleetPartialDatabaseFailureBandTests	Darling	no, name rule
DarlingPeerDisclosureTests	Darling	no, name rule

Test plan (Lane B's own runs, on the Census version)

• Both test projects built with 0 warnings. GuardStageCensusTests + GuardStageWorkflowTests (Darling) 28/28, GuardStageCensusTests (Lite) 12/12.
• Red-plant: untagging DarlingWebEndpointsTests made EveryCatalogWideClass_... fail and name it.
• *Census* + *Ratchet*: Darling 581 pass, 1 skipped (needs a store). Lite *CiSelect* + *Census* + *Ratchet* + *TestMap*: 144/144.

Lane C reran all of these on the final version. Its numbers are below.

Lane C

Design correction: no new trait. Pushed as e2696a2eb.

What changes

• C1. The 122 classes Lane B tagged Stage=Census (87 Darling, 35 Lite) now carry [Trait("Stage", "Guard")]. A byte-level script did the retag. The three slow classes carry no Stage trait, the same as on dev. git grep finds no Stage=Census and no Trait("Stage", "Census") in the tree.

• C2. These files are back to dev or to Lane A's text: build.yml's guard job, GuardStageWorkflowTests, the ci-select.py replay and map-select Stage=Census arms, and the lane-orders line (Lane A's -trait Stage=Guard line is kept). The scanner's CensusTraitText, Row.TaggedCensus and Row.AlwaysRun are removed, and HasGuardTrait reads Guard only again. One comment in ci-select.py (the local-classes blind spot) now names the slow list.

• C3. Slow list. GuardStageScanner.SlowCatalogClasses is a pinned list of R4 classes that stay off the guard stage because they take 50 s or more in the nightly map. Each entry gives its reason as slow: <N> s in the nightly map of 2026-10-13:

  Class	Map seconds
  WebDataStartNoteTests	257
  ConsumedTimestampFrameDisciplineTests	188
  QueryStatsWriterCensusPins	56

  The brief said "more than 60 s" but also named QueryStatsWriterCensusPins, which is 55.6 s. The coordinator ruled to keep all three and to write the threshold as 50 s or more. Problems(rows, exempt, slow) exempts a listed class only when rule R4 alone flags it (Row.CatalogOnly). An entry is STALE when its class is tagged, gone, or also flagged by R1 or R2. An entry is REASON when it does not start with slow: . Lite passes no slow list. The 25-class store list is as Lane B left it.

• C4. Census tests. Darling's TheExemptionList_HasItsPinnedSize pins the slow list at 3. The synthetic case checks four things: an R4-only slow entry is honoured; a slow entry for a guard-style-named class is MISSING and STALE; an untagged R4 class is MISSING; a tagged store class is STORE. The real-tree R4 case allows the slow list and requires each listed class to be an untagged, R4-only class. The probe-text case is unchanged.

C5. Measured

Always-run time locally (Debug, this laptop, -trait Stage=Guard, wall clock including startup):

Project	origin/dev e7db662b9	this PR e2696a2eb	Lane B's Guard + Census, for comparison
Darling.Tests	1656 tests, 65 s (runner 54.2 s)	3847 tests, 104 s (runner 92.7 s)	3948 tests, 192.6 s
Lite.Tests	502 tests, 27 s (runner 24.1 s)	1093 tests, 32 s (runner 27.2 s)	1093 tests, 27.7 s

Nightly test map (run 38356641507, built 2026-10-13T07:05Z, sha 5805c0e): the Guard-tagged R4 classes sum to 44.1 s in Darling.Tests (87 classes, 1 not yet in the map) and 38.1 s in Lite.Tests (35 classes, 1 not in the map). Before the slow list, the Darling sum was 545.1 s.

Test plan

• Darling.Tests and Lite.Tests build: 0 errors, 0 warnings.
• -trait Stage=Guard: Darling 3847 total, 0 failed. Lite 1093 total, 0 failed.
• Darling *Census*, *Ratchet*, GuardStageWorkflowTests, LiteReadsTraitGuardTests (it names GuardStageScanner) and the three slow classes: 683 total, 0 failed, 1 skipped (needs a store).
• Lite *Census*, *Ratchet*, *CiSelect*: 110 total, 0 failed.
• python .github/scripts/ci-select.py --replay: 731 rows, 0 misses.
• C6 red-plants, both planted in one build and then restored:
  • Removing the Guard tag from WebPlainTextTests made EveryCatalogWideClass_CarriesTheStageGuardTrait_OrIsExemptOrSlow fail. The full census failed too, with MISSING WebPlainTextTests.cs: WebPlainTextTests (reaches a whole catalog ... (R4); store: none).
  • Adding a fourth slow entry (DarlingWebEndpointsTests) made TheExemptionList_HasItsPinnedSize fail (Expected 3, Actual 4). The full census also reported STALE DarlingWebEndpointsTests: is on the slow list but now carries the trait.
• Live store classes: not run. No class this PR tags needs a store, and R3 holds over the tagged set.

For the coordinator

• The shards run with -trait- Stage=Guard. Under Lane B's Census version the 122 classes ran on both the guard job and the shards. Now they run in the guard job only, the same as every Guard class. The three slow classes still run on the shards.
• The guard job's Darling stage grows by about 39 s locally (44 s in the map), against about 107 s with Census.
• I did not call the advisor.

SECTION: None
ENTRY: none (CI and test infrastructure only)