Skip to content

CI: run slow classes that have not failed in five weeks off the PR path: 101 Darling classes (35% of shard time) and 35 Lite classes #5946

Closed

Description

@erikdarlingdata

Claude posting for Erik Darling.

The Darling PostgreSQL shards decide how long a PR run takes, and about a third of their time goes to slow test classes that have not failed in five weeks. This issue proposes to run those classes in the nightly and on dev pushes, and on a PR only when the change reaches them. That is the mechanism #5510 already uses for 27 classes. The candidates are 101 Darling classes (35% of the Darling shard time) and 35 Lite classes (27% of the Lite test time).

Source: GitHub Actions data for build.yml from 2026-10-09 to 2026-10-16, CI failure logs from 2026-09-07 to 2026-10-07 and from 2026-10-09 to 2026-10-16, .github/scripts/darling-shard-weights.json at dev 67835f03d, and one full Lite run on Windows from 2026-10-07.

The long pole (measured)

  • A Darling PostgreSQL shard takes a median of 11.3 minutes: 1.6 minutes of setup and 9.3 of tests. There are 6 shards per run.
  • A Darling shard is the last test job to finish in 270 of 405 PR runs that ran tests.
  • PR runs take a median of 22.6 minutes of wall time.

The rule for a candidate

A class is a candidate when all of these are true:

  • It takes 5 seconds or more: weighted seconds from darling-shard-weights.json for Darling, and seconds in one full Windows run for Lite.
  • It failed no CI job in either failure window above. October 8 is not covered.
  • It does not carry [Trait("Cost", "Slow")] already.
  • It is not a Guard class.
Suite Candidates Their seconds Share of the suite
Darling.Tests 101 1,447 of 4,139 weighted 35%
Lite.Tests 35 329 of 1,222 27%

Expected effect (inferred, not measured)

The Darling weights model each class's share of a shard's wall time. So taking 35% of them out of PR runs would cut the Darling shards' test time by about a third. That is about 19 of the 55 runner-minutes of Darling test time per PR run, and about 3 minutes of wall time. The Lite saving is smaller, about 4 runner-minutes. Measure both after the change.

Proposal

  1. Add [Trait("Cost", "Slow")] to the candidates below, so the existing Pull requests skip 27 slow, never-failed test classes unless the change reaches them (#5459) #5510 rule skips them on a PR unless the change reaches them.
  2. Re-run the rule every month, from the latest weights and failure history. Take the trait off any class that failed in the window.
  3. The nightly and dev pushes keep running everything.

Risk

  • A regression in one of these classes fails on the dev push after the merge, not in the PR. Dev pushes run everything today, so that is one merge later.
  • Somebody must read nightly and dev push failures every day.
  • "Has not failed" is not "has no value". These classes stay. They only move off the PR path.
Darling.Tests candidates (101)
Class Seconds
ComposeQueryStoreStampExactnessLiveTests 66.0
PgIoStatsHourlyLiveTests 65.3
ServerEditPasswordRuleViewerTests 64.1
ServerIdentityLockLiveTests 48.5
ViewerFleetAzureMasterScopeLiveTests 43.3
StoreQueryHistoryToolLiveTests 42.9
ViewerBlockedProcessReportsCoverageLiveTests 35.1
ServerEditLiveTests 33.9
PerfmonRegroupTimingLiveTests 33.2
QueryStoreTopDailyBuilderLiveTests 32.8
DarlingModuleMapGroupedSourceLiveTests 31.9
DarlingModuleMapRecentRefreshLiveTests 29.2
StoreServerPasswordRulesLiveTests 26.3
ExternalAlertGrantLiveTests 26.3
ViewerSystemEventsDataStartLiveTests 22.7
QueryStoreIntervalConvergenceLiveTests 22.7
ViewerShortWindowDataStartTests 22.2
ViewerClutterAndMemoryPressureDataStartLiveTests 22.0
ForcePlanFailuresMemoLiveTests 20.8
TrendStaleStatsLiveTests 19.9
DiagnosticsBundleStoreHistoryLiveTests 19.8
ServerEditViewerRoleLiveTests 19.8
QueryStoreIntervalPurgeRowCappedTests 19.0
ProcedureStatsTotalsWriterLiveTests 18.3
BlockingDatabaseFilterLiveTests 17.8
QueryStoreIntervalFloorWarmUpTests 17.0
QueryStoreWatermarkReaderLiveTests 16.0
QueryStoreIntervalPartitionsLiveTests 16.0
QueryStoreTopDailyReadLiveTests 15.6
SecondaryReplicaScopeLiveTests 13.8
PgDeadlockRemaskAlertPagingLiveTests 13.8
ViewerEventDataStartLiveTests 13.1
ViewerJobHistoryDataStartLiveTests 12.8
FinOpsStorageGrowthViewLiveTests 12.8
ScratchPostgresExitDrainTests 12.4
AlertNotebookEndpointTests 12.0
DatabaseWatermarkCacheRunnerLiveTests 11.9
StoreStatementHistoryLiveTests 11.8
FinOpsUtilizationViewLiveTests 11.7
WebServerPageRangeTests 11.2
McpCollectionCaveatGrantLiveTests 11.2
WebhookPostTimeoutTests 10.2
QueryStoreWatermarksWriterLiveTests 10.2
TuningHourlyCreateLiveTests 9.9
FinOpsRecommendationsToolParityLiveTests 9.8
QueryStoreBackfillDroppedDatabaseTests 9.6
PasswordKeyTablesLiveTests 9.5
WebRangeKeptHistoryBehaviourTests 9.4
DarlingMcpServerTrendToolsLiveTests 9.0
ExternalAlertStoreLiveTests 8.9
BlockingListPlanFlagsLivePostgresTests 8.8
QueryStoreBackfillCandidateCacheLiveTests 8.5
BoundedDdlBusyStreakLiveTests 8.4
PlanViewerBehaviourTests 8.4
CollectionHealthRunTimeFieldsLiveTests 8.3
SlowReadsLiveTests 8.1
PgDeadlocksResumeLiveTests 8.0
ViewerEventCapDataStartLiveTests 7.9
QueryStoreIntervalRetentionLiveTests 7.9
MaterializationHoleProbeLiveTests 7.9
WebBlockingSaveXmlBehaviourTests 7.9
QueryStatsHourLedgerTests 7.9
GridToolsBehaviourTests 7.8
DarlingMcpHealthParserToolsWindowNoticeLiveTests 7.8
FinOpsOptimizationViewLiveTests 7.7
ComposeHourlyRawEdgesModuleMapLiveTests 7.7
FinOpsRecommendationsSecondaryReplicaLiveTests 7.6
ReadAdmissionWebTests 7.6
ViewerQueryStoreRegressionsDataStartLiveTests 7.4
ProvisioningBatchLockedTransactionLiveTests 7.3
DiagnosticsBundleSeedFailureLiveTests 7.3
QueryStoreBackfillSkipFailingDatabaseTests 7.2
ViewerPlanCorrectionsDataStartLiveTests 7.0
QueryStoreWideFleetEligibilityLiveTests 6.9
WebPerfmonPerSecondBehaviourTests 6.8
BoundedCompressionDdlTests 6.7
QueryStoreIntervalPartitionRungTests 6.7
PlanRegressionDailyTriggerLiveTests 6.5
QueryStoreIntervalWideGapCacheLiveTests 6.4
WebChartWidthBehaviourTests 6.4
ComposeQueryStoreRankedSingleScanLiveTests 6.3
ViewerConfigChangesDataStartTests 6.1
RetiredBaselineSweepRecoveryLiveTests 6.1
QueryStoreBackfillCutChunkLiveTests 6.0
FinOpsPostgresTargetTests 6.0
CollectorRunTimeViewerLiveTests 5.9
FinOpsRecommendationsReadsLiveTests 5.9
ProcedureStatsIdleHoursLiveTests 5.9
FleetCollectionHealthRecoveryLiveTests 5.8
DroppedDatabaseScopeLiveTests 5.6
FinOpsApplicationConnectionsViewLiveTests 5.4
RetentionCollectionYieldLiveTests 5.4
AnomalyGateNullWindowMonteCarloTests 5.3
AlertHistoryDismissTests 5.3
CollectorRunTimeCliVerbLiveTests 5.3
SensitiveStatementsTests 5.3
FinOpsIdleCoverageLiveTests 5.2
ProcedureStatsFloorAndHistoryLiveTests 5.2
McpArgumentTypeTests 5.1
QueryStoreIntervalMaintenancePassLiveTests 5.1
ProcedureTrendTotalsStitchLiveTests 5.1
Lite.Tests candidates (35)
Class Seconds
ConfigChangeAttributionPipelineTests 24.6
ArchiveQuerySnapshotsDailyCompactionTests 21.9
WebhookCooldownSeedTests 20.4
FactScorerTests 14.6
ArchiveCompactionLowDiskTests 14.0
DuckDbSentinelResetRaceTests 13.5
QueryWindowTruncationTests 12.6
QueryStoreDedupReadTests 12.3
DeadlockAndBlockedProcessEnsureLogLevelLiteTests 10.2
AnalysisCoverageTests 9.8
SystemEventsReaderTests 9.4
ArchiveCompactionSwapTests 9.1
CollectionHealthReadBehaviourTests 9.0
ArchiveCompactionSwapLockTests 9.0
LocalDatabaseReopenTests 8.7
LongQueryTraceReadOnlyIntentLiteTests 8.5
ServerTimeHelperCollectionTests 8.2
AnalysisAsOfAnchorTests 8.0
QueryStoreSliceRepairTests 7.4
DatabaseStateExpectedStoreTests 7.3
DataStartBannerBlockingEventsTests 7.3
QueryStoreTextAfterRankingTests 7.2
OverviewBlockingLaneDataStartLiveTests 7.1
DismissedArchiveSidecarTests 6.7
LatestValueLookbackTests 6.2
QueryStoreBackfillSkipFailingDatabaseTests 6.0
LiteMcpServerClockTests 6.0
EventTimeAnalysisReadsTests 6.0
IndexObjectStatsTests 5.8
ObjectLockingBudgetTests 5.6
ReplayedIndexRebuildTests 5.4
FactCollectorMiseryTests 5.3
StoredEventCopiesDeadlockTests 5.2
AnalysisServiceTests 5.2
McpServerLocalOffsetTests 5.1

Activity

  1. erikdarlingdata commented on Oct 16, 2026

    @erikdarlingdata
    OwnerAuthor

    Claude posting for Erik Darling. Adversarial review before any lane, at Erik's ask (an Opus reviewer re-derived the numbers from the run history and the repo).

    Verdict: floats with changes. Held for tomorrow, Darling half only.

    • In 33 of the last 60 dev merges the skip would have dropped live tests that run code the PR changed (998 such skips): all 101 Darling classes run shared-project code and 95 run migration files. 86 of the 101 were added inside the five-week window, so "has not failed in five weeks" mostly means "is new".
    • The Lite half saves 0 wall minutes: Lite shards finish 7 to 9 minutes before the Darling shards in every run read.
    • The Darling half is worth about 2 to 3 minutes per run.

    Change: skip a slow class only when the test map says the PR does not reach it (today the skip applies only to FULL selections, run-darling-pg-shard.ps1:85-112). Drop the Lite half.

  2. erikdarlingdata commented on Oct 16, 2026

    @erikdarlingdata
    OwnerAuthor

    Claude posting for Erik Darling.

    Closing as measured: the gate would skip nothing.

    The lane on #5967 built the gate the review asked for (a Cost=Slow Darling class is skipped on a pull request only when it has not failed in 35 days and the test map says the change does not reach it) and ran it against the last five merged pull requests (#5958, #5954, #5952, #5939, #5937) with the newest nightly map of 2,545 Darling classes.

    • With the 101 candidate classes tagged, it skipped 0 classes on every one of the five.
    • Three were FULL selections because the map could not speak for the change (a keep-full file such as build.yml). The gate must not fire there.
    • Two were already narrowed by the selection, so the gate had nothing to add.
    • The gate can fire only on a FULL selection the map can still speak about, which means a stale map or drift over the limit, and PR test selection takes the newest complete map, and a map is built mid-day (#5945) #5954 made those rare.

    The gate was reverted out of #5967, which now carries only #5955 (the test projects build in parallel). The Lite half was dropped at review. The shard-time lever on the pull request path is the selection and the build order, not this tag.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions