Require a population large enough for a maximum to bound anything (#3193)

[erikdarlingdata]

Closes #3193.

RefreshCeilingProvenancePinTests required each ceiling constant to equal the maximum of a population its doc comment publishes, and had no opinion about whether that population could support a maximum. Both constants could be re-derived DOWNWARD on a handful of runs and satisfy every relationship in the file.

What was measured before choosing a floor

Both halves of #3193's claim were tested against the shipped file rather than reasoned about, and they do not agree with each other.

• OtherHourlyRefreshObservedCeilingSeconds: The ceiling provenance pin accepts any population size, so a constant can be TIGHTENED on three observations with nothing going red #3193 is right. Its census cut from 874 runs to 4 — the count sentence and the filter-control sentence rewritten consistently — was green across the whole file. That population is deliberately not republished; only its count is, so every figure the paragraph states is consistent with any count whatever.
• HeaviestHourlyRefreshObservedCeilingSeconds: The ceiling provenance pin accepts any population size, so a constant can be TIGHTENED on three observations with nothing going red #3193 is wrong that nothing goes red, and the detail matters. A consistent 4-run re-derivation with the constant tightened to 697 trips the pre-existing NearestRank(sorted, 95) < max clause, because below 20 draws the nearest-rank 95th percentile IS the maximum. But that clause is a guard on the estimator paragraph's premise, not on the population, and its message says "the sample-size half of the estimator argument is live again. Re-read HeaviestHourlyRefreshObservedCeilingSeconds belongs to neither regime: too fast for the wide window, too slow for the narrow one, and the clean maximum is ~594s #3101's trade rather than editing this assertion" — which sends the reader to re-take the maximum-versus-percentile decision, never mentions population size, and never mentions that tightening is the direction that costs. It also sits behind the summary clauses in a fail-fast chain, so it is reached only after every other prose figure has been made consistent.

Also worth recording: the published lists are pinned as [0-9]+(?:, [0-9]+)+, so the smallest publishable heaviest population is four runs, not three.

The floor, and why this one

PopulationFloor is the first n at which the nearest-rank 95th percentile of an ascending population is a different reading from that population's maximum. It computes to 20, and it is searched with NearestRank itself rather than written down — change the percentile the summaries quote, or the definition of nearest rank, and the floor moves with them. It is also asserted minimal: at one draw below it the 95th percentile must BE the maximum, so it cannot be quietly raised either.

The derivation is the constants' own estimator argument. Both summaries take the maximum rather than a percentile on the ground that the two answers have DIVERGED. Below 20 draws that divergence cannot exist at any values, so a population that thin cannot support the argument its own doc makes for taking a maximum over it.

Why this beats the other two candidates #3193 weighed:

• A minimum span in whole days needs new extraction. The heaviest paragraph states its per-day run counts as spelled-out words (nine, twenty-two, twenty-four) and the light one says over three days, all deliberately invisible to Numbers(). It would also not have blocked the defect: The ceiling provenance pin accepts any population size, so a constant can be TIGHTENED on three observations with nothing going red #3193's three runs sit inside a 23:00–01:00 window that straddles two calendar days.
• Midnight-regime coverage does not block the defect at all, and this is the one place The ceiling provenance pin accepts any population size, so a constant can be TIGHTENED on three observations with nothing going red #3193's ranking has to be reversed. Those three runs are from the 23:00–01:00 midnight window, so a midnight-coverage floor accepts precisely the population the issue says must be rejected. It is a composition condition and the defect is a size defect. Worse, for the constant whose doc supplies the midnight mechanism it would be a clause that cannot fail: Verify already requires the light maximum to BE the 00:30 run its paragraph names, with an earlier midnight run strictly below it, and the date is inside that pin's own pattern — so a population with no midnight in it is already a parse miss. For the heaviest ceiling the mechanism does not transfer: its published population is a list of durations carrying no clock, and its summary attributes its maximum to a per-day growth trend rather than to an hour of the day.

It is a conjunction, and the other half was already here

The honest requirement is size AND composition. The two fail differently and neither implies the other — a run count can be met by a hundred draws from one quiet stretch, and midnight coverage can be met by a two-hour population. But the composition conjunct was already in Verify at both sites, in the form each constant's own evidence can support:

• the heaviest ceiling's population must hold runs from the boundary day AND from at least one day after it (the two non-empty requirements on its separately published halves), so it spans two calendar days at minimum;
• the light ceiling's maximum must BE the midnight run its paragraph names, which is stronger than requiring the population to merely contain a midnight.

So what neither site had was the size conjunct, and that is all this adds. The added strictness is priced rather than hidden: any future census under 20 runs is now rejected, including an honest early reading of a fresh regime — which is why the failure message's whole job is to tell that reader to widen the population rather than to lower the floor or move the constant.

Where it is asserted, and what the message says

At the two points where a run count is already in hand — the heaviest population's parsed length and the light census sentence's third captured group — so this is an assertion, not a new extraction. It is placed ahead of the derivation clauses at both sites: Verify is fail-fast, and clause order decides whether the next reader is told to move the constant or to widen the population. On a population re-derived downward those are the same edit read two ways, and the wrong one of them is the cheap one.

The message states that a maximum over n draws is a prefix maximum and therefore a weak lower bound; that the weakness has a direction, because loosening a sizing constant on thin evidence is conservative while tightening one narrows the clearance the compression grid is asserted against; and that the repair is one of exactly two things — widen the population and re-derive, or leave the constant where it is. It also names, per constant, what widening actually means: extending the published lists for the heaviest, a fresh read at a later instant for the light one, whose population is not republished.

The stated bound, said plainly rather than left looking covered

The two sites are not equally well guarded, and the difference is inherent rather than deferred.

The heaviest ceiling's floor reads a parsed population — population.Length over the readings the comment actually lists — so it cannot be satisfied by a claim. The light ceiling's floor reads its census count, which is a number in prose over a population the doc deliberately does not republish. So a re-derivation that states over 900 runs while having read four satisfies this floor.

That gap cannot be closed here. OtherHourlyRefreshObservedCeilingSeconds' own summary declines to republish its 874 readings, with a reason — "a list that long stops being re-derivable by reading and starts being a wall of digits" — and re-deriving the estimator from a transcription of 874 numbers would trade this weakness for a worse one. What the floor buys at that site is that the count is now load-bearing: it was previously consistent with any value at all, and a thin census had to be asserted as wide before it can pass. Corroborated, not proved, by the exclusion-control sentence, which states the same number a second time.

Stated here because the alternative is a PR that reads as if both sites were closed to the same degree.

Red-proofing

EveryNumericPin_ReportsAnInjectedDrift enumerates Pins() and bumps captured numbers, so it covers extractions. The floor adds no pattern and consumes no pin — a digit bump moves a stated value and never a population's size — so it is unreachable from that sweep, and no equivalent harness for a non-pin assertion exists beyond the file's own idiom: a dedicated mutating fact, which is what TheInadmissibleReadings_CannotBeFoldedIntoTheStatusVerifiedSeries, TheInadmissibilityClaim_MatchesTheShippedSql and TheOpenPopulationGuard_ReportsEachForbiddenShape already use. ThePopulationFloor_ReportsAPopulationTooThinToSupportAMaximum follows it: baseline verified green first and again at the end, the floor pinned minimal in both directions, both real populations required to clear it, and both sites mutated on copies with the failure required to carry a PIN POPULATION FLOOR marker — because shrinking a population falsifies several claims at once, and without the marker the test would pass on whichever of them Verify reached first.

Mutations run against the real TimescaleSupport.cs on disk, each confirmed present by content and restoration confirmed by content:

#	mutation	result
0	none (baseline)	green, 11/11
1	heaviest published lists cut to two readings each (4 runs)	red — floor fires, names the heaviest constant, 4 runs, under the 20-draw floor
2	restored	green, 11/11
3	light census 874 → 4	red — floor fires, names the light constant (this exact mutation was green 10/10 before this change)
4	restored	green, 11/11
5	floor derivation weakened (95th → 50th, collapsing the floor to 2)	red — minimality assertion: "at 2 draws the nearest-rank 95th percentile is still the maximum"
6	light site's floor call deleted	red — the light mutation no longer carries PIN POPULATION FLOOR
7	restored	green, 11/11

Run on macOS by compiling the real shipped RefreshCeilingProvenancePinTests.cs into a net10.0 xunit.v3 harness named Darling.Tests, so InternalsVisibleTo applies and [CallerFilePath] still resolves to the real TimescaleSupport.cs. PopulationFloor = 20 and the 19-versus-20 boundary were read out of the shipped NearestRank by reflection rather than from a transcription of its formula.

Base

Based on dev at 723ae0cbe. PR #3190 is still open and touches the same file; the two conflict in exactly one place — both insert a <para> at the same anchor in the class summary, and everything else auto-merges. Resolution is to keep both paragraphs, once, after #3190 lands.

CHANGELOG entry text — deliberately not in this diff

CHANGELOG.md is not part of this change, and that is the lane convention rather than an omission. Several lanes are working this queue concurrently and every one of them would append to the same [Unreleased] block, so a per-PR edit turns a batched, conflict-free append into N conflicting ones. The entry text is therefore reported here for the coordinator to batch, which is where the reference link gets added with it.

Under Changed:

RefreshCeilingProvenancePinTests now requires each hourly-refresh ceiling's published population to clear a floor derived from its own nearest-rank percentile definition, so a sizing constant cannot be tightened on a population too thin for a maximum over it to bound anything (#3193).

[claude]

Reviewed. This PR only touches Darling/Darling.Tests/RefreshCeilingProvenancePinTests.cs (test-only, +303/-0) — no production code, SQL, or migrations, so there's nothing here that touches the T-SQL style rules or introduces a Lite/Darling parity gap (TimescaleSupport is PostgreSQL/TimescaleDB-specific to Darling; Lite has no counterpart to keep in sync).

Traced the new logic against the base commit's diff by hand:

• DerivePopulationFloor / NearestRank math checks out. NearestRank uses rank = (95n + 99) / 100 (integer ceiling division), 1-based. For n = 19, rank = 19 = n, so the 95th percentile equals the max — floor not yet reached. For n = 20, rank = 19 < 20, so it diverges. That confirms the floor really does compute to the smallest n where max-vs-95th-percentile disagree, matching the PR's claim of 20.
• Fail-fast clause ordering is correct at both call sites. For the light ceiling in particular, if RequireEnoughDrawsForAMaximum were placed even one clause later, a census shrunk to 4 runs would trip the exclusion-count consistency check (lightKept[0] == lightRuns) instead of the floor, with a misleading message ("two sentences are about different reads") rather than the intended "population too thin" one. The PR puts the floor check immediately after lightRuns is read, before that clause — verified this is necessary, not just tidy.
• ShrinkPublishedPopulation's shape-based regex is scoped correctly. It only touches <c>...</c> blocks with 3+ comma-separated integers within the ceiling's own doc-run span. Checked the surrounding doc comment by hand for other bracketed number lists that could collide (e.g. the snapshot-reading parenthetical (342 s, 348 s, 286 s)) — none are wrapped in <c> tags, so there's no risk of the mutation helper shrinking the wrong list.
• The light-site mutation's ordinal arithmetic (firstCaptured + 2) is right — the three numbers in "the maximum is X.Y s over N runs" are consecutive in the doc run with no intervening unrelated digits, so the offset lands on the run-count capture group, and the test re-verifies this by parsing the mutated prose back rather than trusting the arithmetic.

No correctness, security, or performance concerns. The heavy prose/ALL-CAPS comment style matches this file's and CONTRIBUTING.md's already-established convention ("comments explain WHY, at length"), so no style flags either.

[claude]

Review

Scope: this PR touches exactly one file, Darling/Darling.Tests/RefreshCeilingProvenancePinTests.cs — a Darling-only test file with no Lite counterpart (TimescaleSupport is a Postgres/TimescaleDB concept that doesn't exist in Lite's DuckDB storage layer), and no T-SQL, so the collector/style/parity conventions in CONTRIBUTING.md don't apply here.

Correctness of the new logic — checked out independently rather than taken on faith:

• NearestRank's rank formula is rank = ceil(percentile * n / 100) via (percentile*n+99)/100. ceil(0.95n) == n exactly when n < 20 (since that requires 0.05n < 1), and first fails to equal n at n == 20. So DerivePopulationFloor()'s search correctly lands on PopulationFloor == 20, matching the PR description, and the minimality assertions in ThePopulationFloor_ReportsAPopulationTooThinToSupportAMaximum are consistent with that derivation.
• The new RequireEnoughDrawsForAMaximum call sites are placed before their respective max == Ceiling derivation clauses in Verify, as the fail-fast-ordering comment says they must be, and both real populations (57 heaviest, 874 light) clear the floor by a wide margin, so this doesn't risk turning the suite red on the current tree.
• The EveryNumericPin_ReportsAnInjectedDrift sweep only bumps individual captured digits width-preserving; it never changes a published list's length or the light census run-count in a way that would drop below the floor, so the new checks don't interfere with that sweep's existing mutation cases.
• No production code changed — both HeaviestHourlyRefreshObservedCeilingSeconds and OtherHourlyRefreshObservedCeilingSeconds already sit well above the new floor, so this is a pure test-hardening change with no runtime behavior difference.

I don't see a correctness bug, a parity gap, or a security/performance concern here.

One process gap: the PR description includes a fully-written "CHANGELOG entry text" section, but CHANGELOG.md isn't part of the diff (gh pr diff --name-only shows only the test file). Every prior hardening pass on this same file (#3133, #3107, #3166, #3174, #3182, etc.) has a corresponding ### Changed entry under [Unreleased] with a [#NNNN] reference link at the bottom of the file. Worth adding that entry (with a [#3193] link) before merge to keep the established convention, rather than leaving it as prose in the PR description for someone else to transcribe later.

[claude]

Reviewed. This is a test-only change to Darling/Darling.Tests/RefreshCeilingProvenancePinTests.cs (no Lite counterpart exists for this file/concept, so parity is not a concern here), and no T-SQL is touched.

Correctness — verified by hand:

• DerivePopulationFloor's search over Enumerable.Range(1, draws) correctly reproduces the nearest-rank formula (rank = ceil(95·n/100), 1-based). For n=19, rank=19=n, so the 95th percentile always equals the max regardless of values; at n=20, rank=19<n, so it can diverge. PopulationFloor == 20 is right, and the minimality assertion in ThePopulationFloor_ReportsAPopulationTooThinToSupportAMaximum checks exactly that boundary.
• RequireEnoughDrawsForAMaximum is inserted as the first population-size-reading clause in Verify at both call sites (heaviest: population.Length right after tail/after are parsed and concatenated; light: lightRuns right after lightCensus is read), both ahead of the existing derivation clauses — matches the fail-fast-ordering claim in the PR description.
• ShrinkPublishedPopulation's regex ({2,}, i.e. 3+ comma-separated numbers inside a <c>…</c> block) only matches the two real published lists in the file today (confirmed no other <c> block with 3+ comma-separated numbers exists in TimescaleSupport.cs), so it isn't at risk of a false match elsewhere in the doc run.
• The light-ceiling mutation path (OrdinalOfFirstNumberInGroups + RewriteNumberInDocRun with firstCaptured + 2) correctly lands on the run-count group ("the maximum is 226.8 s over 874 runs of …" → group 3), since there's no intervening digit run between the captured groups.
• Both new-floor failure paths are asserted to carry the PIN POPULATION FLOOR marker and name the correct constant, and the source is re-verified clean after both mutations — good non-vacuity coverage.

One minor (non-blocking) note: the floor is a necessary condition derived from a strictly-ascending synthetic population, not a guarantee for every real population of size ≥20 — if the two largest real readings were ever tied, nearest-rank 95th could still equal the max at n=20. That's an inherent property of a count-only floor (the PR text already acknowledges the floor "cannot see composition"), not a code bug, and not worth guarding given real timing data essentially never ties exactly.

No functional or security concerns found.