Skip to content

Commit c3291f6

Browse files
Seam repair walks newest-first and stops at the first failed range (#4186 round-3 H1)
A seam wider than one start's repair cap, or with two or more ranges, could move the successor's floor past a still-unrepaired seam bucket: oldest-first capping and repair let the OLDEST buckets close first, which drags the floor (a bare min(bucket)) down to them even while newer seam buckets in between stay holes, and RetentionArmSafetySql's probe stops looking above the new floor. The raw purge could then arm and drop rows neither rollup ever held. Fix: the seam window's holes are now scanned, capped and walked separately from the ordinary window's. The seam takes its cap from the newest end (closest to the successor's floor) and walks descending; the ordinary window is unchanged (oldest-first, same cap, same continue-past-a-remainder handling, since an interior repair can never move the floor). The seam walk stops at the first range that throws (propagates to the existing per-target catch) or leaves buckets standing (remaining > 0) rather than moving on to an older range. Both windows share one cap budget per aggregate per start, seam first, so "a start never re-materializes more for one aggregate than an ordinary policy run does" stays true. CapMaterializationHoleRepairs gains a newestFirst parameter (default false, unchanged behavior) that also splits a straddling range at its older edge instead of its newer one, so the kept portion stays adjacent to whatever is already materialized. Also corrects RetentionArmSafetySql's doc: it claimed an over-horizon seam "still gets repaired" and the gate "releases automatically, with no manual step" unconditionally. Both are true only when the raw purge was already held when the store stopped (#4299) and the outage did not cross the upgrade (#4300) respectively. Adds the H2-parity note: the stitch trusts the frozen legacy's own span without checking it, which can read Covered over a pre-upgrade hole nothing here repairs -- accepted because 3.8.0 loses the same rows on the same schedule, and the A6 backfill (--backfill-rollups) covers it for an operator who runs it. Two live tests against a real TimescaleDB rig: - a 35-bucket seam (over the 24-bucket cap) stays Short after one walk (top 24 repaired, bottom 11 still a hole) and reads Covered only after a second walk closes the rest; revert-proved by temporarily restoring oldest-first, which fails at the same assertion. - a two-range seam where the newer range's refresh is forced to raise 23514 via a CHECK constraint on its own materialization chunk: the older range is never touched and the gate still reads not-safe. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TszxYhJJbTEh4LrZ56NYo3
1 parent 35a0aef commit c3291f6

4 files changed

Lines changed: 361 additions & 23 deletions

File tree

‎Darling/Darling.Tests/FrozenRollupLiveTests.cs‎

Lines changed: 195 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -635,6 +635,201 @@ await LiveStoreCleanup.RunAsync(scratch.ConnectionString, bodySucceeded, async (
635635
}
636636
}
637637

638+
/// <summary>
639+
/// #4186 round-3 H1: a seam wider than one start's repair cap (24 hourly buckets) used to release the gate
640+
/// early. Oldest-first repaired the buckets FARTHEST from the successor's floor first, which still moved
641+
/// the floor (a bare <c>min(bucket)</c>) all the way down to them — stranding the un-repaired NEWER seam
642+
/// buckets above the new floor and outside <see cref="TimescaleSupport.RetentionArmSafetySql"/>'s probe.
643+
/// Newest-first must NOT do that: repairing the top 24 of a 35-bucket seam should leave the floor exactly
644+
/// adjacent to the still-open 11-bucket remainder, so the probe keeps finding it and the gate stays Short
645+
/// until a second walk closes the rest.
646+
/// </summary>
647+
[Fact]
648+
public async Task Outage_SeamWiderThanTheCap_NewestFirstRepairsTheTopAndKeepsTheGateHeldUntilFullyRepaired()
649+
{
650+
var baseConnectionString = Environment.GetEnvironmentVariable("DARLING_TEST_PG");
651+
Assert.SkipWhen(string.IsNullOrEmpty(baseConnectionString),
652+
"Set DARLING_TEST_PG to a Postgres connection string (with TimescaleDB installed) to run the live A6 freeze test.");
653+
654+
var ct = TestContext.Current.CancellationToken;
655+
656+
await using var scratch = await ScratchPostgres.CreateAsync(baseConnectionString!, ct);
657+
await using var connection = new NpgsqlConnection(scratch.ConnectionString);
658+
await connection.OpenAsync(ct);
659+
await PgMigrations.MigrateAsync(connection, ct);
660+
661+
var timescaleEnabled = await TimescaleSupport.TryEnableAsync(connection, null, ct);
662+
Assert.SkipWhen(!timescaleEnabled, "The live A6 freeze test needs TimescaleDB.");
663+
await TimescaleSupport.ConvertToHypertablesAsync(connection, null, ct);
664+
Assert.True(await TimescaleSupport.EnsureCollectionLogHypertableAsync(connection, null, ct));
665+
666+
await using (var stop = new NpgsqlCommand("SELECT _timescaledb_functions.stop_background_workers()", connection))
667+
{
668+
await stop.ExecuteNonQueryAsync(ct);
669+
}
670+
671+
await DarlingMcpTestData.RegisterServerAsync(connection, ServerId, ServerName, ct);
672+
await TimescaleSupport.EnsureContinuousAggregatesAsync(connection, null, ct);
673+
674+
var bodySucceeded = false;
675+
try
676+
{
677+
/* S is the stop. The legacy materializes only ONE bucket, 35 hours back, so l.mx = S-35h and the
678+
seam floor is S-34h. Raw carries an unbroken run of 35 hourly buckets from S-34h through S —
679+
wider than the 24-bucket cap, so ONE walk cannot close it in one pass. */
680+
var s = D0.AddDays(3);
681+
682+
for (var hour = 0; hour <= 35; hour++)
683+
{
684+
await InsertProcedureStatsAsync(connection, s.AddHours(-hour), $"seam35_proc_{hour}", 900, 9, 3600, ct);
685+
}
686+
687+
await RefreshAsync(connection, TimescaleSupport.ProcedureStatsHourlyView, s.AddHours(-35), s.AddHours(-34), ct);
688+
689+
var u = s.AddDays(2);
690+
await InsertProcedureStatsAsync(connection, u.AddHours(-1), "seam35_proc_successor_floor", 900, 9, 3600, ct);
691+
await RefreshAsync(connection, TimescaleSupport.ProcedureStatsIntervalHourlyView, u.AddDays(-1), u, ct);
692+
693+
Assert.False(await TimescaleSupport.IsRawTierDropSafeAsync(connection, "procedure_stats", ct));
694+
695+
/* Walk 1: the cap takes the NEWEST 24 of the 35 seam buckets (S-23h..S), leaving the OLDER 11
696+
(S-34h..S-24h) as a hole immediately below the new floor. The product's own start-path entry
697+
point (DarlingWorker calls this exact method). */
698+
var summary1 = await TimescaleSupport.RepairMaterializationHolesAsync(connection, null, u, ct);
699+
Assert.Equal(24, summary1.BucketsRepaired);
700+
701+
/* The regression this pins: with the old oldest-first order, this walk would instead have
702+
repaired S-34h..S-11h (the OLDEST 24) and left S-10h..S as holes ABOVE the new floor — outside
703+
the probe window entirely, and the gate below would have read true (Covered) after only one
704+
walk. */
705+
Assert.False(await TimescaleSupport.IsRawTierDropSafeAsync(connection, "procedure_stats", ct));
706+
707+
/* Walk 2: the remaining 11-bucket range is now the whole seam (S-34h..S-24h, under the cap), and
708+
closes it completely. */
709+
var summary2 = await TimescaleSupport.RepairMaterializationHolesAsync(connection, null, u, ct);
710+
Assert.Equal(11, summary2.BucketsRepaired);
711+
712+
Assert.True(await TimescaleSupport.IsRawTierDropSafeAsync(connection, "procedure_stats", ct));
713+
714+
bodySucceeded = true;
715+
}
716+
finally
717+
{
718+
await LiveStoreCleanup.RunAsync(scratch.ConnectionString, bodySucceeded, async (cleanup, cleanupCt) =>
719+
{
720+
await using var probe = new NpgsqlCommand(
721+
"SELECT count(*) FROM pg_catalog.pg_stat_activity WHERE datname = pg_catalog.current_database() " +
722+
"AND backend_type LIKE 'TimescaleDB Background Worker Scheduler%'", cleanup);
723+
var schedulers = Convert.ToInt64(await probe.ExecuteScalarAsync(cleanupCt));
724+
Assert.Equal(0L, schedulers);
725+
});
726+
}
727+
}
728+
729+
/// <summary>
730+
/// #4186 round-3 H1, the failure arm: a seam with TWO separate ranges, where the NEWER one fails. A CHECK
731+
/// constraint added directly to the successor's own materialization chunk (the mechanism verified against
732+
/// this rig's TimescaleDB before this test was written — <c>ALTER TABLE</c> against the continuous
733+
/// aggregate's VIEW or its parent materialization hypertable is refused, but a concrete chunk table takes
734+
/// one) makes the refresh over the newer range raise 23514, the same way a real refresh failure would.
735+
/// Newest-first must stop there: the older range must be left completely untouched, and the gate must
736+
/// still read not-safe, exactly as if neither range had been attempted.
737+
/// </summary>
738+
[Fact]
739+
public async Task Outage_SeamTwoRanges_NewerRangeFails_OlderRangeUntouchedAndGateStillNotSafe()
740+
{
741+
var baseConnectionString = Environment.GetEnvironmentVariable("DARLING_TEST_PG");
742+
Assert.SkipWhen(string.IsNullOrEmpty(baseConnectionString),
743+
"Set DARLING_TEST_PG to a Postgres connection string (with TimescaleDB installed) to run the live A6 freeze test.");
744+
745+
var ct = TestContext.Current.CancellationToken;
746+
747+
await using var scratch = await ScratchPostgres.CreateAsync(baseConnectionString!, ct);
748+
await using var connection = new NpgsqlConnection(scratch.ConnectionString);
749+
await connection.OpenAsync(ct);
750+
await PgMigrations.MigrateAsync(connection, ct);
751+
752+
var timescaleEnabled = await TimescaleSupport.TryEnableAsync(connection, null, ct);
753+
Assert.SkipWhen(!timescaleEnabled, "The live A6 freeze test needs TimescaleDB.");
754+
await TimescaleSupport.ConvertToHypertablesAsync(connection, null, ct);
755+
Assert.True(await TimescaleSupport.EnsureCollectionLogHypertableAsync(connection, null, ct));
756+
757+
await using (var stop = new NpgsqlCommand("SELECT _timescaledb_functions.stop_background_workers()", connection))
758+
{
759+
await stop.ExecuteNonQueryAsync(ct);
760+
}
761+
762+
await DarlingMcpTestData.RegisterServerAsync(connection, ServerId, ServerName, ct);
763+
await TimescaleSupport.EnsureContinuousAggregatesAsync(connection, null, ct);
764+
765+
var bodySucceeded = false;
766+
try
767+
{
768+
/* S is the stop. The legacy materializes one bucket 20 hours back (l.mx = S-20h, seam floor
769+
S-19h). Range A sits right above the seam floor — the OLDEST part of the seam. */
770+
var s = D0.AddDays(3);
771+
await InsertProcedureStatsAsync(connection, s.AddHours(-20), "seam2_proc_legacy_floor", 900, 9, 3600, ct);
772+
await RefreshAsync(connection, TimescaleSupport.ProcedureStatsHourlyView, s.AddHours(-20), s.AddHours(-19), ct);
773+
774+
await InsertProcedureStatsAsync(connection, s.AddHours(-19), "seam2_proc_a0", 900, 9, 3600, ct);
775+
await InsertProcedureStatsAsync(connection, s.AddHours(-18), "seam2_proc_a1", 900, 9, 3600, ct);
776+
777+
/* The successor's own first refresh sets floor = U-1h and, in the same call, creates the
778+
materialization chunk for that whole calendar day — the chunk range B (below) also falls in. */
779+
var u = s.AddDays(2);
780+
await InsertProcedureStatsAsync(connection, u.AddHours(-1), "seam2_proc_floor", 900, 9, 3600, ct);
781+
await RefreshAsync(connection, TimescaleSupport.ProcedureStatsIntervalHourlyView, u.AddDays(-1), u, ct);
782+
783+
/* Range B: two buckets below the floor, on the SAME calendar day as it, separated from range A by
784+
a multi-day gap where raw holds nothing (so A and B are two DISTINCT merged ranges, not one). A
785+
CHECK constraint on B's exact bucket span, added to the chunk the floor refresh just created,
786+
blocks it before it ever holds a row. */
787+
var chunks = await ReadMaterializationChunksAsync(connection, TimescaleSupport.ProcedureStatsIntervalHourlyView, ct);
788+
Assert.Single(chunks);
789+
await using (var block = new NpgsqlCommand(
790+
$"ALTER TABLE {chunks[0]} ADD CONSTRAINT seam2_block_b CHECK (bucket NOT BETWEEN '{u.AddHours(-5):O}'::timestamp AND '{u.AddHours(-4):O}'::timestamp)",
791+
connection))
792+
{
793+
await block.ExecuteNonQueryAsync(ct);
794+
}
795+
796+
await InsertProcedureStatsAsync(connection, u.AddHours(-5), "seam2_proc_b0", 900, 9, 3600, ct);
797+
await InsertProcedureStatsAsync(connection, u.AddHours(-4), "seam2_proc_b1", 900, 9, 3600, ct);
798+
799+
Assert.False(await TimescaleSupport.IsRawTierDropSafeAsync(connection, "procedure_stats", ct));
800+
801+
/* Newest-first tries B (closer to the floor) before A. B's refresh raises 23514, caught by this
802+
target's own per-aggregate catch — the walk must stop there, never reaching A. */
803+
var summary = await TimescaleSupport.RepairMaterializationHolesAsync(connection, null, u, ct);
804+
Assert.True(summary.Failures >= 1, $"expected B's constraint violation to be caught as a failure, got {summary.Failures}");
805+
806+
/* A is untouched: no row near it exists in the successor at all. */
807+
await using (var probeA = new NpgsqlCommand(
808+
$"SELECT count(*) FROM collect.{TimescaleSupport.ProcedureStatsIntervalHourlyView} WHERE bucket >= '{s.AddHours(-19):O}'::timestamp AND bucket < '{s.AddHours(-17):O}'::timestamp",
809+
connection))
810+
{
811+
Assert.Equal(0L, (long)(await probeA.ExecuteScalarAsync(ct))!);
812+
}
813+
814+
/* The floor never moved off U-1h — B's insert rolled back with its transaction, so both A and B
815+
still sit inside the probe window, and the gate must still read not-safe. */
816+
Assert.False(await TimescaleSupport.IsRawTierDropSafeAsync(connection, "procedure_stats", ct));
817+
818+
bodySucceeded = true;
819+
}
820+
finally
821+
{
822+
await LiveStoreCleanup.RunAsync(scratch.ConnectionString, bodySucceeded, async (cleanup, cleanupCt) =>
823+
{
824+
await using var probe = new NpgsqlCommand(
825+
"SELECT count(*) FROM pg_catalog.pg_stat_activity WHERE datname = pg_catalog.current_database() " +
826+
"AND backend_type LIKE 'TimescaleDB Background Worker Scheduler%'", cleanup);
827+
var schedulers = Convert.ToInt64(await probe.ExecuteScalarAsync(cleanupCt));
828+
Assert.Equal(0L, schedulers);
829+
});
830+
}
831+
}
832+
638833
/// <summary>
639834
/// #4186: the no-deadlock twin of <see cref="Outage_SeamBetweenFrozenLegacyAndSuccessor_HoleWalkRepairsItAndGateReleases"/> —
640835
/// same frozen-legacy-plus-outage shape, but the seam itself holds NO raw rows (an outage that began

‎Darling/Darling.Tests/MaterializationHoleRepairTests.cs‎

Lines changed: 38 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -202,6 +202,39 @@ public void TheCap_TakesTheOldestFirst_SplitsAStraddlingRangeExactly_AndDefersTh
202202
Assert.Throws<ArgumentNullException>(() => TimescaleSupport.CapMaterializationHoleRepairs(null!, 24, width));
203203
}
204204

205+
/// <summary>
206+
/// #4186 round-3 H1: the seam window's own direction. Mirrors
207+
/// <see cref="TheCap_TakesTheOldestFirst_SplitsAStraddlingRangeExactly_AndDefersTheRest"/> exactly, with
208+
/// <c>newestFirst: true</c> — same three ranges, same cap, but the NEWEST 24 buckets are kept and a
209+
/// straddling range splits at its OLDER edge instead of its newer one, so the kept portion stays adjacent
210+
/// to whatever sits above it (the successor's already-materialized span in the real caller).
211+
/// </summary>
212+
[Fact]
213+
public void TheCap_NewestFirst_TakesTheNewestFirst_SplitsAStraddlingRangeAtItsOlderEdge_AndDefersTheRest()
214+
{
215+
var width = TimescaleSupport.HourlyBucket;
216+
var ranges = new List<(DateTime Start, DateTime End)>
217+
{
218+
(Hour.AddHours(30), Hour.AddHours(40)), /* newest, 10 buckets */
219+
(Hour, Hour.AddHours(20)), /* oldest, 20 buckets */
220+
(Hour.AddHours(22), Hour.AddHours(28)), /* middle, 6 buckets */
221+
};
222+
223+
var (repair, deferred) = TimescaleSupport.CapMaterializationHoleRepairs(ranges, 24, width, newestFirst: true);
224+
225+
/* Newest 10 whole, then the middle's 6 whole (16 spent), then 8 of the oldest's 20 — split at its NEWER
226+
edge, since newestFirst keeps the portion adjacent to what is already above it (24 = 10 + 6 + 8) —
227+
the older remaining 12 hours of the oldest range deferred. */
228+
Assert.Equal(new[] { (Hour.AddHours(30), Hour.AddHours(40)), (Hour.AddHours(22), Hour.AddHours(28)), (Hour.AddHours(12), Hour.AddHours(20)) }, repair);
229+
Assert.Equal(new[] { (Hour, Hour.AddHours(12)) }, deferred);
230+
Assert.Equal(24, repair.Sum(r => (int)((r.End - r.Start).Ticks / width.Ticks)));
231+
232+
/* Under the cap and exactly at it: everything repaired, nothing deferred, same as oldest-first. */
233+
var small = new List<(DateTime Start, DateTime End)> { (Hour, Hour.AddHours(2)) };
234+
Assert.Equal(small, TimescaleSupport.CapMaterializationHoleRepairs(small, 24, width, newestFirst: true).Repair);
235+
Assert.Empty(TimescaleSupport.CapMaterializationHoleRepairs(small, 24, width, newestFirst: true).Deferred);
236+
}
237+
205238
[Fact]
206239
public void AlignDown_LandsOnABucketBoundary_ForBothWidths()
207240
{
@@ -408,8 +441,11 @@ and per-failure lines stay — they are the detail the summary counts. */
408441
Assert.DoesNotContain("aggregate(s) scanned", storage, StringComparison.Ordinal);
409442
Assert.DoesNotContain("no holes.", storage, StringComparison.Ordinal);
410443
Assert.Contains("passClock.Elapsed", storage, StringComparison.Ordinal);
411-
Assert.Contains("holesFound += ranges.Count;", storage, StringComparison.Ordinal);
412-
Assert.Contains("bucketsFound += holes.Count;", storage, StringComparison.Ordinal);
444+
/* #4186 round-3 H1: found is now the seam and ordinary windows' ranges/buckets summed, since the two
445+
are scanned and capped separately (the seam newest-first, the ordinary oldest-first) rather than
446+
merged into one list before counting. */
447+
Assert.Contains("holesFound += seamRanges.Count + ordinaryRanges.Count;", storage, StringComparison.Ordinal);
448+
Assert.Contains("bucketsFound += seamHoles.Count + ordinaryHoles.Count;", storage, StringComparison.Ordinal);
413449
Assert.Contains("had {Buckets} bucket(s) in [{Start}, {End})", storage, StringComparison.Ordinal);
414450
Assert.Contains("left for the next start", storage, StringComparison.Ordinal);
415451
Assert.Contains("could not scan or repair {View} this start", storage, StringComparison.Ordinal);

0 commit comments

Comments
 (0)