Repository navigation
Lite: the size-triggered archive reset pauses collection for its whole length #5733
Description
Claude posting for Erik Darling.
Split from #5731. #5731 moves Lite's housekeeping off the collection loop, but one step it cannot move without a redesign.
What happens. When the local database reaches 512 MB, Lite runs a size-triggered reset (ArchiveService.ArchiveAllAndResetAsync). It exports every table to parquet under the store's write lock and holds CollectionResetGate, so every collection round waits at RemoteCollectorService.BeginCollectionAsync until the reset ends. Collection pauses for the whole reset, and the slots it covers are skipped. After #5730 the skip report names the reset, but the gap is still there.
What to find out first. How long a reset takes on a real store: its length against the database size and table count, and how often it fires. A reset that takes a few seconds once a day may need nothing more than the report. One that takes minutes needs a redesign: for example, exporting under the read lock and taking the write lock only for each table's swap, as periodic archival already does.
Component. Lite. Darling: not affected (no size-triggered reset; retention runs as TimescaleDB jobs).
Activity
Reset gate docs: the gate is not held through compaction, and the exa…
Lite's size-triggered reset no longer holds collection while it compa…
erikdarlingdata commentedon Oct 13, 2026
Claude posting for Erik Darling.
Fixed by #5771, merged to dev as 9c48bac.
Measured first: on a field Lite store, 38 size-triggered resets over 16 days paused collection for 7 to 194 seconds each. The time did not follow the row count. It dropped to about 7 s at the start of the month and grew by 4 to 5 s a day.
The cause: after the database was reset, the reset went on to compact the archive while still holding the gate collection waits on. Compaction merges each month's new files with that month's existing archive file, so it grows through the month. The reset now releases the gate as soon as the fresh database is ready and compacts afterwards. A new log line, "Database reset held collection for N s (T tables, R rows)", records the remaining pause.
The export itself still runs inside the gate. At the start of the month, the export, swap and reopen together took 7 to 15 s, which is less than one 60 s collection slot. If the new log line shows holds over about 30 s, the next step is to export most rows before taking the gate.