Repository navigation
perf(migration): bound processing snapshot memory while preserving frozen digests #2156
Description
Problem
Follow-up to #2084 R12. At master 88899876b2ac7b38a11420f34f4ecf6da14dd65b, _processing_snapshot in src/powercontext/builtin/persistence/migrations/atomic_memory_v1.py retains canonical hex-encoded JSON rows to sort and hash processing tables, including pc_sources.
A local CPython 3.14/SQLite tracemalloc experiment with 64 KiB synthetic payloads per row measured:
| Raw payload bytes | Existing Python allocation peak |
|---|---|
| 8388608 | 17223854 |
| 33554432 | 67634734 |
| 67108864 | 134837599 |
This measures one snapshot function, not full migration RSS or a maximum deployment memory requirement. Pre/post snapshots run sequentially.
Replacing buffered SQL execution with streaming while retaining the encoding and sorting produced the same digest but increased the 64 MiB case to 135376917 bytes. Do not treat streaming alone as a demonstrated fix.
Required design and implementation
- Establish an explicit working-set budget and failure behavior before starting destructive migration work.
- Preserve the frozen digest definition: canonical full JSON bytes sorted lexicographically. Primary-key order or sorted per-row hashes are not compatible substitutes.
- Evaluate bounded-memory computation against the measured encoded-row working set.
- If external storage is proposed, explicitly decide content confidentiality, permissions, cleanup on crash, and disk capacity before storing raw Source material outside its database. No implicit plaintext spill.
- Keep read-only planning and the existing migration evidence/recovery model intact.
Acceptance criteria
- Golden fixtures produce byte-identical digests, including BLOBs, NULLs, non-ASCII values, and rows whose JSON order differs from primary-key order.
- Measure allocation peak and process RSS across increasing payload sizes and state the supported budget.
- Cancellation, resource exhaustion, and restart do not report a verified migration or leave accepted partial evidence.
- A failed resource preflight does not mutate the database.
Non-goals
Skipping embedding during force projection repair. Matching metadata and vector length cannot establish that a vector is correct; the existing force repair contract must remain able to repair same-length corruption.
Activity
AlexStocks commentedon Oct 21, 2026
Additional feasibility evidence for #2084 R12 / #2156, using CPython 3.11.15
under WSL Linux, real SQLite and synthetic data only.
I evaluated an in-memory alternative that streams 16 rows, retains level-1 zlib
compressed canonical JSON, sorts by comparing decompressed complete JSON bytes,
then hashes decompressed bytes in that order. This avoids plaintext spill and
preserves the frozen ordering, but trades memory for repeated decompression.
It is an experimental external probe, not a production patch.
Each case ran in its own subprocess. In the 64 MiB random case, process maximum
RSS was 292,708,352 bytes for the baseline and 198,881,280 for the compressed
variant; their pre-snapshot maximum RSS values were 89,919,488 / 89,903,104.
These are whole-process high-water measurements, not incremental snapshot RSS
or complete migration memory. The data are local single observations, not
latency promises.
All six payload pairs and an additional fixture containing BLOB, NULL, CJK,
decomposed Unicode, empty bytes and JSON text produced identical digests and
validation errors. The fixture digest is
461721ed81b06924e6450f6d6736de3ff1932ec100a9246bb37fd21fa47c1b38.The inspected baseline function source SHA-256 is
10079a919d59073c6ba1b6d9c6487d10aa65ec42c135e00b89ffa238f0796933;source is unchanged in the local eight-branch integration tree.
Decision: do not submit this as the bounded-memory solution. Random-input
working set still grows linearly; the 64 MiB case is about 9.8 times slower.
Compression is not an admission budget, and favorable repeated-byte results
must not be generalized to real Source payloads.
The required next implementation still needs a defined supported working-set
budget, a pre-mutation failure boundary, and cancellation/exhaustion recovery.
If an external sort is chosen, its confidentiality, ephemeral key ownership,
disk budget and crash cleanup must be explicit before any Source bytes spill.
The experiment does not change the CLI, database, frozen digest or force repair.
AlexStocks commentedon Oct 21, 2026
Implemented in #2172 (
159c784c10274b4a54aae3c5171c014d438004e6); its current-Head CI checks have completed without failures and review/integration remain pending.The default 256 MiB per-table conservative working-set admission estimate is checked with database-side scalar aggregates before payload materialization.
--snapshot-budget-mibis an explicit offline override. SQLite apply first performs the read-only preflight before writable-open/journal changes. Existing full canonical-JSON byte sorting and digest semantics are preserved; there is no plaintext spill.The standalone and nine-branch migration suites each passed 89 cases with 88 external-backend variants explicitly skipped locally. For a 64 MiB random-payload fixture, default-budget rejection had about 23 KiB of Python peak allocation; a 512 MiB override succeeded with the unchanged baseline digest. This is an admission estimate for the snapshot working set, not a hard total-process RSS cap. Forced projection repair still re-embeds rather than trusting metadata/length checks.