[AlexStocks]

Which issue or RFC does this PR close?

Closes #2156. Refs #2084 (R12).

Rationale for this change

Frozen migration snapshots sort complete canonical JSON rows, including Source payloads. Streaming the database result alone still retains a linear sorting buffer. An offline migration needs to reject an oversized working set before accepting destructive work while preserving the existing evidence digest.

What changes are included in this PR?

• Admit each processing table using database-side scalar byte counts, row count and largest-row size before reading payloads. Reserve for JSON expansion, row objects and encoding intermediates; check retained encoded bytes during streaming as well.
• Default to a 256 MiB processing snapshot budget and expose --snapshot-budget-mib on migration plan/apply/verify. Initial rejection returns ready: false without a snapshot digest. SQLite apply preflights through a read-only connection before writable opening can change journal mode.
• Retain the frozen digest definition: full canonical JSON bytes sorted lexicographically. No plaintext temporary-file spill, compression or hash substitution.
• Document the budget, explicit override and recovery in both locales. Force projection repair and its embedding behavior are unchanged.

Are there any user-facing changes?

Maintenance inputs exceeding the conservative default estimate are rejected. Operators can explicitly raise the budget after measuring available memory on a stopped backup copy. The budget is per processing table, not a total-process RSS hard cap; legacy inventory, embedding, database driver buffers and other maintenance allocations remain outside it. All writers must remain stopped throughout maintenance.

How was this change tested?

• pytest tests/builtin/persistence/test_atomic_memory_snapshot.py tests/test_atomic_memory_migration_cli.py tests/e2e/test_atomic_memory_migration_evidence.py tests/e2e/test_atomic_memory_grant_migration.py tests/e2e/test_atomic_memory_migration_vector.py tests/e2e/test_atomic_memory_projection_rebuild.py -q: 89 passed, 88 skipped on Linux/Python 3.11.15.
• Full ty check --python /tmp/pc2167-venv --python-version 3.11 --python-platform linux: passed.
• Changed-file Ruff lint/format, pre-commit checks and git diff --check: passed. The native Windows Ty hook was replaced by the complete Linux check because the baseline has POSIX-only typing diagnostics on Windows.
• RED: the new budget CLI test fails on the base implementation because --snapshot-budget-mib is unrecognized. GREEN: byte-preservation and explicit larger-budget retry cases pass with the implementation.

The first complete run found a test environment setup gap: the dependency-only Linux virtual environment had no installed PowerContext metadata, so 16 Server-startup tests failed with PackageNotFoundError. Installing the project without changing runtime dependencies resolved that setup error; the complete rerun above passed. The WSL install uses synthetic package version 0.0.0 because Windows worktree Git paths are not valid Linux Git paths.

CPython 3.11.15/Linux, real SQLite, independent subprocess per synthetic random-payload case:

Raw payload	Variant	Budget	Result	Python peak bytes	Process maxRSS bytes
8 MiB	original	none	accepted	17150023	115519488
8 MiB	budget	256 MiB	accepted	17150899	106045440
32 MiB	original	none	accepted	67574791	193064960
32 MiB	budget	256 MiB	accepted	67541779	157925376
64 MiB	original	none	accepted	134808256	294240256
64 MiB	budget	256 MiB	rejected before payload read	22685	90066944
64 MiB	budget	512 MiB	accepted	134730060	225460224

All admitted pairs have byte-identical digests. The 64 MiB rejection estimates 408104188 bytes and returns no digest. This measures predictable admission, not constant-memory sorting or production backend performance.

The golden fixture covers BLOB/NULL/CJK/decomposed Unicode and JSON ordering different from identity ordering. CLI regressions exercise plan/apply/verify with ordinary paths and SQLite file URIs, byte-for-byte database preservation on rejection, and successful explicit larger-budget retry. Fault injection exercises allocation failure/cancellation before initial snapshot acceptance and successful restart without archived or imported partial evidence.

External backend acceptance is left to the real OceanBase/SeekDB CI jobs; local skipped cases do not establish those results.

AI usage statement

Codex was used to implement and review this change and run the reported validation.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.

[hidb4ai]

LGTM

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.