Repository navigation
--session-state-codec bf16: a saved/parked conversation's running state at half the bytes - #2247
Closed
sumomomomomo wants to merge 18 commits into
Closed
sumomomomomo wants to merge 18 commits into
sumomomomomo wants to merge 18 commits into
Conversation
…alf the bytes (the file says which codec it used; the live state stays fp32)
--serve keeps a conversation's running state - the GDN recurrence, the PLE history, the QSA indexer's tails and dead
keys - as fp32 wherever it stores it: 118,063,152 B per state, once for the live conversation and once per 16,384
tokens, in parked images and in session files beside the K/V. --session-state-codec bf16 (an engine argument, default
fp32) stores those arrays as bfloat16 instead. bf16 is the top half of a float32, so the encoder is a shift and a
round-to-nearest-even and the decoder is the exact shift the repo already uses for BF16 rows
(include/strata/kernels/ngram.hpp:134): no scale to choose, no value that can overflow, no subnormal cliff, and every
value lands within 2^-8 (0.39%) of itself. block_pos is int32 positions and is never coded.
Measured on one Tesla V100 32 GB plus one RTX 2080 Ti, 93 GB RAM, Linux, CUDA 12.8, IQ3_S, --kv int8, 262144 context,
needle-bench filler read once and answered greedily, the same conversation both ways:
63,459 tokens 1,205,289,012 B -> 1,087,225,924 B (-9.8%) save_ms 546.4 -> 503.2 restore_ms 759.5 -> 714.3
32,650 tokens 734,789,228 B -> 616,726,140 B (-16.1%) save_ms 444.1 -> 315.5 restore_ms 468.5 -> 400.3
37 tokens 236,738,756 B -> 118.7 MB (-50%)
The saving is the checkpoint spacing and not the length, so short conversations gain the most. Coding costs nothing on
either side: half the state's bytes never reach the disk, and the same 63,455-token prompt read in 44.8 s configured
fp32 against 44.6 s configured bf16, so nothing landed on the token path.
Over the 29,417,472 floats of a 63K conversation's recurrent state - bf16 encode and decode of the file's own values,
no model and no GPU in the number - the worst value moves 0.3891% (the bound is 0.3906%) and the typical one 0.166%,
with no sign flips and none of them going to zero. Two separate fp32 runs of this engine move that same state 26 times
further, which is the adaptive expert tier and a shared PCIe link moving the rounding (Niko1221#152, Niko1221#410; DETAILS.md names
--adapt-swaps 0 --pcie-frac 0 as the way to get repeatable runs). With those pinned, two runs leave states differing by
exactly the codec and nothing else, and one engine restored both files, alternating twice each: the same text on three
of four continuations, and the fourth - free creative generation - varied between repeats of the uncoded file too.
Needle recall at 32K, depths 10/50/90, asked after a save and a restore so the answer comes off disk: 3 of 3 fp32,
3 of 3 bf16. A coded file restores on an engine configured fp32 and a version-1 file on one configured bf16, both
answering right.
A restore widens the state back to fp32 before the engine uses it, so the live checkpoint chain, the kernels and every
consumer of a resumed conversation are untouched. Which codec was used is recorded per conversation in the file, which
is written as version 2 only when something in it is coded: an unknown codec code is refused, a build older than this
refuses v2 by name, and v1 files read exactly as before. With the flag off nothing changes - the file a 63,459-token
conversation writes is byte-identical to the one a0116b2's engine wrote for the same conversation, same 232-byte
header, same token ids, same bytes in all five state arrays.
Tests: state_codec_test (59 checks - the bound, every bf16 code, specials, underflow, fp32 identity), coded file
format in conversation_file_test (v2 write, codec preserved on read-back, unknown code refused, v2 downgraded to v1
refused, fp32 limits bounding a coded file), and coded capture / validate / restore / store / widen on GPU in
conversation_snapshot_test (half-size capture, refusal, the 2^-8 tolerance, restore-then-recapture idempotence, and the
streamed and captured paths writing the same bytes). 123 of 124 checks pass here; ple_parity fails before and after
this change for a model artifact it looks for at a relative path.
Built CUDA only (V100 sm_70); no backend file is touched and the new code is host code beside copies that already use
this file's stream helper, but HIP and SYCL were not compiled - this box has no ROCm or oneAPI.
docs/STATE_SNAPSHOT_BF16.md is the plan with the measurements filled in, docs/DETAILS.md the short version.
SYCL was missing the codec. The shared conversation_file.cpp and conversation_state.cpp both call
state_codec_by_code, and sycl/CMakeLists.txt keeps its own engine source list, which had conversation_snapshot.cpp,
conversation_state.cpp and conversation_file.cpp but not state_codec.cpp - an unresolved symbol on every SYCL build,
including one that never turns the flag on. Added to the list. No SYCL toolchain on this box, so this is the list being
made complete, not a build that was run.
Widening could end the engine. conversation_checkpoints_widen allocates the fp32 array beside the coded one, and the
parked mount calls it with nothing around it (the serve loop's try covers the SAVE branch, not this one), so a
std::bad_alloc on a memory-tight box escaped into the serve loop. Both widen and store now report a refusal instead,
and the refusal itself is built without allocating, because this is the path where there is nothing to allocate. The
caller's answer to `false` is already to drop the older resume points, which is a conversation that keeps working and
reads its long prompt again. conversation_snapshot_test caps a forked child's address space and widens 256 MiB of bf16
through it: refused to its caller, chain still labelled correctly, nothing thrown past it.
A coded restore followed by an fp32 save asked for less RAM than it then took. session_save_peak_bytes priced both
copies of the deepest checkpoint at the size the chain holds, but when the chain came off a coded file and this engine
writes fp32, the copy inside the image is widened - a host fixture admitted 242.2 MiB for allocations needing at
297.5 MiB. The peak now adds the widening on top of the figure it has always returned, and that figure is still
returned unchanged when the chain and the file agree, which is every save with the flag off; session_save_reclaim_test
pins the all-fp32 sum term by term so the default path cannot drift.
Parking charged the live state at fp32 while capturing bf16. conversation_snapshot_bytes applied the codec to the
checkpoints and left the live arrays at full size, then added the slice on top, so a park needing 116.6 MiB was
estimated at 172.9 MiB and turned away from a 128 MiB cache. The live gdn, PLE and indexer tails and dead keys are now
estimated the way they are written; block_pos stays int32 positions. conversation_snapshot_test checks the estimate
drops by the live state's own half and not merely by the checkpoints'.
Also found by the existing suite while fixing the above: the new checkpoint-size helper built
`for (const auto& v : {c.gdn, c.ple, ...})`, which copied every array - inside an estimate, of the bytes it exists to
price. conversation_file_test's forked low-address-space child caught it. It takes pointers now.
Tests here: state_codec_test 59, conversation_file_test 212, session_save_reclaim_test 124 (119 before: five for the
peak and the default-path figure), conversation_snapshot_test 4434 (one more: the widening under pressure),
conversation_validation_test 972, conversation_cache_test 4219, conversation_memory_test 23. CUDA only (V100 sm_70);
HIP and SYCL still not compiled here - this box has no ROCm or oneAPI.
sycl/src/program/generate.cpp kept its own copy of the --serve loop and still called session_save_checkpoints(checks, sl, admit, ...); the shared declaration now takes the codec the file will be written with as its third argument, so the SYCL build failed on converting `admit` to StateCodec. Pass StateCodec::Fp32. This port has no --session-state-codec and its hand-ported copies of conversation_state.cpp and conversation_snapshot.cpp contain no codec code at all, so the file holds the state exactly as the chain does and the peak is the figure this call has always been priced at - nothing about a SYCL save changes. Checked the rest of the port while here: nothing else calls the changed API (no session_save_reclaim, no conversation_checkpoints_widen/store), strata_resolve() prefers the migrated conversation_state.cpp and conversation_snapshot.cpp so those builds never see the codec code, and the migrated conversation_snapshot_test.cpp is codec-free too. The one thing they do share is src/core/conversation_file.cpp, which reads the codec byte - that is what needs src/core/state_codec.cpp in the engine list, added at 725714a. docs/STATE_SNAPSHOT_BF16.md now says outright that the port does not implement the flag, and that what a SYCL build does with a file written bf16 elsewhere has not been run here: no oneAPI on this box. Its own conversation_checkpoint_validate still compares the state arrays against fp32 sizes (sycl/src/core/conversation_state.cpp:158), which is the check that would have to catch such a file. Not compiled: no oneAPI here. Host tests unchanged by this commit - no file the CUDA build touches.
sumomomomomo
marked this pull request as ready for review
October 13, 2026 09:46
`convert_state` allocated the destination and then went through an fp32 scratch for every slice: decode into it, encode out of it. Neither direction needs that. bf16 -> fp32 is a shift, so the second operation was a full-size memcpy; fp32 -> bf16 reads one value and writes one value, so the first was a memcpy in. `state_transcode_bytes` converts between the two byte buffers directly - fixed-size memcpy loads and stores, because these are byte buffers whose alignment is the allocator's, `bf16_of` for the round-to-nearest-even and a new `bf16_bits_of` for the shift, which `f32_of_bf16` now uses too so the shift has one home. An array already under the codec asked for is copied, never converted twice, and a codec with no pair of paths here is refused rather than treated as one of the two. Kept: the destination built beside the source and swapped in only once every slice has converted (a mount calls this with one vector on both sides), the slice size, the watchdog beats and STRATA_TEST_SLOW_COPY_US at the slice boundary, the refusal that reports an allocation failure instead of throwing into the serve loop, the codec metadata, and the staging buffers in copy_state_in/copy_state_out - those exist because a coded array cannot cross the device boundary in the shape the engine runs on. The destination is still a sized vector, so its bytes are still initialised before the conversion writes them. Measured on this box (Tesla V100 32 GB, 93 GB RAM, Linux, CUDA 12.8, IQ3_S, --kv int8), one real conversation's state, allocation inside the timed region, both paths in one binary and alternating who goes first (order bias on a build where both paths are the same code: at most 2.3 ms): one whole checkpoint stored bf16 36.4 -> 28.4 ms, widened to fp32 44.1 -> 39.9 ms. Both engine builds in one session, arms alternating, --adapt-swaps 0 --pcie-frac 0, 2 warm-ups and 10 reps, 2 rounds: park 233.7 -> 202.1 ms (-13.5%) and resume 328.0 -> 307.4 ms (-6.3%) at 63K/32K, 233.5 -> 201.8 and 374.4 -> 354.7 at 121K/63K, the same gap in both order positions. RESTORE unchanged within noise: it is dominated by reading 0.6-2 GB off the drive. VmRSS windows per operation unchanged (save +240.8 -> +236.8 MiB, restore +351.0 -> +350.2), which is what a 4 MiB scratch removing itself looks like. Output is byte-for-byte what the scratch path wrote: 20,023 values covering rounding ties on both sides of the even/odd rule, signed zeros, subnormals at both ends of their range, both infinities, float max, NaNs whose payload truncates away, and 20,000 random bit patterns, plus the whole 29.4M-value real state. Both builds write the same session file for the same conversation (sha256 2f92a766f002 fp32, e7eb5ea81f11 and 31572cc9f877 bf16, cmp clean in every round). Tests: state_codec_test 59 -> 95 checks, conversation_snapshot_test 4,434 -> 4,476; conversation_file_test 212, session_save_reclaim_test 124, conversation_validation_test 972, conversation_cache_test 4,219 and conversation_memory_test 23 all pass. HIP and SYCL not built here: no ROCm or oneAPI on this box.
… asked for, and the one-pass conversion Fills in every row `Measurements the PR must report` left outstanding, on the same box and build (Tesla V100 32 GB, RTX 2080 Ti idle, 93 GB RAM, Linux Mint 22.1 kernel 6.14.0-33, CUDA 12.8, IQ3_S, --kv int8, --conversation-cache-mib 16384 --conversation-cache-slots 12, --prompt-cache 6 and --prompt-cache-every 16384 at their defaults). Each configuration is its own engine process, because the codec is read once at start-up; same prompts, same files, same order, medians of 10 measured operations after 2 warm-ups, normal build, memory sampled from outside the process. Session files at 4,223 / 31,797 / 63,026 / 121,614 tokens: 300,697,908 -> 182,634,820 B (-39.3%), 721,838,564 -> 603,775,476 (-16.4%), 1,198,752,588 -> 1,080,689,500 (-9.8%), 2,093,508,528 -> 1,975,445,440 (-5.6%). Every saving is exactly two stored states at 59,031,544 B, which is why the percentage shrinks with length. Cold page cache, measured separately with posix_fadvise(DONTNEED) on that one file: 2,093,508,528 B read in 1423.5 ms configured fp32 against 1,975,445,440 B in 986.1 ms configured bf16. Parked images, two conversations alternated so every turn parks one and resumes the other, 20/20 switches answered right in every size in both configurations: 1.414 -> 1.194 GiB (-15.5%) at 31,798 tokens, 2.192 -> 1.862 (-15.0%) at 63,027, 3.189 -> 2.804 GiB (-12.1%) at 121,615. Divided by one state's saving the deltas are 4, 6 and 7 coded states - the image's live state plus its retained chain of 3, 5 and 6, the last one the --prompt-cache 6 cap - and predicted and measured bytes agree to 8. That is also where the cost is: a resume widens the whole chain, so resume went 52 -> 186 ms and the first turn's prompt_ms 253 -> 381 ms at 32K. At long context K/V (15,271 B/token here) is 79% of an image, so the byte saving flattens toward -16%. Quality gate: ab_engine A/B'd on the same prompts, 2 rounds, identical answer text in 7 of 8 paired requests and decode medians within 2.8 tok/s (the one differing pair is this box's own run-to-run drift, 26x the codec's error); needle recall 9/9 at 32K across a disk restore and the same 9 across a parked resume, 3/3 at 121,581 tokens, older checkpoint ages recalled from the checkpoints at 16,384 and 32,768 tokens, 12/12 park -> resume -> generate cycles, both configurations. 4K was not run as a needle case: a 4K conversation holds no checkpoint older than its live state. The error-vs-tokens-since-resume curve still needs a host-side harness - /v1/chat/completions ignores logprobs. New section 6 documents the one-pass conversion and its numbers: on the host, allocation inside the timed region and both paths alternating who goes first, one whole checkpoint stored 36.4 -> 28.4 ms and widened 44.1 -> 39.9 ms; in the engine, with both builds in one session and the arms' order reversed between rounds, park 233.7 -> 202.1 ms and resume 328.0 -> 307.4 ms at 63K/32K, 233.5 -> 201.8 and 374.4 -> 354.7 at 121K/63K, RESTORE unchanged within noise, and the fp32 default path byte-identical and as fast as before. VmRSS windows per operation unchanged, which is what a 4 MiB scratch removing itself looks like.
DamirLukina88
pushed a commit
to DamirLukina88/D.L.-AI---Strata
that referenced
this pull request
Oct 13, 2026
…conversation's running state at half the bytes Source: Niko1221#2247 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
DamirLukina88
pushed a commit
to DamirLukina88/D.L.-AI---Strata
that referenced
this pull request
Oct 13, 2026
…up robustness, conversation cache memory) Serve: Niko1221#2290 Niko1221#2284 Niko1221#2285 Niko1221#2170 Niko1221#2140 Niko1221#2278. Setup: Niko1221#2161 Niko1221#2287. Engine: Niko1221#2253 Niko1221#2265 Niko1221#2247 Niko1221#2217 Niko1221#2243. New opt-in engine flags: --session-state-codec (enabled in the local configs), --conversation-cache-park-gain and --drop-file-cache (left off: no measured benefit on the RX 7600 XT box). Bench: bench/results/upstream-prs-2026-10-13. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…n the codec they were parked in (resuming no longer widens the whole retained chain; only the one checkpoint a rewind selects is decoded into the session's fp32 buffers, and its stored bytes stay as they are) A conversation that merely continues now pays nothing for the older resume points it never uses: they stay at half the bytes, beside the fp32 ones this session captures itself. Every reader of a checkpoint reads the codec that checkpoint names, never the engine's configured option, which only decides how the next parked image or session file is written. File formats, fingerprints, version-1 serialisation, newly captured checkpoints and the checkpoint interval are unchanged; conversation_checkpoints_widen() stays for callers that want a whole chain in fp32 up front. The deferred decode allocates where the old path did not, so checkpoint_restore() and the batch slot restore report an allocation failure through the existing ERR contract instead of throwing past the serve loop. Re-parking a mixed chain is tested refused under a real address-space limit. Measured on a Tesla V100 32 GB, 93 GB RAM, Linux, CUDA 12.8, IQ3_S, --kv int8, both arms --session-state-codec bf16, --adapt-swaps 0 --pcie-frac 0, two rounds alternating which build goes first, 2 warm-ups + 10 measured repetitions: append-only parked resume 174.3 -> 53.5 ms at 31,821 tokens over 3 retained checkpoints, 229.6 -> 76.8 at 63,050 over 5, 266.4 -> 126.8 at 121,462 over 6, with the chain held at 177.4 / 296.1 / 356.4 MB instead of 354.5 / 591.2 / 710.6 MB; older-checkpoint resume 124.2 -> 40.2, 75.0 -> 39.4, 71.5 -> 37.4 ms. The one deferred decode costs 14.6-17.2 ms at every size, and the mount still restores the image's live state, so the old resume penalty is reduced, not gone. Both builds answered byte-identically in every size and workload, needle recall 20/20 per arm. ctest: 123/124 (ple_parity needs a Q2_0 table this box does not have). No HIP or SYCL build on this box; the only change outside a header is host C++ in generate.cpp.
…essage does not throw Both handlers around a deferred running-state restore built their error string - "checkpoint restore failed: " + e.what() - which wants a heap block on the one path that has none, so a second bad_alloc escaped the catch and ended the engine for a reason that had nothing to do with the state. The message is now a literal or what() read while the exception is still alive, and the copy into the caller's string is attempted inside a try (strata/core/conversation_report.hpp): a throw there is swallowed, the caller still gets its false, and the log line prints from a pointer that needs no heap. A catch-all joins both sites, and the batch slot restore reports the same way instead of assembling a message in its handlers. conversation_report_test arms the allocator to refuse requests of a given size and checks both halves: the pattern the handlers used really does throw, and the one they use now really does not. It also pins the pointer identity of the fallback, the selection between a message and the literal, and that a message short enough for a string's inline buffer asks the allocator for nothing at all. Host-only, no device, no model. ctest: 124/125 (ple_parity needs a Q2_0 table this box does not have).
…d what is still unmeasured Three claims in docs/STATE_SNAPSHOT_BF16.md were the design as written before the code was, and two more were wrong about the format: - a session file carries the live state plus the deepest checkpoint in the chain (session_save_checkpoints), not one checkpoint per 16,384 tokens; DETAILS.md said the same in the session-file note - a coded capture does not stage a 118 MB copy of the running state. copy_state_out and copy_state_in cross the device boundary in slices of kStateCodecChunkFloats floats - 4 MiB - coding or widening in the slice, and the only full-size allocation is the destination array the image or file has to hold anyway. A coded restore adds the widened tails and dead arrays, 24 KiB at the measured geometry. The RAM sections now say that, and say which preflight admits which figure: conversation_snapshot_bytes charges the state at the write codec plus the slice, session_save_peak_bytes charges the deepest checkpoint twice over plus the widening when the file holds it wider plus the 16 MiB write buffer - the report of a failed restore does not allocate (conversation_report.hpp), which is now described where the deferred decode's cost is described Also: the harness note at the head of Measured - bench-state-codec/ is not committed yet, so every script the document names is local and the tables are the record; the files-the-commits-touch table for the two new commits; the stale generate.cpp line numbers refreshed; and a plain statement that FP32 parity at 63K and 121K is NOT yet measured in one matched run, with the command that would produce it. The claim this document makes for bf16 is half the bytes with the codec's parked-resume penalty gone against the eager build, not millisecond-for-millisecond equality with fp32.
… pays the decode too Three claims in the measured section did not survive being read against the JSON they came from. The fp32 resume figures (50.2 ms append-only, 52.2 ms rewind) were 3 repetitions on one arm of a cold process and sat in the document where a reader would take them for fp32's cost on this path. Over 10 repetitions the same fp32 rewind is 29.8 ms, so those numbers read about 20 ms high and are now labelled as what they were. In their place is the closest comparison that exists: a table of fp32 against bf16 parked resume at all three sizes, with its provenance in the same breath - the fp32 column is one round of a pass whose arms were one binary and two configs, the bf16 column is the eager-against-lazy pass, and the builds differ only in the restore-failure handler. Same workload, same prompts, different process runs, so only the two 32K rows are matched inside one run and they say bf16 costs about 11 ms on a parked resume. The run-to-run movement measured here on the same workload is 0.6-2.7 ms, which is the same size as the 63K and 121K deltas, so those four rows are reported as a juxtaposition and explicitly cannot separate "slightly slower" from "equal". What the document claims for bf16 is half the bytes with the whole-chain resume penalty gone and parked resume 3-11 ms slower than fp32 - not millisecond parity. And "nothing is paid at all when the conversation only continues" was wrong. The append-only rows carry selected=31821: the continuing turn restores the newest checkpoint, which in a parked bf16 image is coded, so it pays the one deferred decode like any rewind. Lazy restoration removes the widen of the snapshots the conversation never goes back to, not the decode of the one it does - which is exactly the +11 ms the continuing turn is behind fp32 at 32K. The workload intro and the decode paragraph now say that. No code change. The byte tables are untouched: the interrupted fp32 pass carried another load on the box, which shows in its prompt_ms, park and image columns and not in resume_ms, so no byte figure is taken from it.
sumomomomomo
changed the title
…apart The section that said FP32 parity was unmeasured now has the measurement. One binary (strata.parity, built from this tree) serves both arms; the two arm configs differ in --session-state-codec alone; same prompts, same token counts, 2 warm-ups and 10 measured repetitions per workload, two rounds with the arm order reversed so fp32 goes first in one round and bf16 in the other, --adapt-swaps 0 --pcie-frac 0 pinned. Median of the two rounds' medians, and the round-to-round movement of the same arm printed next to it as the noise floor instead of an assumed one. Parked resume, fp32 against bf16: 49.55 / 52.55 at 32K, 71.15 / 77.68 at 63K, 122.9 / 127.95 at 121K on a continuing turn, and 39.8 / 39.8, 36.45 / 39.63, 41.3 / 37.38 on a rewind to an older checkpoint. So bf16 costs 3-6.5 ms on a parked resume - 2% at 32K, 9% at 63K - while the same arm moved 0.2-1.0 ms between its two rounds. The 121K rewind row is reported as not separable: fp32's own two rounds there moved 3.2 ms and its repetitions ranged 29.6-45.0. The cost is the one deferred decode, which the engine times at 14.7-16.2 ms at every size and in both workloads; the mount that carries it is cheaper by most of that because the image's live state arrives as 59,031,600 B instead of 118,063,152 B. What it gives back on the same requests, from the same run: the park on a rewind turn 141.7 -> 90.5 / 164.7 -> 101.5 / 197.9 -> 95.8 ms, prompt_ms 243.8 -> 196.8 / 262.4 -> 208.9 / 299.5 -> 203.2, client wall 0.330 -> 0.282 / 0.346 -> 0.290 / 0.433 -> 0.332 s, parked image -15.5 / -15.0 / -12.1% on that turn and -28.8% on the continuing one, retained chain 177,414,796 / 296,055,504 / 356,406,176 B for 3 / 5 / 6 checkpoints, observed VmRSS step 449.0 -> 224.8 / 449.7 -> 224.2 / 561.0 -> 138.1 MiB. Recall 20/20 in every cell of both arms but one, named in the text with its reason. A correction is recorded where the wrong number was: a partial pass interrupted mid-run logged fp32 rewind resume at 29.8 / 30.55 / 38.95 ms and carried another load on the box. The matched figures are 39.8 / 36.45 / 41.3, so that pass read the fp32 rewind about 10 ms fast and made bf16 look worse against it than it is. The 3-repetition default-path resume figures are labelled as what they were and no longer read as fp32's cost. No code change.
…ed when it is captured, not when it is parked `--session-state-codec` decided how a conversation was written down and nothing else: every checkpoint the serve loop captured while a conversation ran was held in the cache as fp32, and only became bf16 when that conversation was parked or SAVE'd. A conversation that never parks - the common case in a chat - therefore held full-price snapshots of a state the engine had already agreed to store at half price, and a chain of N retained checkpoints cost N times what the same chain costs once parked. This makes a newly captured retained checkpoint take the configured codec at the moment it is captured. Four things change, and the boundary of each is where the word "retained" stops. `checkpoint_save()` takes the codec the copy is to hold. `checkpoint_at()` passes the resolved `session_codec` for the checkpoint itself and for each layer-split stage part, and `on_stage_chunk` passes it for the parts the pipelined split captures ahead of the join; the parts path of `checkpoint_at` needs no change because it already names the joined checkpoint by `parts[0].state_codec`, which is now the configured codec for a different reason. `copy_to_slot` and `copy_from_slot` are left at fp32 on purpose: they move the working state between a session and a batch slot and restore it straight back, so coding them would round a state the slot is still computing on. The option is read at capture and nowhere near inference - no kernel, no `ss.gdn_state`/`ss.ple_hist`/`idx_tail`/`idx_dead`/`idx_block_pos` and no attention or gated-state path is touched, the live state stays fp32, and a restore decodes a coded snapshot into those fp32 buffers without rewriting the snapshot. `block_pos` is int32 positions and keeps its own size under either codec. The SAVE preflight priced the live state at fp32 while the capture now takes bf16, so it asked for twice what the save then took and refused a SAVE that fit. It prices the state at `session_codec` through `state_codec_size()`, which is the same fix a park already had in `conversation_snapshot_bytes()`. `conversation_checkpoint_save()` built its arrays by resizing the checkpoint's own vectors and filling them, so a capture that ran out of memory left a truncated snapshot published and threw `std::bad_alloc` through `checkpoint_save()`, which has no handler, and ended the engine. A coded capture allocates more than the old fp32 one - the stored arrays plus one bounded 4 MiB staging slice per array to cross the device boundary in - so that hole is now on the path this feature actually uses. The arrays are built beside the checkpoint and moved in once every one of them has been copied: a capture that cannot allocate is a refusal with the checkpoint exactly as it was, and the session keeps what it held. This changes no size, byte or order on the fp32 path. With the option off `state_codec_size()` is the identity and `checkpoint_save()`'s new argument defaults to fp32, so the default path writes the same bytes in the same order as the last release. The SYCL port is untouched: it has its own `sycl/src/` copies of both files and no codec option, so it keeps capturing fp32. The two header edits are comments - `conversation_cache.hpp` said a checkpoint captured by the running session is always fp32, which is no longer true here. Checked on a V100-PCIE-32GB (CUDA arch 70, CUDA 12.8) and an RTX 2080 Ti (arch 75, CUDA 13.0); both build clean and both pass the conversation tests. `conversation_snapshot_test` is 5245 checks (was 5177): `fresh_capture()` captures at 400, 900 and 1400 tokens with a park in between and reads back the exact `block_pos` of pre-park device content under either codec, checks the live buffers still hold their fp32 values after a bf16 capture, and runs a mixed fp32/bf16/fp32 chain through restore, park and a file round-trip. Park and SAVE stay byte-for-byte identical - 120 checks comparing the live state against the parked image and against a re-read. Split/merge with both codecs mixed is 42 checks. The new `conversation_capture_codec_test` is host-only and arms the allocator: 28 checks that a capture which cannot allocate refuses, publishes no partial checkpoint, and leaves a later good capture exact; reverting `conversation_checkpoint_save()` to resize-first fails it with `std::bad_alloc` where it is meant to refuse, so the test bites. Full ctest 125/126, the one failure being `ple_parity`, which fails the same way on unmodified HEAD here (it wants `Q2_0/parities.txt` relative to the directory it is run from). Not measured, and the PR should not claim it until it is: capture latency and prefill latency against the previous build at `--prompt-cache-every` 16384 and 8192, retained-checkpoint bytes before the first park, repeated-turn latency, needle recall after a restore with checkpoints captured in bf16, and a memory-tight conversation that used to refuse and now fits. HIP is unbuilt - no ROCm on this box - and `src/core/conversation_state.cpp` and `src/program/generate.cpp` are shared with it, so a HIP build is owed before review.
|
i know this guy IRL, please disclose AI usage and which model youre using for this cuz im curious and too lazy to ask in the group chat edit: its probably qwen flash next |
…ng-buffer check that never reached staging
Two tests in the capture commit claimed more than they checked. No production behaviour changes; both files are tests.
`what_survives_a_failing_allocator()` kept the `const char*` that `report_restore` returns and compared it with
`strcmp` after the catch handler ended. That pointer is `e.what()`, and `e` is destroyed when the handler ends, so the
comparison read freed memory. AddressSanitizer says it exactly, on gcc 13.3 and on clang 18, from the same source with
only that function put back:
ERROR: AddressSanitizer: heap-use-after-free ... READ of size 14
Niko1221#1 what_survives_a_failing_allocator /tmp/old_report.cpp:215
freed by thread T0 here:
Niko1221#2 std::runtime_error::~runtime_error()
The text is now compared inside the handler, where the exception is alive, and only four booleans leave it; nothing is
copied out to keep the message alive, because a `std::string` built for that purpose would itself want the heap the
test is refusing and would then be measuring its own allocation. The message is 86 characters, past libstdc++'s
15-byte and libc++'s 22-byte small-string buffer, so constructing the exception takes a heap block before the
allocator is armed and what() hands out storage the exception owns rather than a literal - which is also what makes
`refused > 0` a real assertion instead of a free one. The allocator is disarmed before any `check()`, since `check()`
prints and printing may allocate. 21 checks become 25.
`conversation_capture_codec_test` armed its allocator at `st.z.gdn` and called the refusal a failure of the capture's
bounded staging buffer. It never reached it. At the fixture's `ssm_state_size = 2` the fp32 GDN state is 768 bytes, so
the bf16 GDN output array - half of that, 384 bytes - is served and the bf16 PLE array after it, half of a 368,640-byte
PLE history, is what the allocator says no to. Counting refusals cannot tell that apart from a staging refusal, so the
armed allocator now records the sizes it saw (`refused_at[]`, `refused_n`, `served_max`) - fixed-size and
allocation-free, because recording a refusal must not itself want the heap being refused, which with
`fail_from_bytes = 1` would mean recording nothing at all.
The mislabelled block keeps its coverage and says what it tests: it failed at the PLE output array, the GDN one having
already been served. A second fixture that moves only `ssm_state_size`, to 256, supplies the staging check that was
missing: there the fp32 GDN state is 3,146,304 bytes, bigger than any array the capture sizes - the largest is the
1,573,152-byte bf16 GDN one - and under the 4 MiB staging limit, so the first slice is the whole state and the bar at
`z.gdn` lands on it and on nothing before it. That block asserts the one refusal was at exactly the staging size and
that the largest stored array had already been served by then. Every pressure block now also asserts that a refused
capture leaves all five arrays and the checkpoint's own codec as they were, leaves the working state byte-identical,
and is followed by a capture that works byte for byte. 28 checks become 43.
Each new assertion was shown to bite by breaking the thing it checks: the staging fixture at the old size fails its
own sizing check; moving the bar down onto the largest stored array fails "the one refusal was the staging slice";
putting `conversation_checkpoint_save` back to resizing the checkpoint's own vectors fails "is not thrown past its
caller"; raising the report test's bar to 4096, where the copy succeeds, fails "the copy really was refused".
Both tests pass normally and under AddressSanitizer, UndefinedBehaviorSanitizer and LeakSanitizer on gcc 13.3 and
clang 18.0 - in the sanitizer builds the capture path itself is instrumented (`conversation_state.cpp`,
`conversation_snapshot.cpp`, `conversation_file.cpp`, `conversation_memory.cpp`, `state_codec.cpp` compiled with the
sanitizers and linked ahead of the archives; `conversation_checkpoint_save` carries 88 sanitizer call sites), so this
is not an ordinary run relabelled. The rest of `strata_engine`, `strata_kernels` and `ggml` stays uninstrumented: a
fully sanitized build would rebuild the CUDA kernels. Only libstdc++ is installed here, so the libc++ small-string
path is guarded by a length check rather than exercised. Full ctest 125/126 on the V100 (CUDA 12.8, arch 70), the one
failure being `ple_parity`, which fails the same way on unmodified HEAD; the conversation, codec and SAVE tests pass
11/11 on the 2080 Ti (CUDA 13.0, arch 75).
…e it `3f04cc72` moved the codec to the moment a retained checkpoint is captured, and the documents still described the engine before that. Four claims were affected and the line references in the same sections had drifted by the new commits' insertions. "The configured option decides only how the NEXT parked image or session file is written" and "newly captured checkpoints are still fp32" are now said about the commit they were true at, with the later change named: the option decides how a NEW copy is written - the next retained checkpoint, the next parked image, the next session file - and a retained checkpoint is coded when it is taken, so a chat that never parks pays half price for its resume points too. `docs/DETAILS.md` said the coding covered "parked conversations and in session files"; the retained chain in RAM is in scope now and is named. §4 described `conversation_checkpoint_save` as resizing the checkpoint's own vectors and filling them, which is the shape that published a truncated snapshot when a capture ran out of memory. It builds beside and moves in now, and one call serves a retained capture, a park and a SAVE. Test 19 said the continuation "captures its own fp32 checkpoint": left at the API default a capture is fp32 and only then, and the serve loop asks for its own coding by setting `state_codec` first, which `fresh_capture()` checks under both codecs. Every engine measurement in the lazy-restoration and fp32-against-bf16 sections was taken on `e2983e98`, before capture-time coding, and says so: a never-parked conversation now holds a coded chain as well, a park converts nothing, and capture itself codes, so the retained-byte, `VmRSS` and `park` columns read high against today's build while the deferred decode and the image and file bytes stand. That commit's own unmeasured list - capture and prefill latency at `--prompt-cache-every` 16384 and 8192, retained bytes before the first park, repeated-turn latency, needle recall after a restore with checkpoints captured in bf16, a memory-tight conversation that used to refuse and now fits - is recorded with it, along with the HIP build still owed on the two shared files. Line references re-taken against HEAD: `conversation_state_sizes` 155-177, `metadata_bytes` 138-152, `conversation_snapshot_bytes` 330-369, `conversation_checkpoint_restore` 250, `convert_state` 89-110, `generate.cpp` 7992 / 1513 / 2119. Docs only; no code, no test, no number changed.
…he capture path `3f04cc72` moved coding to the moment a retained checkpoint is captured, and every engine number in this document's FP32-against-BF16 comparison was taken on `e2983e98`, before it. Both arms now run on `engine-cuda12/strata.head` (`e79abfc0`), configs differing in `--session-state-codec` alone, 2 warm-ups + 10 measured repetitions, two rounds with the arm order reversed, `--adapt-swaps 0 --pcie-frac 0` pinned. The resume path did not move: **+2.2 / +0.2 / +5.9 / +4.5 / +5.6 / −2.3 ms** against the previous build's **+3.0 / +0.0 / +6.5 / +3.2 / +5.1 / −3.9** on the same six cells. Four of the five positive deltas clear the cell's own round-to-round spread; the 32K rewind (+0.2 against 1.55) does not, and the 121K rewind reads −2.3 ms with fp32's repetitions ranging 29.3–48.4, so neither is separable. Deferred decode 14.95 / 16.2 / 15.7 ms. What did move is the park, and that is the commit doing what it says. At `e2983e98` a bf16 park still converted the retained chain on the way into the image, and at 121K - six checkpoints - that cancelled the smaller write: fp32 78.85 against bf16 78.95 ms, dead even. Coding at capture leaves nothing to convert and the same cell reads 89.2 against 68.45 ms. The gains table is replaced with the HEAD figures and a continuing-turn park row added. New subsection: capture and turn latency at `--prompt-cache-every` 16384 and 8192, which `3f04cc72` listed as unmeasured. The engine has no timing line for a capture, so it measures what a client can see around it - a 31,813-token prefill and 12 short turns per cell, every turn carrying one capture, chain of 3 checkpoints at the default spacing and 5 at 8192. Prefill 21,716.5 ms fp32 against 21,702.5 bf16 (21,796 against 21,867.5 at 8192), inside the arms' own 3-22 ms round spread: coding does not reach the prefill. A turn is 49 ms cheaper under bf16 at the default spacing and 84 ms at 8192, both distributions non-overlapping. Halving the spacing costs a turn 69.5 ms in fp32 and 33.5 ms in bf16 - 48 % of the penalty, on bytes that scale with chain length. Recall after restores from checkpoints captured in bf16: 20/20 in every cell of both arms on the two workloads that ask for the code word. The eager-against-lazy provenance note is scoped to the table it describes rather than the whole section: that table compares two restoration builds and `3f04cc72` does not undo its result, but its retained-byte, VmRSS and park columns are the pre-capture-coding case and say so. Retained bytes before the first park and a memory-tight conversation that used to refuse and now fits stay unmeasured. Docs only; no code, no test, no kernel.
…round medians The paragraph credited `3f04cc72` with a 121K park going from "78.85 against 78.95 ms, dead even" to "89.2 against 68.45 ms". Both figures were single rounds read off the first matching record instead of the median of the two rounds the rest of the section uses. Re-taken as medians, the old build's 121K cell is fp32 84.15 against bf16 79.15 - a 5 ms gap, not a tie - and fp32's own two rounds there were 78.85 and 89.45, an 11 ms spread wide enough that this one cell separates nothing on its own. The claim survives, but only in the form that was always right: compare each arm against itself across the two builds. Coding at capture removes the park's conversion, so the bf16 park must get faster and the fp32 park must not move. Continuing-turn medians, 32K / 63K / 121K: fp32 190.52 -> 190.5, 136.9 -> 136.32, 84.15 -> 87.95 against round spreads of 3.2 / 1.85 / 2.5 ms, i.e. flat; bf16 164.77 -> 154.82, 112.6 -> 102.75, 79.15 -> 68.15, i.e. about 10 ms faster at every size. Three sizes moving together is the evidence; one cell is not.
…he bf16 RESTORE delta is a constant 56.2 MiB Four figures in this section could not be reproduced from any surviving artifact, so both arms were run again on the current build: fresh engine per arm, two configs differing only in --session-state-codec, VmRSS polled at 20 ms, median of 9, 32K and 63K. SAVE is not one operation. The first save after a prefill admits a fixed amount of RSS (+353.2 MiB fp32 against +180.5 bf16 at 32K, +354.0 against +181.2 at 63K) that every later save reuses, so those read ~0 - or negative at 63K, as page cache is written back while the image goes to disk. The old +353.2/+240.8 pair came from a median of 3 that mixed the two cases; its per-rep values were 353.2, -107.2, -12.2. The run behind the bf16 half of it lived in a file the harness keyed on codec alone and a second size overwrote. RESTORE is bigger under bf16, and by a constant: +406.5 against +350.3 at 32K, +861.3 against +805.1 at 63K, so +56.2 MiB at both. The constancy is the mechanism - widening allocates the fp32 array beside the coded original, and the delta is that coded state, 59,031,600 B, to 0.1 MiB. Parked steady RSS and the cache's own accounting are restated with the parked count actually reached (4, not 3). Also flagged, in the pre-change A/B above, that its SAVE column is a matched pair rather than a level: same median-of-3 statistic.
Owner
|
Thanks. This makes saved states smaller but does not make tokens faster, and it changes stored numerics, so it is outside this speed round. It would need a rebase and a quality check on our side before it can be an opt-in. |
Interpause
added a commit
to Interpause/fork-Strata
that referenced
this pull request
Oct 17, 2026
--session-state-codec bf16: a saved or parked conversation's running state at half the bytes (opt-in; fp32 by default, v1 files byte-identical). From Niko1221#2247 (sumomomomomo): - StateCodec (strata/core/state_codec.hpp) with exact bf16 widening, a bounded fp32 slice across the device boundary, and per-checkpoint codec labels - v2 session files carry the codec per checkpoint; v1 files are unchanged and a coded file read as v1 is refused - capture/restore size the stored copy (session_save_peak_bytes takes the codec) - a resumed chain keeps imported snapshots in the codec they were parked in; only the snapshot a rewind selects is widened Merged onto main and extended: - the pooled/threaded checkpoint capture (new on main) takes the resolved codec, so a pooled retained checkpoint is labelled the way the option says - metal/CMakeLists.txt links state_codec.cpp (conversation_file/state call it) - the codec block of conversation_snapshot_test runs before the borrow block, each from the fixture it was written against Tests: 12/12 conversation/session/codec/expert-layout tests.
Interpause
added a commit
to Interpause/fork-Strata
that referenced
this pull request
Oct 18, 2026
APR support: Agention's Gyro quants, decoded by the engine itself - the trellis experts TQ2_T / TQK6 / TQK7 (Gyro-S / -M) and the pair trellis TQP5 / TQP6 / TQP7 (Gyro-L), with Hadamard-folded weights. From agentionai/Strata feat/tqp-types on GitLab (rc3 = v0.1.43 + Gyro, upstream v0.1.44.1 merged, plus the pair trellis): - the trellis block formats and codebook (third_party/ggml/ggml-common-tq.h, tq_lut_i8.cuh) and CPU scalar / AVX2 / AVX-512 decode-and-dot kernels (tq_*.cpp; a TQP type on an AVX-512 CPU takes the AVX2 kernel, which measured faster) - trellis decode and the block-Hadamard rotation on the CUDA/HIP decode, FP16 and MMQ prompt paths, the peer, the layer split's helper, remote GPUs and verify - hadamard.txt beside native_experts.txt (now v5), so an engine that cannot rotate refuses the pack instead of computing garbage; tools/iq_pack.py writes it, tools/gyro_setup.sh and docs/GYRO.md cover setup (--model L included) Merged onto the custom stack and extended to the paths main grew since 0.1.40.3: - the format lists carry main's NVFP4 (40) and Q5_K/Q6_K downs beside 144-149; dq_nvfp4 and dq_tq coexist; moe_mmq keeps its KQ-DSO switch and supported() - quantize_scatter and the NVFP4 per-row-scale quantize take the rotated copy (xsrc) - the peer / layer-split helper rotates its own input copy on its own device (had_sx in PeerPrefill and peer_had_signs) - hadamard.cpp and the tq kernels are in the SYCL and Metal source lists so those builds link Startup refusal: Metal and SYCL cannot rotate; a folded pack is refused at start with the build that can run it named (CUDA/HIP unchanged). Extended to Niko1221#1937 (disk tier) and Niko1221#2247 (bf16 state codec): the streamed park and read use the codec and finish with conversation_kv_finish, unchanged by the trellis types. Built with the pinned ggml (agentionai-group/llama.cpp 9aa51ce4, ggml 0.25.3, TQP MMQ instances); Gyro-L loads (--load-check: 45.12 GiB, 48 folded layers, PLE Q8_0 direct I/O, 0 failures) and 156 of 160 tests pass - the four that fail (quantize_act_parity, q8k_quant_parity, q8k_quant_parity_avx, ple_parity) fail on the untouched v0.1.45 tree here too.
Owner
|
Thanks for the pull request. We're resetting the tracker for 0.2.0-beta. There are more than 1,300 open issues and pull requests, too many for us to review properly, so we're closing everything that isn't being worked on right now. Closed here does not mean rejected. To reopen it takes about two minutes:
The rules are in CONTRIBUTING.md. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #2377
Overview
Adds opt-in
--session-state-codec bf16to reduce the RAM and disk needed for conversation checkpoints. For the tested model, checkpoint state is approximately 50% smaller and session files are 5.6–39.3% smaller, providing headroom for more cached conversations or denser resume points.In the 32K workload, BF16 at 8K checkpoint spacing delivered comparable complete-turn latency to FP32 at the default 16K spacing. Parking was faster across the tested context sizes, while continuing resumes added approximately 2–6 ms.
The benchmark harness, raw results and engine logs are now public.
Architecture
The codec covers GDN recurrence/convolution history, PLE history, and QSA indexer tails/dead keys. Integer positions and K/V storage are unchanged. Encoding uses round-to-nearest-even and is lossy; widening is exact.
Validation
Benchmarks used Linux, CUDA 12.8, a Tesla V100 32 GB as the single model GPU, a 24-core/32-thread Intel CPU, 93 GB reported RAM, IQ3_S, INT8 K/V, and a Samsung 990 EVO Plus SSD.
Latency comparisons used the same
strata.headbinary, with runs recorded ate79abfc0: two rounds with reversed arm order, two warm-ups and ten measured repetitions per round. Figures below are medians of round medians. RSS measurements were re-run at79955461.Checkpoint spacing and turn latency
A fresh 31,813-token document followed by short turns:
Complete-turn timings include parking, resumption and generation; they are not TTFT or isolated capture timings. This sweep measures latency; recall is assessed separately below.
Parked conversation performance
At 16K checkpoint spacing, FP32 → BF16:
In a separate 32K run with four parked images, cache-accounted allocation fell from 3.73 GB to 3.08 GB (17.4%). These figures include allocator capacity; they are not serialized payload sizes or a guarantee of proportionally more conversations per cache budget.
SAVE / RESTORE memory
Observed RSS growth, FP32 → BF16:
First SAVE was measured separately from later saves, which reuse allocations. RESTORE figures are medians of nine repetitions. BF16 RESTORE grew RSS by approximately 56.2 MiB more at both sizes.
RSS was sampled every 20 ms on fresh processes; these are observed increments, not guaranteed peak-memory bounds.
Functional and failure-path checks
ple_parityrequires a missing artifact and also fails before this change.Compatibility and limitations
Implementation and detailed results:
docs/STATE_SNAPSHOT_BF16.md.Public evidence: artifact manifest, harness and reproduction instructions, and raw results. The manifest identifies historical runs, superseded measurements and provenance gaps; earlier session-file timings are not presented as current-build performance.
AI Disclosure
Created with Qwen3.8-Flash-Next-exl3 3.05bpw.