Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,9 @@ __pycache__/

# plan v0.3 P1 (repository hygiene, review E1): profiler captures and large diagnostic dumps stay out of git.
*.nsys-rep

# Pi harness metadata (local session state, not project source)
.pi/
*.sqlite
*.npz
*.occ1
Expand Down
15 changes: 15 additions & 0 deletions src/program/generate.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -1515,6 +1515,21 @@ int main(int argc, char** argv) {
std::fprintf(stderr, "strata generate: session_init failed\n");
return 1;
}
// **A GENERATE RUN ON A NATIVE PACK NEVER ZEROED THE SESSION STATE.** The canonical path zeroes it from
// `put_input` at position 0 (`session_zero`, with the embedding as `R`), but a native pack breaks out of that
// loop before the first `put_input` - it runs verify windows instead - and neither the batched prompt path
// nor the verifier zeroes anything. `sbuf` is `cudaMalloc`'d, so the GDN recurrence, the QSA KV/indexer
// state, the PLE history and `R` all started from whatever the allocator last held: finite on a fresh
// allocation and overflow/NaN after reuse, which surfaced as the whole layer stack saturating and every
// prompt decoding to the same token. `--serve` zeroes exactly this state when `resume == 0`; generate mode
// always starts from an empty sequence, so it must too.
if (native_pack) {
strata::core::session_zero(ss, g, nullptr, main_cs);
if (cudaDeviceSynchronize() != cudaSuccess) {
std::fprintf(stderr, "strata generate: zeroing the session state failed\n");
return 1;
}
}
if (g.n_qsa_layers() > 0 && ss.qsa_states[0].kv_mode == 1)
std::fprintf(stderr, "strata generate: KV streaming: %lld of %lld cells per QSA layer in VRAM, the K/V in "
"%.2f GiB of pinned RAM\n", (long long) (ss.qsa_states[0].n_slots * 4),
Expand Down