Repository navigation
generate: zero session state for native packs before the prompt path - #167
Closed
samuelishida wants to merge 1 commit into
Closed
samuelishida wants to merge 1 commit into
samuelishida wants to merge 1 commit into
Conversation
The canonical path zeroes session state from put_input at position 0, but a native pack breaks out of that loop before the first put_input and runs verify windows instead, leaving cudaMalloc'd state uninitialized. On a reused allocation that produced NaN/overflow and degenerate identical-token output. Zero the state explicitly when native_pack is set. Also removes an unused argmax() helper that emitted a compiler warning, and ignores the local .pi/ harness directory.
Niko1221
added a commit
that referenced
this pull request
Sep 30, 2026
…mode (#167 follow-up); no .pi/ ignore line Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
Owner
|
Released in 0.1.29: and extended to every pack type (a canonical pack on the batched prompt path had the same gap). Merged under your name (cherry-picked, so GitHub doesn't show it as merged). Thank you! |
gputier
pushed a commit
to gputier/StrataGP
that referenced
this pull request
Sep 30, 2026
…mode (Niko1221#167 follow-up); no .pi/ ignore line (cherry picked from commit ec64796)
asp345
pushed a commit
to asp345/Strata-dual-3060
that referenced
this pull request
Sep 30, 2026
…mode (Niko1221#167 follow-up); no .pi/ ignore line Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
perronemirko
pushed a commit
to perronemirko/Strata
that referenced
this pull request
Sep 30, 2026
New qwen35 attention kernel + core/qwen35 glue; wire into strata_engine and CMake (engine lib, build tests). gr.cu block_sum double->float to cut register pressure on the norm path; f32 sum of 2560 squares is close enough to ref/gr.py's float64 that the shuffle-order difference no longer matters. fused_gdn now owns the GDN step, so gdn.cu drops its softplus helper and the parity test trims unused layout code. generate: zero session state on startup (Niko1221#167) - native packs skip put_input at pos 0 and left R/KV/PLE uninitialized, saturating every decode to one token after allocator reuse; also accept ple_io=ram for a locked PLE table and print its load time. serve fast-exits (std::_Exit) on CUDA fault (Niko1221#224) instead of unwinding destructors that hang past the watchdog.
perronemirko
pushed a commit
to perronemirko/Strata
that referenced
this pull request
Oct 1, 2026
New qwen35 attention kernel + core/qwen35 glue; wire into strata_engine and CMake (engine lib, build tests). gr.cu block_sum double->float to cut register pressure on the norm path; f32 sum of 2560 squares is close enough to ref/gr.py's float64 that the shuffle-order difference no longer matters. fused_gdn now owns the GDN step, so gdn.cu drops its softplus helper and the parity test trims unused layout code. generate: zero session state on startup (Niko1221#167) - native packs skip put_input at pos 0 and left R/KV/PLE uninitialized, saturating every decode to one token after allocator reuse; also accept ple_io=ram for a locked PLE table and print its load time. serve fast-exits (std::_Exit) on CUDA fault (Niko1221#224) instead of unwinding destructors that hang past the watchdog.
eddoursul
added a commit
to eddoursul/Strata
that referenced
this pull request
Oct 1, 2026
…g, the split sampler, fixes Upstream's 103 commits since a790805, with this branch's engine kept wherever both sides do the same thing (the prompt path, the verify window's kernels, the QSA block scores, the prompt cache, --ple-io ram). Taken: - serve/: upstream's server: optional idle unload, POST /unload and /load, a free-VRAM guard and a before_load hook (Niko1221#208); /status behind the API key (Niko1221#212) and a constant-time key check (Niko1221#213); an engine exit quotes its last log line (Niko1221#215); tool calls end at their own </parameter> (Niko1221#210); the engine, the image encoder and the MCP servers end with the server on Windows; no stack trace when a client hangs up; the prompt speed on the dashboard; SIGTERM. Without a "gpu" key in the config, the Monitor and the VRAM guard read the cards of --main-gpu and --second-gpu. - Sampling: top_k by a lexicographic threshold, split over the whole GPU (Niko1221#197, the same tokens). - Fixes: the session state zeroed at start in generate mode (Niko1221#167); a serve request that fails on a CUDA fault exits at once (Niko1221#224), and every failed request's reason goes to the log; the error string cleared per request; the expert cache shrinks by 1 GiB steps when the free VRAM reads 0 (Niko1221#153), and says so when no VRAM is left for it (Niko1221#174); s_gemv's q8 split kernel reaches its barrier; a NaN stays a quiet NaN in f32 -> bf16; NOMINMAX; iq_pack writes native_experts.txt under a temporary name (Niko1221#172). - STRATA_IQ_MT_MIN=1 (opt-in, Niko1221#152), the AVX-2 i-quant prefetch, the table indexer's spare rotated at position 0 (bitwise unscaled), version 0.1.30; tests: mmvq_multi_parity, bf16_bits_test, the sampler's three paths. - setup.py, docs, Docker, the AMD RDNA4 and Pascal files (not built here), data/draft_vocab_en.bin (byte-identical to this branch's data/draft_vocab.bin). DETAILS.md gains the GPU-sharing options, STRATA_IQ_MT_MIN and this branch's draft vocabulary; examples/README.md lists the upstream options this branch's engine leaves out. Not taken: rope scaling (contexts past 262K; the QSA RoPE kernels keep their form), the conversation cache (the prompt cache stashes replaced conversations), the low-RAM resident variant, the multi-GPU session carve, coupled draft sampling, the shared expert arena (Linux), the WSL2 context order, upstream's GDN recurrence pipelining and its 1024-token expert streaming (this branch's prompt path has its own forms), STATE_HASH. Their files are in the tree, not built. Window hashes on fixed text (--adapt-every 0, the second GPU's tier pinned) equal the previous build's on both models; the API tests (prompt cache, streaming, keep-alives, disconnect) pass. Claude-Session: https://claude.ai/code/session_01GgM6qCebHZfQmgkcxNY1U5
perronemirko
pushed a commit
to perronemirko/Strata
that referenced
this pull request
Oct 1, 2026
New qwen35 attention kernel + core/qwen35 glue; wire into strata_engine and CMake (engine lib, build tests). gr.cu block_sum double->float to cut register pressure on the norm path; f32 sum of 2560 squares is close enough to ref/gr.py's float64 that the shuffle-order difference no longer matters. fused_gdn now owns the GDN step, so gdn.cu drops its softplus helper and the parity test trims unused layout code. generate: zero session state on startup (Niko1221#167) - native packs skip put_input at pos 0 and left R/KV/PLE uninitialized, saturating every decode to one token after allocator reuse; also accept ple_io=ram for a locked PLE table and print its load time. serve fast-exits (std::_Exit) on CUDA fault (Niko1221#224) instead of unwinding destructors that hang past the watchdog.
perronemirko
pushed a commit
to perronemirko/Strata
that referenced
this pull request
Oct 2, 2026
New qwen35 attention kernel + core/qwen35 glue; wire into strata_engine and CMake (engine lib, build tests). gr.cu block_sum double->float to cut register pressure on the norm path; f32 sum of 2560 squares is close enough to ref/gr.py's float64 that the shuffle-order difference no longer matters. fused_gdn now owns the GDN step, so gdn.cu drops its softplus helper and the parity test trims unused layout code. generate: zero session state on startup (Niko1221#167) - native packs skip put_input at pos 0 and left R/KV/PLE uninitialized, saturating every decode to one token after allocator reuse; also accept ple_io=ram for a locked PLE table and print its load time. serve fast-exits (std::_Exit) on CUDA fault (Niko1221#224) instead of unwinding destructors that hang past the watchdog.
perronemirko
pushed a commit
to perronemirko/Strata
that referenced
this pull request
Oct 2, 2026
New qwen35 attention kernel + core/qwen35 glue; wire into strata_engine and CMake (engine lib, build tests). gr.cu block_sum double->float to cut register pressure on the norm path; f32 sum of 2560 squares is close enough to ref/gr.py's float64 that the shuffle-order difference no longer matters. fused_gdn now owns the GDN step, so gdn.cu drops its softplus helper and the parity test trims unused layout code. generate: zero session state on startup (Niko1221#167) - native packs skip put_input at pos 0 and left R/KV/PLE uninitialized, saturating every decode to one token after allocator reuse; also accept ple_io=ram for a locked PLE table and print its load time. serve fast-exits (std::_Exit) on CUDA fault (Niko1221#224) instead of unwinding destructors that hang past the watchdog.
perronemirko
pushed a commit
to perronemirko/Strata
that referenced
this pull request
Oct 2, 2026
New qwen35 attention kernel + core/qwen35 glue; wire into strata_engine and CMake (engine lib, build tests). gr.cu block_sum double->float to cut register pressure on the norm path; f32 sum of 2560 squares is close enough to ref/gr.py's float64 that the shuffle-order difference no longer matters. fused_gdn now owns the GDN step, so gdn.cu drops its softplus helper and the parity test trims unused layout code. generate: zero session state on startup (Niko1221#167) - native packs skip put_input at pos 0 and left R/KV/PLE uninitialized, saturating every decode to one token after allocator reuse; also accept ple_io=ram for a locked PLE table and print its load time. serve fast-exits (std::_Exit) on CUDA fault (Niko1221#224) instead of unwinding destructors that hang past the watchdog.
perronemirko
pushed a commit
to perronemirko/Strata
that referenced
this pull request
Oct 3, 2026
New qwen35 attention kernel + core/qwen35 glue; wire into strata_engine and CMake (engine lib, build tests). gr.cu block_sum double->float to cut register pressure on the norm path; f32 sum of 2560 squares is close enough to ref/gr.py's float64 that the shuffle-order difference no longer matters. fused_gdn now owns the GDN step, so gdn.cu drops its softplus helper and the parity test trims unused layout code. generate: zero session state on startup (Niko1221#167) - native packs skip put_input at pos 0 and left R/KV/PLE uninitialized, saturating every decode to one token after allocator reuse; also accept ple_io=ram for a locked PLE table and print its load time. serve fast-exits (std::_Exit) on CUDA fault (Niko1221#224) instead of unwinding destructors that hang past the watchdog.
MAHDI-AQ
pushed a commit
to MAHDI-AQ/Strata-AGX
that referenced
this pull request
Oct 4, 2026
…mode (Niko1221#167 follow-up); no .pi/ ignore line Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN (cherry picked from commit ec64796)
MAHDI-AQ
pushed a commit
to MAHDI-AQ/Strata-AGX
that referenced
this pull request
Oct 4, 2026
…mode (Niko1221#167 follow-up); no .pi/ ignore line Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN (cherry picked from commit ec64796)
MAHDI-AQ
pushed a commit
to MAHDI-AQ/Strata-AGX
that referenced
this pull request
Oct 4, 2026
…mode (Niko1221#167 follow-up); no .pi/ ignore line Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN (cherry picked from commit ec64796)
perronemirko
pushed a commit
to perronemirko/Strata
that referenced
this pull request
Oct 4, 2026
New qwen35 attention kernel + core/qwen35 glue; wire into strata_engine and CMake (engine lib, build tests). gr.cu block_sum double->float to cut register pressure on the norm path; f32 sum of 2560 squares is close enough to ref/gr.py's float64 that the shuffle-order difference no longer matters. fused_gdn now owns the GDN step, so gdn.cu drops its softplus helper and the parity test trims unused layout code. generate: zero session state on startup (Niko1221#167) - native packs skip put_input at pos 0 and left R/KV/PLE uninitialized, saturating every decode to one token after allocator reuse; also accept ple_io=ram for a locked PLE table and print its load time. serve fast-exits (std::_Exit) on CUDA fault (Niko1221#224) instead of unwinding destructors that hang past the watchdog.
Hugua700
pushed a commit
to Hugua700/Strata
that referenced
this pull request
Oct 6, 2026
…mode (Niko1221#167 follow-up); no .pi/ ignore line
perronemirko
pushed a commit
to perronemirko/Strata
that referenced
this pull request
Oct 6, 2026
New qwen35 attention kernel + core/qwen35 glue; wire into strata_engine and CMake (engine lib, build tests). gr.cu block_sum double->float to cut register pressure on the norm path; f32 sum of 2560 squares is close enough to ref/gr.py's float64 that the shuffle-order difference no longer matters. fused_gdn now owns the GDN step, so gdn.cu drops its softplus helper and the parity test trims unused layout code. generate: zero session state on startup (Niko1221#167) - native packs skip put_input at pos 0 and left R/KV/PLE uninitialized, saturating every decode to one token after allocator reuse; also accept ple_io=ram for a locked PLE table and print its load time. serve fast-exits (std::_Exit) on CUDA fault (Niko1221#224) instead of unwinding destructors that hang past the watchdog.
anon761
pushed a commit
to anon761/Strata-upstream
that referenced
this pull request
Oct 6, 2026
…mode (Niko1221#167 follow-up); no .pi/ ignore line
anon761
pushed a commit
to anon761/Strata-upstream
that referenced
this pull request
Oct 6, 2026
…mode (Niko1221#167 follow-up); no .pi/ ignore line Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0154bwiad8gY7wRehCXy27pN
perronemirko
pushed a commit
to perronemirko/Strata
that referenced
this pull request
Oct 8, 2026
New qwen35 attention kernel + core/qwen35 glue; wire into strata_engine and CMake (engine lib, build tests). gr.cu block_sum double->float to cut register pressure on the norm path; f32 sum of 2560 squares is close enough to ref/gr.py's float64 that the shuffle-order difference no longer matters. fused_gdn now owns the GDN step, so gdn.cu drops its softplus helper and the parity test trims unused layout code. generate: zero session state on startup (Niko1221#167) - native packs skip put_input at pos 0 and left R/KV/PLE uninitialized, saturating every decode to one token after allocator reuse; also accept ple_io=ram for a locked PLE table and print its load time. serve fast-exits (std::_Exit) on CUDA fault (Niko1221#224) instead of unwinding destructors that hang past the watchdog. 1. Introduced the dense path (dense_model, kernels, and --serve binary) tailored for the Qwen3.8-27B model. 2. The architecture utilizes the same line protocol as strata --serve through the unchanged StrataEngine. 3. Cleaned up CMakeLists by reverting parts of a prior commit to drop obsolete files and tests. 4. Implemented quantized KV cache support featuring fp16, int8, q4_0, and k8v4 formats. 5. Integrated a 256-dimension Walsh-Hadamard rotation to handle Q8 and Q4 quantized formats. 6. Updated chunked prefill kernels (kv_append_rows and attn_chunk) to natively read the quantized cache. 7. Removed the restriction that previously disabled chunked prefill when using a quantized KV cache. 8. Optimized the decode path by walking columns through chunk kernels, drastically reducing driver launches. 9. Fixed a pre-existing memory overrun bug within the bounds of the positions_kernel component. 10. Gated the MMQ prompt path verification on relative error instead of strict absolute bit parity. 11. Added GDN state checkpointing along with conversation history and last hidden states to pinned host RAM. 12. The GDN checkpoints prevent full prompt re-reading from token 0 when a chat template prefix breaks. 13. Introduced a Tensor Core (WMMA) chunk attention kernel for GQA 6:1 and head_dim 256. 14. The WMMA kernel implements a two-pass online softmax and handles Walsh-Hadamard rotations for Q4 keys/values. 15. Verified that the final conflict-resolution commits kept the crucial checkpointing logic fully intact.
perronemirko
pushed a commit
to perronemirko/Strata
that referenced
this pull request
Oct 10, 2026
New qwen35 attention kernel + core/qwen35 glue; wire into strata_engine and CMake (engine lib, build tests). gr.cu block_sum double->float to cut register pressure on the norm path; f32 sum of 2560 squares is close enough to ref/gr.py's float64 that the shuffle-order difference no longer matters. fused_gdn now owns the GDN step, so gdn.cu drops its softplus helper and the parity test trims unused layout code. generate: zero session state on startup (Niko1221#167) - native packs skip put_input at pos 0 and left R/KV/PLE uninitialized, saturating every decode to one token after allocator reuse; also accept ple_io=ram for a locked PLE table and print its load time. serve fast-exits (std::_Exit) on CUDA fault (Niko1221#224) instead of unwinding destructors that hang past the watchdog. 1. Introduced the dense path (dense_model, kernels, and --serve binary) tailored for the Qwen3.8-27B model. 2. The architecture utilizes the same line protocol as strata --serve through the unchanged StrataEngine. 3. Cleaned up CMakeLists by reverting parts of a prior commit to drop obsolete files and tests. 4. Implemented quantized KV cache support featuring fp16, int8, q4_0, and k8v4 formats. 5. Integrated a 256-dimension Walsh-Hadamard rotation to handle Q8 and Q4 quantized formats. 6. Updated chunked prefill kernels (kv_append_rows and attn_chunk) to natively read the quantized cache. 7. Removed the restriction that previously disabled chunked prefill when using a quantized KV cache. 8. Optimized the decode path by walking columns through chunk kernels, drastically reducing driver launches. 9. Fixed a pre-existing memory overrun bug within the bounds of the positions_kernel component. 10. Gated the MMQ prompt path verification on relative error instead of strict absolute bit parity. 11. Added GDN state checkpointing along with conversation history and last hidden states to pinned host RAM. 12. The GDN checkpoints prevent full prompt re-reading from token 0 when a chat template prefix breaks. 13. Introduced a Tensor Core (WMMA) chunk attention kernel for GQA 6:1 and head_dim 256. 14. The WMMA kernel implements a two-pass online softmax and handles Walsh-Hadamard rotations for Q4 keys/values. 15. Verified that the final conflict-resolution commits kept the crucial checkpointing logic fully intact.
perronemirko
pushed a commit
to perronemirko/Strata
that referenced
this pull request
Oct 12, 2026
New qwen35 attention kernel + core/qwen35 glue; wire into strata_engine and CMake (engine lib, build tests). gr.cu block_sum double->float to cut register pressure on the norm path; f32 sum of 2560 squares is close enough to ref/gr.py's float64 that the shuffle-order difference no longer matters. fused_gdn now owns the GDN step, so gdn.cu drops its softplus helper and the parity test trims unused layout code. generate: zero session state on startup (Niko1221#167) - native packs skip put_input at pos 0 and left R/KV/PLE uninitialized, saturating every decode to one token after allocator reuse; also accept ple_io=ram for a locked PLE table and print its load time. serve fast-exits (std::_Exit) on CUDA fault (Niko1221#224) instead of unwinding destructors that hang past the watchdog. 1. Introduced the dense path (dense_model, kernels, and --serve binary) tailored for the Qwen3.8-27B model. 2. The architecture utilizes the same line protocol as strata --serve through the unchanged StrataEngine. 3. Cleaned up CMakeLists by reverting parts of a prior commit to drop obsolete files and tests. 4. Implemented quantized KV cache support featuring fp16, int8, q4_0, and k8v4 formats. 5. Integrated a 256-dimension Walsh-Hadamard rotation to handle Q8 and Q4 quantized formats. 6. Updated chunked prefill kernels (kv_append_rows and attn_chunk) to natively read the quantized cache. 7. Removed the restriction that previously disabled chunked prefill when using a quantized KV cache. 8. Optimized the decode path by walking columns through chunk kernels, drastically reducing driver launches. 9. Fixed a pre-existing memory overrun bug within the bounds of the positions_kernel component. 10. Gated the MMQ prompt path verification on relative error instead of strict absolute bit parity. 11. Added GDN state checkpointing along with conversation history and last hidden states to pinned host RAM. 12. The GDN checkpoints prevent full prompt re-reading from token 0 when a chat template prefix breaks. 13. Introduced a Tensor Core (WMMA) chunk attention kernel for GQA 6:1 and head_dim 256. 14. The WMMA kernel implements a two-pass online softmax and handles Walsh-Hadamard rotations for Q4 keys/values. 15. Verified that the final conflict-resolution commits kept the crucial checkpointing logic fully intact.
perronemirko
pushed a commit
to perronemirko/Strata
that referenced
this pull request
Oct 14, 2026
New qwen35 attention kernel + core/qwen35 glue; wire into strata_engine and CMake (engine lib, build tests). gr.cu block_sum double->float to cut register pressure on the norm path; f32 sum of 2560 squares is close enough to ref/gr.py's float64 that the shuffle-order difference no longer matters. fused_gdn now owns the GDN step, so gdn.cu drops its softplus helper and the parity test trims unused layout code. generate: zero session state on startup (Niko1221#167) - native packs skip put_input at pos 0 and left R/KV/PLE uninitialized, saturating every decode to one token after allocator reuse; also accept ple_io=ram for a locked PLE table and print its load time. serve fast-exits (std::_Exit) on CUDA fault (Niko1221#224) instead of unwinding destructors that hang past the watchdog. 1. Introduced the dense path (dense_model, kernels, and --serve binary) tailored for the Qwen3.8-27B model. 2. The architecture utilizes the same line protocol as strata --serve through the unchanged StrataEngine. 3. Cleaned up CMakeLists by reverting parts of a prior commit to drop obsolete files and tests. 4. Implemented quantized KV cache support featuring fp16, int8, q4_0, and k8v4 formats. 5. Integrated a 256-dimension Walsh-Hadamard rotation to handle Q8 and Q4 quantized formats. 6. Updated chunked prefill kernels (kv_append_rows and attn_chunk) to natively read the quantized cache. 7. Removed the restriction that previously disabled chunked prefill when using a quantized KV cache. 8. Optimized the decode path by walking columns through chunk kernels, drastically reducing driver launches. 9. Fixed a pre-existing memory overrun bug within the bounds of the positions_kernel component. 10. Gated the MMQ prompt path verification on relative error instead of strict absolute bit parity. 11. Added GDN state checkpointing along with conversation history and last hidden states to pinned host RAM. 12. The GDN checkpoints prevent full prompt re-reading from token 0 when a chat template prefix breaks. 13. Introduced a Tensor Core (WMMA) chunk attention kernel for GQA 6:1 and head_dim 256. 14. The WMMA kernel implements a two-pass online softmax and handles Walsh-Hadamard rotations for Q4 keys/values. 15. Verified that the final conflict-resolution commits kept the crucial checkpointing logic fully intact.
perronemirko
pushed a commit
to perronemirko/Strata
that referenced
this pull request
Oct 17, 2026
New qwen35 attention kernel + core/qwen35 glue; wire into strata_engine and CMake (engine lib, build tests). gr.cu block_sum double->float to cut register pressure on the norm path; f32 sum of 2560 squares is close enough to ref/gr.py's float64 that the shuffle-order difference no longer matters. fused_gdn now owns the GDN step, so gdn.cu drops its softplus helper and the parity test trims unused layout code. generate: zero session state on startup (Niko1221#167) - native packs skip put_input at pos 0 and left R/KV/PLE uninitialized, saturating every decode to one token after allocator reuse; also accept ple_io=ram for a locked PLE table and print its load time. serve fast-exits (std::_Exit) on CUDA fault (Niko1221#224) instead of unwinding destructors that hang past the watchdog. 1. Introduced the dense path (dense_model, kernels, and --serve binary) tailored for the Qwen3.8-27B model. 2. The architecture utilizes the same line protocol as strata --serve through the unchanged StrataEngine. 3. Cleaned up CMakeLists by reverting parts of a prior commit to drop obsolete files and tests. 4. Implemented quantized KV cache support featuring fp16, int8, q4_0, and k8v4 formats. 5. Integrated a 256-dimension Walsh-Hadamard rotation to handle Q8 and Q4 quantized formats. 6. Updated chunked prefill kernels (kv_append_rows and attn_chunk) to natively read the quantized cache. 7. Removed the restriction that previously disabled chunked prefill when using a quantized KV cache. 8. Optimized the decode path by walking columns through chunk kernels, drastically reducing driver launches. 9. Fixed a pre-existing memory overrun bug within the bounds of the positions_kernel component. 10. Gated the MMQ prompt path verification on relative error instead of strict absolute bit parity. 11. Added GDN state checkpointing along with conversation history and last hidden states to pinned host RAM. 12. The GDN checkpoints prevent full prompt re-reading from token 0 when a chat template prefix breaks. 13. Introduced a Tensor Core (WMMA) chunk attention kernel for GQA 6:1 and head_dim 256. 14. The WMMA kernel implements a two-pass online softmax and handles Walsh-Hadamard rotations for Q4 keys/values. 15. Verified that the final conflict-resolution commits kept the crucial checkpointing logic fully intact.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A native (IQ) pack breaks out of the canonical
put_inputloop before position 0 and runs verify windows instead, sogeneratemode never reached thesession_zerocall that the canonical path uses. On a reusedcudaMallocallocation that left GDN, QSA KV/indexer, PLE history, andRuninitialized, producing NaN/overflow and degenerate identical-token output.Zero the session state explicitly when
native_packis set.Also removes an unused
argmax()helper that emitted a compiler warning, and ignores the local.pi/harness directory.