Repository navigation
Tokenizer: byte-level BPE from GGUF, exact against HF tokenizers - #2
Merged
Merged
Conversation
…zers - engine/core/tokenizer: added-token split, NFC, hand-ported Qwen2 split regex, GPT-2 byte mapping, BPE merges, decode. Unverified tokenizer.ggml.pre values are refused. - engine/core/unicode: UTF-8 codec, letter/number/whitespace classes, NFC. Tables generated from Python unicodedata 16.0.0 by tools/gen_unicode_tables.py (no ICU). - tests: test_unicode (CI), golden_tokenizer checking pre-token pieces, ids and decoded bytes. The Qwen3 vocab-only GGUF (5.9 MB) and 481 cases are committed so it runs in CI. 481 committed cases and 20,081 random cases: 0 mismatches on split, encode and decode. Nine of ten single-rule mutations fail the test; the tenth is equivalent. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Sep 22, 2026
bong-water-water-bong
pushed a commit
that referenced
this pull request
Sep 26, 2026
…ama.cpp #2, #5, #6) The pin adds the zaya architecture and its converter (#2, #5) on top of #104's #95 fixes, and carries #106's scalar-scatter SET_ROWS onto the tracked branch (#6): #106 pinned c358334, which is on no branch of the fork. registry/architectures.json, regenerated: ZayaForCausalLM maps to hrx and vulkan (266 architectures, all mapped). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
bong-water-water-bong
added a commit
that referenced
this pull request
Sep 26, 2026
…nly architectures (#107) * ZAYA1 (Zyphra) on Vulkan: serve routes fork-only architectures to our llama.cpp `1bit serve --device vulkan` reads the GGUF's general.architecture (app/gguf_meta.h) and runs an architecture only our llama.cpp implements (zaya) on the HRX build's Vulkan0, even when the upstream Vulkan build is present. registry_build counts such architectures for vulkan the same way, and check_models.tsv checks ZAYA1-8B Q4_K_M on vulkan. docs/vulkan.md: converting and serving ZAYA1, with the checks against transformers and the measured speeds on Strix Halo. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ZAYA1 on HRX and ROCm: docs, and serve_e2e on vulkan and hrx in the registry Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Pin llama.cpp 5556bf2: ZAYA1 on Vulkan, HRX and ROCm (1bit-MONSTER/llama.cpp #2, #5, #6) The pin adds the zaya architecture and its converter (#2, #5) on top of #104's #95 fixes, and carries #106's scalar-scatter SET_ROWS onto the tracked branch (#6): #106 pinned c358334, which is on no branch of the fork. registry/architectures.json, regenerated: ZayaForCausalLM maps to hrx and vulkan (266 architectures, all mapped). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * registry: vulkan and hrx rows re-checked at b9812e0 (16/16 pass, ZAYA1-8B included) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ci: build gguf_meta_test Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 3, 2026
bong-water-water-bong
pushed a commit
that referenced
this pull request
Oct 12, 2026
…ontains Read-only; no arms, no box. The sentinel instrument cannot support the inference drawn from it. GGML_HRX_SENTINEL fills the 619520-byte host-visible region inside hrx_buffer_init, i.e. once per allocation at load: the faulting arm has exactly one fill line at 0.44s and faults at 2.02s after ~1087 tokens wrote that region, so sentinel=0 only says an earlier token overwrote the pattern. Section 22 finding #2's 'not one is our payload, so the write is present' over-reads it. The conclusion survives on sounder grounds: call 1086 was finite with hash f38cc8b5b54dbb8f, call 1087 is nan=154880, and no constant NaN payload reproduces 91a04b03818e7083 - so the faulting graph did write all 154880 elements as NaN. Face A stands, for a different reason. Fork research (1bit-MONSTER/llama.cpp and AMD-Ecosystem/llama.cpp): - e6187096b (Oct 2) fixed exactly this class: 'masked KV rows reach flash-attention P*V as V = +0', with tests/test-hrx-fa-masked-v (27 cases). - That fix is ALREADY IN THE PIN: e6187096b is an ancestor of e44c9d01a, as are both 314 commits, and the zeroing comment plus the test are in the tree. - 314's predicate rewrite is semantically equivalent on the source (same >-1e30 test, same LDS array, published by subgroup 0 before a workgroup barrier), so it is not a regression. - fb4dff7bb (engine-315/host-visible-coherence, Oct 9) is forward-only but section 13 already falsified coherence as a sufficient mechanism. - Nothing post-pin on either remote fixes an FA/visibility/NaN defect. Surviving hypothesis: the zeroing predicate is one mask value per key while the accumulation covers several query rows, so a tile masked for the accumulating row can be judged all-visible and read directly - finite stale V gives Face B, non-finite gives tiny_P * NaN = Face A. Cheapest discriminator is the existing 27-case test extended with non-finite masked V and multi-row differing-key masks: seconds, not 60 in-regime arms.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Text in, token ids out, with no Python and no ICU. It reads the vocabulary from GGUF metadata.
What
engine/core/tokenizer.*: the same steps as HFtokenizers, in order:tokenizer.ggml.prevalues it has no golden for.engine/core/unicode.*: UTF-8 codec, character classes and NFC. The tables come from Pythonunicodedata16.0.0 viatools/gen_unicode_tables.py.docs/tokenizer.mdexplains the design, the verification and the known limits.Verification
golden_tokenizerchecks three things per case, all exact against HFtokenizers0.22.2:Mutation sweep: ten single-rule deletions, nine caught (for example, dropping NFC gives 187 mismatches). The tenth, the
\p{N}branch, is an equivalent mutant. An earlier version of the cases missed contraction rules, because" 'd"is split by the punctuation branch before the contraction branch runs. The cases now put contractions after letters.Test plan
cteston strixhalo: 5/5 (incl.golden_cpu)ctestlocally: 4/4 (no model needed)-Wall -Wextra -Wpedantic🤖 Generated with Claude Code