serve: tokenise the prompt from the last shared prefix, not from the start

[gputier]

Based on v0.1.39 since 5 October (it was first written on v0.1.37). The rebase takes blange48's port: on 0.1.39 the plain spans of #537 (a </think> quoted in a message is text) also decide which special matches are taken, so the shared prefix also ends where the two prompts' plain spans first differ, and the rest is encoded with its spans shifted by the resume point.

A chat client sends the whole conversation every turn, and Service.prepare runs BPE over all of it each time (serve/server.py, self.tok.encode(prompt, parse_special=True)), even when the engine's conversation cache already holds the whole prefix. /v1/messages/count_tokens does the same.

This PR adds PromptEncoder in tools/strata_tokenizer.py. It keeps the last few rendered prompts with their ids. For a new prompt it finds the last special-token boundary shared with one of them, reuses the ids up to that boundary, and sends only the text after it through BPE.

The ids match a full encode by construction. Encoding splits the text at special-token matches and encodes each stretch between two matches on its own, so nothing crosses a match. Whether a match is taken at position q depends only on text[q : q + max_special_len] and on whether q lies inside a plain span, so the cut sits at least max_special_len - 1 characters before the end of the common prefix. The class docstring has the reasoning.

The tokenizer gains max_special_len and encode_marked() (encode() plus the offset and id count after each special-token match). ByteTokenizer gets the same two, so the mock-engine tests exercise the encoder. 0.1.39's Service.encode_prompt(), which prepare and count_tokens already use, now calls the encoder. encode() itself is untouched.

Diff: 5 files, 3 of them tests.

Measurement

A conversation of 100,893 tokens (433 K characters) that grows by one turn, encoding time for the new prompt, 5 repetitions:

• full encode: 235 ms median (231 to 241)
• PromptEncoder: 1.5 ms median (1.05 to 2.07)

Container with 2 cores, Python 3.12, llama.cpp's ggml-vocab-qwen35.gguf (with the template's added tokens), synthetic text mixing English, code and CJK. This is not the model pack's tokenizer and not the target machine, so the absolute numbers will differ there.

Evidence that the ids are identical

On v0.1.39, in a Linux container (Python 3.13, llama.cpp's qwen35 vocabulary): serve.test_prompt_encoder and serve.test_server pass 146 tests, tools/test_strata_tokenizer.py passes 11 with 1 skipped (the pack tokenizer). The rest of this section was measured on the v0.1.37 version.

• serve/test_prompt_encoder.py (blange48's, added with the rebase): a fuzz against the full encode with random plain spans, spans that change inside the shared part, and edited histories, plus quoted <think> / </think> across turns through Service.encode_prompt.

• tools/test_strata_tokenizer.py (new, 11 tests): llama.cpp's qwen35 vectors, encode_marked against encode, chat_golden.json, and conversations compared token for token with tok.encode. Those cover several turns with unicode and literal special tokens in the text, tool definitions, calls and responses, images, a changed reasoning effort or system message, an edited earlier turn, a conversation that gets shorter, an empty message, and interleaved conversations, with 1, 2 and 4 kept prompts. Separate tests use byte-level vocabularies whose special tokens overlap each other, which is where a cut without the look-ahead margin breaks.

• serve/test_server.py, IncrementalPrompts: six turns over HTTP against the mock engine, the same conversations through Service.encode_prompt with both the byte tokenizer and qwen35, and a tokenizer without encode_marked.

• Mutations: with the margin set to 0, 5 tests fail. With the cut one id too late, 305 fail.

• Fuzz run outside the test suite: 40,000 cases, no difference in ids. 8,000 on qwen35 (seeds 1000 to 1099) and 32,000 on byte vocabularies with overlapping literals.

• Full Python suite (tools/test_*.py, serve/test_*.py) in a Linux container: 454 passed, 9 skipped, 0 failed on this branch, against 440 passed, 9 skipped on v0.1.37. That run uses scratch copies of both trees where setup.EXE is forced to strata.exe. Without it, 47 tests fail on Linux, the same 47 on a clean v0.1.37 (test_setup_golden and one test_setup_amd test), because they expect the Windows executable name. The change is not in this PR. The skips are the pack tokenizer, jsonschema and the Windows job objects. No existing test is modified.

Limits

• No fallback switch. The default path changes, with the same ids, and there is no environment variable to go back to the full encode. I can add one if you want it.
• Service now imports strata_tokenizer, and so regex, whenever the tokenizer has encode_marked. That includes the byte tokenizer used in tests without a pack. regex is already pinned in requirements.txt.
• The server keeps up to 4 rendered prompts in memory. Measured at 239,670 tokens (572,670 characters): one entry holds 3.6 MB of ids on top of its text, so the cap of 4 entries is about 15 MB of ids.
• /v1/messages/count_tokens shares the same encoder, so a count request can evict another client's entry. The cost is speed only, the ids stay the same.
• Not tested against the real engine or the model pack's vocabulary, only the mock engine and llama.cpp's qwen35 vocabulary.

[Niko1221]

Thank you, and thanks @sergqwer for the test with the pack's own tokenizer. This is independent of the work in 0.1.39, so it goes to a later release. We will review it then together with the server's other prompt-cache changes.

[blange48]

In case it helps the review: 0.1.39's plain spans (#537, a </think> quoted in a message is text) also decide which special matches are taken, so the resume point has to respect them. I rebased this on 0.1.39 with that one addition: the shared prefix ends where the two prompts' plain spans first differ, and the tail is encoded with its spans shifted by the resume point. Branch: https://github.com/blange48/Strata/tree/port-567 (commit d07043e, your authorship kept as co-author).

Checks:

• serve/test_prompt_encoder.py fuzzes it against the full encode: random plain spans, spans that change inside the shared part, and edited histories. It also covers quoted <think> / </think> across turns through Service.encode_prompt.
• With the Qwen tokenizer, a 12-turn agent conversation up to 145K tokens gives identical ids on every turn. Encoding takes ~83 ms per turn instead of 485 ms at the end.

[gputier]

Thanks @sergqwer for the run with the pack's own tokenizer, and @blange48 for the port. You're right about #537: on 0.1.39 the resume point has to respect the plain spans, and my branch didn't.

The PR branch is now on v0.1.39 (bfe4bcf), with your change and your fuzz test (serve/test_prompt_encoder.py), and you as co-author. The original tests stay: tools/test_strata_tokenizer.py and IncrementalPrompts in serve/test_server.py, adapted to 0.1.39's encode_prompt(messages, tools, kwargs).

On v0.1.39, Linux container, Python 3.13, llama.cpp's qwen35 vocabulary: serve.test_prompt_encoder and serve.test_server pass 146 tests, and tools/test_strata_tokenizer.py passes 11 with 1 skipped (the pack tokenizer, which I don't have).

@Niko1221 nothing more to do on my side until your prompt-cache review.

[blange48]

Thanks @gputier, great that it's on 0.1.39 with both test sets.

One heads-up for the review: #931 (control-token text inside a message stays text) changes the same spot in encode_prompt. It replaces the <think>-only plain spans with spans for every control token (mark_literals / unmark_literals), so the two PRs conflict textually. They do combine cleanly, because the encoder already takes arbitrary plain spans. The merge just passes #931's spans to PromptEncoder.encode:

plain = ()
if changed:
    prompt, plain = unmark_literals(prompt, self.literals)
if self.prompts is not None:
    return self.prompts.encode(prompt, plain)
return self.tok.encode(prompt, parse_special=True, plain=plain) if plain else \
    self.tok.encode(prompt, parse_special=True)

We run exactly that in production since this afternoon: 0.1.39 with both PRs on a 4-GPU split, agents (OpenCode, Claude Code) and chat. The server tests pass (193, including #931's and the fuzz test). Follow-up turns of a long conversation now reach their first token in 0.22-0.24 s.

[gputier]

Thanks @blange48, I merged #931 into this branch locally with your resolution and it holds for the code. Two of this PR's own tests did not, though: IncrementalPrompts.test_turns and test_a_tokenizer_without_resume_points_encodes_in_full in serve/test_server.py. Your port (d07043e) doesn't carry them, only test_prompt_encoder.py, so your 193 wouldn't have run them.

• test_turns puts <|im_end|> in a user message and compared the ids with tok.encode(prompt, parse_special=True), which reads it as a control token. With serve: control-token text inside a message stays text (<|im_end|> in a file no longer ends the turn) #931 it stays text, so the test failed on the expectation, not on the encoder.
• The stub tokenizer had no control_tokens, which serve: control-token text inside a message stays text (<|im_end|> in a file no longer ends the turn) #931's Service.__init__ reads.

Pushed d55679c: test_turns now takes its reference from encode_prompt on a Service without the encoder, and the stub forwards everything but encode_marked / max_special_len to ByteTokenizer. serve.test_server and serve.test_prompt_encoder in a Linux container, Python 3.13: 146 tests on this branch alone, 152 with #931 merged, all passing. So the two PRs can go in either order; whichever is second only needs your encode_prompt resolution.

[sergqwer]

Ran d55679c on Windows 11 (Python 3.13) with the tokenizer you were missing: the pack's own (Qwen3.8-Flash-Next, 248,320 tokens, STRATA_TOKENIZER=<pack>/tokenizer).

• tools/test_strata_tokenizer.py: 10 of 11 pass with the pack tokenizer. The one failure is the test, not the encoder. test_llama_cpp_vectors gets its tokenizer from load_tokenizer(), which prefers $STRATA_TOKENIZER, and then compares it with llama.cpp's qwen35 vectors. With both set, it encodes those vectors with another vocabulary. The first one differs at once: [1086, 220, 19, 220, 25229, 3818] against [1122, 220, 19, 220, 26062, 3951]. With only STRATA_GGUF_PY all 11 pass. The oracle could build its tokenizer from vocab_gguf() directly.
• The serve suite with the pack tokenizer: 274 of 275. The error is test_responses.OverHttp.test_json_schema_text_format, which 0.1.39 has on its own wherever jsonschema is not installed (serve tests: test_json_schema_text_format's schema failure only where jsonschema is installed #785).

[gputier]

Thanks @sergqwer, you're right: the oracle took load_tokenizer(), so with STRATA_TOKENIZER set it checked qwen35's vectors against the pack's vocabulary. Fixed in 76b916e: test_llama_cpp_vectors builds its tokenizer from vocab_gguf() (a new gguf_tokenizer(path), which load_tokenizer() also uses), and the other tests still prefer the pack.

Checked with a stand-in pack, since I don't have the real one: the qwen35 vocabulary with the template's added tokens and half its merges. With STRATA_TOKENIZER pointing at it, test_llama_cpp_vectors fails before the change and passes after; the whole file passes 11 of 11 with and without the pack, the vectors included.

[Niko1221]

Heads-up: today we cleaned up this repository's commit history, and GitHub closed this PR automatically when main was force-pushed (that is the "base branch was force-pushed" event above). It is not a rejection.

GitHub cannot reopen it, because its branch is built on the old history. If you want to keep it going: start a new branch from the new main, cherry-pick your own commits onto it (git fetch upstream && git switch -c my-fix upstream/main && git cherry-pick <your commits>), and open a new PR. Or simply leave it: when we take your change for a release we port it onto the new main ourselves and keep you as its author.

Sorry for the noise, and thank you for the contribution.