serve: tokenise the prompt from the last shared prefix, not from the start by gputier · Pull Request #1852 · Niko1221/Strata

gputier · 2026-10-10T14:22:42Z

This reopens #567 on the new main, as Niko1221 suggested after the history rewrite. It supersedes #567 and closes nothing.

Every request re-encoded the whole conversation before the engine saw it. PromptEncoder keeps the last prompts with their ids and resume points, and encodes only the text that follows the longest shared prefix. The ids are the ones a full encode gives, token for token.

Changes against #567

Rebased on v0.1.42 (61b3fb5). #931's literals (control-token text inside a message stays text) go through the incremental encoder: the shared prefix ends where the two prompts' plain spans first differ. blange48 merged #567 with #931 the same way on port-567, and stays credited as co-author. f346dc6's piece cache is kept, and the incremental encoder sits on top of it. Upstream's tools/test_strata_tokenizer.py tests are all kept, and ours are added.

Two fixes of #567's own behaviour:

Every rendered prompt starts with the template's header, a special-token boundary, so a prompt from another conversation counted as extending the kept one and replaced it. A subagent's request cost the main conversation its reuse on the next turn (8,276 characters reused, then 236). Now a prompt replaces an entry only when it reuses more than half of that entry's text. Otherwise it is a new entry, and the server keeps one per slot of the engine's conversation cache (--conversation-cache-slots, 4 without it). Token counts (/v1/messages/count_tokens) read the kept prompts without storing one.
A prompt refused because it does not fit the context is no longer kept by the encoder. Other refusals keep it, since the client resends the same prompt.

SECURITY.md now says that the server keeps the text and ids of its last few prompts in memory, and that no endpoint shows them.

Numbers and tests

Measured on v0.1.41 with llama.cpp's qwen35 vocabulary, Service.encode_prompt per turn, 12 measured turns per pass, in a 10-CPU Linux container on an Apple M5 Pro:

125K-token agent conversation resent with one more ~500-token turn: 42.5 ms per turn on v0.1.41, 11.0 ms with this PR (3 passes, spread between passes 0.5 ms). Final branch, one pass: 11.7 ms.
89K-token prompt resent identically: 29.5 ms on v0.1.41, 6.6 ms with this PR. Final branch: 6.6 ms.
Ids identical to v0.1.41 on all 84 compared turns, and on every turn of the final-branch run.
On a slower CPU the absolute times are higher: sergqwer measured 221 ms to 5 ms at 125K on serve: tokenise the prompt from the last shared prefix, not from the start #567, before the piece cache.

python -m pytest serve tools/test_strata_tokenizer.py: 598 passed, 18 skipped, 0 failed (v0.1.41 alone: 584 passed, 11 skipped). The 7 extra skips are the qwen35-vocabulary tests, which need STRATA_TOKENIZER or llama.cpp's vocab file. Fuzz: 2,400 random plain/literal-span cases, incremental against full, and 2,400 random message prompts through Service.encode_prompt with control-token and think text quoted. All equal.

Five commits on v0.1.41:

c319b11 serve: the prompt is tokenised from the last shared prefix, not from the start (co-author blange48)
b98660e serve: serve: tokenise the prompt from the last shared prefix, not from the start #567's server tests take their reference from a full encode_prompt
854564d tools: llama.cpp's qwen35 vectors are checked with the qwen35 vocabulary
8941f1f serve, tools: serve: tokenise the prompt from the last shared prefix, not from the start #567's tests and ByteTokenizer without repeated code
96e6139 serve: serve: tokenise the prompt from the last shared prefix, not from the start #567's kept prompts, one per conversation, none refused for the context

…the start Every request re-encoded the whole conversation: 235 ms for a 100K-token prompt on a two-core container, each turn. PromptEncoder keeps the last prompts with their ids and resume points, and encodes only what follows the longest shared prefix: 1.5 ms for the same turn. On 0.1.41 the plain spans of Niko1221#537 and Niko1221#931 (a </think> or an <|im_end|> quoted in a message is text) also decide which special matches are taken. The shared prefix now ends where the two prompts' plain spans first differ, and the rest is encoded with its spans shifted by the resume point (blange48's port). The ids are the ones tok.encode gives, token for token. The tests compare both paths on growing, edited and shortened conversations, with tools, images and special-token literals in the text, on the byte tokenizer and on the qwen35 vocabulary, and fuzz random plain spans against the full encode. A tokenizer without resume points keeps the full encode. Co-Authored-By: benoit lange <benoit@lange.xyz>

…ncode_prompt test_turns compared the ids with tok.encode of the rendered prompt, which reads <|im_end|> in a message as a control token. With Niko1221#931 that text stays text, so the reference is now encode_prompt on a Service without the encoder: the same check, whatever encode_prompt does with literals. The tokenizer without resume points forwards everything else to ByteTokenizer, so it also has Niko1221#931's control_tokens.

test_llama_cpp_vectors took its tokenizer from load_tokenizer(), which prefers $STRATA_TOKENIZER: with a pack set, it encoded qwen35's vectors with the pack's vocabulary and failed on the first one (sergqwer's run). It now builds the tokenizer from vocab_gguf() itself. With a pack whose merges differ, the test fails before the change and passes after.

… code ByteTokenizer.max_special_len reads the class's SPECIALS, so ThinkTokenizer no longer repeats it. spans_of keeps finditer's own non-overlapping scan. The tests load the qwen35 tokenizer once and encode each reference prompt once for the three keep values. No change to what is encoded.

…d for the context Every rendered prompt starts with the template's header, a special-token boundary, so a prompt from another conversation counted as extending the kept one and replaced it: a subagent's request cost the main conversation its reuse on the next turn (8,276 characters reused, then 236). A prompt now replaces only the entry whose text it reuses for more than half; otherwise it is a new entry (keep 4). The ids are unchanged: they are always the full encode's. A prompt refused because it does not fit the context is dropped from the encoder, as its ids were before Niko1221#567. The other refusals keep it: the client sends the same prompt again. SECURITY.md says the server keeps the last few prompts in memory, whatever "api_monitor" says.

PromptEncoder kept four prompts whatever --conversation-cache-slots was. With eight slots, a fifth conversation evicted a kept prompt while the engine still held its cache entry. The count now comes from the INFO conversation_cache_slots the engine reports, and stays at four without it.

/v1/messages/count_tokens went through the same encode as a request, so a count on any other conversation added an entry and could evict the main conversation's kept prompt. A count now reuses a kept prefix but leaves the kept prompts untouched, and returns the same ids as a full encode.

…y start too Service read the slot count from engine.info, which stays empty until the engine has started, so a lazy start kept four prompts whatever the setting. The count now comes from the engine's own --conversation-cache-slots argument, known at construction, then from the INFO line, and is four without any setting.

gputier · 2026-10-11T03:21:19Z

Rebased on v0.1.42, with two review fixes and a test for each.

The conflicts were all in serve/server.py: the tokenizer caches, the "request exceeds the context window:" prefix (two places) and check_request_sampling. v0.1.42's code is kept as is in each of them, and this change sits on top.

New commits:

The encoder keeps one prompt per slot of the engine's conversation cache (--conversation-cache-slots), instead of a fixed 4. The value comes from the engine's INFO line, or from its arguments when the engine starts lazily. Without the setting it stays 4.
/v1/messages/count_tokens encodes without storing anything, so a count no longer replaces the prompt kept for a running conversation.

Tests: python -m pytest serve tools/test_strata_tokenizer.py gives 715 passed and 8 skipped, with jsonschema installed and llama.cpp's ggml-vocab-qwen35.gguf found through STRATA_GGUF_PY. Plain v0.1.42 gives 688 passed in the same container. This branch has served an RTX 5090 (IQ3_S, 1M context, 8 conversation slots) since October 11.

…ltimo prefisso condiviso (STRATA_PROMPT_ENCODER) Upstream Niko1221#1852 (Guillaume Putier, pr/prompt-encoder-0.1.41, 8 commit da e13bb1f... a 9b92530, schiacciati qui): encode_prompt ritokenizza solo cio' che segue l'ultimo token speciale condiviso con un prompt recente (PromptEncoder in tools/strata_tokenizer.py), un prompt tenuto per slot di --conversation-cache-slots. Stessi id di un encode completo (test della PR). Stima della PR: -55 ms a turno a 125k. Fork: opt-in con STRATA_PROMPT_ENCODER=1 nell'ambiente del server finche' il banco TTFT non lo misura; test_server.py lo accende per i test della PR. Test: test_prompt_encoder 7 OK, test_strata_tokenizer 11 OK (3 saltati), test_server 304 OK. Nessun codice del motore. Co-Authored-By: Guillaume Putier <guillaume@putier.fr> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…1852, banco TTFT, penalita' e build a2111f8 Stato di lab/ttft43: i tre commit di lab/ttft portati puliti sulla 0.1.43, STRATA_CONV_DISK, Niko1221#1852 opt-in, il banco TTFT e i suoi casi, il controllo delle penalita' di ripetizione nei log (nessuna richiesta seriale), la build 22a2eb85. Compilato si', provato no. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…start (#1852, #567) PromptEncoder keeps the last prompts with their ids and resume points (one per conversation-cache slot) and encodes only the text after the longest shared special-token boundary; the ids equal a full encode. Rebased onto 0.1.44.1 (prepare() keeps the new `cancel` argument; forget_prompt on a context refusal; count_tokens reads without storing). SECURITY.md notes the in-memory prompts. Based on gputier's #1852 (8 commits) and blange48's #567. Co-authored-by: Guillaume PUTIER <guillaume@putier.fr> Co-authored-by: benoit lange <benoit@lange.xyz>

Niko1221 · 2026-10-17T07:13:22Z

Thank you, this is in for 0.1.45 (f13a870, co-authored with you and blange48). I rebased it onto 0.1.44.1 by hand because of the new cancel argument in prepare(). On a synthetic 140K-token conversation the warm encode went from 22.6 ms to 5.4 ms with identical ids. #1455 and #1823 do the same job and are covered by this one.

gputier · 2026-10-17T11:40:07Z

Thank you, and thanks for the rebase by hand. I checked f13a870 against this branch: tools/strata_tokenizer.py, serve/test_prompt_encoder.py, tools/test_strata_tokenizer.py and SECURITY.md are byte-identical, and serve/server.py carries the same changes on top of 0.1.44.1 (forget_prompt on a context refusal, store=False for count_tokens). Closing this one since it is in 0.1.45.

Niko1221#1934 non reprise)

…ancements - Automatic thinking budget protection (Niko1221#984, commits d0e990b, 194c268) - Anthropic thinking.budget_tokens API mapping in frontend.py - Prompt tokenizer prefix caching and resume points (Niko1221#567, Niko1221#1852, commit f13a870) - Dynamic conversation slot tracking in PromptEncoder matching engine settings - Windowed Prometheus throughput gauges covering scrape intervals (Niko1221#2341, commit a091cdf) - Preserve explicit False reasoning effort against model defaults (commit e69cbfa) - Unit test suites in test_prompt_encoder, test_strata_tokenizer, test_server, test_effort_paths - Custom feature benchmark tool tools/custom_tokenizer_bench.py

gputier added a commit to gputier/StrataGP that referenced this pull request Oct 10, 2026

docs: FORK.md, the fork restarts from v0.1.41 with Niko1221#1852 only

ca56071

gputier and others added 8 commits October 10, 2026 23:05

gputier force-pushed the pr/prompt-encoder-0.1.41 branch from 96e6139 to 9b92530 Compare October 11, 2026 03:21

k0valik mentioned this pull request Oct 15, 2026

replay (review only): hub.lumenfield.work-gputier-StrataGP-main — filtered squash of 1138 commit(s) from https://github.com/gputier/StrataGP k0valik/Strata#1

Open

gputier closed this Oct 17, 2026

gputier mentioned this pull request Oct 17, 2026

Passer la .99 en 0.1.45 : A/B à 8 paires contre la 0.1.42, puis bascule gputier/StrataGP#66

Open

6 tasks

gputier added a commit to gputier/StrataGP that referenced this pull request Oct 17, 2026

docs: FORK.md pour la base v0.1.45 (Niko1221#1852 reprise dans l'amont,

b7c227a

Niko1221#1934 non reprise)

gputier deleted the pr/prompt-encoder-0.1.41 branch October 17, 2026 12:51

Navigation Menu

Repository navigation

serve: tokenise the prompt from the last shared prefix, not from the start - #1852

serve: tokenise the prompt from the last shared prefix, not from the start#1852
gputier wants to merge 8 commits into
Niko1221:mainfrom
gputier:pr/prompt-encoder-0.1.41

gputier commented Oct 10, 2026 •

edited

Loading

Uh oh!

gputier commented Oct 11, 2026

Uh oh!

Niko1221 commented Oct 17, 2026

Uh oh!

gputier commented Oct 17, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

gputier commented Oct 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes against #567

Numbers and tests

Uh oh!

gputier commented Oct 11, 2026

Uh oh!

Niko1221 commented Oct 17, 2026

Uh oh!

gputier commented Oct 17, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

gputier commented Oct 10, 2026 •

edited

Loading