[sergqwer]

Service.prepare() renders the chat template and tokenizes the whole prompt with parse_special=True. The template writes <|vision_start|><|image_pad|><|vision_end|> for each image item, and the #150 guard turns only an <|image_pad|> not preceded by <|vision_start|> back into text.

Message text that contains the whole marker pair becomes the same special ids as a real image. That happens with an agent reading serve/chat_template.jinja, these docs, or a tool result quoting them. If such text comes before a real picture:

• picture 1's embedding rows go to the text's marker;
• every later picture moves up one;
• the last real marker becomes text.

The counts still match, so nothing reports it. With Claude Code reading a repository and a screenshot in one conversation, this is an ordinary case.

Fix (serve/server.py, Service.prompt_ids()):

• The vision markers inside the messages' and tools' strings are swapped for a unique placeholder before rendering.
• After rendering, each placeholder becomes its marker's literal (parse_special=False) tokens. The pieces between placeholders are tokenized as before, since a special token split the text there anyway.
• A prompt without such text takes the old path: byte-identical ids, checked on the real tokenizer too.
• count_tokens uses the same ids.
• If text still yields more marker pairs than there are pictures (a marker cut between two text parts), that is now a 400 instead of a silent shift.

Test test_the_whole_marker_in_text_before_images: each picture's rows follow its own marker, and the text keeps its literal tokens. Without the fix it finds 3 vision_start where 2 belong. The #150 test is unchanged. Serve tests: 173 OK (7 skipped).

This touches the first lines of prepare(), as #529 does; whichever goes in second needs a one-line rebase.

🤖 Generated with Claude Code

https://claude.ai/code/session_01VZy1yKaDDiA8a7svdwaHio

[sergqwer]

Rebased onto 0.1.39 as a new implementation (1273345). 0.1.39's #537 already solves the same problem for <think> / </think>: it marks literal tags in the messages with private-use characters, renders, and encodes those spans as text through the tokenizer's plain spans. So the vision markers now take that path:

• The fix: frontend.VISION_TAGS joins THINK_TAGS as LITERAL_TAGS. A <|vision_start|>, <|image_pad|>, <|vision_end|> or <|video_pad|> written in a message's or a tool's text is encoded as text. The markers the template writes for an image item stay special.
• Removed: this PR's own placeholder tokenizing (prompt_ids, hide_markers). The server keeps encode_prompt from 0.1.39 and calls nothing new.
• Unchanged: the refusal of a marker pair cut apart across text parts ("the prompt and its images do not match").

The test (ImageMarkers.test_the_whole_marker_in_text_before_images) fails on a clean v0.1.39 (3 failures) and passes with this branch. A prompt without the marker strings gets exactly the ids it had. I dropped <think> from that case, since #537 now encodes it as text by design.

[sergqwer]

Reopened as #1045, the same branch and commit. GitHub closed this one when my fork was made private by mistake, which took it out of the fork network.

[Niko1221]

Merged for 0.1.40: the vision markers inside a message's text stay text and no longer take a picture's place.