[tokinasin]

Overview

Nemotron-3.5-Embed inserts U+200C immediately after [ and < in user input (Nemotron-3 does not). The official implementation removes these characters after identifying special tokens, but #30877 left them intact. This fix adds the missing normalization step.

Sorry for the oversight! 🙏

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: YES, I used Codex for research, code generation and reviews, etc.

[CISC]

Nemotron-3.5-Embed inserts U+200C immediately after [ and < in user input (Nemotron-3 does not). The official implementation removes these characters after identifying special tokens

Sorry, what? Where does it do that?

[tokinasin]

Sorry, what? Where does it do that?

The U+200C insertion is done at

https://huggingface.co/nvidia/Nemotron-3.5-Embed-1B-BF16/blob/main/chat_template.jinja#L7-L9

and the normalization is defined at

https://huggingface.co/nvidia/Nemotron-3.5-Embed-1B-BF16/blob/main/tokenizer.json#L9007

[CISC]

and the normalization is defined at
https://huggingface.co/nvidia/Nemotron-3.5-Embed-1B-BF16/blob/main/tokenizer.json#L9007

Oh wow, that's, hm, special...

"normalizer": {
    "type": "Sequence",
    "normalizers": [
      {
        "type": "Replace",
        "pattern": {
          "String": "[\u200c"
        },
        "content": "["
      },
      {
        "type": "Replace",
        "pattern": {
          "String": "<\u200c"
        },
        "content": "<"
      }
    ]
  }

[CISC]

Well, this sucks, but not likely we can generalize it within reason.

[CISC]

Rebase?