[lunosia]

Documents the requirement reported in #7715.

BpeOptions.ByteLevel only documented the byte <-> unicode mapping (for example Space -> 'Ġ').
It did not state that a pre-tokenizer is required. Without one, byte-level encoding maps the space
character to 'Ġ' but does not keep the whitespace attached to the following token, so spaces and
newlines are dropped during encoding and cannot be recovered by decoding - with no exception and
no warning.

This adds a <remarks> block to the ByteLevel property that:

• states that PreTokenizer must be set when ByteLevel is true;
• describes the failure mode (silent whitespace loss, not recoverable on decode);
• points at RegexPreTokenizer built from the GPT-2 pattern as a configuration that round-trips losslessly.

Measured against HuggingFace tokenizers as the reference, a properly configured pre-tokenizer makes
encoding match token-for-token and decoding exact, while without one the lossless round-trip rate drops
to 1/5 on a small mixed-language sample.

This is a documentation-only change: no code and no public API surface change.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 69.95%. Comparing base (f1c6972) to head (bc05887).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #7716      +/-   ##
==========================================
- Coverage   69.95%   69.95%   -0.01%     
==========================================
  Files        1419     1419              
  Lines      272603   272603              
  Branches    27562    27562              
==========================================
- Hits       190707   190706       -1     
  Misses      73701    73701              
- Partials     8195     8196       +1     

Flag	Coverage Δ
Debug	69.95% <ø> (-0.01%)	⬇️
production	64.14% <ø> (-0.01%)	⬇️
test	89.69% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
src/Microsoft.ML.Tokenizers/Model/BpeOptions.cs	68.42% <ø> (ø)

... and 11 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[rosebyte]

Could we limit this claim to preserving whitespace during pre-tokenisation? Normalisation and missing vocabulary entries can still change or discard input, so the GPT-2 regex alone cannot guarantee a lossless round trip.

[rosebyte]

Could we describe the default pre-tokeniser’s behaviour rather than make this a requirement of ByteLevel?

[lunosia]

Thanks for the review — I've reworded the remarks accordingly:

• Dropped the "lossless round trip" claim; the statement is now limited to whitespace during pre-tokenization.
• Replaced the "PreTokenizer must be set" requirement with a description of the default pre-tokenizer (CreateWordOrNonWord), whose pattern doesn't cover whitespace characters.

Please take another look when you have a moment.