[koush]

adds TokenizersBackend: PreTrainedTokenizer to the registry, so tokenizer configs declaring tokenizer_class: "TokenizersBackend" (the generic HF tokenizers Rust backend alias) resolve cleanly to the base class instead of tripping the unknown-class warning.

Add tokenizer-only package entry point so it can be used without loading onnx.

[xenova]

Hi there 👋 if you would like to use the tokenizers.js backend by itself, you can use it via https://www.npmjs.com/package/@huggingface/tokenizers, which is what Transformers.js uses as a dependency.
Repo: https://github.com/huggingface/tokenizers.js

[xenova]

(sorry, accidentally closed when I posted that comment)

Perhaps there is a particular use-case that you are trying to achieve? If so, and the above usage of tokenizers.js directly is insufficient, do let us know and we'd be happy to help :)

[koush]

my use case is a custom cuda inference engine that just needs the tokenizer stuff. from_pretrained and apply_chat_template specifically. which the raw @huggingface/tokenizers package doesn't have, it's in the transformers.js wrapper layer. the tokenizer-only entry point in this PR gives me that without onnxruntime getting pulled in via the main entry point.

[koush]

@xenova hi, any other way to address my use case?