[nico-martin]

Adds Gradium Phonon TTS (gradium_phonon_tts), ported from the Python Transformers implementation. It includes text preparation via a JS port of PocketTTSProcessor, mixed-precision model variants (fp32, q4, q4f16), flat float32 voices, and PCM streaming at the model level.

Non-streaming

import { pipeline } from "@huggingface/transformers";

const tts = await pipeline("text-to-speech", "onnx-community/phonon-31m-en-ONNX", { dtype: "q4" });

const audio = await tts("Hello from Phonon!");
await audio.save("hello.wav");

Custom voice

Voices are flat float32 files in [1, voice_dim, frames] order, the same approach as Supertonic. The model repo ships all 102 Phonon voices under voices/, and voices/voices.jsonl lists their name, language, gender and description. Without speaker_embeddings, the pipeline uses the checkpoint's config.phonon.default_voice (voices/default.bin).

// Easton, en-us, male
const audio = await tts("Hello from Phonon!", { speaker_embeddings: "voices/LMDIYE0Oa377Qh6Z.bin" });
await audio.save("hello-easton.wav");

speaker_embeddings accepts a URL, a Float32Array, or a Tensor. Nothing is cached, so to reuse a voice without downloading it again, load it once and pass the tensor.

Streaming

The pipeline always returns complete audio. If you want streaming, you can use generate_speech(), which yields 1920-sample PCM frames at 24 kHz as soon as each one is decoded:

import { AutoProcessor, GradiumPhononTTSForConditionalGeneration } from "@huggingface/transformers";

const id = "onnx-community/phonon-31m-en-ONNX";
const processor = await AutoProcessor.from_pretrained(id);
const model = await GradiumPhononTTSForConditionalGeneration.from_pretrained(id, { dtype: "q4" });

const speaker_embeddings = await model.load_voice(); // or model.load_voice("voices/LMDIYE0Oa377Qh6Z.bin")
const chunks = await processor("Hello from Phonon!");

for await (const pcm of model.generate_speech({ chunks, speaker_embeddings })) {
  // Enqueue pcm (Float32Array, 1920 samples) for playback at 24 kHz.
}

The voice is processed once per generate_speech() call and shared across all text chunks. It is freed when the loop finishes, throws, or exits early with break.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[xenova]

very cool!

not a full review, but it should hopefully provide some guidance before I can review it fully.

[xenova]

generate_speech does not exist on GradiumPhononTTSForConditionalGeneration

[xenova]

do you have a reference implementation for streamed version in python transformers library?

[nico-martin]

Yes, you're right: Python uses generate() and returns latents and the processor decodes them. I went with generate_speech because SpeechT5/Supertonic do it that way in JS.

And no, there's no streaming reference in Python. The PocketTTS mixin just ignores streamer with a warning. Streaming is a JS-only extra. It works because the decoder is causal, so we can decode frame by frame.

What do you think would be a good way forward?
I could keep the pipeline returning full audio only and leave streaming as a JS-only generator on the model, or drop streaming from this PR and do it in a follow-up.

[xenova]

this should be fixed in our tokenizer.json for now -- no need to add this custom behaviour.

[nico-martin]

The gap is actually in tokenizers.js: BPE supports byte_fallback, but Unigram ignores the flag, so I can't fix this in tokenizer.json alone. There's already an issue for it: huggingface/tokenizers.js#26

I can pick that up next. Until it lands, should I keep this small adapter, or drop it now and accept for rare characters (emoji, CJK, etc.)?

[nico-martin]

@xenova, Thanks for the early pass, super helpful! I went through all the comments. Individual replies are in the threads.