Training acoustic model with predictions from the variance model

[blueyred]

Once a variance model has completed training, would it be possible to use it's output, rather than the ground truth for training the acoustic model?

I would expect to get better results at inference, compared to the detatched method where they both get trained on the ground truth.

[yqzhishen]

How can you train acoustic models with generated pitch but without the corresponding mel-spectrogram? We only have the ground-truth mel-spectrograms.

[blueyred]

I was imagining using the f0 prediction & duration predictions to replace the ground truth in the acoustic training dataset (which has the mel-spectogram).

So train the acoustic network to learn to generate the mel-spectrogram from the output of the variance model, rather than the ground truth f0 & duration data.

The process could be a branch in the acoustic binarizer to take the raw training data, pass in fields to a trained variance model to predict the ph_dur & f0_seq, which would replace the ground truth ph_dur & f0_seq into the training data.

[yqzhishen]

No, the goal of acoustic models is to be as accurate as possible. Not using the ground truth duration and pitch will be misleading to the networks.