[Bug] YourTTS alignment is weird

[thivux]

Describe the bug

I am training YourTTS model on VCTK using the recipe provided in this repo. But the alignment seems off, and the audio output is not great, as it often skips over words.

Below is the alignment plot. I tried both using VITS checkpoint trained on LJSpeech for 1M steps (like in the paper), and starting from scratch, but the alignment is still the same. Can someone confirm if the current code for training YourTTS is working or not? Thanks.

eval losses:

To Reproduce

• install TTS
• download checkpoint from https://github.com/coqui-ai/TTS/releases/download/v0.5.0_models/tts_models--multilingual--multi-dataset--your_tts.zip
• in recipes/vctk/yourtts/train_yourtts.py, change SPEAKER_ENCODER_CHECKPOINT_PATH and SPEAKER_ENCODER_CONFIG_PATH to your downloaded checkpoint
• (optional) change RESTORE_PATH to model_file.pth to start training from pretrained VITS checkpoint
• run train_yourtts.py

Expected behavior

• good audio, good alignment

Logs

No response

Environment

{
    "CUDA": {
        "GPU": [
            "A100-SXM4-40GB"
        ],
        "available": true,
        "version": "11.8"
    },
    "Packages": {
        "PyTorch_debug": false,
        "PyTorch_version": "2.1.1",
        "TTS": "0.22.0",
        "numpy": "1.22.0"
    },
    "System": {
        "OS": "Linux",
        "architecture": [
            "64bit",
            "ELF"
        ],
        "processor": "x86_64",
        "python": "3.9.18",
        "version": "#152-Ubuntu SMP Wed Nov 23 20:19:22 UTC 2022"
    }
}

Additional context

No response