ivan provalov

Results 51 comments of ivan provalov

"RuntimeError: [!] NaN loss with loss" on GlowTTS introduction example - mailabs dataset

Also, I did resample this dataset before the training: ``` python3 -c "import librosa;print(librosa.get_samplerate('northandsouth_01_f000008.wav'))" 22050 ```

"RuntimeError: [!] NaN loss with loss" on GlowTTS introduction example - mailabs dataset

@erogol Nothing is posted on the tensorboard, since it is failing after 75 steps. Here is a reduced log (without the entire pre-computing phonemes output) - [glow_tts_en.log](https://github.com/coqui-ai/TTS/files/8993188/glow_tts_en.log)

"RuntimeError: [!] NaN loss with loss" on GlowTTS introduction example - mailabs dataset

> Is this duplicate #1750 ? I think the two issues are related, but not a duplicate. This #1683 issue is about MAILABS format causing glow tts exception, while #1750...

Address Already in Use Error When Training on 2 GPUS and Starting a New Job on the Remaining 2 GPUS

Thank you, @erogol . Is it `distributed_url` parameter?

Address Already in Use Error When Training on 2 GPUS and Starting a New Job on the Remaining 2 GPUS

Thank you, @erogol , works great!

Address Already in Use Error When Training on 2 GPUS and Starting a New Job on the Remaining 2 GPUS

@erogol here is what happens when I use the distributed training with multiple GPUs. At first everything looks OK, but then the second job fails. I ran this on two...

Address Already in Use Error When Training on 2 GPUS and Starting a New Job on the Remaining 2 GPUS

@lexkoro I am not sure about the phoneme preprocessing in this case. My second job was a vocoder, which starts training loops right away and fails on the first epoch...

Address Already in Use Error When Training on 2 GPUS and Starting a New Job on the Remaining 2 GPUS

@lexkoro I think you are right for the model side. I re-ran the phoneme preprocessing on a single GPU, then restarted the process on multiple: `nohup python3 -m trainer.distribute --script...

Address Already in Use Error When Training on 2 GPUS and Starting a New Job on the Remaining 2 GPUS

@lexkoro @erogol @Dapwner I ended up debugging this more at runtime and tested a fix. It turns out that the current Trainer `distributed.py` behavior is using a GPU id for...

Address Already in Use Error When Training on 2 GPUS and Starting a New Job on the Remaining 2 GPUS

@erogol @Dapwner If you request GPU `0` in your list of GPUs, it will be used correctly, as in the case with my first command, but otherwise, it's the first...