RVC-Project
Filled up my SSD (storage leak issue)
Hello! First of all, thank you for this incredible software. This is the first voice cloning AI that works for me! Sadly, I met a pretty huge issue. After a couple of hours, I realized the software was filling up my SSD completely. I had 40GB left after install, and a mysterious bug took all of it. I investigated, and realized the "log" folder was 40GB big!
I deleted it and things got back to normal, except I lost the "indexes" (I don't know what these do, but my tests without them work fine so I don't mind).
I suggest people check out their drive regularly while using this. I installed the software on a secondary SSD so it didn't brick my PC, but it could have been terrible had it been my main...
confirm if you choose to save the ckpt file for each epoch. You can select to save only the latest ckpt file in the training web to reduce the storage space occupied by training files
I have the save frequency set to 5 epochs. I had tried training a model with 100 epochs right before I noticed the bug, and it did fail. So is that the issue? Did the back ups fill up the SSD and I only noticed it afterwards? That would explain it!
The issue is that you did not choose 'save only latest ckpt' which means for every save interval it wasnt deleting the old epoch
Thank you! I no longer have this issue! It might be a good idea to set "save only latest ckpt" to True by default ^^.
Maybe you need "save only latest ckpt". Also if you want to test the result of intermediate epoch model, you can also set "Save a small finished model to the 'weights' directory for every epoch matching the specified 'save frequency' :"to true.
If your hard drive space is tight, when the training is completed and there is no need to continue training, you can delete files other than the index in the folder corresponding to the log/experiment name.
