Checkpoint set and restore

Open rspiewak47 opened this issue 4 years ago • 2 comments

args.restore_step is referenced in train.py and model.py. Is there any documentation on recommended use for this argument in checkpointing? Thanks!

May 12 '21 14:05 rspiewak47

@rspiewak47 It is for the case you want to restart the training from an available checkpoint.

May 26 '21 07:05 ming024

args.restore_step is referenced in train.py and model.py. Is there any documentation on recommended use for this argument in checkpointing? Thanks!

it will save checkpoint automatically each 100000 steps, you can change save_step in /config/LJSpeech/train.yaml to make it save earlier or later to restore and continue training, put the checkpoint file (ex: 15000.pth.tar) in /output/ckpt/LJSpeech and add --restore_step 15000 at the end of train command

May 26 '21 09:05 EuphoriaCelestial