Ranger21
Ranger21 copied to clipboard
lessw2020
Ranger deep learning optimizer rewrite to use newest components
Are there any recommended settings for Transformer Language modeling?
To developer: Thanks a lot for developing such a nice project. There are many parameters to be set in Ranger21, but I don't know what these parameters do. If possible,...
Hello, should I use what kind of hyper-paramter for the first try? For example, learning rate, AdamW or Madgrad?
Can you provide a sample notebook on how to use ranger in fastai? Fastai has a ranger optimizer but how do I replace fastai's version with this version? and also...
As I learned ranger21 does internal lr scheduling etc. How should training be resumed? Is there a state dict to be loaded etc.?
Just wanted to say thank you. I've been "playing" with deep learning libraries and zoneminder dvr solution and really love that sphere as hobby. I will delete this comment later...
I found this problem while training ResNet18 on cifar100 for some experiment. I still haven't looked into this issue enough to find out what the cause is.
To developer: Thank you for developing such a grateful optimizer. I have used it with pytorch_1.8 and pytorch_1.9 successfully. When I use the pytorch_1.3.1, ranger21 reports some errors. I think...
Calling Ranger21 with mostly default parameters: ``` optimizer = ranger21.Ranger21( net.parameters(), lr=0.001, num_epochs=50, weight_decay=1e-5, num_batches_per_epoch=len(train_loader) ) ``` Training seems fine for half a day with decent progress on all loss...
Hi @lessw2020, thanks for the very nice work! I noticed that in this Ranger21, the optimizer is tightly coupled with the lr scheduler, could you guide me how I can...
