MegaCocos

@rockerBOO Could you help giving an example of SequentialLR command (with two different schedulers as arguments)? By some reason it didn’t work for me 😒 (for example, CosineAnnealingWarmRestarts command that...

> Hi everybody, > > I want to define my own custom optimizer. How can I do that? Can you explain this or provide an example? > > [@rockerBOO](https://github.com/rockerBOO) [@kohya-ss](https://github.com/kohya-ss)...

> > SequentialLR > > Those require instances of the other schedulers, so that wouldn't work with the current system unless you made a custom one that initialized the 2...

@rockerBOO have one more question with lr_scheduler options for sd-scripts, could you help to clear it? When i use ` --optimizer_type adafactor --optimizer_args "relative_step=False" "scale_parameter=False" "warmup_init=False" --lr_warmup_steps 0.1 --lr_scheduler cosine...

> > How can i get this constant max lr between warmup and cosine start? > > you would need to use another lr scheduler as cosine doesn't have a...

> This is very interesting... I've been using `network_dim = network_alpha` for a long time (Flux.1 dev), it gets good results but some times the output comes out degraded and...