Deepspeed support CosineAnnealingLR scheduler

[hahchenchen]

Now Deepspeed do not support CosineAnnealingLR scheduler.

So i want to support it by meself. question is how to develop a custom scheduler, Are there any tutorials available?

[DavdGao]

The same question for me

[tjruwase]

@hahchenchen and @DavdGao, unfortunately we don't have a tutorial for this. However, there are two options available for this issue:

1. You can directly pass the torch implementation into deepspeed.initialize() as documented here.
2. You can implement directly in deepspeed by following existing custom implementations, such as WarmupLR or OneCycle.

[HsuWanTing]

Hi @tjruwase ,

You can directly pass the torch implementation into deepspeed.initialize() as documented here.

For this option, do you mean like the following?

model, optimizer, _, lr_scheduler = deepspeed.initialize(
        model=model,
        args=args,
        lr_scheduler=torch.optim.lr_scheduler.CosineAnnealingLR,
        config_params=ds_config
    )

If yes, there is a T_max argument in CosineAnnealingLR that is required. How to pass this argument?

[tjruwase]

@HsuWanTing, you can also pass lr scheduler as a Callable, which should work for your case. Please see the following example https://github.com/microsoft/DeepSpeed/blob/3dd7ccff8103be60c31d963dd2278d43abb68fd1/tests/unit/runtime/test_ds_initialize.py#L254