hal-314

Thank you! Another option is document it. Once you know how to do it, it isn't a great deal. However, it took me some hours to figure out. First, discover...

Thank you to look into :)

@saudet Currently, I can't give it a try. However, from the commit code, I think it would be enough. Thanks!

@hankyul2 I don't think that @mathemusician solution is equivalent. `avg_fn` is called once per epoch while EMA updates happens every training step. I don't think that EMA can be implemented...

@hankyul2 I like more the approach from timm as it doesn't involve changing forward method. Users could not use the forward method in `xxx_step` method. So, I would recommend your...

@hankyul2 Here is the code. Be aware that you need `overrides` package installed (`pip install overrides`). If you don't want it, comment the import and the `@overrides` decorator. I only...

@flukeskywalker Glad to de that it's useful for you :) If you fix the multi gpu code, could you mind to share the fix? So, others can use it.

@hankyul2 Sorry but I don't have experience with multi-gpu in lightning. From the stack, it seems that the OOM occurs when broadcasting the state. [Here ](https://pytorch-lightning.readthedocs.io/en/1.5.4/api/pytorch_lightning.plugins.training_type.DDPPlugin.html#pytorch_lightning.plugins.training_type.DDPPlugin) is where I find...

@SeanNaren Be aware that #5542 prevents to load automatically EMA weights on validation/testing only (trainer.validate / trainer.test). On those situations, PL doesn't call `callbacks.on_load_checkpoint`. To fix it, you will need...

Validation and test sure. Finetune, I don't think so although I didn't check