contrib
contrib copied to clipboard
Add COCOB Optimizer
This PR adds pytorch implementation of Cocob Optimizer, introduced in this paper. Feature request to add in pytorch/contrib is here.
Original Tensorflow implementation is here and this PR is based on this repo.
Apart of adding COCOB and it's tests, there is a small code refactor. A lot of code in TestSWA wasn't specific to testing SWA optimizer. So I have plucked that out into a general class TestOptim, and made TestSWA to inherit from TestOptim. TestCocob also inherits TestOptim to avoid code duplication.