Metalhead.jl icon indicating copy to clipboard operation
Metalhead.jl copied to clipboard

Potential enhancements for `DropBlock`

Open theabhirath opened this issue 3 years ago • 0 comments

DropBlock is a type of regularisation that tries to replace dropout. The original paper describes it as best used with a linear scaling rate across blocks in a model, as has been implemented in #174. However, timm adds several experimental features, including:

  • [x] a scaling value for the gamma calculated (already included in #174).
  • [ ] a certain configuration of DropBlock rates picked by experimentation to give better results on ResNets - these however extend only to four-stage ResNets i.e. those similar to the models in the original ResNet paper.

theabhirath avatar Jul 26 '22 14:07 theabhirath