Metalhead.jl
Metalhead.jl copied to clipboard
Potential enhancements for `DropBlock`
DropBlock is a type of regularisation that tries to replace dropout. The original paper describes it as best used with a linear scaling rate across blocks in a model, as has been implemented in #174. However, timm adds several experimental features, including:
- [x] a scaling value for the
gammacalculated (already included in #174). - [ ] a certain configuration of
DropBlockrates picked by experimentation to give better results on ResNets - these however extend only to four-stage ResNets i.e. those similar to the models in the original ResNet paper.