mapbox
Allow arbitrary number of input channels in ResNet encoder (not only RGB)
With https://github.com/mapbox/robosat/pull/46 we are changing our model architecture from training the encoder and decoder from scratch to using a pre-trained ResNet for the encoder. The pre-trained ResNet uses three channels (RGB) for the input layer through.
We need to be able to add arbitrary channels, say, RGB + water mask + elevation + lidar. To to this we need to construct a wrapper module architecture extending the ResNet architecture, copying weights over, and initializing the new channels with zero. In addition the channel-wise mean and std dev needs to be adapted.
Tasks
- [ ] Figure out how to extend the ResNet input channels
- [ ] Figure out how to copy over the pre-trained ResNet parts
- [ ] Let users construct a model with arbitrary channels
- [ ] Adapt mean and std dev
Is there any concern with initializing channels with 0 instead of a value that implies 'missing'? Or is this just me misunderstanding the process?
Looking forward to seeing this develop, regardless!
If you check out Kaggle competitions with multi-spectral data or in general more channels most of them either initialize additional channels with zero, randomly initialize, or copy the rgb channels over. I've seen zero initialization working best for most winning solutions.