apache
3D Upsampling
Description
Currently MXNet does not support upsampling on 5D tensors (3D images). There is more and more work being done in the medical field where 3D images are very common. As MXNet supports 3D transposed convolutions, 3D Upsampling would be very helpful as well, since there are instances where upsampling has been shown to be as effective if not more so than a transposed convolutions.
References
PyTorch, TF both support 3D upsampling.
If it's supported by cuDNN or Intel oneDNN, you may only need to edit a few lines to expose the feature in MXNet when built with either of the backends.
I'd be happy to give it a go, just not sure how to go about doing that. I'm using the Julia bindings, if that makes a difference.
Also, I think there's a way of using average pooling backwards to accomplish the same thing?
Great to have this use-case. @andevellicus Thanks for bringing it up. While your experience in Julie will be handy, we would still need work to be done on MXNet Backend [C/C++] because those few lines that @leezu mentioned would go somewhere here
For Upsampling Forward & Backward https://github.com/apache/incubator-mxnet/blob/3b23c2de950fb0e4d44560f4c7ea933a520c526c/src/operator/nn/upsampling-inl.h#L100
CPU-specific implementation For e.g. Shape currently checks for 4D input [2D image] https://github.com/apache/incubator-mxnet/blob/eceb5f2c1c494094c1a697286f2c4560b7ca472e/src/operator/nn/upsampling.cc#L44-L45 https://github.com/apache/incubator-mxnet/blob/eceb5f2c1c494094c1a697286f2c4560b7ca472e/src/operator/nn/upsampling.cc#L61-L62
There don't seem to be GPU-specific implementations at the moment. So we are good on that.
You can take a stab at updating the forward & backward implementations. Additionally, we could add a test for this 3D image use-case Similar to https://github.com/apache/incubator-mxnet/blob/f1f3f44166e2e47afad6c65025fb48dd47efeb65/tests/python/gpu/test_operator_gpu.py#L1481-L1489
I can help review your PR. Feel free to ping me if you need any help / have specific doubts related to contributing code to the MXNet Backend.
I see this was created a while ago. Was it implemented? If not, I'd like to give it a try.
Hey @r3stl355 I don't believe any update was made. Feel free to implement it! Thanks!
I managed to get a basic 3D sampling implemented (still only covering one straightforward case) but for that to work I had to make a modification to mshadow https://github.com/apache/incubator-mxnet/blob/da247654371664fb906ad286053c63e6fc2f5b5a/3rdparty/mshadow/mshadow/extension/spatial_upsampling_nearest.h#L52 as at the moment it scales only the last 2 dimensions whereas for 3D sampling it needs to do that for 3 last dims. I feel like I'm biting more than I can chew and would appreciate some guidance, would I have to create a separate PR to mshadow for this?
As per https://github.com/dmlc/mshadow
This code base has been donated to the Apache MXNet project per #373, and repo is deprecated. Future development should continue in Apache MXNet.
Feel free to make changes to that header file, test it out on your end. You don't have to create a separate PR for it. Update it within the same one.
Nice catch, thank you. That statement about being donated only appears under DMLC repo.
I'm about to resume my work on this one. I paused my work due to job change. I've implemented forward for Nearest upsampling but came across a problem while trying to implement the same for Bilinear -the 3rd party library only supports 2D - any advise would be appreciated, I will provide more info shortly after I recreate my dev environment
hey @ChaiBapchya can i work on it?
One approach you can take is to reshape your 5D tensor into a 4D tensor with a batch dimension and a channel dimension, and then apply 2D upsampling to the resulting 4D tensor. This can be done using the MXNet UpSampling operator.While this approach may not be as efficient as directly upsampling a 5D tensor, it can still be effective in some cases and may be worth trying out
@Pheewww I'm no longer actively contributing. So feel free to build on your approach and raise a pull request. I'm sure relevant folks will chime on with feedback.
@r3stl355 Looks like you have been closely implementing and tracking down the issue. Let me know if it needs to be taken care of, happy to collaborate, Thanks
