apple
Auto-Fuse ReLU with BatchNorm
When constructing my model in tf.keras, I have a repeated sequence of layers that goes:
tf.keras.layers.Conv2D(...)
tf.keras.layers.BatchNormalization(...)
tf.keras.layers.ReLU(...)
Due to the BatchNorm layer in between the convolution and activation function, I cannot specify aFused ReLUdirectly within the Conv2D layer. Therefore, I lose out on reaping the benefits of Fused ReLU.
When converting the model to CoreML, BatchNorm layers are automatically fused in so we are just left with: Conv2D -> ReLU after model conversion.
I would like to see coremltools automatically fuse ReLU into the Fused Conv2D+BN layer (currently, when inspecting the model with Netron, I see that "Convolution" and "activation' are separate layers).
This would be pretty convenient for a lot of ML practitioners looking to optimize their models.
Any reference to fusing relu? Maybe you are talking about inplace relu like https://towardsdatascience.com/in-place-operations-in-pytorch-f91d493e970e ?
I have no official knowledge of the insides of Core ML but just because something is multiple layers in the mlmodel, doesn't mean it's also executed like this at runtime.
I'm pretty sure Core ML folds the batchnorm weights into the conv layer when it loads the model, and then at runtime uses a fused ReLU. (You can verify this by running the model on the GPU and using Metal frame capture.)
So you don't need to worry about this. 😄
@hollance is right in sprit but this is something that is more of a case by case basis. @aseemw Can we confirm that this does not impact performance? If not, can we add a graph-pass to do this?
