Not all CNN blocks are converted

[WWEISONG]

self.conv_embedding = conv_bn_relu(in_channels, channels[0], kernel_size=patch_size, stride=patch_size, padding=0)

The first embedding layer is still CNN without conversion to MLP finally, have you tried to convert this embedding layer to MLP as well?