saint icon indicating copy to clipboard operation
saint copied to clipboard

MLP is equivalent to a single linear layer

Open YishGene opened this issue 2 years ago • 0 comments

One more issue - your implementation of MLP, in model.py, is just a bunch of stacked linear layers with no non-linearities between them. This is mathematically equivalent to just a linear layer with in_dim = dims[0] and out_dim=dims[-1].

Why not use the activation between the layers?

YishGene avatar Mar 05 '23 17:03 YishGene