Supporting non-vision models

[Laubeee]

Hi, could you explain how to use this for non-vision models? Trying to apply this to an audio model I am getting an error because get_previous_layer never finds a Conv2d or a BatchNorm2d and hence, after recursing through all the modules, just returns None, which then fails at re.sub.

Could you explain the logic behind "going back to the previous layer of exactly this type"?

the immediate earlier layers are: (add, causing the search) -> [transpose] -> gelu -> transpose -> layer_norm -> transpose -> conv1d -> ...

which layer would you expect to find here? Are you looking for the last layer that learns anything (which would be layer norm) or with actual learnable parameters (then it's conv1d)

---

there is a second case where this happens where the chain is: (add, causing the search) -> [dropout] -> linear -> layer_norm -> transpose -> gelu -> transpose -> layer_norm -> transpose -> conv1d -> ...

again, please help me out which layer should be found. I would guess the linear layer?

---

on a side note: instead of deep recursion, checking the whole list each time, building a tree or doubly-linked-list-like structure seems more appropriate (and easier to debug)

[carloalbertobarbano]

Hi @Laubeee

Thanks for the interest! Actually, when we developed simplify we mainly had vision models in mind, so Conv1d/BatchNorm1d layers are not supported out-of-the-box now (they will be completely ignored). However, it should be fairly straightforward to support them based on the operations shown here https://www.sciencedirect.com/science/article/pii/S2352711021001576 (check the appendix)

However, simplification on 1d models should not fail I guess, but just be a no-op.

Unfortunately I do not have time to draft a PR at the moment for it (not sure about @AndreaBrg). Feel free to start one if you feel like working on it!

[Laubeee]

Thanks for clarifying. For the next few months I won't have capacity to put significant efforts, unfortunately. I'll probably look at https://github.com/VainF/Torch-Pruning and perhaps OpenVINO / OneDNN for improved inference in the meantime. What I like about Simplify is that it's focusing on one task and does not provide a whole other framework.