WIP: some optimizations

[thomasw21]

TODO:

• [ ] Run an actual benchmark testing that it's faster
• [ ] Maybe Linear doesn't make sense as running Conv1d would remove quite a few reshape/contiguous etc ...

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

[patrickvonplaten]

@thomasw21 let me know if this is ready for review :-)

[thomasw21]

Yep sorry tested on a benchmark, there wasn't much improvement, so thinking of dropping it. I'll close when I decide to quit on that PR if that's alright with you

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.