huggingface
Fix a shape annotation and typos in `mamba` slow forward
What does this PR do?
It only addresses typos and a wrong shape annotation in the comments of mamba's slow forward call. There's no change in the logic or anything.
Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the contributor guideline, Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the forum? Please add a link to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the documentation guidelines, and here are tips on formatting docstrings.
- [ ] Did you write any new necessary tests?
Who can review?
Anyone in the community is free to review the PR once the tests have passed. Feel free to tag members/contributors who may be interested in your PR.
@ArthurZucker
Yup, same typo. The shape annotation is correct tho.
Another thing I've noticed in Jamba are these line;
https://github.com/huggingface/transformers/blob/5962d62bac850cd01ee830ffba880469338c96fd/src/transformers/models/jamba/modeling_jamba.py#L916-L920
If you remember my issue from the past ( #29526 ), I've added something similar but not differentiating between training and eval. Might be worth to change in base mamba as well.
Down to change, but with a bench / something that shows it does produce improvements!
