James Fleming comments

Repositories
Issues
Comments

Results 4 comments of


                                            James Fleming

AQLM CUDA support

> Looks good from my testing @jaemzfleming, just some formatting cleanup. Load time is now only a few seconds at most - the issue was the initial model inference for...

AQLM CUDA support

Thanks for doing all this work @mgoin, much appreciated.

Very long delay when a breakpoint is hit and many (150+) threads are created by the program

you got my vote too, this is a horrible issue that makes vscode much less useful than it should be. We should be able to debug in a timely manner....

Remove post flattening CUDA `clone()`s, for 2% speedup in a 1x16 7B llama2

> Hi! It has to do with backward pass calculation. For some reason it doesn't work without the `.clone` [#26 (comment)](https://github.com/Vahe1994/AQLM/pull/26#discussion_r1496273980) We should probably investigate it further. Wow, that's really...