Please do not abandon this project!

Open oobabooga opened this issue 2 years ago • 3 comments

Earlier this year I was impressed with the offloading performance of FlexGen, and I wonder how it would compare with the performance currently provided by llama.cpp for Llama and Llama-2 models in a CPU offloading scenario.

Any chance Llama support could be added to FlexGen @Ying1123 @keroro824?

Nov 27 '23 02:11 oobabooga

We are pushing a refactoring of the current implementation to support most HF models, we will release that soon under a fork of this repo and will keep you informed.

Nov 27 '23 02:11 BinhangYuan

That's exciting news @BinhangYuan! I look forward to testing the new release and incorporating it in my text-generation-webui project. Cheers :)

Nov 27 '23 02:11 oobabooga

Are there any news of this fork?

May 12 '24 12:05 arnfaldur