LMFlow icon indicating copy to clipboard operation
LMFlow copied to clipboard

Support for QLORA

Open louisoutin opened this issue 2 years ago • 1 comments

Hello, Can we finetune model using qlora with LMFlow? (Cf: https://github.com/artidoro/qlora) Thanks

louisoutin avatar Jun 23 '23 00:06 louisoutin

Thanks for your interest in LMFlow! Currently we haven't supported qlora yet, but it is in our schedule. By the way, we encourage any contributions to accelerate the develop progress. Thanks very much 😄

research4pan avatar Jun 27 '23 12:06 research4pan

Sorry, i've been offline the last week. I'll try to have a look for QLORA integration, but i need to check more LMFlow codebase first (i'm new to it). If anyone want to help in this effort, I think we can take example on that PR, should be something pretty similar: https://github.com/tloen/alpaca-lora/pull/487/files

louisoutin avatar Jul 07 '23 12:07 louisoutin

Hi, thanks. Agree. I feel that we can do very similar modifications to our code. If you have any questions about our repo and need help, feel free to let us know.

shizhediao avatar Jul 07 '23 12:07 shizhediao

interested in this work, 资瓷。

Haihsu avatar Jul 19 '23 08:07 Haihsu

I have created a PR for support for QLoRA (Added QLoRA support for Decoder transformers with tune_strategy "Normal" #613)

Once it is reviewed hopefully it get's merged.

TensorBlast avatar Aug 09 '23 21:08 TensorBlast

It has been merged a few weeks ago. Thanks for @ankitpasi 's contribution!

shizhediao avatar Aug 25 '23 19:08 shizhediao