Muhammad Abdullah

PeftModelForCausalLM( (base_model): LoraModel( (model): MistralForCausalLM( (model): MistralModel( (embed_tokens): Embedding(32000, 4096, padding_idx=2) (layers): ModuleList( (0-31): 32 x MistralDecoderLayer( (self_attn): MistralAttention( (q_proj): Linear4bit(in_features=4096, out_features=4096, bias=False) (k_proj): lora.Linear4bit( (base_layer): Linear4bit(in_features=4096, out_features=1024, bias=False) (lora_dropout):...

I have tested your updated code https://github.com/EricLBuehler/xlora/pull/25 currently quantized model are trained using xlora , it start working with quantized model but facing issue when I want to make inference...

You can review my notebook at : https://colab.research.google.com/drive/1_B1ualsMbRfYWy0gdjdMi9RSDU-qmPHf#scrollTo=I4UZaqDAnnB6

Also, Checkout this notebook : https://colab.research.google.com/drive/1Eyh-mBd0LpcJwyzBHjGKhwNLQ9R74eLl?usp=drive_open Verify that a few lines are being repeated in the output.

What adjustments should we make if we wish to upgrade XLora for IA^3?

I am using your branch [alexrs:multi-ia3](https://github.com/alexrs/peft/tree/multi-ia3) to test the addd_weighted_adapter for ia3 FIRST_ADAPTER_PATH = "/content/drive/MyDrive/FedMl_test_llm/TrainedModels/WestLake_SFT_IA3_1/Weights/Epoch_1/Job_1" SECOND_ADAPTER_PATH = "/content/drive/MyDrive/FedMl_test_llm/TrainedModels/WestLake_SFT_IA3_2/Weights/Epoch_1/Job_1" FIRST_ADAPTER_NAME = "first" SECOND_ADAPTER_NAME = "second" model = PeftModel.from_pretrained(quantized_model_4bit, FIRST_ADAPTER_PATH, FIRST_ADAPTER_NAME) _...

It is good to start with a simpler linear approach.

I have tested this, it is working   it is working now, the next step could be for svd like strategies

i am also facing some issue relate ia3 that, Cannot merge ia3 layers when the model is loaded in 4-bit mention in this https://github.com/huggingface/peft/issues/1704   @BenjaminBossan please...

@Arnav0400 please help me to solve this issue