PeftModel.from_pretrained and merge_and_unload don't work good for me

[shahelaojieraozhi]

System Info

peft==0.9.0; 0.13.0; 0.17.0 (All have made attempts.) LLM : vicuna-7b-v1.5 (https://huggingface.co/lmsys/vicuna-7b-v1.5)

Who can help?

No response

Reproduction

When I used PeftModel.from_pretrained() to load the imported Lora weights, I couldn't obtain the desired results during model inference. However, by using the following method, it was possible to achieve the desired outcome.

    # "q_proj,v_proj"
    lora_modules = "q_proj,v_proj".split(",")
    lora_config = LoraConfig(
        r=128,
        lora_alpha=256,
        target_modules=lora_modules,
        lora_dropout=0.3,  # 0.3
        bias="none",
        task_type="CAUSAL_LM",
    )
    # model = get_peft_model(model, lora_config, adapter_name="default")
    model = get_peft_model(model, lora_config)
    from collections import OrderedDict
    from safetensors import safe_open

    lora_weight_path = "xxxxx/adapter_model.safetensors"
    lora_state_dict = OrderedDict()
    with safe_open(lora_weight_path, framework="pt", device="cpu") as f:
        for key in f.keys():
            value = f.get_tensor(key)
            lora_state_dict[key] = value
    ret = model.load_state_dict(lora_state_dict, strict=False)
    print("Missing keys: \n", ret.missing_keys)
    print("Unexpected keys: \n", ret.unexpected_keys)

During the test, it was also discovered that even though I correctly loaded the LoRA parameters using the aforementioned method. When I wanted to combine the LoRA parameters, I used the merge_and_unload function. However, when the final merged model was used for inference, it still did not yield the result I desired. I think this is a very serious bug.

It should be noted that if LLM is replaced with Qwen3-8B(https://huggingface.co/Qwen/Qwen3-8B), the aforementioned bug will no longer exist.

Expected behavior

work it out

[Aznix07]

Hi @shahelaojieraozhi thank you for reporting this issue! How was this LoRA adapter trained?

• Training script or framework used?
• Were there any warnings during training?
• Also can you share the transformers version ?

[BenjaminBossan]

@shahelaojieraozhi Could you please show the exact code that you used with PeftModel.from_pretrained and merge_and_unload which does not work for you? Please include how you tested that the model does not work. Finally, if it's possible to share the adapter checkpoint, that would be ideal to try to reproduce it.

[shahelaojieraozhi]

Hi @shahelaojieraozhi thank you for reporting this issue! How was this LoRA adapter trained?

• Training script or framework used?
• Were there any warnings during training?
• Also can you share the transformers version ?

Hi! Aznix07, nice to meet you!

1. Due to the uniqueness of the task, I did not use training frameworks like trainer. Instead, I referred to llava(https://github.com/haotian-liu/LLaVA/tree/main) to extract LORA parameters, and then used save_pretrained to save the LORA parameters.

# Borrowed from peft.utils.get_peft_model_state_dict
def get_peft_state_maybe_zero_3(named_params, bias):
    if bias == "none":
        to_return = {k: t for k, t in named_params if "lora_" in k}
    elif bias == "all":
        to_return = {k: t for k, t in named_params if "lora_" in k or "bias" in k}
    elif bias == "lora_only":
        to_return = {}
        maybe_lora_bias = {}
        lora_bias_names = set()
        for k, t in named_params:
            if "lora_" in k:
                to_return[k] = t
                bias_name = k.split("lora_")[0] + "bias"
                lora_bias_names.add(bias_name)
            elif "bias" in k:
                maybe_lora_bias[k] = t
        for k, t in maybe_lora_bias:
            if bias_name in lora_bias_names:
                to_return[bias_name] = t
    else:
        raise NotImplementedError
    to_return = {k: maybe_zero_3(v, ignore_status=True) for k, v in to_return.items()}
    return to_return

2. Were there any warnings during training? ----don't have any warnings.
3. transformers version==4.51.3

[shahelaojieraozhi]

@shahelaojieraozhi Could you please show the exact code that you used with PeftModel.from_pretrained and merge_and_unload which does not work for you? Please include how you tested that the model does not work. Finally, if it's possible to share the adapter checkpoint, that would be ideal to try to reproduce it.

Hi! BenjaminBossan, nice to meet you! This will require some time to sort out. I will ensure to complete it as soon as possible.