How to specify lora parameters

[01miaom]

When running an entire bloom model in local environment, I can view the information of all layers and specify the query_key_value module in lora. But in petals, the (h) layer becomes a remote sequential. How should I specify the target module in lora like this:

config = LoraConfig(
    r=16,
    lora_alpha=16,
    target_modules=["query_key_value"],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM"
)

print bloom in device:

print bloom in petals:

[borzunov]

Hi @01miaom,

You're right - LoRAs can't be loaded in this way. Right now, we only support pre-loading LoRA adapters when you start your own server. See "Pre-loading LoRA adapters" here for details.

We plan to add a way to use/fine-tune custom LoRA adapters from clients (without pre-loading anything on the server) in future, but it's not done yet.