I trained bloom7b model but for inference they give me error

[imrankh46]

This is the code

batch = tokenizer("Two things are infinite: ", return_tensors="pt")

with torch.cuda.amp.autocast():
    output_tokens = model.generate(**batch, max_new_tokens=50)

print("\n\n", tokenizer.decode(output_tokens[0], skip_special_tokens=True))

The error is

AttributeError: 'NoneType' object has no attribute 'device'

[paolorechia]

Please see https://github.com/huggingface/peft/issues/115, it's probably the same thing.

In my case (using alpaca model) I had to add a device_map parameter:

model = PeftModel.from_pretrained(
    model,
    alpaca_path,
    torch_dtype=torch.float16,
    device_map={"": 0}
)

[imrankh46]

Please see #115, it's probably the same thing.

In my case (using alpaca model) I had to add a device_map parameter:

model = LlamaForCausalLM.from_pretrained(
    model_path,
    load_in_8bit=True,
    torch_dtype=torch.float16,
    device_map="auto",
)
model = PeftModel.from_pretrained(
    model,
    alpaca_path,
    torch_dtype=torch.float16,
    device_map={"": 0}
)

Thank you.

Please I m confused about the second one. You load two model. Which one should need to follow. !!

[paolorechia]

Sorry, should be the second one, I’ll edit the original comment.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.