cgd-bot

Results 2 comments of cgd-bot

[BUG] Inference fail with "mat1 and mat2 shapes cannot be multiplied" for Llama model.

> @satpalsr Thanks, TP inference works now! So my next question: is it possible to split-load this checkpoint using the meta device? I find if I use `AutoModelForCausalLM.from_pretrained` to load...

when using huggingface pretrained model with multi-gpu, model parameters were duplicate for every gpu in ram

> Please let me know if that helps :) I had the same problem, but this method didn't solve it > Hey @linyubupa > > This is expected in the...