vicuna 13B not answering

[tdguchi]

Hi I have downloaded llama 13B, "updated" weights with given instructions. (fresh install, 1.1 weights)

I have a ryzen 9 7950x and 64gb of ram, also 500gb 980 pro ssd.

I tried he cpu only command, and i got this:

Vicuna is not answering anything, i have waited 30 min, cpu is almost sleep, how can I debug what is happening? there is free RAM, so thats not the problem...

thanks a lot

[tdguchi]

also 7B gives me the same result...

[sirus20x6]

same here. it's only using 10% of my cpu, but a lot of memory and not spitting anything out.

[sirus20x6]

it finally spit out a couple of words. using --load-8bit seems to have improved things

[zhisbug]

Seem like a bug with CPU support? CC @merrymercy

[surak]

@tdguchi any news here? Did you try with the latest versions of everything? It was quite a while ago, perhaps the underlying libraries fixed it.

Did you try with vllm? (I'm not sure you can do cpu inference with it, but worth a try)