Repeated tokens generated from 'generate.py' running on GPU

Open shengzhelyu65 opened this issue 8 months ago • 2 comments

Dear Authors,

Thanks for introducing the amazing project. When I tested the BitNet Inference Kernel on RTX 3090 with Ubuntu system, I followed the commands in README.md, but I got repeated tokens as the output. For example:

Could you help me explain Python?

OfOfOfOfOfOfOfOfOfOfOfOfOfOfOfOfOfOfOfOfOfOfOfOfOfOfOfOfOfOfOfOf

Could you help me check if anything could be wrong here? Thanks.

May 26 '25 11:05 shengzhelyu65

I am facing the same problem.

Hello! GGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGG

Jun 04 '25 19:06 prapti02

Similar issue, the model just outputs repeated nonsensical letters:

python ./run_inference.py -m ./ggml-model-i2_s.gguf -p "When people say \"I'm terrified of the future\", the primary emotion expressed is: " -n 10

When people say "I'm terrified of the future", the primary emotion expressed is:  �zuônimersimerszuimersuggyimershands

Jun 05 '25 08:06 itripodi-ctl