mylinfh

Results 2 comments of mylinfh

Llama-3-8B run inference error occurred：InternalError: Check failed: (offset + needed_size <= this->buffer.size) is false: storage allocation failure, attempted to allocate 513024 at offset 0 in region that is 163840bytes

Okay, thank you. I'll try again.llama.cpp/ollama can be used, but the inference time seems to be longer

Llama-3-8B run inference error occurred：InternalError: Check failed: (offset + needed_size <= this->buffer.size) is false: storage allocation failure, attempted to allocate 513024 at offset 0 in region that is 163840bytes

Mmm, yes, Thanks for your reply. I can run llama3 using nanoLLM. But I also tried deploying inference on nanollm and mlc using Llama2-7B, both of which were fast, but...