bebopkim

Low GPU usage of quantized Mixtral 8x22B for prompt processing on Metal

1

My computer is M1 Max Mac Studio with 32 Cores of GPU with 64 GB of RAM. macOS version is Sonoma 14.4.1. I run `llama-bench` from commit 4cc120c7443cf9dab898736f3c3b45dc8f14672b and it...

BUG: Cannot import charater cards from local disks since 1.95.0

2

I am not sure whether it is a macOS specific issue or not, but I have faced the problem. Since 1.95.0 8c0fe0d, I cannot import character cards from local disks....

infinity_emb failed at startup using `torch.compile` when installed via pip

8

commit hash: 296472eefaa93c361f086ea26bd7cd7e3c6e9a3e I tried it on my Linux machne - Ubuntu 22.04 with CUDA 12.3, and it was failed. ``` % infinity_emb --device cuda --engine torch 2024-03-03 11:05:28.807 |...

When I tried to use models named starting convnext_xxlarge, the webui did not work and shown only 'Unknown model (convnext_xxlarge)' error message in the terminal console window. Does the version...