manageseverin

too slow?

30

I'm using it under Windwos 11 with alpaca 7B Ok, it's great overall, but I have a native cpp version (chat.exe) and it's running 2 times faster than your docker...

Running a Vicuna-13B 4it model ?

25

I found this model : [[ggml-vicuna-13b-4bit](https://huggingface.co/eachadea/ggml-vicuna-13b-4bit)](https://huggingface.co/eachadea/ggml-vicuna-13b-4bit/tree/main) and judging by their online demo it's very impressive. I tried to run it with llama.cpp latest version - the model loads fine, but...

can't encode character '\u2212'

2

It often excepts as the following : ``` return codecs.charmap_encode(input,self.errors,encoding_table)[0] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ UnicodeEncodeError: 'charmap' codec can't encode character '\u2212' in position 2402: character maps to ``` of course the character '\u2212'...

If I access it through the web interface and other people are accessing at the same time it would mirror the answers to all the users, right ? I guess,...