LocalAIVoiceChat use existing llama.cpp install

I've been using llama.cpp for quite a while (M1 Mac). Is there a way I can get ai_voicetalk_local.py to point to that installation instead of reinstalling it here? Sorry, newbie question...

Mar 24 '24 21:03 scalar27

Just leave out step 2 of installation. I think coqui engine does not run in realtime on a Mac though.

Mar 25 '24 09:03 KoljaB

I did leave out step 2 but then I get an error when I try to run: ModuleNotFoundError: No module named 'llama_cpp'

Mar 27 '24 15:03 scalar27

Python import of llama_cpp fails, that means your environment does not have working python bindings for your llama.cpp. Please look here for Mac bindings, probably Metal (MPS).

Mar 27 '24 15:03 KoljaB

Thank you. I did get it to work following your comment. Like the other M1 person, I do get stuttering. It's a shame because the voice quality is excellent and the latency is rather short. Hope a future update might solve this for us!

Mar 27 '24 20:03 scalar27

I managed to get this working with the Gemma 2 model. However, I am having trouble setting the parameters. It's working but doesn't seem optimal. I see them in creation_params.json, and also in coqui_engine.py. Would it be possible for LocalAiVoiceChat to utilize Llama.cpp's server endpoint (instead)? Or would that require a lot of rewriting of the code?

Jul 17 '24 19:07 scalar27

I like that idea, I'll have to look into that.

Jul 17 '24 19:07 KoljaB

Great. It seems like a more standard approach these days. I'd be happy to test whatever. As mentioned above I'm on a M1 Mac so this isn't the fastest setup but it's now working pretty well with no stuttering.

Jul 18 '24 00:07 scalar27