Adding parameter support for Local LLMs and GROQ

[gpapp]

Some locally served LLMs need more fine-tune in their parameters, than the OpenAI API.

GROQ also opened their API-s, which has a limited parameters, that can be passed over in the API

The prompt example containts a timeout quoted, while it expects numeric value. This is a waste of many tokens to resolve.

[senko]

Thanks for this @gpapp !

We have an experimental groq branch, which uses their official SDK instead of making manual requests (which is what we'd like to do in the future with the default openai compatible client function).

Do you think you could adapt your changes to work on that? This would also make them less risk (in terms of how different people have set up environments wihch may inadvertenly cause problems with extra env flags), as it would only apply to Groq so far.

Btw the prompt mismatch is a good catch, I was convinced that was a shortcoming of Mixtral.

[gpapp]

Updated to use the SDK (which in fact uses the openAI compatible API for now) readded all the modifiers that can be passed as parameters to groq.