ChatFred icon indicating copy to clipboard operation
ChatFred copied to clipboard

Implemented support for more LLM models

Open April-June-August opened this issue 2 years ago • 3 comments

This PR implements support for more LLM models like Claude 2 from Anthropic, PaLM from Google, Llama v2 70B from Meta, and GPT-4 from OpenAI (more models are being added) by routing the request using OpenRouter with change of code and affected compatibility kept to a minimum.

  1. Requests to OpenRouter use the same openai module as those sent to OpenAI, no additional dependency is required. The only difference is that it requires a headers argument for tracking usages from different apps. User can fill out whatever header they like for potentially supporting more routing services.

  2. The option to use OpenRouter (or other services supported) is opt-in, means that users who doesn't know or don't want to use it won't feel a thing after they upgrade the workflow. And if they do use it, it will only apply to the chat completion part, instruct GPT part and image generation part will still use models from OpenAI.

  3. Unsupported parameter will be ignored for models that do not support it, such as stream for meta-llama/llama-2-70b-chat, in which case a window will show after the full reply is returned. No code modification for each parameter based on each model.

April-June-August avatar Aug 19 '23 22:08 April-June-August

I'll add the missing documentation once if it seems reasonable for you to include this in ChatFred, I'm curious about what you think! @chrislemke

You can get your own key at https://openrouter.ai, $1 of credit is granted with each new account.

April-June-August avatar Aug 19 '23 22:08 April-June-August

Hey @Sponge-bink! Thanks for this. A really cool idea! I will checkout https://openrouter.ai - looks interesting.

chrislemke avatar Aug 20 '23 10:08 chrislemke

I'm now discouraging this PR since the service of OpenRouter has been, to say at least, unstable recently.

April-June-August avatar Aug 31 '23 16:08 April-June-August