The Tools Only Work With OpenAI Models

[MirzaAreebBaig]

Self Checks

• [X] I have searched for existing issues search for existing issues, including closed ones.
• [X] I confirm that I am using English to file this report (我已阅读并同意 Language Policy).

1. Is this request related to a challenge you're experiencing?

The tools section used in Agent Assistant works only with OpenAI Models i have tried using other models like Claude then AWS titan & Llama but it seems like the tools only work with OpenAI Models.

2. Describe the feature you'd like to see

It would be great if tools functionality work with other models.

3. How will this feature improve your workflow or experience?

As per my scenario the OpenAI model costs are little bit high compared to other models would like to use other models with the toolset provided.

4. Additional context or comments

No response

5. Can you help us with this feature?

• [ ] I am interested in contributing to this feature.

[dosubot[bot]]

🤖

Hey @MirzaAreebBaig! Great to see you back and diving into new challenges. Hope you're doing well!

Thank you for your request. The current implementation of the dify repository is indeed tightly coupled with OpenAI's GPT models. The codebase uses specific methods and classes such as ChatCompletion, ChatCompletionChunk, ChatCompletionMessageToolCall, FunctionCall, etc., which are specific to OpenAI's GPT models and may not be compatible with other models like Claude, AWS titan, and Llama.

To make the tools functionality compatible with other models, modifications to the code would be required to handle the specific input and output formats of those models. This could involve changing the way the prompt messages are formatted, the way the model is invoked, and the way the model's response is handled.

However, without specific information about the implementation details of the Claude, AWS titan, and Llama models, it's not possible to provide a definitive answer or code example. You would need to refer to the documentation or source code of those models to understand how to integrate them with the dify repository.

In the dify codebase, the AzureOpenAILargeLanguageModel class, validate_credentials method, invoke method, _llm_result_to_stream method, LLMResult, LLMResultChunk, and LLMResultChunkDelta classes, AssistantPromptMessage, TextPromptMessageContent, SystemPromptMessage, ImagePromptMessageContent, PromptMessageTool, and UserPromptMessage classes, CredentialsValidateFailedError class, and setup_openai_mock function are all tightly coupled with OpenAI's GPT models. These would likely need to be adapted or replaced to work with other models.

I hope this information is helpful. If you have any further questions or need more specific guidance, please don't hesitate to ask.

Sources

• api/core/model_runtime/model_providers/xinference/llm/llm.py
• api/tests/integration_tests/model_runtime/azure_openai/test_llm.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[Yeuoly]

Tools dose work with Almost all models, but the real behavior of LLM is related to instruction and align of model. For example, Claude is not trained to follow instruction but summary of texts, just try your instruction on Anthropic, you can see that Claude will not follow your instruction. Maybe each model should have a custom react prompt, but it brings a complex to normal user. BTW, you can search langchain react to learn about what's react

[MirzaAreebBaig]

Hello, Thank you for your reply on this. But whenever i'm using other models the tools are not even loading or initializing to get the answer.

[Yeuoly]

You can ask Gemini Pro what time is it as a test just like this And see if there is a tool call

[MirzaAreebBaig]

Yes, i have tested this & its working no doubt, But its not working with custom tool which is created by OpenAI Schema JSON. That is only working with OpenAI Models.

[Yeuoly]

Maybe a detailed description of your custom tool is what you need, other models cannot understand what your tool is designed for and how to use, you can read the standard OpenAPI schema docs to find a field named "summary", it helps LLM understand your apis.

[github-actions[bot]]

Close due to it's no longer active, if you have any questions, you can reopen it.