Function calling

[xumeng12321]

Describe the feature

Hi, I wonder if it is possible to add custom tools to the LLM to make it do something like, create a new cube in the scene.

[amakropoulos]

hi, not at the moment! you could do it somehow manually: prompt the LLM to generate some code, then parse the response to extract it and call the code.

[3inary]

Function Calling, as i understand it, would be similar to command detection maybe through sentence similarity or intend recognition, like for ex. the input "Hello" -> callfunc WaveHand()

[amakropoulos]

Yes, that's correct @3inary ! I was trying to answer the request by @xumeng12321 . At the moment, one way to do function calling is the following:

• Specify the commands available at the LLMCharacter (system) prompt
• Ask the LLM to select an appropriate function based on the user input
• Use a grammar for the response that allows only the function names and parse the function name
• Call the function

I will implement a sample that does that and then close the issue. Let me know if you have other thoughts!

[ringorjo]

Hi, I would like to know if, by using the method suggested by @amakropoulos for function calling, this approach can return parameters such as, for instance, the weather in NY. Specifically, can the output return the temperature in degrees Celsius?

[amakropoulos]

You can specify it in a grammar. I will add an example for that as well, hopefully this week.