microsoft
[Bug]: [autogenstudio] llm forget to use skills.py functions
Describe the bug
when an agent have several skills
and despite having the following instructions:
While solving the task you may use functions below which are available in a file called skills.py .\nTo use a function skill.py in code, IMPORT THE FUNCTION FROM skills.py and then use the function.\nIf you need to install python packages, write shell code to\ninstall via pip and use --quiet option.\n\n
The LLM don't use the skills.py function call but copies or recreates the function.
Is it possible to pass only function descriptions rather than entire functions?
Steps to reproduce
No response
Model Used
gpt-4 or mistral-7b
Expected Behavior
use from skills import ...
Screenshots and logs
No response
Additional Information
No response
Hi @nicho2 ,
Smaller models like Mistral 7B sometimes struggle with using skills. Similar to your thinking, this might be because the context is too long (hence your suggestion to use the function signature).
I think your proposed suggesting is a good feature to add.
- [ ] Modify the get_skills_from_prompt util to add only function signature . This probably can be done using AST parsing of the skill code.
- [ ] Verify and test that general behavior still holds.
Please let me know if you'd be willing to contribute to implementing this.
fyi @jackgerrits @gagb
Great! Please refer to the AGS contribution guide here, let me know if you have any questions https://github.com/microsoft/autogen/tree/autogenstudio/samples/apps/autogen-studio#contribution-guide
I was reviewing the history of this issue and looked interesting to me to try and solve. Wanted to check if this is still available for contribution or if it’s considered stale/closed. I’d love to help if it's still open, but I wanted to confirm before proceeding.
@saldanhad 0.2 is certainly open for contributions if you are interested. However, AutoGen Studio has also been updated to work with AutoGen 0.4.5 and I'd encourage you to try that.
