[BUG] GEMINI 2 and Memory support issue leading to BadRequestError 400 from Gemini endpoint

[dchevenement]

Description

When a crew is set with memory=True, there is a task evaluation going on to store relevant information into long term memory. This LLM call is made using the same model as the agent used for handling its task. When using Gemini (1.5 and 2.0 have the same issue), the liteLLM call structure is not conform to the Gemini expectations, see evidences

}

Here is the POST made to the endpoint: POST Request Sent from LiteLLM: curl -X POST
https://generativelanguage.googleapis.com/v1beta/models/gemini-2.0-flash-exp:generateContent?key=XXXXXXXX
-H 'Content-Type: *****'
-d '{'contents': [{'role': 'user', 'parts': [{'text': "Assess the quality of the task completed based on the description, expected output, and actual results.\n\nTask Description:\nHi Alcie how are you ?\n\nExpected Output:\nA response to the user's input or query\n\nActual Output:\nThis is a simple greeting. I can respond directly.\nFinal Answer: Hi there! I'm doing well, thank you for asking. As Alice, your AI Organization Manager, I'm ready to assist you. What can I do for you today?\n\n\nPlease provide:\n- Bullet points suggestions to improve future similar tasks\n- A score from 0 to 10 evaluating on completion, quality, and overall performance- Entities extracted from the task output, if any, their type, description, and relationships"}]}], 'system_instruction': {'parts': [{'text': 'Convert all responses into valid JSON output.'}]}, 'tools': [{'function_declarations': [{'name': 'TaskEvaluation', 'description': 'Correctly extracted TaskEvaluation with all the required parameters with correct types', 'parameters': {'$defs': {'Entity': {'properties': {'name': {'description': 'The name of the entity.', 'title': 'Name', 'type': 'string'}, 'type': {'description': 'The type of the entity.', 'title': 'Type', 'type': 'string'}, 'description': {'description': 'Description of the entity.', 'title': 'Description', 'type': 'string'}, 'relationships': {'description': 'Relationships of the entity.', 'items': {'type': 'string'}, 'title': 'Relationships', 'type': 'array'}}, 'required': ['name', 'type', 'description', 'relationships'], 'title': 'Entity', 'type': 'object'}}, 'properties': {'suggestions': {'description': 'Suggestions to improve future similar tasks.', 'items': {'type': 'string'}, 'title': 'Suggestions', 'type': 'array'}, 'quality': {'description': 'A score from 0 to 10 evaluating on completion, quality, and overall performance, all taking into account the task description, expected output, and the result of the task.', 'title': 'Quality', 'type': 'number'}, 'entities': {'description': 'Entities extracted from the task output.', 'items': {'$ref': '#/$defs/Entity'}, 'title': 'Entities', 'type': 'array'}}, 'required': ['entities', 'quality', 'suggestions'], 'type': 'object'}}]}], 'toolConfig': {'functionCallingConfig': {'mode': 'ANY', 'allowed_function_names': ['TaskEvaluation']}}, 'generationConfig': {}}'

Steps to Reproduce

1. Start a crew with a simple how are you task.
2. Make Sure you've enabled the memory part of the crew.
3. This memory settings will trigger a call to task_evaluator evaluate function which creates a malformed request to Gemini endpoint. Works fine with OpenAI and Anthropic endpoints.

Expected behavior

There should be no errors when running with Gemini schemas.

Screenshots/Code snippets

Did not want to dig to far into specific LLM support so ended up solving the issue right now using an .env variable evaluation_model and creating the evaluation LLM from it.

def evaluate(self, task, output) -> TaskEvaluation:
    evaluation_query = (
        f"Assess the quality of the task completed based on the description, expected output, and actual results.\n\n"
        f"Task Description:\n{task.description}\n\n"
        f"Expected Output:\n{task.expected_output}\n\n"
        f"Actual Output:\n{output}\n\n"
        "Please provide:\n"
        "- Bullet points suggestions to improve future similar tasks\n"
        "- A score from 0 to 10 evaluating on completion, quality, and overall performance"
        "- Entities extracted from the task output, if any, their type, description, and relationships"
    )

    instructions = "Convert all responses into valid JSON output."

    if not self.llm.supports_function_calling():
        model_schema = PydanticSchemaParser(model=TaskEvaluation).get_schema()
        instructions = f"{instructions}\n\nReturn only valid JSON with the following schema:\n```json\n{model_schema}\n```"

    converter = Converter(
        llm=LLM(model=evaluation_model, api_key=os.getenv("ANTHROPIC_API_KEY")),
        text=evaluation_query,
        model=TaskEvaluation,
        instructions=instructions,
    )

    return converter.to_pydantic()

Operating System

macOS Sonoma

Python Version

3.11

crewAI Version

0.86.0

crewAI Tools Version

0/17/0

Virtual Environment

Venv

Evidence

{ "error": { "code": 400, "message": "Invalid JSON payload received. Unknown name "$defs" at 'tools[0].function_declarations[0].parameters': Cannot find field.\nInvalid JSON payload received. Unknown name "title" at 'tools[0].function_declarations[0].parameters.properties[0].value': Cannot find field.\nInvalid JSON payload received. Unknown name "title" at 'tools[0].function_declarations[0].parameters.properties[1].value': Cannot find field.\nInvalid JSON payload received. Unknown name "$ref" at 'tools[0].function_declarations[0].parameters.properties[2].value.items': Cannot find field.\nInvalid JSON payload received. Unknown name "title" at 'tools[0].function_declarations[0].parameters.properties[2].value': Cannot find field.", "status": "INVALID_ARGUMENT", "details": [ { "@type": "type.googleapis.com/google.rpc.BadRequest", "fieldViolations": [ { "field": "tools[0].function_declarations[0].parameters", "description": "Invalid JSON payload received. Unknown name "$defs" at 'tools[0].function_declarations[0].parameters': Cannot find field." }, { "field": "tools[0].function_declarations[0].parameters.properties[0].value", "description": "Invalid JSON payload received. Unknown name "title" at 'tools[0].function_declarations[0].parameters.properties[0].value': Cannot find field." }, { "field": "tools[0].function_declarations[0].parameters.properties[1].value", "description": "Invalid JSON payload received. Unknown name "title" at 'tools[0].function_declarations[0].parameters.properties[1].value': Cannot find field." }, { "field": "tools[0].function_declarations[0].parameters.properties[2].value.items", "description": "Invalid JSON payload received. Unknown name "$ref" at 'tools[0].function_declarations[0].parameters.properties[2].value.items': Cannot find field." }, { "field": "tools[0].function_declarations[0].parameters.properties[2].value", "description": "Invalid JSON payload received. Unknown name "title" at 'tools[0].function_declarations[0].parameters.properties[2].value': Cannot find field." } ] } ] }

Possible Solution

build a valid schema for Gemini endpoints, which do not seem to support everything.

Additional context

thanks for everything to the Team :)