.Net: SK not working correctly with gemini

[ShreyasJejurkar]

I am trying SK for the first time and as Gemini is free, so I am trying gemini models. Below is the code that I am trying.

using System.ComponentModel;
using Microsoft.SemanticKernel;
using Microsoft.SemanticKernel.Agents;

const string api_key = "__API_KEY__";
const string model = "gemini-2.5-pro";

var builder = Kernel.CreateBuilder()
    .AddGoogleAIGeminiChatCompletion(model, api_key);

var kernel = builder.Build();

kernel.Plugins.Add(KernelPluginFactory.CreateFromType<LightsController>());

ChatCompletionAgent agent = new()
{
    Kernel = kernel,
    Name = "Lights Agent",
    Description = "An agent that can provide information about available lights.",
    Arguments = new KernelArguments(new PromptExecutionSettings() { FunctionChoiceBehavior = FunctionChoiceBehavior.Auto()})
};

await foreach (var response in agent.InvokeAsync("Get a list of available lights"))
{
    Console.WriteLine(response.Message);
}

public class LightsController
{
    private readonly Dictionary<int,string> _lights = new()
    {
        { 1, "Table Lamp" },
        { 2, "Floor Lamp" },
        { 3, "Ceiling Light" }
    };
    
    [KernelFunction]
    [Description("Get a list of available lights")]
    public string GetLights()
    {
        return string.Join(", ", _lights.Select(kv => $"{kv.Key}: {kv.Value}"));
    }
}

And I am getting the output below. Of course. Here is a list of your available lights and their current status:

### **Living Room**
*   **Living Room Lamp**
    *   **Status:** On
    *   **Brightness:** 70%
*   **Ceiling Fan Light**
    *   **Status:** Off

### **Kitchen**
*   **Kitchen Downlights** (Group)
    *   **Status:** On
    *   **Brightness:** 100%
*   **Under Cabinet Lights**
    *   **Status:** On
    *   **Brightness:** 80%

### **Bedroom**
*   **Bedroom Mood Light**
    *   **Status:** On
    *   **Brightness:** 40%
    *   **Color:** Warm White
*   **Bedside Lamp**
    *   **Status:** Off

### **Office**
*   **Office Desk Lamp**
    *   **Status:** Off

You can control these by name, such as *"Turn off the Living Room Lamp"* or *"Set the Bedroom Mood Light to blue."*

If you see output, it's not matching with the lights that I have in my code. Looks like hallucination.

[futugyou]

I also encountered a problem when using gemini.

When using AddGoogleAIGeminiChatCompletion directly, gemini will say I don't have direct control over physical devices like lights...... But if I use AddOpenAIChatCompletion and configure the model/key/endpoint, gemini can make function calls. (The toolCallId problem needs to be solved)

It seems that when using AddGoogleAIGeminiChatCompletion, the Kernel function is not correctly passed to gemini

[rogerbarreto]

Function calling is not yet fully supported (multiple function calls) in the Google Connector.

For best experience of function calling as of now as @futugyou rightly mentioned is using their compatible OpenAI endpoints with our OpenAI Connector.

More info here:

https://ai.google.dev/gemini-api/docs/openai

https://cloud.google.com/vertex-ai/generative-ai/docs/migrate/openai/overview

[kaspermarstal]

@rogerbarreto Do you have an ETA on this? Like, is this a priority for you right now or are there other things that takes precedence?

[HillPhelmuth]

@ShreyasJejurkar Unless something's changed recently, I believe you can still use function calling with the Google Gemini connector, but FunctionChoiceBehavior doesn't work. Instead, I've had to use the old ToolCallBehavior property in GeminiPromptExecutionSettings, so your chat completion agent would look like:

ChatCompletionAgent agent = new()
{
    Kernel = kernel,
    Name = "Lights Agent",
    Description = "An agent that can provide information about available lights.",
    Arguments = new KernelArguments(new GeminiPromptExecutionSettings() { ToolCallBehavior = GeminiToolCallBehavior.AutoInvokeKernelFunctions })
};

It's definitely limited compared to FunctionChoiceBehavior, but should work with your demo code.

[verdie-g]

For best experience of function calling as of now as @futugyou rightly mentioned is using their compatible OpenAI endpoints with our OpenAI Connector.

@rogerbarretoI've tried that

var kernelBuilder = Kernel.CreateBuilder()
    .AddOpenAIChatCompletion(
        modelId: "gemini-2.5-flash",
        endpoint: new Uri("https://generativelanguage.googleapis.com/v1beta/openai/"),
        apiKey: Environment.GetEnvironmentVariable("GOOGLE_API_KEY")!);
kernelBuilder.Plugins.AddFromType<OrderPizzaPlugin>("OrderPizza");
var kernel = kernelBuilder.Build();

but I'm getting the same exception @futugyou mentioned

> I'd like to make a pizza order
Unhandled exception. System.ArgumentException: Value cannot be an empty string. (Parameter 'toolCallId')
   at OpenAI.Argument.AssertNotNullOrEmpty(String value, String name)
   at OpenAI.Chat.ToolChatMessage..ctor(String toolCallId, String content)
   at Microsoft.SemanticKernel.Connectors.OpenAI.ClientCore.CreateRequestMessages(ChatMessageContent message)
   at Microsoft.SemanticKernel.Connectors.OpenAI.ClientCore.CreateChatCompletionMessages(OpenAIPromptExecutionSettings executionSettings, ChatHistory chatHistory)
   at Microsoft.SemanticKernel.Connectors.OpenAI.ClientCore.GetChatMessageContentsAsync(String targetModel, ChatHistory chatHistory, PromptExecutionSettings executionSettings, Kernel kernel, CancellationToken cancellationToken)
   at Microsoft.SemanticKernel.ChatCompletion.ChatCompletionServiceExtensions.GetChatMessageContentAsync(IChatCompletionService chatCompletionService, ChatHistory chatHistory, PromptExecutionSettings executionSettings, Kernel kernel, CancellationToken cancellationToken)
   at Program.<Main>$(String[] args)
   at Program.<Main>(String[] args)

Full Code

```csharp var kernelBuilder = Kernel.CreateBuilder() .AddOpenAIChatCompletion( modelId: "gemini-2.5-flash", endpoint: new Uri("https://generativelanguage.googleapis.com/v1beta/openai/"), apiKey: Environment.GetEnvironmentVariable("GOOGLE_API_KEY")!); // .AddGoogleAIGeminiChatCompletion(modelId: "gemini-2.5-flash", Environment.GetEnvironmentVariable("GOOGLE_API_KEY")!); kernelBuilder.Plugins.AddFromType("OrderPizza"); var kernel = kernelBuilder.Build(); PromptExecutionSettings executionSettings = new GeminiPromptExecutionSettings { FunctionChoiceBehavior = FunctionChoiceBehavior.Auto(), };

var chat = kernel.GetRequiredService<IChatCompletionService>(); ChatHistory history = new();

while (true) { Console.Write("> "); string? userInput = Console.ReadLine(); if (string.IsNullOrWhiteSpace(userInput)) { break; }

history.AddUserMessage(userInput);

var result = await chat.GetChatMessageContentAsync(history, executionSettings, kernel);

Console.WriteLine(result);

history.AddAssistantMessage(result.Content ?? "");

}

class OrderPizzaPlugin { [KernelFunction("get_pizza_menu")] public Task<string[]> GetPizzaMenuAsync() { return Task.FromResult(new[] { "Cheese and Herb", "Pepperoni", "Pepperoni/Mushroom", "Chicken, Bacon, Ranch", }); } }

</details>

[futugyou]

@verdie-g There is a temporary solution in the link I replied to before, but this solution can only be used for non-stream requests

I guess the reason why 'toolCallId' appears in this code

The id in the toolCalls obtained by openai connectors from gemini is not set correctly, so an error is reported in the subsequent ToolChatMessage.ctor.

This code cannot be affected by IAutoFunctionInvocationFilter or IFunctionInvocationFilter. It seems that it can only be corrected by DelegatingHandler at present.

[github-actions[bot]]

This issue is stale because it has been open for 90 days with no activity.