Azure Foundry models don't work (Mistral Codestral 2501 with Azure)

[BastinFlorian]

Issue Category

Undocumented feature or missing documentation

Affected Documentation Page URL

https://docs.continue.dev/customize/model-providers/azure

Issue Description

Using Codestral 2501 available in the Azure Hub through Azure Foundry or Azure ML products have a different target URI that OpenAI Azure models.

Azure Open AI Target URI: https://just-an-example.openai.azure.com/openai/deployments/gpt-4o-july/chat/completions?api-version=2023-03-15-preview Foundry Codestral URL example (same process for AzureML deployed models): https://just-an-example.openai.azure.com/chat/completions?api-version=2023-03-15-preview

The difference is just : openai/deployments/gpt-4o-july

How to configure such models ? Can you please update the Codestral documentation thats seems to be lights/wrong for Autocomplete Codestral in Azure ?

Adding this feature allows both Azure Foundry and Azure ML models to be deployed with continue dev because the URI are the same for both products (I know azure is messy).

RESOLUTION: just add a new value for ApiType = 'foundry' to set the goold URI.

Version: 0.8.66 IDE: VS Code

Thank you,

Expected Content

No response

[Patrick-Erichsen]

Discord thread that this issue began in: https://discord.com/channels/1108621136150929458/1328759921512288316

[BastinFlorian]

Any news @sestinj 🙏

[George-Const]

Same for deepseek-R1 in azure foundry.

[BastinFlorian]

Can you provide an URI exemple for DeepSeek ? Is it in the same format that the codestral one ? If yes, adding this feature could be very helpful.

[George-Const]

Can you provide an URI exemple for DeepSeek ? Is it in the same format that the codestral one ? If yes, adding this feature could be very helpful.

for deepseek-R1 deployments, following urls are provided:

Target URI: https://.eastus.models.ai.azure.com

API Routes

  Azure AI model inference: Chat Completion
  https://<deployment-name-here>.eastus.models.ai.azure.com/chat/completions

  DeepSeek-R1: Chat Completion
  https://<deployment-name-here>.eastus.models.ai.azure.com/v1/chat/completions

[Patrick-Erichsen]

Fix here: https://github.com/continuedev/continue/pull/4027

@BastinFlorian @George-Const - mind sharing your configs, with sensitive info removed? I know we had some mixups with the apiType and provider fields for Azure in the past, so want to make sure I'm not breaking any existing configs with that PR.

[XYYuFish]

Fix here: #4027

@BastinFlorian @George-Const - mind sharing your configs, with sensitive info removed? I know we had some mixups with the apiType and provider fields for Azure in the past, so want to make sure I'm not breaking any existing configs with that PR.

Model deployment rounting: https://learn.microsoft.com/en-us/azure/ai-foundry/model-inference/concepts/endpoints?tabs=python Hi, Patrick. This picture shows my deployment information, and the link explains model deployment routing. I hope it helps.

[BastinFlorian]

Fix here: #4027 @BastinFlorian @George-Const - mind sharing your configs, with sensitive info removed? I know we had some mixups with the apiType and provider fields for Azure in the past, so want to make sure I'm not breaking any existing configs with that PR.

Model deployment rounting: https://learn.microsoft.com/en-us/azure/ai-foundry/model-inference/concepts/endpoints?tabs=python Hi, Patrick. This picture shows my deployment information, and the link explains model deployment routing. I hope it helps.

You have it on the discord dicussion

[Patrick-Erichsen]

Merged that PR after confirming the new logic should work with your configs 👍 thanks for your patience here @BastinFlorian and @XYYuFish !

[Patrick-Erichsen]

Although @BastinFlorian , to your original question regarding Codestral w/ Azure, we're still awaiting a reply as to whether or not this is possible. The Azure docs seem to imply that Codestral can only be used for Chat: https://learn.microsoft.com/en-us/azure/ai-studio/how-to/deploy-models-mistral-codestral?pivots=programming-language-python

If this is indeed the case we'll update the docs to make it clear that Codestral can't be used for Autocomplete with Azure.

[fuglede]

I also wanted to play with Codestral for completions, so following along on the side on this one, and while waiting for the PR to be completed, I had just manually patched the

    if (this.apiType === "azure") {
      return new URL(
        `openai/deployments/${this.deployment}/${endpoint}?api-version=${this.apiVersion}`,
        this.apiBase,
      );
    }

issue in the built extension, i.e. ~/.vscode/extensions/continue.continue-0.8.68-linux-x64/out/extension.js.

And here, that was enough to get completions out of it in VS Code, so it looks like it is at least possible, but maybe not intended? The quality did strike me as surprisingly poor; infinite repetitions being common for instance.

[grosjeang]

Although @BastinFlorian , to your original question regarding Codestral w/ Azure, we're still awaiting a reply as to whether or not this is possible. The Azure docs seem to imply that Codestral can only be used for Chat: https://learn.microsoft.com/en-us/azure/ai-studio/how-to/deploy-models-mistral-codestral?pivots=programming-language-python

If this is indeed the case we'll update the docs to make it clear that Codestral can't be used for Autocomplete with Azure.

In Azure AI Foundry, in my Codestral deployment details, I have 3 API routes including one route for FIM. When sending a post request to that route with a prompt and a suffix, it returns the code completion needed for Autocomplete. So using Codestral with Azure should be possible for Autocomplete. I don't know how API calls are made in Continue.dev for Azure AI Foundry models but I am not sure the azure-ai-inference package includes an easy way to do FIM for now.

[2mas]

I got the code completions (and chat) working by deploying the Codestral model in Machine Learning studio (Foundry deployment + endpoint didn't work) and by using the provider: "mistral" in the model config

      "provider": "mistral",
      "apiKey": "",
      "deployment": "x",
      "apiBase": "https://x.models.ai.azure.com/v1",
      "model": "Codestral-2501",
      "title": "Codestral-2501"

[ulo]

The documentation on using Azure APIs is really rather outdated and it took me quite some try and error to properly configure Continue.

This is my config to get Chat + Autocompletion working with Azure AI Foundry models:

{
  "models": [
    {
      "title": "GPT-4o Azure",
      "provider": "azure",
      "model": "gpt-4o",
      "apiBase": "https://***.openai.azure.com",
      "deployment": "***",
      "apiKey": "***"
    }
  ],
  "tabAutocompleteModel": {
    "title": "Codestral Azure",
    "provider": "mistral",
    "model": "codestral-latest",
    "apiBase": "https://***.models.ai.azure.com/v1",
    "apiKey": "***"
  }
}

Azure AI Foundry also offers the Cohere reranker, but I would not know how the config section should look like to use it...

[BastinFlorian]

@Patrick-Erichsen, @ulo's solution working like a charm for your information (autocomplete + chat).

I opened this Pull Request to update the docs accordingly: https://github.com/continuedev/continue/pull/4526

[George-Const]

Although @BastinFlorian , to your original question regarding Codestral w/ Azure, we're still awaiting a reply as to whether or not this is possible. The Azure docs seem to imply that Codestral can only be used for Chat: https://learn.microsoft.com/en-us/azure/ai-studio/how-to/deploy-models-mistral-codestral?pivots=programming-language-python If this is indeed the case we'll update the docs to make it clear that Codestral can't be used for Autocomplete with Azure.

In Azure AI Foundry, in my Codestral deployment details, I have 3 API routes including one route for FIM. When sending a post request to that route with a prompt and a suffix, it returns the code completion needed for Autocomplete. So using Codestral with Azure should be possible for Autocomplete. I don't know how API calls are made in Continue.dev for Azure AI Foundry models but I am not sure the azure-ai-inference package includes an easy way to do FIM for now.

For which region? I recently deployed codestral to sweden central and cannot find the fim endpoint.

[nickthecook]

I recently deployed codestral to sweden central and cannot find the fim endpoint.

I just deployed to canadaeast via AI Foundry and see the same: just one endpoint, for models/chat/completions. No FIM. I can access the chat endpoint in curl, but when I try v1/fim/completions I get a 404.

Following the docs, but my apiBase doesn't match that format. I have this:

https://xxxxxxxx-resource.services.ai.azure.com/models/chat/completions?api-version=2024-05-01-preview

Does Continue support Codestral the way I've deployed it?

[George-Const]

I recently deployed codestral to sweden central and cannot find the fim endpoint.

I just deployed to canadaeast via AI Foundry and see the same: just one endpoint, for models/chat/completions. No FIM. I can access the chat endpoint in curl, but when I try v1/fim/completions I get a 404.

Following the docs, but my apiBase doesn't match that format. I have this:

https://xxxxxxxx-resource.services.ai.azure.com/models/chat/completions?api-version=2024-05-01-preview

Does Continue support Codestral the way I've deployed it?

I had the same. Deployed a second time to sweden, not sure why but the second deployment has the fim endpoint as well and all works fine. The only difference I see is that I specifically had to diselect the content filter when deploying model in azure foundry.

[nickthecook]

Interesting, I'll try deploying again.

Do you mean "all works fine" as in Continue will use the FIM successfully? Is that just following these docs?

[George-Const]

Interesting, I'll try deploying again.

Do you mean "all works fine" as in Continue will use the FIM successfully? Is that just following these docs?

Yes works fine. Both for fim and chat.

[George-Const]

Interesting, I'll try deploying again.

Do you mean "all works fine" as in Continue will use the FIM successfully? Is that just following these docs?

Yes works fine. Both for fim and chat.

I double-checked. For model I used Codestral-2501 and not codestral-latest as indicated in the article you provided. Codestral-2501 is the actual model name in azure.

[bebeo92]

This config seem work to me, at least I can see request in metric tab on azure

"tabAutocompleteModel": {
    "title": "Codestral",
    "provider": "azure",
    "model": "Codestral-2501",
    "apiBase": "azure api base",
    "apiType": "azure-foundry",
    "apiKey": "key",
    "apiVersion": "2024-05-01-preview",
    "deployment": "deploymentName"
  },

[George-Const]

This config seem work to me, at least I can see request in metric tab on azure

"tabAutocompleteModel": {
    "title": "Codestral",
    "provider": "azure",
    "model": "Codestral-2501",
    "apiBase": "azure api base",
    "apiType": "azure-foundry",
    "apiKey": "key",
    "apiVersion": "2024-05-01-preview",
    "deployment": "deploymentName"
  },

I think for FIM to work you still need to use mistral for provider and your apiBase should end with /v1 :)

[BastinFlorian]

Azure does not offer autocomplete model for codestral, only chat completion codestral To use codestral as an autocomplete, you need to deploy it in Mistral. It is very disappointing from Azure by the way

[Patrick-Erichsen]

@BastinFlorian - are these docs just flat-out wrong then? https://docs.continue.dev/customize/model-providers/azure#autocomplete-model

Wondering if they should be updated to just state that there is no autocomplete offering on Azure.

[njaku21]

@BastinFlorian - are these docs just flat-out wrong then? https://docs.continue.dev/customize/model-providers/azure#autocomplete-model

Wondering if they should be updated to just state that there is no autocomplete offering on Azure.

Codestral autocomplete via Foundry has definitely worked for me up until (assumedly) a recent Continue.dev update. No change to API endpoints or keys on my end, just not working as of recent. I decided to rewrite my config in yaml to see if there is any difference in behavior but it appears not.

I get "JSON.parse(...)?.error?.replace is not a function" with apiType: azure-foundry

Sometimes I get "Unexpected token 'N', "NOT FOUND" is not valid JSON" instead

I will be digging into troubleshooting further but it hasn't been an annoying enough issue until today for me on my work assets.

A possibly important thing to note is codestral currently works via chat but not for autocomplete. So the API is handshaking and presumably my config is correct(especially since it worked great before). I have also dabbled with Azure AI services and serverless deployments in Foundry and both seem to have issues. They each seem to have slightly different endpoint configurations but I am not sure of the downstream effect with continue.dev.

Azure AI Services Hosted URL: https://[Project].services.ai.azure.com/models/chat/completions?api-version=2024-05-01-preview

Serverless URL: https://[deployment name].eastus.models.ai.azure.com(also exposes a /swagger.json)

Some kind of test-all-configured-model-methods button would be nice to have in the settings for more obvious outputs on this kind of troubleshooting.

I at least can use chat with this config:

- name: Codestral
    provider: azure # azure-foundry does not work here
    model: codestral-2501
    apiBase: https://[project].services.ai.azure.com
    apiKey: [key] # If you use subscription key, try using Azure gateway to rename it apiKey
    requestOptions:
      caBundlePath: [optional_cert]
    env:
      deployment: Codestral-2501
      apiType: azure # azure here works for chat but not code completion. azure-foundry does not work for either, gives the error noted above.
    roles:
      - autocomplete
      - chat

further testing with modified model settings(projectTemplates>autocomplete) gives(note the different api-version date) :

HTTP 404 Resource Not Found from https://[project].services.ai.azure.com/v1/completions?api-version=2023-07-01-preview {"error":{"code":"404","message": "Resource not found"}}

if I add autocomplete to gpt-4o-mini it does attempt to autocomplete so it seems to be isolated to the config with codestral.

- name: GPT-4o-Mini
    provider: azure
    model: gpt-4o-mini
    apiBase: https://[project]openai.azure.com
    apiKey: [key] # If you use subscription key, try using Azure gateway to rename it apiKey
    env:
      deployment: gpt-4o-mini
      apiType: azure-openai # Or "azure-openai" if using OpenAI models
    requestOptions:
      caBundlePath: [optional_cert]
    roles:
      - autocomplete
      - chat

[Patrick-Erichsen]

@2mas can you try downgrading to an older version of Continue and see if that same config that was working previously is still working? That would help determine if its us or Azure

I think this is what your Codestral config should look like though

- name: Codestral
    provider: azure 
    model: codestral-2501
    apiBase: https://[project].services.ai.azure.com
    apiKey: [key] # If you use subscription key, try using Azure gateway to rename it apiKey
    requestOptions:
      caBundlePath: [optional_cert]
    env:
      apiType: azure-foundry
    roles:
      - autocomplete
      - chat

Here's where we use those env configs to determine the endpoint: https://github.com/continuedev/continue/blob/main/core/llm/llms/OpenAI.ts#L199-L222

[George-Const]

@Patrick-Erichsen @BastinFlorian FIM works just fine but the azure foundry deployment must have the fim endpoint like so:

It used to not have it before, at least for sweden from what I've noticed, I just tried deploying another instance and as you can see the fim endpoint is there. Then I had to use mistral as provider instead of azure (otherwise it's not working indeed):

  - name: Codestral azure new
    provider: mistral
    model: Codestral-2501
    apiBase: https://Codestral-2501-rkfug.swedencentral.models.ai.azure.com/v1
    apiKey: ******
    roles:
      - chat
      - edit
      - apply
      - autocomplete

With this config, fim and chat work fine.

[nickthecook]

Thanks for the config and the screenshot.

When I deployed Codestral in canadacentral and canadaeast a couple weeks ago I didn't get the FIM endpoint, just the chat endpoint.

I know Azure's feature support varies by region, especially with new stuff like AI, so I assume this is just a per-region thing. Hopefully they roll it out to more regions soon.

[njaku21]

@Patrick-Erichsen @BastinFlorian FIM works just fine but the azure foundry deployment must have the fim endpoint like so:

It used to not have it before, at least for sweden from what I've noticed, I just tried deploying another instance and as you can see the fim endpoint is there. Then I had to use mistral as provider instead of azure (otherwise it's not working indeed):

  - name: Codestral azure new
    provider: mistral
    model: Codestral-2501
    apiBase: https://Codestral-2501-rkfug.swedencentral.models.ai.azure.com/v1
    apiKey: ******
    roles:
      - chat
      - edit
      - apply
      - autocomplete

With this config, fim and chat work fine.

Microsoft must have been having an outage with their serverless endpoints in some regions. I know that I definitely did not have these routes last week(I must have prior as it was working for me prior), but I do have them today in Foundry and FIM is properly working for me now. Something to note in the future since I looked and didn't see any obvious reports of issues with Codestral in Foundry.

I guess I should have tried interacting with the API directly before assuming it was caused by a Continue update😅

my slightly-different config using the Serverless that also works fwiw

  - name: Codestral
    provider: mistral
    model: codestral-latest
    apiBase: https://[deployment-name].eastus.models.ai.azure.com/v1
    apiKey: [key] # If you use subscription key, try using Azure gateway to rename it apiKey
    requestOptions:
      caBundlePath: [optional cert]
    roles:
      - autocomplete
      - chat

using the Azure AI Services endpoint(both Serverless and Azure AI services endpoints are created automatically when deploying) does not work for autocomplete, only chat.

  - name: Codestral 2
    provider: azure
    model: codestral-2501
    apiBase: https://[project].services.ai.azure.com/models
    api-version: 2024-05-01-preview
    apiKey: [key] # If you use subscription key, try using Azure gateway to rename it apiKey
    requestOptions:
      caBundlePath: [optional cert]
    env:
      deployment: codestral-2501
      apiType: azure
    roles:
      - autocomplete
      - chat