[BUG] Can't add GitHub Models as a Connection

[mor10]

Describe the bug I've tried adding GitHub Models as a connection, both as a variant of Azure OpenAI and as a custom connection. I can't get it to work.

For reference, GitHub Models are loaded by passing a GH token to a custom Azure endpoint like this:

token = os.environ["GITHUB_TOKEN"]
endpoint = "https://models.inference.ai.azure.com"
model_name = "gpt-4o"

How To Reproduce the bug Steps to reproduce the behavior, how frequent can you experience the bug:

1. Try using the existing Connections options to connect to GitHub Models
2. Connection does not work

Expected behavior Connection should be possible, and the connection should appear as an option within the flow.

Running Information(please complete the following information):

• Promptflow Package Version using pf -v: 1.14.0
• Operating System: Codespaces
• Python Version using python --version: python==3.9.19

Additional context Related issue in Prompty

[brynn-code]

Hi @mor10, thanks for reaching us. The 'token' looks like something we can't save to db. Could you please show the full usage by a dummy example? Include connection yaml and how you are trying to use the connection in tool.

[mor10]

Here's the Codespaces they've provided with various code examples:

https://github.com/github/codespaces-models/blob/main/samples%2Fpython%2Fazure_ai_inference%2Fbasic.py

Here's a cookbook with a different approach: https://github.com/github/codespaces-models/blob/main/cookbooks%2Fpython%2Fopenai%2FHow_to_stream_completions.ipynb

GitHub Models is accessed via an Azure endpoint being authenticated with a GitHub token.

More here: https://azure.microsoft.com/en-us/bloglaccelerating-ai-app-development-with-azure-ai-and-github/

[mor10]

Specifically on the token, it can be saved anywhere. That example used OS, but you could just as easily save it to a DB or use .env or a local variable.

[brynn-code]

Thanks, I got the usage by their example, I mean you've said you can't get connection work, we'd like to see the way you are using the promptflow connections and get more clues about the trouble you met.

[mor10]

Ah, I see. I can't get a connection to work at all, either as a variant of the AzureOpenAIConnection or as a Custom Connection. I've tried various permutations, adding the endpoint and token, but the system does not recognize the connecting and in the UI I get red boxes around the connection name and option box.

I hypothesize the challenges here are the unusual auth flow (GitHub Personal Token) and that the endpoint for GitHub Models is multi-model - it takes a "models" parameter from which you can pick form a huge list of available models (see list in the linked examples earlier). My understanding is the current Connections options assume one model type (OpenAI in particular) while GH Models offers up Minstral and Phi and a bunch of other ones, all with different properties and supporting libraries.

Off the top of my head, the ideal situation here would be to add GitHub Models as a Connection type, then in the implementation have a property / drop-down that lists all the available models and the dev chooses the one they want to use in the node. At that point the relevant properties for that model are activated.

Worth noting: I could be doing something wrong here, so if I'm just in error and there's an obvious path forward please let me know!

Here are snippets and a screenshot of what I have that isn't working.

Trying AzureOpenAIConnection (doesn't work):

$schema: https://azuremlschemas.azureedge.net/promptflow/latest/AzureOpenAIConnection.schema.json
name: gh_models_connection
type: azure_open_ai
api_key: "[from .env]" # .env works for the same setup using OpenAI key
api_base: "https://models.inference.ai.azure.com"
api_type: "azure"
auth_mode: key

Trying CustomConnection (doesn't work):

$schema: https://azuremlschemas.azureedge.net/promptflow/latest/CustomConnection.schema.json
name: gh_models_connection
type: custom
configs:
  endpoint: "https://models.inference.ai.azure.com"
secrets:  # required
  token: "from .env" # .env works for the same setup using OpenAI key

Relevant section in flow.dag.yaml:

nodes:
- name: chat
  type: llm
  source:
    type: code
    path: chat.jinja2
  inputs:
    max_tokens: 256
    temperature: 0.7
    model: gpt-4
    response_format:
      type: text
  connection: gh_models_connection
  api: completion # "completion" and "chat" both cause the same error

Error displayed in PromptFlow VS Code extension:

[brynn-code]

Thank you for the detail, I got it.

TL:DR - Use custom connection and custom python tool can achieve this.

About Connection If the github token can be set as never expire, then it could work as a connection key. The yaml looks like:

$schema: https://azuremlschemas.azureedge.net/promptflow/latest/CustomConnection.schema.json
name: gh_models_connection
type: custom
configs:
  endpoint: "https://models.inference.ai.azure.com"
secrets:  # required
  token: "<your_token>"

We don't support expression as a yaml value so the token: "from .env" will not be resolved and doen't work.

About Tool The type: llm api:completion in flow yaml means this node will use promptflow built in llm tool, this type of llm tool supports AzureOpenAI and OpenAI, you could found all built-in tool reference here: https://microsoft.github.io/promptflow/reference/tools-reference/llm-tool.html

As we don't have github model as a built-in tool for now, you could write your own python tool to achieve that, the python tool will look like:

from promptflow import tool
from azure.ai.inference import ChatCompletionsClient
from azure.ai.inference.models import SystemMessage, UserMessage
from azure.core.credentials import AzureKeyCredential

@tool
def my_tool(
    connection: CustomConnection,
    messages: list,
    model_name: str,
    temperature: float = None,
    max_tokens: int = None,
    top_p: float = None
):
    client = ChatCompletionsClient(
        endpoint=connection.configs['endpoint'],
        credential=connection.secrets['token'],
    )

    response = client.complete(
        messages=messages,
        model=model_name,
        # Optional parameters
        temperature=temperature,
        max_tokens=max_tokens,
        top_p=top_p
    )

    return response.choices[0].message.content

and use this python tool in yaml:

nodes:
- name: call_github
  type: python
  source:
    type: code
    path: my_tool.py
  inputs:
    connection: gh_models_connection
    messages: ....
    model: ....