chattr
chattr copied to clipboard
mlverse
Feature Request: Support Databricks Foundation Model & Pay-Per-Token APIs
Perhaps it's user error but I can't pass a custom OpenAI base_url to redirect the requests to a Databricks serving endpoint. This would be ideal for using {chattr} to interact with Databricks foundation model APIs and pay-per-token LLMs.
Below is an example of how to query these APIs from the Databricks docs:
import os
import openai
from openai import OpenAI
client = OpenAI(
api_key="dapi-your-databricks-token",
base_url="https://example.staging.cloud.databricks.com/serving-endpoints"
)
response = client.chat.completions.create(
model="databricks-dbrx-instruct",
messages=[
{
"role": "system",
"content": "You are a helpful assistant."
},
{
"role": "user",
"content": "What is a mixture of experts model?",
}
],
max_tokens=256
)
Appreciate the help and awesome package!
@zacdav-db - Is this the endpoint that you referred to when we discussed the enhancement to chattr?
@edgararuiz Yep! Haven't yet had a chance to tinker and add it, might do soon.
