Support for Anyscale models with Log Probabilities

[chemeris]

Is your feature request related to a problem? Please describe. OpenAI support at guidance doesn't rely on logrpobs, as far as I understand. Anyscale seems to mimic OpenAI API but in addition they also provide log probabilities for the generated output (https://docs.endpoints.anyscale.com/guides/logprobs). It looks like this could significantly improve guidance performance when working with these models.

Describe the solution you'd like Better performance when working with Anyscale models with logprob output enabled.

[Harsha-Nori]

HI @chemeris, great suggestion! Do you happen to know if Anyscale models support partial Assistant completions? For technical reasons, we'd be able to leverage logprobs much better if we can send up Chat messages that end in an assistant role, which the model then continues. This would enable us to cut the generation stream and switch to new, grammar-compliant tokens midway through an assistant generation. Unfortunately the OpenAI API doesn't allow for this, but if Anyscale does, it is worth looking into.

I don't have a deployment, but you can hopefully test this easily with something like the following (borrowed most of the code from the link you sent above):

import os
import requests

s = requests.Session()

api_base = os.getenv("OPENAI_BASE_URL")
token = os.getenv("OPENAI_API_KEY")
url = f"{api_base}/chat/completions"
body = {
    "model": "mistralai/Mixtral-8x7B-Instruct-v0.1",
    "messages": [
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Count to 10, starting from 1."},
        {"role": "assistant", "content": "1,2,3,"}
    ],
    "temperature": 0.2,
    "logprobs": True,
    "top_logprobs": 1
}

with s.post(url, headers={"Authorization": f"Bearer {token}"}, json=body) as resp:
    print(resp.json())