RuntimeError: Bad response to Guidance request (phi-3.5-mini-instruct serverless api)

[Remilistrasza]

The bug Received the following error when using gen() with phi-3.5-mini

RuntimeError: Bad response to Guidance request
Request: https://model_name.region.models.ai.azure.com/guidance (authorization: Bearer XX...XX)
Response: 404 Not Found
NOT FOUND

Traceback

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
Cell In[2], line 4
      2     phi3_lm += "What is the capital of Australia?"
      3 with assistant():
----> 4     phi3_lm += "The capital of Australia is " + gen("answer", stop="\n", temperature=0)

File ...\anaconda3\envs\guidance\lib\site-packages\guidance\models\_model.py:486, in Model.__add__(self, value)
    484 # run stateless functions (grammar nodes)
    485 elif isinstance(value, GrammarFunction):
--> 486     out = lm._run_stateless(value)
    488 # run stateful functions
    489 else:
    490     out = value(lm)

File ...\anaconda3\envs\guidance\lib\site-packages\guidance\models\_model.py:692, in Model._run_stateless(self, stateless_function, temperature, top_p, n)
    690 delayed_bytes = b""
    691 # last_is_generated = False
--> 692 for chunk in gen_obj:
    693 
    694     # we make everything full probability if we are not computing uncertainty
    695     # if not self.engine.compute_log_probs:
    696     #     chunk.new_bytes_prob = 1.0
    697 
    698     # convert the bytes to a string (delaying if we don't yet have a valid unicode string)
    699     lm.token_count += chunk.new_token_count
    700     chunk.new_bytes = delayed_bytes + chunk.new_bytes

File ...\anaconda3\envs\guidance\lib\site-packages\guidance\models\_azure_guidance.py:73, in AzureGuidanceEngine.__call__(self, parser, grammar, ensure_bos_token)
     71     except:
     72         pass
---> 73     raise RuntimeError(
     74         f"Bad response to Guidance request\nRequest: {info}\n"
     75         + f"Response: {resp.status_code} {resp.reason}\n{text}"
     76     )
     78 for line in resp.iter_lines():
     79     if not line:

RuntimeError: Bad response to Guidance request
Request: https://model_name.region.models.ai.azure.com/guidance (authorization: Bearer XX...XX)
Response: 404 Not Found
NOT FOUND

It was working back in Oct, but after 3 weeks I cannot re-run the same code I had before. Found a discussion of the similar topic: https://github.com/guidance-ai/guidance/discussions/1041#discussioncomment-11141599, where other people are having the same issue, but I didn't find any working solution.

To Reproduce

1. Create a serverless phi-3.5-mini-instruct endpoint following this guide: https://github.com/microsoft/Phi-3CookBook/blob/main/code/01.Introduce/guidance.ipynb
2. Run the following code:

from guidance import gen, user, assistant
from guidance.models import AzureGuidance

phi3_url = "https://model-name.region.models.ai.azure.com"
phi3_api_key = "api_key"
phi3_lm = AzureGuidance(f"{phi3_url}/guidance#auth={phi3_api_key}")

with user():
    phi3_lm += "What is the capital of Australia?"
with assistant():
    phi3_lm += "The capital of Australia is " + gen("answer", stop="\n", temperature=0)

I also tried to add /chat/completions to the route, but got the same error

System info (please complete the following information):

• OS: Windows 11
• Guidance Version: 0.2.0rc1

[jmandivarapu]

I am facing similar issue. Create issued on microsoft phi3 cookbook too. Still no resolution (https://github.com/microsoft/Phi-3CookBook/issues/222)

[Harsha-Nori]

Hi @Remilistrasza and @jmandivarapu --

So sorry about that! We had to temporarily take down the experimental Phi-3.5 integration, but we will be coming back + on more Azure AI models in the very near future. I will update this thread when we are live again on Azure AI. It's early days for us on testing how this experimental integration will go into Azure AI, so really appreciate your patience with us here!

-Harsha

[jmandivarapu]

Thank you for your response @Harsha-Nori. Can I know if there is any timeline for this? So it will give us an idea of waiting or looking for other solutions.

[salekh]

Hi @Harsha-Nori, any update on guidance integration for Azure AI hosted Phi models?