Running with Azure supported?

[tanujjain]

I've been trying to use Nemo-Guardrails with Azure but keep getting the dreaded "openai.error.InvalidRequestError: Resource not found" error. Outside of Nemo-Guardrails, my Azure credentials work with openai python api as well as langchain Azure apis (both AzureOpenAI and AzureChatOpenAI).

I tried the helloworld example and ran the cli command: nemoguardrails chat --config=config/hello_world. Also tried running the basic usage example for python api. Have tried setting engine parameter in config.yml to openai and azure.

Checking the engine parameter in the config from the documentation- engine: the LLM provider; currently, only "openai" is supported.

But also from the documentation- You can use any LLM provider that is supported by LangChain, e.g., ai21, aleph_alpha, anthropic, anyscale, azure ..

So, is Azure supported?

My specifics- Python 3.10.11 nemoguardrails==0.3.0 openai==0.27.7 langchain==0.0.167 conda 22.9.0 macOS Ventura(M1 Max chip)

[chilicrabcakes]

Hey, I found a workaround for this. It can be solved using LangChain in the Python API. You can define your LLM in LangChain using the Azure Chat OpenAI object and pass that as an argument to the LLMRails class.

For example,

from langchain.chat_models import AzureChatOpenAI
llm = AzureChatOpenAI(<your-parameters-here>)

from nemoguardrails import LLMRails, RailsConfig

config = RailsConfig.from_path("path/to/config")

app = LLMRails(config, llm=llm)
new_message = app.generate(messages=[{
    "role": "user",
    "content": "Hello! What can you do for me?"
}])

An additional error I was noticing here is that app.generate kept telling me that I can't run synchronous calls inside async code, not really sure why. I got it running by replacing app.generate with app.generate_async but that's probably not the best way to solve this problem.

[ansumanparija007]

@tanujjain @chilicrabcakes facing a similar issue while trying to work with guardrails . Parameter temperature does not exist for NoneType Error 'NoneType' object has no attribute 'agenerate_prompt' while execution check_jailbreak Traceback (most recent call last): File "C:\Python311\Lib\site-packages\nemoguardrails\actions\action_dispatcher.py", line 125, in execute_action result = await fn(**params) ^^^^^^^^^^^^^^^^^^ File "C:\Python311\Lib\site-packages\nemoguardrails\actions\jailbreak_check.py", line 50, in check_jailbreak check = await llm_call(llm, prompt) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Python311\Lib\site-packages\nemoguardrails\actions\llm\utils.py", line 31, in llm_call result = await llm.agenerate_prompt(

Any suggestions ,have you faced this issue

[drazvan]

@ansumanparija007 : looks like llm is set to None in your case. Can you provide more details on the config?

[ansumanparija007]

Hi @drazvan i am try in to use the following config to connect to azure open ai- chat_model = AzureChatOpenAI( openai_api_type="azure", openai_api_version="2023-03-15-preview", openai_api_key=azure_openai_key, deployment_name=azure_openai_model, openai_api_base=azure_openai_endpoint ) app = LLMRails(config=config, llm=chat_model)

new_message = app.generate(messages=[{ "role": "user", "content": "What's the latest fashion trend?" }])

i hope currently guradrails supports azure open ai ,pls let me know

[ansumanparija007]

any luck with this @drazvan ?

[drazvan]

I can't test this directly as I don't have an Azure Key. Can you confirm the chat_model instance works correctly? i.e. completion = chat_model("some text"). And if it does, can you share the complete error stack trace? Thanks.

[ansumanparija007]

Yeah chat_model works fine .but while working with guardrails following error is thrown - Parameter temperature does not exist for NoneType Error 'NoneType' object has no attribute 'agenerate_prompt' while execution check_jailbreak Traceback (most recent call last): File "C:\Users\testcopilot2\guardrails\chatbot-guardrails\env\Lib\site-packages\nemoguardrails\actions\action_dispatcher.py", line 125, in execute_action result = await fn(**params) ^^^^^^^^^^^^^^^^^^ File "C:\Users\testcopilot2\guardrails\chatbot-guardrails\env\Lib\site-packages\nemoguardrails\actions\jailbreak_check.py", line 50, in check_jailbreak check = await llm_call(llm, prompt) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\testcopilot2\guardrails\chatbot-guardrails\env\Lib\site-packages\nemoguardrails\actions\llm\utils.py", line 31, in llm_call result = await llm.agenerate_prompt( ^^^^^^^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'agenerate_prompt' new_message: {'role': 'assistant', 'content': "I'm sorry, an internal error has occurred."}

[drazvan]

Ok, I think I'm starting to understand the issue. Can't test, so bear with me. Can you try adding the following:

app = LLMRails(config=config, llm=chat_model)
app.runtime.register_action_param("llm", chat_model)

[ansumanparija007]

thanks @drazvan this error is gone.ll try to play with it and let you know ..

[ansumanparija007]

Hi @drazvan while running hallucination cheack it says Hallucination rail can only be used with OpenAI LLM engines.Current LLM engine is AzureChatOpenAI. Any ETA when it will be supported for other LLM models?

[ishaan-jaff]

Hi @drazvan @ansumanparija007 @tanujjain I believe we can help with this issue. I’m the maintainer of LiteLLM https://github.com/BerriAI/litellm

TLDR: We allow you to use any LLM as a drop in replacement for gpt-3.5-turbo. If you don't have access to the LLM you can use the LiteLLM proxy to make requests to the LLM

You can use LiteLLM in the following ways:

With your own API KEY:

This calls the provider API directly

from litellm import completion
import os
## set ENV variables 
os.environ["OPENAI_API_KEY"] = "your-key" # 
os.environ["COHERE_API_KEY"] = "your-key" # 

messages = [{ "content": "Hello, how are you?","role": "user"}]

# openai call
response = completion(model="gpt-3.5-turbo", messages=messages)

# cohere call
response = completion(model="command-nightly", messages=messages)

Using the LiteLLM Proxy with a LiteLLM Key

this is great if you don’t have access to claude but want to use the open source LiteLLM proxy to access claude

from litellm import completion
import os

## set ENV variables 
os.environ["OPENAI_API_KEY"] = "sk-litellm-5b46387675a944d2" # [OPTIONAL] replace with your openai key
os.environ["COHERE_API_KEY"] = "sk-litellm-5b46387675a944d2" # [OPTIONAL] replace with your cohere key

messages = [{ "content": "Hello, how are you?","role": "user"}]

# openai call
response = completion(model="gpt-3.5-turbo", messages=messages)

# cohere call
response = completion(model="command-nightly", messages=messages)

[ayoubazaouyat]

if anyone is still looking for an answer here is what you should do :

first init the env variables : os.environ["AZURE_OPENAI_API_KEY"] = os.getenv("AZURE_OAI_KEY") os.environ["AZURE_OPENAI_ENDPOINT"] = os.getenv("AZURE_OAI_ENDPOINT")

in your config.yml :

models:

• type: main engine: azure model: gpt-4o
  parameters: deployment_name: xxxxx api_version: xxxxxx

  this did work for me.

[Ryan-ZL-Lin]

@ishaan-jaff and @drazvan Any idea how to use Litellm in NeMo Guardrails?

Here is my attempt which is not working with error: An error occurred: LLM Call Exception: Connection error.

but the connection and api key is actually correct by using pure Litellm approach

BTW, I can use Azure OpenAI API in NeMo Guardrails without issues.