Error When Prompting Mistral/Mixtral Model from AWS Bedrock

[bahamasbahamas]

Description: I successfully cloned the repository from the main branch and tested the application with LLMs deployed on Azure using the OpenAI resource, and everything worked as expected. However, I encountered an issue when attempting to prompt a Mistral model hosted on AWS Bedrock.

Steps to Reproduce:

• Cloned the repository from the main branch.
• Created temporary credentials on AWS for accessing the AWS Bedrock.
• Attempted to send prompts to the Mistral/Mixtral LLM models.
• Received the following error message for both model types.

Expected Behavior: I expected that the prompts would be successfully processed by the Mistral/Mixtral models without any errors.

Observed Behavior: When attempting to send prompts, the application failed to collect responses and suggested re-running the prompt node. The specific error logged was:

Errors collecting responses. Re-run prompt node to retry.
Mistral Mixtral: "c.get is not a function"

Environment: Operating System: Ubuntu 22.04 + docker

[ianarawjo]

Thanks for this issue. @massi-ang is the point person for Bedrock integrations. massi-ang

@massi-ang might you know what is going on?

c.get doesn't sound like any code in the ChainForge code base (there is no "get" function or "c.get" call). So, it sounds like this might actually be a bug on the Bedrock side with loading the model. Not sure.

[bahamasbahamas]

Hi @ianarawjo, do you use the boto3 AWS client to query the Bedrock models? If not, I'm curious—why don't you use it? I've noticed it's more straightforward since you can just use regular API keys without needing a session key.

[ianarawjo]

I didn’t write the Bedrock integration, @massi-ang did. He would have to comment here.

[massi-ang]

Hi @bahamasbahamas, we use the boto3 client. The @mirai73/bedrock-fm library is just a wrapper around the boto3 client to provide the correct prompt formatting for the different models. The reason why we require temporary credentials and a session token is to safeguard you against the possibility of accidentally leaking long term credentials.

I'll need to look how allow to use just an AccessKeyId and SecretAccessKey when running locally, that is when using chainforge/app.py serve.

[massi-ang]

@bahamasbahamas regarding the issue you are facing, I tested the latest main commit and I could not reproduce the error. Does this happen with any flow and prompt or just specific configurations?

[massi-ang]

HI @bahamasbahamas. I reproduced this problem both on chainforge.ai/play site and when using chainforge serve from a local install. On the other hand I do not get this error when I clone the repo locally and build the react backend from chainforge/react-server with npm run build and then serve it via python chainforge/app.py serve. Similarly there is no error when serving the react app with other means, as you can see by accessing this url https://d1sozkr3w0qe91.cloudfront.net/.

@ianarawjo can you share how the react app is built so that I can get to the root cause of the problem?

[ianarawjo]

There is no difference with how the react app is built —npm run build is what is done in all cases. The sole difference is just adding /play the HTML index page on the /play site. There are no other differences.