feat:Add Amazon Bedrock to support the leading large language models such as Amazon Nova, Claude, Llama, Mistral, and others.

[glayyiyi]

💻 变更类型 | Change Type

• [x] feat

🔀 变更说明 | Description of Change

This PR introduces AWS Bedrock to support the leading large language models such as Claude, Llama, Mistral, and others.

1. API Enhancements:

   • Add Bedrock API integration (app/api/bedrock.ts)
   • Updated client API integration (app/client/api.ts)
   • Add Bedrock platform support (app/client/platforms/bedrock.ts)
   • Add Bedrock platform util (app/utils/aws.ts)

2. Localization:

   • Updated Chinese translations (app/locales/cn.ts)
   • Updated English translations (app/locales/en.ts)

3. Configuration:

   • Modified constants and configurations (app/constant.ts)

4. Documentation:

   • Add the description of multiple Bedrock model responses format ('docs/bedrock-response-format.md')

📝 补充信息 | Additional Information

These changes leveraging Amazon Bedrock provides access to cutting-edge language models, offering several advantages:

Access to State-of-the-Art Models: Users can now utilize the latest versions of Llama 3.2, Mistral Large, and Claude 3.5, which are known for their advanced natural language understanding and generation capabilities.

Summary by CodeRabbit

Release Notes

• New Features

  • Integrated comprehensive support for AWS Bedrock, including authentication, request handling, streaming, and error management.
  • Added Bedrock-specific configuration options in Settings for AWS credentials with validation and encryption.
  • Enhanced chat and API client to support Bedrock models and streaming responses.
  • Introduced a new FullScreen component for toggling fullscreen mode.
  • Expanded localization for Bedrock settings in English and Chinese.
  • Added Bedrock-related icons and plugin display support.
  • Improved PasswordInput component with optional masking of sensitive values.

• Bug Fixes

  • Improved error handling and validation for AWS credentials and Bedrock API interactions.

• Documentation

  • Added detailed documentation on AWS Bedrock streaming response format.
  • Updated localization files with Bedrock configuration entries.

• Chores

  • Updated environment template with Bedrock-related variables.
  • Added cryptographic utilities for encryption, decryption, and signing of AWS requests.

[vercel[bot]]

@glayyiyi is attempting to deploy a commit to the NextChat Team on Vercel.

A member of the Team first needs to authorize it.

[coderabbitai[bot]]

## Walkthrough

This pull request introduces extensive changes to integrate support for the Bedrock API across various components of the application. Key modifications include the addition of new handlers, validation logic, and utility functions for AWS credentials. The routing logic is updated to accommodate Bedrock-specific requests, while the authentication and configuration components are enhanced to manage AWS credentials securely. Additionally, localization files are modified to support Bedrock service settings, and new constants are defined to facilitate API interactions.

## Changes

| File Path                           | Change Summary                                                                                                                                                                                                 |
|-----------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| app/api/[provider]/[...path]/route.ts | Added `bedrockHandler` for handling Bedrock API requests in the routing logic.                                                                                                                               |
| app/api/auth.ts                   | Added Bedrock case in auth function to construct system API key from AWS credentials for authorization.                                                                                                        |
| app/api/bedrock.ts                | Added functions for credential retrieval, request handling, validation, timeout, and error handling for Bedrock API requests.                                                                                  |
| app/client/api.ts                 | Updated `ClientApi` constructor and header logic to support Bedrock model provider and service provider.                                                                                                        |
| app/client/platforms/bedrock.ts   | Enhanced `BedrockApi` class with methods for chat, speech (not implemented), request formatting, streaming response handling, usage, and models. Added `bedrockStream` function for streaming.                  |
| app/components/chat.module.scss   | Added nested `.icon` class inside `.chat-input-action` for SVG alignment and adjusted SCSS hierarchy.                                                                                                          |
| app/components/settings.tsx       | Added Bedrock configuration section with input fields for AWS region, access key, secret key, and encryption key, including validation and state binding.                                                     |
| app/config/server.ts              | Added optional environment variables for AWS Bedrock credentials and updated server-side config to include Bedrock-related properties.                                                                        |
| app/constant.ts                   | Added Bedrock API path, service and model provider enums, Bedrock constants with endpoint logic, Bedrock model list, and included Bedrock models in default models.                                            |
| app/locales/cn.ts                 | Added Bedrock configuration localization strings for AWS credentials under Access section.                                                                                                                    |
| app/locales/en.ts                 | Added Bedrock configuration localization strings for AWS credentials under Access section.                                                                                                                    |
| app/store/access.ts               | Extended access state with Bedrock AWS credentials, added encryption/decryption on set/get, added validation method for Bedrock credentials, and updated authorization logic.                                  |
| app/utils.ts                     | Updated `showPlugins` function to include Bedrock provider with specific model conditions for plugin display.                                                                                                  |
| package.json                    | Added newline formatting change at end of file, no dependency or config changes.                                                                                                                               |
| .env.template                   | Added new optional environment variables for AWS Bedrock configuration: AWS_REGION, AWS_ACCESS_KEY, AWS_SECRET_KEY.                                                                                            |
| app/components/ui-lib.tsx         | Enhanced `PasswordInput` with `maskWhenShow` prop and editing state; added new `FullScreen` component for toggling fullscreen mode.                                                                             |
| app/utils/aws.ts                | Added AWS-related utilities including encryption/decryption, signing requests, parsing event data, processing chunks, generating Bedrock endpoints, and extracting messages with robust error handling.         |
| docs/bedrock-response-format.md  | Added documentation detailing AWS Bedrock streaming response format, SSE chunk structure, model-specific response formats, and processing logic for streaming data.                                          |
| app/components/emoji.tsx          | Added import for `BotIconBedrock` and updated icon selection logic to assign it for models including "nova".                                                                                                   |

## Suggested reviewers

- Dogtiti
- lloydzhou

Possibly related PRs

• ChatGPTNextWeb/NextChat#5741: Adds a new handler case for ApiPath.ChatGLM in the same routing file, similar to the Bedrock handler addition here, indicating related integration patterns.

Suggested labels

planned

Suggested reviewers

• Dogtiti
• lloydzhou

🐇
New Bedrock paths we weave,
AWS keys we safely keep.
Chat streams flow, errors caught,
In code and config, all is taught.
With hops and bounds, the API sings,
Secure and swift on digital wings!
🐇

---

📜 Recent review details

Configuration used: CodeRabbit UI Review profile: CHILL Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 63707077634a6f97aa37f067c025a5a1151b957a and 121b0954b05b82dad44f7d63e760a51ddbb15fc1.

📒 Files selected for processing (1)

• app/constant.ts (8 hunks)

🚧 Files skipped from review as they are similar to previous changes (1)

• app/constant.ts

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

• Review comments: Directly reply to a review comment made by CodeRabbit. Example:
  • I pushed a fix in commit <commit_id>, please review it.
  • Generate unit testing code for this file.
  • Open a follow-up GitHub issue for this discussion.
• Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
  • @coderabbitai generate unit testing code for this file.
  • @coderabbitai modularize this function.
• PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
  • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
  • @coderabbitai read src/utils.ts and generate unit testing code.
  • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
  • @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

• @coderabbitai pause to pause the reviews on a PR.
• @coderabbitai resume to resume the paused reviews.
• @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
• @coderabbitai full review to do a full review from scratch and review all the files again.
• @coderabbitai summary to regenerate the summary of the PR.
• @coderabbitai generate docstrings to generate docstrings for this PR.
• @coderabbitai resolve resolve all the CodeRabbit review comments.
• @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
• @coderabbitai help to get help.

Other keywords and placeholders

• Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
• Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
• Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

• You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
• Please see the configuration documentation for more information.
• If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

• Visit our Documentation for detailed information on how to use CodeRabbit.
• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.

[Dogtiti]

1.一个pr不要包含太多不同的功能 2.上传附件要通盘考虑其他模型，#5585 目前我们也在考虑这块怎么处理，可以先看看这个pr

[Issues-translate-bot]

Bot detected the issue body's language is not English, translate it automatically.

---

1. A pr should not contain too many different functions
2. When uploading attachments, you must consider all other models. #5585 We are currently also considering how to handle this. You can take a look at this PR first.

[glayyiyi]

Based on the feedback from the previous PR, I have revised and optimized this PR. The scope has been narrowed and focused as suggested, with the following key improvements:

1. Streamlined PR scope:
   • Removed document-related functionality, which will be addressed in a separate PR
   • Kept the PR focused on core Bedrock API functionality improvements
2. Optimized Llama model streaming:
   • Modified streaming response handling logic according to official examples
   • Correctly processed contentBlockDelta.delta.text format messages
3. Enhanced error handling and message processing:
   • Improved logic for handling message formats
   • Strengthened error handling mechanisms
4. Code refinements:
   • Implemented suggested modifications and optimizations These changes make the PR more focused, primarily addressing core Bedrock API functionality improvements, especially support for the Llama model. Document-related features will be handled in a subsequent PR. I appreciate your previous feedback and have resubmitted this PR accordingly. Please review the changes and let me know if any further adjustments are needed.

Enhancements for tool use in both the server-side and client-side implementations for Bedrock using the Bedrock Converse API. Here is a summary of the key changes made: Server-side (bedrock.ts and models.ts):

• Added proper tool use and tool result type definitions
• Enhanced stream response handling for tool use events
• Added proper content conversion for tool use and results
• Implemented proper validation and error handling Client-side (bedrock.ts):
• Added tool and function type definitions
• Integrated with the plugin store to retrieve available tools
• Included proper tool use event handling in stream processing
• Implemented tool execution and result handling
• Added proper type safety and null checks
• Followed Anthropic's pattern for tool use implementation The implementation now supports:
• Tool use events in the stream
• Accumulation of tool arguments
• Tool execution with proper error handling
• Integration of tool results into the message stream
• Ensured proper type safety throughout

[glayyiyi]

1.一个pr不要包含太多不同的功能 2.上传附件要通盘考虑其他模型，#5585 目前我们也在考虑这块怎么处理，可以先看看这个pr

已经按建议修改和调整，请再帮忙评审下，多谢！

[Issues-translate-bot]

Bot detected the issue body's language is not English, translate it automatically.

---

1. One PR should not contain too many different functions. 2. When uploading attachments, all other models must be considered. #5585 Currently we are also considering how to deal with this issue. You can take a look at this PR first.

It has been modified and adjusted as suggested. Please help me review it again. Thank you!

[Dogtiti]

我们内部交换了一下意见,这个pr在功能上还要拆,建议拆分成两个pr，一个专门处理增加bedrock这个service provider,另一个是文档 原因如下: 1.文档这块的处理暂时不太合适,文档应该是一个独立的功能，应该是通过类似plugin那边一个读取文档内容的tool去处理。由LLM决定是否需要使用文档内容。 至于这里的上传文档，也不一定放aws，可以有很多的存文档（storage）的位置以及读取文档的tool（每一个storage会配套一个read content的tool）,现在直接扩充数据结构是不太合适的做法,可以等后续我们有一个比较确定的方案了再回头来处理这里 2.sdk的引入体积稍微有点大,体积过大在cf那边可能部署不了,之前也有人提过issue 3.sdk的接入方式可能跟目前整个项目的设计不太契合,按之前的逻辑，需要客户端也能直连，不经过/api转发,现在打包成桌面端是不会经过nextjs server

结合上述,建议看看bedrock是否能按照现有其他模型接入的方式接入,pr的范围越精简越独立越好,否则会增加review的工作量 和测试的工作量,导致pr合并时间越来越长

[Issues-translate-bot]

Bot detected the issue body's language is not English, translate it automatically.

---

We exchanged opinions internally, and this PR still needs to be split in terms of functionality. It is recommended to split it into two PRs, one specifically for adding bedrock service provider, and the other for documentation. The reasons are as follows:

1. The processing of the document is not suitable for the time being. The document should be an independent function and should be processed through a tool similar to the plugin that reads the document content. It is up to LLM to decide whether the document content needs to be used. As for the uploaded documents here, it does not necessarily have to be in AWS. There can be many storage locations and tools for reading documents (each storage will be equipped with a read content tool). Now it is not feasible to directly expand the data structure. The appropriate approach is to wait until we have a more definite plan later and then come back to deal with this.
2. The introduction volume of sdk is a bit large. If the volume is too large, it may not be deployed on cf. Some people have had issues before.
3. The access method of SDK may not be in line with the current design of the entire project. According to the previous logic, the client needs to be able to connect directly without forwarding through /api. Now it is packaged into the desktop and will not go through nextjs server.

[vercel[bot]]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
nextchat	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Dec 10, 2024 3:58am

[yilololo]

may I ask when it can be merged?

[glayyiyi]

Waiting for another author(https://github.com/Dogtiti) to review. your can get EarlyBird version at https://github.com/Dogtiti/ChatGPT-Next-Web-EarlyBird

glay @.***

 

------------------ 原始邮件 ------------------ 发件人: "ChatGPTNextWeb/ChatGPT-Next-Web" @.>; 发送时间: 2025年1月13日(星期一) 中午1:51 @.>; @.@.>; 主题: Re: [ChatGPTNextWeb/ChatGPT-Next-Web] feat:Add Amazon Bedrock to support the leading large language models such as Amazon Nova, Claude, Llama, Mistral, and others. (PR #5746)

may I ask when it can be merged?

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.***>

[silenceroom]

Waiting for another author(https://github.com/Dogtiti) to review. your can get EarlyBird version at https://github.com/Dogtiti/ChatGPT-Next-Web-EarlyBird glay @.***   … ------------------ 原始邮件 ------------------ 发件人: "ChatGPTNextWeb/ChatGPT-Next-Web" @.>; 发送时间: 2025年1月13日(星期一) 中午1:51 @.>; @.@.>; 主题: Re: [ChatGPTNextWeb/ChatGPT-Next-Web] feat:Add Amazon Bedrock to support the leading large language models such as Amazon Nova, Claude, Llama, Mistral, and others. (PR #5746) may I ask when it can be merged? — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.***>

@glayyiyi Thanks for the update, tried to pull the docker image but it seems the change did not update to the image?