400 for saving text to knowledge base, just for extra '\' before exisiting '\', '\\' cause 400

[hanwsf]

Self Checks

• [X] This is only for bug report, if you would like to ask a question, please head to Discussions.
• [X] I have searched for existing issues search for existing issues, including closed ones.
• [X] I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
• [X] Please do not modify this template :) and fill in all the required fields.

Dify version

0.68

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

{ "request": "POST http://docker-nginx-1/v1/datasets/3ssss/document/create_by_text HTTP/1.1\nContent-Type: application/json\nAuthorization: ***************************************\n\n{\n "name": "java如何进行AIGC应用开发与python开发有什么不同？",\n\n\n "text": "Java和Python是两种常用的编程语言，一些人工智能应用程序可能需要使用这两种语言来开发。虽然AIGC（人工智能和全球社区）可以在Java中工作，但它通常更适用于Python，因为Python的简单性和易于学习的语法使其非常适合AI开发。\n\nJava如何进行AIGC应用开发：\n1. 首先，确保你已经安装了Java Development Kit(JDK)并配置了环境变量。\n2. 接下来，你需要选择一个合适的Java库或框架来处理AI任务。例如，您可以使用DeepLearning4J进行深度学习，或使用Weka进行机器学习。\n3. 编写代码，按照所需的AI应用程序设计流程进行操作。",\n\n "indexing_technique": "high_quality",\n "process_rule": {\n "mode": "automatic"\n }\n}" } success for manual debug. { "request": "POST http://docker-nginx-1/v1/datasets/3sss/document/create_by_text HTTP/1.1\nContent-Type: application/json\nAuthorization: ***************************************\n\n{\n "name": "java如何进行AIGC应用开发与python开发有什么不同？",\n\n\n "text": "Java和Python是两种常用的编程语言，一些人工智能应用程序可能需要使用这两种语言来开发。虽然AIGC（人工智能和全球社区）可以在Java中工作，但它通常更适用于Python，因为Python的简单性和易于学习的语法使其非常适合AI开发。\n\nJava如何进行AIGC应用开发：\n1. 首先，确保你已经安装了Java Development Kit(JDK)并配置了环境变量。\n2. 接下来，你需要选择一个合适的Java库或框架来处理AI任务。例如，您可以使用DeepLearning4J进行深度学习，或使用Weka进行机器学习。\n3. 编写代码，按照所需的AI应用程序设计流程进行操作。",\n\n "indexing_technique": "high_quality",\n "process_rule": {\n "mode": "automatic"\n }\n}" } 400 for workflow, because of '\n'.

✔️ Expected Behavior

200

❌ Actual Behavior

400

[dosubot[bot]]

To resolve the 400 error caused by extra backslashes in the payload when sending a POST request to save text to the knowledge base, you need to ensure that the payload is properly formatted and does not contain unnecessary escape characters.

Here is a general approach to handle this issue:

1. Sanitize the Input: Ensure that the text being sent in the payload is properly sanitized to remove any extra backslashes.

2. Stringify the Payload Correctly: When converting the payload to a JSON string, make sure it is done correctly to avoid adding extra escape characters.

Here is an example of how you can sanitize and stringify the payload in JavaScript:

function sanitizeInput(input) {
  // Remove unnecessary backslashes
  return input.replace(/\\/g, '');
}

function preparePayload(data) {
  // Sanitize each field in the data object
  for (let key in data) {
    if (typeof data[key] === 'string') {
      data[key] = sanitizeInput(data[key]);
    }
  }
  return JSON.stringify(data);
}

// Example usage
let payload = {
  name: "java如何进行AIGC应用开发与python开发有什么不同？",
  text: "Java和Python是两种常用的编程语言，一些人工智能应用程序可能需要使用这两种语言来开发。虽然AIGC（人工智能和全球社区）可以在Java中工作，但它通常更适用于Python，因为Python的简单性和易于学习的语法使其非常适合AI开发。\n\nJava如何进行AIGC应用开发：\n1. 首先，确保你已经安装了Java Development Kit(JDK)并配置了环境变量。\n2. 接下来，你需要选择一个合适的Java库或框架来处理AI任务。例如，您可以使用DeepLearning4J进行深度学习，或使用Weka进行机器学习。\n3. 编写代码，按照所需的AI应用程序设计流程进行操作。",
  indexing_technique: "high_quality",
  process_rule: {
    mode: "automatic"
  }
};

let sanitizedPayload = preparePayload(payload);

// Send the POST request with the sanitized payload
fetch('http://docker-nginx-1/v1/datasets/3ssss/document/create_by_text', {
  method: 'POST',
  headers: {
    'Content-Type': 'application/json',
    'Authorization': 'Bearer YOUR_API_KEY'
  },
  body: sanitizedPayload
})
.then(response => response.json())
.then(data => console.log(data))
.catch(error => console.error('Error:', error));

This code ensures that any extra backslashes in the input are removed before the payload is sent in the POST request. Additionally, ensure that the payload data is correctly formatted according to the API's requirements ^[1][2][3].

Details

About Dosu This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

To continue the conversation, mention @dosu.

[hanwsf]

Alternative way is adding a code block to replace '\' with ''

[crazywoola]

Right I remember this one. But we would like to remain the source untouched.

function main({ arg1 }) {
    const cleaned = arg1.replace(/[\r\n]+/g, '');
    return {
        result: cleaned
    };
}