Text deduplication

[linnnff]

Self Checks

• [X] I have searched for existing issues search for existing issues, including closed ones.
• [X] I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
• [X] Please do not modify this template :) and fill in all the required fields.

1. Is this request related to a challenge you're experiencing?

When I tried to use the knowledge base function, I found that there was duplicate content in the segmented data, and I hoped to have a deduplication function.

2. Describe the feature you'd like to see

After splitting the data, duplicate data is deduplicated.

3. How will this feature improve your workflow or experience?

Remove useless duplicate data to ensure that more data is obtained during retrieval.

4. Additional context or comments

No response

5. Can you help us with this feature?

• [ ] I am interested in contributing to this feature.