Feature/improved-language-detection

[suzinyou]

Reviewer: @lickem22 Estimate: 1 hr

---

Ticket

Fixes:

Description

Detect both language AND script, and use them in generating the response.

Goal

Changes

1. In /chat, we used to first modify the query (paraphrase) to be optimized as a search query, then run the modified query through the guardrails+search pipeline. In some cases, the initial modification rendered Hinglish sentences as English (e.g. "Portal kahan hai" -> "Portal location"; Note that in Hinglish people would often use English words like "location"). So we do not use the modified search query any more. (TODO: remove that logic)
2. New IdentifiedScript enum and modified prompts
3. Modified test cases
4. Modified language detection guardrail test

Future Tasks (optional)

How has this been tested?

For guardrails, at project root

make setup-llm-proxy
python -m pytest core_backend/tests/rails/test_language_identification.py

For the question-answering endpoints, test various scenarios in the dev environment.

To-do before merge (optional)

Checklist

Fill with x for completed.

• [x] My code follows the style guidelines of this project
• [x] I have reviewed my own code to ensure good quality
• [x] I have tested the functionality of my code to ensure it works as intended
• [x] I have resolved merge conflicts

(Delete any items below that are not relevant)

• [x] I have updated the automated tests