Add Watson X Integration with Granite Models

[PromtEngineer]

Summary

This PR adds support for IBM Watson X AI with Granite models as an alternative LLM backend to Ollama. Users can now choose between running models locally with Ollama or using IBM's hosted Granite models via Watson X.

Changes Made

New Files

• rag_system/utils/watsonx_client.py: New Watson X client class that provides an Ollama-compatible interface for seamless integration
• WATSONX_README.md: Comprehensive documentation for Watson X setup and usage
• env.example.watsonx: Example environment configuration file

Modified Files

• rag_system/main.py:

  • Added LLM_BACKEND configuration variable
  • Added WATSONX_CONFIG dictionary with Watson X settings
  • Updated get_agent() to support backend switching

• rag_system/factory.py:

  • Updated get_agent() and get_indexing_pipeline() to initialize appropriate client based on LLM_BACKEND
  • Added Watson X client initialization logic

• rag_system/requirements.txt:

  • Added ibm-watsonx-ai>=1.3.39 dependency

Key Features

✅ Backend Switching: Simple environment variable (LLM_BACKEND=watsonx) to switch between Ollama and Watson X
✅ Ollama-Compatible Interface: WatsonXClient provides the same API as OllamaClient
✅ Granite Model Support: Works with all IBM Granite models (chat, instruct, code, multilingual)
✅ Full Feature Support: Text generation, async generation, streaming, embeddings
✅ Comprehensive Documentation: Step-by-step setup guide and troubleshooting
✅ Zero Breaking Changes: Existing Ollama functionality remains unchanged

Configuration

Users can enable Watson X by setting environment variables:

LLM_BACKEND=watsonx
WATSONX_API_KEY=your_api_key
WATSONX_PROJECT_ID=your_project_id
WATSONX_GENERATION_MODEL=ibm/granite-13b-chat-v2

See WATSONX_README.md for detailed setup instructions.

Supported Granite Models

• ibm/granite-13b-chat-v2 - General purpose chat model
• ibm/granite-13b-instruct-v2 - Instruction-following model
• ibm/granite-20b-multilingual - Multilingual support
• ibm/granite-8b-japanese - Lightweight model
• ibm/granite-3b-code-instruct - Code generation

Testing

The implementation has been designed to be drop-in compatible with the existing Ollama client interface, requiring no changes to the agent, pipelines, or other components.

Benefits

1. Cloud Flexibility: Users can leverage Watson X's cloud infrastructure without maintaining local models
2. Enterprise Ready: Watson X provides enterprise-grade security and compliance
3. Cost Options: Choose between free local execution (Ollama) or cloud pricing (Watson X)
4. Model Variety: Access to IBM's Granite model family optimized for various tasks

Backward Compatibility

✅ All existing Ollama functionality is preserved
✅ Default backend remains Ollama
✅ No changes required for current users
✅ Watson X is opt-in via configuration

---

Link to Devin run: https://app.devin.ai/sessions/047e10665d1d44268b5b3325d5dc241c

Requested by: Prompt Engineering ([email protected]) / @PromtEngineer

cc: @PromtEngineer

[devin-ai-integration[bot]]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• [ ] Disable automatic comment and CI monitoring