[Question] How to change custom mode name

[landhu]

[ ] I checked the documentation and related resources and couldn't find an answer to my question.

** How do I specify the name of the model** An LLM is deployed locally. All interfaces are the same as openai, but the mode name is different

Code Examples Now I can do this in the following way to change key and url os.environ['OPENAI_API_KEY'] ="xxxx" os.environ['OPENAI_API_BASE'] = "xxxx" But after that, when I run simple code: data_samples = { 'question': [ 'When did Einstein born?', 'Where did Einstein born?', ], 'answer': [ 'Einstein was born in 1879.', 'Einstein was born in Germany.', #'Einstein was born in 1879 in Germany.' ], 'ground_truth': [ 'Einstein was born in 1879 in Germany.', 'Einstein was born in 1879 in Germany.', ] } dataset = Dataset.from_dict(data_samples) score = evaluate(dataset, metrics=[answer_correctness]) print( score.to_pandas() )

it's shows: openai.NotFoundError: Error code: 404 - {'error': {'message': "model 'gpt-3.5-turbo' not found, try pulling it first", 'type': 'api_error', 'param': None, 'code': None}}

[dosubot[bot]]

Hey @landhu! I'm here to help you out with any bugs, questions, or contributions you might have. Let's tackle this together!

To specify the name of the model and avoid the "model 'gpt-3.5-turbo' not found" error when using a locally deployed LLM with the OpenAI interface, you can modify the model parameter in the llm_factory function. Here is the relevant code snippet:

def llm_factory(
    model: str = "gpt-3.5-turbo", run_config: t.Optional[RunConfig] = None
) -> BaseRagasLLM:
    timeout = None
    if run_config is not None:
        timeout = run_config.timeout
    openai_model = ChatOpenAI(model=model, timeout=timeout)
    return LangchainLLMWrapper(openai_model, run_config)

To specify a different model name, simply pass the desired model name as the model argument when calling llm_factory. For example:

llm = llm_factory(model="your-local-model-name")

This will configure the ChatOpenAI instance to use the specified model name.

Additionally, if you are using Langchain, you can pass the Langchain LLM instance directly and Ragas will wrap it with LangchainLLMWrapper. For example:

from langchain_together import Together

together_key = "<your-key-here>"

together_completion = Together(
    model="NousResearch/Nous-Hermes-2-Mixtral-8x7B-SFT",
    temperature=0.7,
    max_tokens=4000,
    top_k=1,
    together_api_key=together_key
)

llm = llm_factory(model=together_completion)

This approach ensures that your custom LLM is properly integrated and avoids the "model not found" error ^[1][2].

To continue talking to Dosu, mention @dosu.

[landhu]

Thanks Very much. If I just want evaluate like simple demo: ` from datasets import Dataset import os from ragas import evaluate from ragas.metrics import faithfulness, answer_correctness

os.environ["OPENAI_API_KEY"] = "your-openai-key"

data_samples = { 'question': ['When was the first super bowl?', 'Who won the most super bowls?'], 'answer': ['The first superbowl was held on Jan 15, 1967', 'The most super bowls have been won by The New England Patriots'], 'contexts' : [['The First AFL–NFL World Championship Game was an American football game played on January 15, 1967, at the Los Angeles Memorial Coliseum in Los Angeles,'], ['The Green Bay Packers...Green Bay, Wisconsin.','The Packers compete...Football Conference']], 'ground_truth': ['The first superbowl was held on January 15, 1967', 'The New England Patriots have won the Super Bowl a record six times'] }

dataset = Dataset.from_dict(data_samples)

score = evaluate(dataset,metrics=[faithfulness,answer_correctness]) score.to_pandas()`

How Can I pass the mode parameter?

[dosubot[bot]]

To pass the model parameter for evaluation in your example, you can follow these steps:

1. Host the custom model using VLLM (if not already done):

   # Run this on your terminal with GPU enabled
   python -m vllm.entrypoints.openai.api_server --model your-local-model-name
   

2. Set up the ChatOpenAI instance with the custom model:

   from ragas import evaluate
   from ragas.metrics import faithfulness, answer_correctness
   from ragas.llms import LangchainLLMWrapper
   from langchain.chat_models import ChatOpenAI
   import os
   
   os.environ["OPENAI_API_KEY"] = "your-openai-key"
   
   inference_server_url = "http://localhost:8000/v1"
   MODEL = "your-local-model-name"
   chat = ChatOpenAI(
       model=MODEL,
       openai_api_key=os.getenv("OPENAI_API_KEY"),
       openai_api_base=inference_server_url,
       max_tokens=2048,
       temperature=0,
   )
   
   langchain_llm = LangchainLLMWrapper(chat)
   
   data_samples = {
       'question': ['When was the first super bowl?', 'Who won the most super bowls?'],
       'answer': ['The first superbowl was held on Jan 15, 1967', 'The most super bowls have been won by The New England Patriots'],
       'contexts' : [['The First AFL–NFL World Championship Game was an American football game played on January 15, 1967, at the Los Angeles Memorial Coliseum in Los Angeles,'], 
       ['The Green Bay Packers...Green Bay, Wisconsin.','The Packers compete...Football Conference']],
       'ground_truth': ['The first superbowl was held on January 15, 1967', 'The New England Patriots have won the Super Bowl a record six times']
   }
   
   dataset = Dataset.from_dict(data_samples)
   
   score = evaluate(dataset, metrics=[faithfulness, answer_correctness], llm=langchain_llm)
   print(score.to_pandas())
   

This code sets up the custom model to be used for evaluation instead of the default model ^[1][2].

To continue talking to Dosu, mention @dosu.

[jjmachan]

@landhu where you able to fix this?

[anistark]

Closing this as resolved in newer versions.

Checkout llm_factory