lighteval icon indicating copy to clipboard operation
lighteval copied to clipboard
verified publisher icon huggingface

Improved stability of litellm models for reasoning models.

Open JoelNiklaus opened this issue 1 year ago • 3 comments

JoelNiklaus avatar Feb 05 '25 14:02 JoelNiklaus

Hey @JoelNiklaus, since you have made most commits around LiteLLM. I have few questions/requests

  1. Why not expose entire LiteLLM config as parameter or something?
  2. I'm actually looking to run open-r1 evals using for instance models deployed at deepinfra with temp=0. Is it possible right now? image

satpalsr avatar Feb 07 '25 11:02 satpalsr

Hi @satpalsr,

  1. When I integrated litellm, I made it very similar to the existing openai_model. But I agree with you that we may want to configure more. I am sure the maintainers are open if you want to open a PR for that :)
  2. I am not familiar with deepinfra, but since it seems supported by litellm it should work out of the box.

JoelNiklaus avatar Feb 07 '25 14:02 JoelNiklaus

@JoelNiklaus Thanks. I'll just drop it as separate issue for anyone to pick. For time being, I just modified the OpenAIClient code & got my evals done.

satpalsr avatar Feb 07 '25 15:02 satpalsr