Transformers model as Judge

[anilaltuner]

#153

Transformers library added to llm as judge.

Only need is when the JudgeLLM class called, change judge model with transformers.

Ex.

JudgeLLM(
            judge_model_name="microsoft/Phi-3-mini-128k-instruct",
            template_path="src/lighteval/tasks/extended/mt_bench/judge_prompts.jsonl",
            multi_turn=True,
        )

[clefourrier]

Hi! Thanks for this PR! Can you fix your PR so that tests are passing?

[anilaltuner]

Yes, I'll fix on a short time but Run tests gives error for

ERROR tests/test_main.py - huggingface_hub.utils._errors.HfHubHTTPError: 500 Server Error: Internal Server Error for url: https://huggingface.co/api/datasets/gsm8k/paths-info/e53f048856ff4f594e959d75785d2c2d37b678ee (Request ID: Root=1-6630cc27-222f10cf5f5b028e1ffcebcc;677b1d40-c6e5-4fc5-9718-6441f20a365c)

Is it about huggingface hub?

[clefourrier]

Hm, let me re-run your tests, maybe you committed when the hub was down

[anilaltuner]

Thanks, I fixed code quality and pushed. We can re-run whenever you want

[clefourrier]

cc @NathanHB if you have the time to do a more in depth review

[NathanHB]

hey ! thanks for the fix, I will have the bandwitdh to test next week and will merge asap :)

[NathanHB]

Hi ! Closing for inactivity. But this PR was used as the base for #223 thanks !