Qingquan Song

> When I run the demo you present, the relative errors of SiLRTC and HaLRTC are consistently 1. > > I am not seeing any errors in the code or...

Hey @YannDubs thank you so much for the prompt response! We're using some in-house supported openAI API call to call GPT-4 as judgement (which is roughly the same as the...

> This is very surprising indeed. Just to understand, why are you not using the default alpaca_eval 2? > > > > i.e. `alpaca_eval evaluate_from_model --model_configs 'mistral-7b-orpo'` > > >...

Is there also a way to not using the logprob api with `alpaca_eval_clf_gpt4_turbo` if we cannot access the logprob such as still using the `alpaca_eval_gpt4` annotator? Thank you! Best regards,...

Hey @YannDubs , it's unfortunately that I cannot adopt the API to access the logprob in our case, and seems like `alpaca_eval evaluate_from_model --model_configs 'mistral-7b-orpo' --annotators_config 'alpaca_eval_clf_gpt4_turbo'` also need to...

Hey @YannDubs wanna reopen this issue. When I'm using `alpaca_eval evaluate_from_model --model_configs 'mistral-7b-orpo' --annotators_config 'alpaca_eval_gpt4_turbo_fn' as suggested. The results is still high ~31-36 length_controlled win rate and ~50 win rate...

Hey @YannDubs Thank you very much for the response! I'm pretty sure we're using the OpenAI API (though is in-house version from Microsoft) and we tried both GPT-4 and GPT-4...

Faced same issue with `0.13.0.dev2024090300` and also have other two issues: 1) master branch example `convert_checkpoint.py` code when doing fp8 quantization (either `use_fp8` or use `use_fpt_rowwise`) has issue as well...

For my case, it seems like the issue happens at here: ``` @property def paged_kv_cache(self): return self._model_config.kv_cache_type == KVCacheType.PAGED ``` the `self._model_config.kv_cache_type` is a string `PAGED` (or others) but the...

@troy1729 Sounds reasonable to me. Assigned and feel free to kick off the implementation and ping us to discuss or review on any issues. Thank you!