Seems that setting logprobs=100 is not useful now.

Open KL4805 opened this issue 2 years ago • 1 comments

Hello authors,

I am really impressed with your efforts in creating this benchmark!

One small thing I notice is that OpenAI seems to limit the 'logprobs' argument to at most 5 (https://platform.openai.com/docs/api-reference/completions/create), while you set to 100. In this case, will your results be affected?

Sep 17 '23 00:09 KL4805

Wondering the same thing here. I'm guessing should have at least some effect.

Sep 19 '23 10:09 don-tpanic