evals icon indicating copy to clipboard operation
evals copied to clipboard
verified publisher icon openai

Is there a list of tasks that chatgpt fails?

Open irthomasthomas opened this issue 2 years ago • 2 comments

Hi, is it possible to see a collection of questions that gpt-4 is failing? I wan't to test some prompts I wrote that improve accuracy, and I thought it would be good to focus on tasks that it is currently failing. Thanks!

irthomasthomas avatar May 10 '23 15:05 irthomasthomas

try the logic evals https://github.com/openai/evals/tree/main/evals/registry/data/logic

They fail even with cot of reasoning

qrdlgit avatar May 16 '23 13:05 qrdlgit