Andrew comments

Results 34 comments of


                                            Andrew

Idea for Evals: Sorting numbers with repeats and negatives

Hey, thanks for the discussion! I agree this is a good eval idea, and I agree with [qrdlgit](https://github.com/qrdlgit) that it seems to be quite representative of common tasks. Also, thanks...

Are not merged PRs the result of irrelevancy to the model?

Hi folks, sorry for the pace of PR reviews, I actually took some time off this week which is why there haven't been many reviews in the past few days....

Chess legal setup

Closing for now, feel free to reopen if you address comments.

Evaluate GPT-4 on classical NLP tasks

Great idea!

Q-Learning Eval

Closing for now, will review #356

Logical reasoning eval | Accuracy 0%

Please make sure your eval passes CI - I think there is a misconfigured path in your YAML

🔀 Relocate issue templates to proper directory

> @andrew-openai Could you have a look at this housekeeping PR? Sorry this got buried, approving!

Dataset hosting, data cards and previews

Thanks for stopping by! HuggingFace datasets is great. Many of our evals are only a few samples long (10-20), which we were worried to be too small to host as...

:sparkles: Added Eval For Sam Altman Degree (he has an honorary degree from University of Waterloo)

Thanks for the submission! Unfortunately we can't merge single line evals, although I definitely learned something new about Sam :)

Building an MMLU Eval issue

It's a bit hard to diagnose the issue - When you switch between python versions, did you ensure to reinstall the CLI - `oaieval` is pointing to the right path?