Building an MMLU Eval issue

[ToadTWP666]

I am trying to execute the Building an MMLU Eval jupyter notebooks all of the cells execute correctly until I execute the following code:

!oaieval gpt-3.5-turbo match_mmlu_anatomy

I receive the following error:

Traceback (most recent call last): File "/anaconda/envs/azureml_py38/bin/oaieval", line 5, in from evals.cli.oaieval import main File "/anaconda/envs/azureml_py38/lib/python3.8/site-packages/evals/init.py", line 1, in from .api import check_sampled_text, completion_query, sample_freeform File "/anaconda/envs/azureml_py38/lib/python3.8/site-packages/evals/api.py", line 9, in from evals.base import ModelSpec File "/anaconda/envs/azureml_py38/lib/python3.8/site-packages/evals/base.py", line 93, in class ModelSpecs: File "/anaconda/envs/azureml_py38/lib/python3.8/site-packages/evals/base.py", line 125, in ModelSpecs def names(self) -> dict[str, Sequence[str]]: TypeError: 'type' object is not subscriptable

[ToadTWP666]

I tried executing using python 3.10 and still had the issue.

[andremafei]

I got several of similar error while trying to run it in Python 3.8, I quit trying to fix it because several other places were also not compatible with 3.8. There is a bug in the file "pyproject.toml" where it says: requires-python = ">=3.8" it should be: requires-python = ">=3.9"

[andrew-openai]

It's a bit hard to diagnose the issue - When you switch between python versions, did you ensure to reinstall the CLI - oaieval is pointing to the right path?

[etr2460]

Closing as stale, please file a new issue if you're still seeing this