Cache results from `Evaluator`

[lewtun]

When running an evaluation with the Evaluator class, it would be great to cache the results (e.g. store them as an Arrow dataset) so that one doesn't have to wait to recompute everything each time.

This would also give evaluate a similar developer experience to datasets and allow users to leverage the intuitions from one lib to the other :)

[ola13]

Can you share code pointers for caching in datasets?

[lewtun]

Can you share code pointers for caching in datasets?

Sure! You can check out the _map_single() method for Dataset objects: https://github.com/huggingface/datasets/blob/747f0c8612d4929dbdd1c72cca201815911ce660/src/datasets/arrow_dataset.py#L2561

Basically, it checks if the Dataset object has a cache_files object, and then returns a Dataset from file instead of re-computing the map function

[ola13]

Cool I'd be happy to take a look in the next week or two