setfit Running Evaluation

Hi, Thanks for sharing this work. I am wondering if it is possible to run evaluation dataset to tune hyperparameters. The SetFitTrainer doesn't seem to accept arguments like 'evaluation_strategy', 'save_strategy', 'compute_metrics', etc. Or perhaps Im doing something wrong? Thanks.

Oct 06 '22 05:10 dhkhey

Hey @dhkhey, thanks for your interest!

The SetFitTrainer is indeed far less feature-complete than the one in transformers. In particular, we don't have an integration to tune hyperparameters with optuna or similar. Is there a type of evaluation that you're having trouble with?

Given how fast one can train SetFit models in the fewshot regime, we found it was quite quick to explore various hyperparameters . Having said that, if you would like to contribute a PR with hyperparameter search, we'd certainly welcome it!

Oct 06 '22 12:10 lewtun

Hi @lewtun I'll be more than happy to add hyperparameter search in a PR.

Oct 06 '22 13:10 Mouhanedg56

Amazing! I loosely modelled SetFitTrainer on transformers.Trainer, so having a similar API could make sense.

I also realised that we'll probably need a model_init arg in SetFitTrainer to be able to re-initialise the SetFitModel with each run

Oct 06 '22 13:10 lewtun

Yeah, I noticed the huge difference between setfit.Trainer and transformers.Trainer. As you mentioned, we need to introduce model_init parameter in args before adding hyperparameter search. Can I lend a hand on this?

Oct 06 '22 14:10 Mouhanedg56

@Mouhanedg56 yes, feel free to start with model_init :)

Oct 06 '22 14:10 lewtun

Thanks @lewtun for approving #87 . Now with model_init supported, we can go forward and add a new PR for hyperparameter search.

Oct 07 '22 09:10 Mouhanedg56