bayesmark icon indicating copy to clipboard operation
bayesmark copied to clipboard

Explanation to the 'visible_to_opt' and 'generalization'

Open Alaya-in-Matrix opened this issue 5 years ago • 2 comments

I found no explanation of the visible_to_opt and generalization in the documentation, are they some kind of linear transformation to the normalized mean score?

Alaya-in-Matrix avatar Jul 30 '20 07:07 Alaya-in-Matrix

The visible_to_opt is the score that the optimizer sees (e.g, CV error in a ML hyper-parameter tuning context). Whereas generalization is meant to be a related metric the optimizer does not get to see (e.g, error on held-out test set in a ML hyper-parameter tuning context).

Does that make sense? If so, a note can be added to the docs.

rdturnermtl avatar Jul 30 '20 22:07 rdturnermtl

Thanks for your explanation! I think would be nice if they are also documented

Alaya-in-Matrix avatar Aug 01 '20 03:08 Alaya-in-Matrix