lorenzo famiglini

An example of how we can compute ECE (for binary settings, but it can be easly extended to the multiclass ones) with automatic binning is the following: ``` import numpy...

While for Brier Score in general settings, both binary and multiclass (this implementation is in PyTorch, but it can be replaced with Numpy): ``` def brierscore(input, target): target = target.view(-1,1)...

Happy to hear that! Yes, my suggestion is to evaluate calibration by exploiting also statistical tests such as https://en.m.wikipedia.org/wiki/Hosmer%E2%80%93Lemeshow_test. Brier Score is not sensitive to the binning scheme (indeed, in...