singa icon indicating copy to clipboard operation
singa copied to clipboard

Update the metric module

Open nudles opened this issue 5 years ago • 4 comments

It would be better to implement a metric as a function in the metric.py as metric typically has not states. Therefore, no need to make it as a class. e.g,

def accuracy(y_pred, y_true):
   """compute the accuracy.

     Args:
        y_pred(numpy array or tensor): each value is a label index
        y_true(numpy array or tensor): each value is a label index
   """
      check shape match
      convert y_pred and y_true to np array     
      return np.sum(y_pred== y_true) / y_true.shape[0]

Refer to https://keras.io/api/metrics/

nudles avatar May 17 '20 10:05 nudles

Or we can convert SINGA Tensor to numpy array and use sklearn.metrics to compute the metrics https://scikit-learn.org/stable/modules/model_evaluation.html.

nudles avatar May 19 '20 14:05 nudles

When we use this accuracy: y_true (each value is a label index) is provided directly by the dataset y_pred (each value is a label index) needs to be obtained by applying max function to the network output (i.e. logit) BEFORE the softmaxCrossEntropy

so we need a max function to turn the logits into pred label index?

chrishkchris avatar May 19 '20 16:05 chrishkchris

I suggest to use sklearn's functions to evaluate the metrics. We can them remove the metric.py module.

nudles avatar Jun 18 '20 02:06 nudles

sklearn.metrics.accuracy_score API: https://scikit-learn.org/stable/modules/generated/sklearn.metrics.accuracy_score.html#sklearn.metrics.accuracy_score

chrishkchris avatar Jul 18 '20 16:07 chrishkchris