Binary Classification Can Not Use F1 Metric

[spot92]

Metric f1 is not allowed in ML task: binary_classification. Use 'logloss'

[pplonski]

@spot92 thank you for reporting. Could you please show the code used? This looks quite strange because f1 metric should be supported. What is more, there is even support for custom evaluation metrics. Do you used latest version 0.10.6?

[spot92]

I'm on the most recent version.

automl = AutoML(mode = "Perform", eval_metric = 'f1')

[pplonski]

@spot92 that's might be some bug. I just run very simple toy example:

import numpy as np
import pandas as pd
from supervised import AutoML

# generate toy data
X = np.random.rand(100,10)
y = np.random.rand(100) * 2
y = y.astype(int)
print(y)
# output
# [0 1 1 1 0 0 0 0 0 1 0 1 0 0 1 1 1 0 1 1 1 1 0 1 1 1 0 1 0 1 1 0 0 1 0 0 0
# 1 0 1 0 0 1 1 0 1 0 1 0 0 1 0 0 1 1 1 1 0 1 0 0 1 1 0 1 0 1 1 0 1 1 1 1 1
# 1 1 1 0 0 0 1 1 1 0 1 0 0 1 1 0 0 0 1 0 1 0 1 1 1 1]

# train automl
automl = AutoML(mode='Perform', eval_metric='f1')
automl.fit(X, y)

Output from AutoML training:

AutoML directory: AutoML_1
The task is binary_classification with evaluation metric f1
AutoML will use algorithms: ['Linear', 'Random Forest', 'LightGBM', 'Xgboost', 'CatBoost', 'Neural Network']
AutoML will ensemble availabe models
AutoML steps: ['simple_algorithms', 'default_algorithms', 'not_so_random', 'golden_features', 'insert_random_feature', 'features_selection', 'hill_climbing_1', 'hill_climbing_2', 'ensemble']
* Step simple_algorithms will try to check up to 1 model
1_Linear f1 0.566667 trained in 10.25 seconds (1-sample predict time 0.0298 seconds)
* Step default_algorithms will try to check up to 5 models
2_Default_LightGBM f1 0.731034 trained in 6.33 seconds (1-sample predict time 0.0189 seconds)
... rest of the output ....

It works on toy example, maybe you can share data and the code, so I can try to reproduce ?