is it possible to run one validation fold per a parameter set?

[vmorozov]

Hi,

is it possible to run one validation fold per a parameter set? Ideally I would like to use a custom constructed training-validation split . Though it is not the issue, because right now i get error with the split made by optunity.cross_validation.generate_folds: Modified bin/examples/python/skleran/svc.py code to use one fold: folds = optunity.cross_validation.generate_folds(data.shape[0], num_folds=3) outer_cv = optunity.cross_validated(x=data, y=labels, num_folds=1, folds=[folds[0:1]], aggregator=optunity.cross_validation.mean)

outer_cv = optunity.cross_validated(x=data, y=labels, num_folds=3)

compute area under ROC curve of default parameters

def compute_roc_standard(x_train, y_train, x_test, y_test): model = sklearn.svm.SVC().fit(x_train, y_train) decision_values = model.decision_function(x_test) auc = optunity.metrics.roc_auc(y_test, decision_values) return auc

decorate with cross-validation

compute_roc_standard = outer_cv(compute_roc_standard) roc_standard = compute_roc_standard()

Traceback (most recent call last): File "/opt/pycharm-community-4.5.4/helpers/pydev/pydevd_comm.py", line 1071, in doIt result = pydevd_vars.evaluateExpression(self.thread_id, self.frame_id, self.expression, self.doExec) File "/opt/pycharm-community-4.5.4/helpers/pydev/pydevd_vars.py", line 344, in evaluateExpression Exec(expression, updated_globals, frame.f_locals) File "/opt/pycharm-community-4.5.4/helpers/pydev/pydevd_exec.py", line 3, in Exec exec exp in global_vars, local_vars File "", line 11, in File "/home/vmorozov/PycharmProjects/optunity/optunity/cross_validation.py", line 485, in wrapper return cross_validated_callable(**args) File "/home/vmorozov/PycharmProjects/optunity/optunity/cross_validation.py", line 306, in init assert (len(folds[0]) == num_folds), 'Number of folds does not match num_folds.' AssertionError: Number of folds does not match num_folds.

Traceback (most recent call last): File "/opt/pycharm-community-4.5.4/helpers/pydev/pydevd_comm.py", line 1071, in doIt result = pydevd_vars.evaluateExpression(self.thread_id, self.frame_id, self.expression, self.doExec) File "/opt/pycharm-community-4.5.4/helpers/pydev/pydevd_vars.py", line 344, in evaluateExpression Exec(expression, updated_globals, frame.f_locals) File "/opt/pycharm-community-4.5.4/helpers/pydev/pydevd_exec.py", line 3, in Exec exec exp in global_vars, local_vars File "", line 12, in File "/home/vmorozov/PycharmProjects/optunity/optunity/cross_validation.py", line 403, in call scores.append(self.f(**kwargs)) File "", line 6, in compute_roc_standard File "/usr/local/lib/python2.7/dist-packages/sklearn/svm/base.py", line 150, in fit X = check_array(X, accept_sparse='csr', dtype=np.float64, order='C') File "/usr/local/lib/python2.7/dist-packages/sklearn/utils/validation.py", line 407, in check_array context)) ValueError: Found array with 0 sample(s) (shape=(0, 64)) while a minimum of 1 is required.

Traceback (most recent call last): File "/opt/pycharm-community-4.5.4/helpers/pydev/pydevd_comm.py", line 1071, in doIt result = pydevd_vars.evaluateExpression(self.thread_id, self.frame_id, self.expression, self.doExec) File "/opt/pycharm-community-4.5.4/helpers/pydev/pydevd_vars.py", line 344, in evaluateExpression Exec(expression, updated_globals, frame.f_locals) File "/opt/pycharm-community-4.5.4/helpers/pydev/pydevd_exec.py", line 3, in Exec exec exp in global_vars, local_vars File "", line 12, in File "/home/vmorozov/PycharmProjects/optunity/optunity/cross_validation.py", line 403, in call scores.append(self.f(**kwargs)) File "", line 6, in compute_roc_standard File "/usr/local/lib/python2.7/dist-packages/sklearn/svm/base.py", line 150, in fit X = check_array(X, accept_sparse='csr', dtype=np.float64, order='C') File "/usr/local/lib/python2.7/dist-packages/sklearn/utils/validation.py", line 407, in check_array context)) ValueError: Found array with 0 sample(s) (shape=(0, 64)) while a minimum of 1 is required.

Traceback (most recent call last): File "/opt/pycharm-community-4.5.4/helpers/pydev/pydevd_comm.py", line 1071, in doIt result = pydevd_vars.evaluateExpression(self.thread_id, self.frame_id, self.expression, self.doExec) File "/opt/pycharm-community-4.5.4/helpers/pydev/pydevd_vars.py", line 344, in evaluateExpression Exec(expression, updated_globals, frame.f_locals) File "/opt/pycharm-community-4.5.4/helpers/pydev/pydevd_exec.py", line 3, in Exec exec exp in global_vars, local_vars File "", line 13, in File "/home/vmorozov/PycharmProjects/optunity/optunity/cross_validation.py", line 403, in call scores.append(self.f(**kwargs)) File "", line 7, in compute_roc_standard File "/usr/local/lib/python2.7/dist-packages/sklearn/svm/base.py", line 150, in fit X = check_array(X, accept_sparse='csr', dtype=np.float64, order='C') File "/usr/local/lib/python2.7/dist-packages/sklearn/utils/validation.py", line 407, in check_array context)) ValueError: Found array with 0 sample(s) (shape=(0, 64)) while a minimum of 1 is required.

[vmorozov]

Cleaner code:

folds = optunity.cross_validation.generate_folds(data.shape[0], num_folds=3)

use one fold:

outer_cv = optunity.cross_validated(x=data, y=labels, num_folds=1, folds=[folds[0:1]], aggregator=optunity.cross_validation.mean)

compute area under ROC curve of default parameters

def compute_roc_standard(x_train, y_train, x_test, y_test): model = sklearn.svm.SVC().fit(x_train, y_train) decision_values = model.decision_function(x_test) auc = optunity.metrics.roc_auc(y_test, decision_values) return auc

decorate with cross-validation

compute_roc_standard = outer_cv(compute_roc_standard) roc_standard = compute_roc_standard()

[claesenm]

Can you provide some more information on why you want to do this? I think there is an XY problem here ...

[vmorozov]

I run a computationally intensive learning (deep neural network). I would like sample biger parameter space in expense of replication (multiple folds) of each parameter space point