dask-ml Drop in replacement for sklearn.model_selection.permutation_test

It'd be great to have a dask wrapper for permutation_test_score so that permutations of gridsearches are easier to run in dask.

Mar 18 '22 13:03 Shotgunosine

Hi Dask-ml folks, just bumping this up. Is there anyone with interest/expertise who could look into this?

Jul 25 '22 15:07 Shotgunosine

I think it'd be possible to use Dask-ML's joblib backend because the implementation of permutation_test_score relies on joblib: github.com/scikit-learn/scikit-learn/.../sklearn/model_selection/_validation.py#L1308

Jul 27 '22 20:07 stsievert

I was hoping to run the permutations across a cluster that uses slurm.

Jul 28 '22 14:07 Shotgunosine

From the docs of Dask-ML's joblib backend:

Dask can scale these Joblib-backed algorithms out to a cluster of machines by providing an alternative Joblib backend. The following video demonstrates how to use Dask to parallelize a grid search across a cluster.

The video indicates that joblib will use Dask-ML's computational resources instead of the local machines computational resources, which should work near seamlessly with the SLURM backend. Are you observing something different?

Jul 29 '22 01:07 stsievert

Drop in replacement for sklearn.model_selection.permutation_test_score