Frameworks for HP optimization

[azev77]

Julia HP optimization packages:

• [ ] Hyperopt.jl @baggepinnen (Random search, Latin hypercube sampling, Bayesian opt)
• [ ] TreeParzen.jl (port of Hyperopt.py to Julia) @IQVIA-ML @iqml
• [ ] NaiveGAflux.jl (helps automate Flux) @DrChainsaw

Other HP optimization packages:

• [ ] Hyperopt.py (Hyperopt-sklearn.py)
• [ ] Optuna.py
• [ ] mlaut.py (Benchmark paper)
• [ ] Ax.py
• [ ] TPOT.py
• [ ] AutoKeras.py
• [ ] AutoML org: SMAC3.py, AutoSKLearn.py, AutoPyTorch.py, AutoWeka.java (unmaintained)
• [ ] H2O (Python & R)
• [ ] mlrMBO.r: AutoXGBoost.r (uses mlrMBO) (paper), tuneRanger.r (uses mlrMBO)
• [ ] liquidSVM (R, Python, MATLAB / Octave, Java, Spark. Paper.)
• [ ] Gama.py
• [ ] oboe.py
• [ ] Spearmint.py Bayesian optimization (unmaintained)
• [ ] Google Vizier (unmaintained)
• [ ] Katib
• [ ] GPyOpt.py
• [ ] Autotune (SAS)
• [ ] Tune

There are projects that benchmark different AutoML systems: https://openml.github.io/automlbenchmark/ From our conversation: https://github.com/alan-turing-institute/MLJ.jl/issues/416#issuecomment-640823116 I wanted to tell you guys about Optuna (repo & paper) a new framework for HP optimization. A nice comparison w/ Hyperopt shows what can be done for HP visualization: https://neptune.ai/blog/optuna-vs-hyperopt

Here are a few snips:

A 3 minute clip: https://www.youtube.com/watch?v=-UeC4MR3PHM

It would really be amazing for MLJ to incorporate this!

[ghost]

Hyperopt.py (also Hyperopt.jl)

To clarify, Hyperopt.jl is not related to the Python hyperopt. It uses different optimisation techniques (random search, latin hypercube sampling and Bayesian optimization) and deserves its own position in the list.

However TreeParzen.jl is a direct port of the Python hyperopt to Julia, uses the same optimisation technique (tree-parzen estimators), and has the same behaviour.

[vollmersj]

Maybe it is worth considering bandit frame works Ax

[azev77]

Thanks. @vollmersj, added. Please let me know if you have other suggestions

[casasgomezuribarri]

Hi, I'm really excited to see a Bayesian optimization method for hyperparameter tuning! I note that RandomSerach() and LatinHypercube() are already possible choices for the tuning = kwarg of TunedModel(), and I see them grouped together with Bayesian opt in the original post of this issue. Is it already possible to implement this method for hyperparameter tuning/is there any notion of whether it will be out soon/how soon? :-)

Thanks a lot for basically everything so far!