tune
tune copied to clipboard
tidymodels
Output results even if computation of some metrics fails
When some of the metrics can't be computed –as in the example below with SVM which can't compute roc_auc–, fitting (e.g. last_fit) and tuning (e.g. tune_grid) methods say that the whole model fails and output NULL. However, predictions and other metrics could be computed anyway.
Should it be possible to set only the failing metric to NA (or something else) and return the other valid outputs?
This is interesting when fitting/tuning workflow sets with workflow_map when metrics are set for all the models at the same time.
Thanks for your consideration, and thanks for the tidymodels framework!
library(tidymodels)
#> Registered S3 method overwritten by 'tune':
#> method from
#> required_pkgs.model_spec parsnip
#> Warning: le package 'broom' a été compilé avec la version R 4.1.2
#> Warning: le package 'tidyr' a été compilé avec la version R 4.1.2
data(ad_data, package = "modeldata")
set.seed(42)
split <- initial_split(ad_data, strata = Class)
train <- training(split)
rec <-
recipe(Class ~ ., data = train) %>%
step_dummy(all_nominal_predictors()) %>%
step_zv(all_nominal_predictors()) %>%
step_normalize(all_nominal_predictors())
spec_svm <- svm_linear(mode = "classification")
wkf_svm <- workflow(rec, spec_svm)
last_fit(wkf_svm, split, metrics = metric_set(accuracy))
#> Warning: le package 'rlang' a été compilé avec la version R 4.1.2
#> # Resampling results
#> # Manual resampling
#> # A tibble: 1 × 6
#> splits id .metrics .notes .predictions .workflow
#> <list> <chr> <list> <list> <list> <list>
#> 1 <split [249/84]> train/test split <tibble> <tibble> <tibble> <workflow>
last_fit(wkf_svm, split, metrics = metric_set(roc_auc, accuracy))
#> x train/test split: preprocessor 1/1, model 1/1 (predictions): Error in `check_spec_pred_typ...
#> Warning: All models failed. See the `.notes` column.
#> Warning: This tuning result has notes. Example notes on model fitting include:
#> preprocessor 1/1, model 1/1 (predictions): Error in `check_spec_pred_type()`:
#> ! No prob prediction method available for this model.
#> • Value for `type` should be one of: 'class', 'raw'
#> # Resampling results
#> # Manual resampling
#> # A tibble: 1 × 6
#> splits id .metrics .notes .predictions .workflow
#> <list> <chr> <list> <list> <list> <list>
#> 1 <split [249/84]> train/test split <NULL> <tibble> <NULL> <workflow>
Created on 2022-02-27 by the reprex package (v2.0.1)
Session info
sessioninfo::session_info()
#> ─ Session info ───────────────────────────────────────────────────────────────
#> setting value
#> version R version 4.1.1 (2021-08-10)
#> os macOS Big Sur 10.16
#> system x86_64, darwin17.0
#> ui X11
#> language (EN)
#> collate fr_FR.UTF-8
#> ctype fr_FR.UTF-8
#> tz Europe/Paris
#> date 2022-02-27
#> pandoc 2.14.0.3 @ /Applications/RStudio.app/Contents/MacOS/pandoc/ (via rmarkdown)
#>
#> ─ Packages ───────────────────────────────────────────────────────────────────
#> package * version date (UTC) lib source
#> assertthat 0.2.1 2019-03-21 [1] CRAN (R 4.1.0)
#> backports 1.4.1 2021-12-13 [1] CRAN (R 4.1.0)
#> broom * 0.7.12 2022-01-28 [1] CRAN (R 4.1.2)
#> class 7.3-20 2022-01-13 [1] CRAN (R 4.1.2)
#> cli 3.1.1 2022-01-20 [1] CRAN (R 4.1.2)
#> codetools 0.2-18 2020-11-04 [1] CRAN (R 4.1.1)
#> colorspace 2.0-2 2021-06-24 [1] CRAN (R 4.1.0)
#> crayon 1.4.2 2021-10-29 [1] CRAN (R 4.1.0)
#> DBI 1.1.2 2021-12-20 [1] CRAN (R 4.1.0)
#> dials * 0.1.0 2022-01-31 [1] CRAN (R 4.1.1)
#> DiceDesign 1.9 2021-02-13 [1] CRAN (R 4.1.0)
#> digest 0.6.29 2021-12-01 [1] CRAN (R 4.1.0)
#> dplyr * 1.0.8 2022-02-08 [1] CRAN (R 4.1.1)
#> ellipsis 0.3.2 2021-04-29 [1] CRAN (R 4.1.0)
#> evaluate 0.14 2019-05-28 [1] CRAN (R 4.1.0)
#> fansi 1.0.2 2022-01-14 [1] CRAN (R 4.1.2)
#> fastmap 1.1.0 2021-01-25 [1] CRAN (R 4.1.0)
#> foreach 1.5.2 2022-02-02 [1] CRAN (R 4.1.2)
#> fs 1.5.2 2021-12-08 [1] CRAN (R 4.1.0)
#> furrr 0.2.3 2021-06-25 [1] CRAN (R 4.1.0)
#> future 1.23.0 2021-10-31 [1] CRAN (R 4.1.0)
#> future.apply 1.8.1 2021-08-10 [1] CRAN (R 4.1.0)
#> generics 0.1.2 2022-01-31 [1] CRAN (R 4.1.1)
#> ggplot2 * 3.3.5 2021-06-25 [1] CRAN (R 4.1.0)
#> globals 0.14.0 2020-11-22 [1] CRAN (R 4.1.0)
#> glue 1.6.1 2022-01-22 [1] CRAN (R 4.1.2)
#> gower 1.0.0 2022-02-03 [1] CRAN (R 4.1.2)
#> GPfit 1.0-8 2019-02-08 [1] CRAN (R 4.1.0)
#> gtable 0.3.0 2019-03-25 [1] CRAN (R 4.1.0)
#> hardhat 0.2.0 2022-01-24 [1] CRAN (R 4.1.2)
#> highr 0.9 2021-04-16 [1] CRAN (R 4.1.0)
#> htmltools 0.5.2 2021-08-25 [1] CRAN (R 4.1.0)
#> infer * 1.0.0 2021-08-13 [1] CRAN (R 4.1.0)
#> ipred 0.9-12 2021-09-15 [1] CRAN (R 4.1.0)
#> iterators 1.0.14 2022-02-05 [1] CRAN (R 4.1.2)
#> knitr 1.37 2021-12-16 [1] CRAN (R 4.1.0)
#> lattice 0.20-45 2021-09-22 [1] CRAN (R 4.1.0)
#> lava 1.6.10 2021-09-02 [1] CRAN (R 4.1.0)
#> lhs 1.1.3 2021-09-08 [1] CRAN (R 4.1.0)
#> LiblineaR * 2.10-12 2021-03-02 [1] CRAN (R 4.1.0)
#> lifecycle 1.0.1 2021-09-24 [1] CRAN (R 4.1.0)
#> listenv 0.8.0 2019-12-05 [1] CRAN (R 4.1.0)
#> lubridate 1.8.0 2021-10-07 [1] CRAN (R 4.1.0)
#> magrittr 2.0.2 2022-01-26 [1] CRAN (R 4.1.2)
#> MASS 7.3-55 2022-01-13 [1] CRAN (R 4.1.2)
#> Matrix 1.4-0 2021-12-08 [1] CRAN (R 4.1.0)
#> modeldata * 0.1.1 2021-07-14 [1] CRAN (R 4.1.0)
#> munsell 0.5.0 2018-06-12 [1] CRAN (R 4.1.0)
#> nnet 7.3-17 2022-01-13 [1] CRAN (R 4.1.2)
#> parallelly 1.30.0 2021-12-17 [1] CRAN (R 4.1.0)
#> parsnip * 0.1.7 2021-07-21 [1] CRAN (R 4.1.0)
#> pillar 1.7.0 2022-02-01 [1] CRAN (R 4.1.1)
#> pkgconfig 2.0.3 2019-09-22 [1] CRAN (R 4.1.0)
#> plyr 1.8.6 2020-03-03 [1] CRAN (R 4.1.0)
#> pROC 1.18.0 2021-09-03 [1] CRAN (R 4.1.0)
#> prodlim 2019.11.13 2019-11-17 [1] CRAN (R 4.1.0)
#> purrr * 0.3.4 2020-04-17 [1] CRAN (R 4.1.0)
#> R.cache 0.15.0 2021-04-30 [1] CRAN (R 4.1.0)
#> R.methodsS3 1.8.1 2020-08-26 [1] CRAN (R 4.1.0)
#> R.oo 1.24.0 2020-08-26 [1] CRAN (R 4.1.0)
#> R.utils 2.11.0 2021-09-26 [1] CRAN (R 4.1.0)
#> R6 2.5.1 2021-08-19 [1] CRAN (R 4.1.0)
#> Rcpp 1.0.8 2022-01-13 [1] CRAN (R 4.1.2)
#> recipes * 0.2.0 2022-02-18 [1] CRAN (R 4.1.1)
#> reprex 2.0.1 2021-08-05 [1] CRAN (R 4.1.0)
#> rlang * 1.0.1 2022-02-03 [1] CRAN (R 4.1.2)
#> rmarkdown 2.11 2021-09-14 [1] CRAN (R 4.1.0)
#> rpart 4.1.16 2022-01-24 [1] CRAN (R 4.1.2)
#> rsample * 0.1.1 2021-11-08 [1] CRAN (R 4.1.0)
#> rstudioapi 0.13 2020-11-12 [1] CRAN (R 4.1.0)
#> scales * 1.1.1 2020-05-11 [1] CRAN (R 4.1.0)
#> sessioninfo 1.2.2 2021-12-06 [1] CRAN (R 4.1.0)
#> stringi 1.7.6 2021-11-29 [1] CRAN (R 4.1.1)
#> stringr 1.4.0 2019-02-10 [1] CRAN (R 4.1.0)
#> styler 1.6.2 2021-09-23 [1] CRAN (R 4.1.0)
#> survival 3.2-13 2021-08-24 [1] CRAN (R 4.1.0)
#> tibble * 3.1.6 2021-11-07 [1] CRAN (R 4.1.0)
#> tidymodels * 0.1.4 2021-10-01 [1] CRAN (R 4.1.0)
#> tidyr * 1.2.0 2022-02-01 [1] CRAN (R 4.1.2)
#> tidyselect 1.1.1 2021-04-30 [1] CRAN (R 4.1.0)
#> timeDate 3043.102 2018-02-21 [1] CRAN (R 4.1.0)
#> tune * 0.1.6 2021-07-21 [1] CRAN (R 4.1.0)
#> utf8 1.2.2 2021-07-24 [1] CRAN (R 4.1.0)
#> vctrs * 0.3.8 2021-04-29 [1] CRAN (R 4.1.0)
#> withr 2.4.3 2021-11-30 [1] CRAN (R 4.1.0)
#> workflows * 0.2.4 2021-10-12 [1] CRAN (R 4.1.1)
#> workflowsets * 0.1.0.9000 2021-10-13 [1] Github (tidymodels/workflowsets@15cfd79)
#> xfun 0.29 2021-12-14 [1] CRAN (R 4.1.0)
#> yaml 2.2.2 2022-01-25 [1] CRAN (R 4.1.2)
#> yardstick * 0.0.9 2021-11-22 [1] CRAN (R 4.1.0)
#>
#> [1] /Library/Frameworks/R.framework/Versions/4.1/Resources/library
#>
#> ──────────────────────────────────────────────────────────────────────────────
I think this is a pretty interesting idea that could make using these functions less frustrating. This likely isn't near the top of our priorities as of today but let's keep this issue and collect feedback on this possibility.
