yardstick
yardstick copied to clipboard
tidymodels
`metric_set()` has a less useful error message when no `estimate` is provided for classification metric sets
The problem
For classification metrics, when truth or estimate is not a factor, metric_set() gives a much less informative error than the classification metric itself.
Reproducible example
yardstick::precision(Orange, age, Tree)
#> Error in `yardstick::precision()`:
#> ! `truth` should be a factor, not a a double vector.
#> Backtrace:
#> ▆
#> 1. ├─yardstick::precision(Orange, age, Tree)
#> 2. └─yardstick:::precision.data.frame(Orange, age, Tree)
#> 3. └─yardstick::class_metric_summarizer(...)
#> 4. ├─rlang::inject(...)
#> 5. ├─base::withCallingHandlers(...)
#> 6. └─yardstick (local) fn(...)
#> 7. └─yardstick::check_class_metric(...)
#> 8. └─yardstick:::validate_factor_truth_factor_estimate(...)
#> 9. └─cli::cli_abort(...)
#> 10. └─rlang::abort(...)
yardstick::metric_set(yardstick::precision)(Orange, age, Tree)
#> Error in `metric_set()`:
#> ! Failed to compute `precision()`.
#> Caused by error:
#> ! Must select at least one item.
#> Backtrace:
#> ▆
#> 1. ├─(yardstick::metric_set(yardstick::precision))(Orange, age, Tree)
#> 2. │ └─base::mapply(...)
#> 3. │ └─yardstick (local) `<fn>`(dots[[1L]][[1L]], dots[[2L]][[1L]])
#> 4. │ ├─base::tryCatch(...)
#> 5. │ │ └─base (local) tryCatchList(expr, classes, parentenv, handlers)
#> 6. │ │ └─base (local) tryCatchOne(expr, names, parentenv, handlers[[1L]])
#> 7. │ │ └─base (local) doTryCatch(return(expr), name, parentenv, handler)
#> 8. │ └─rlang::eval_tidy(expr, data = data, env = env)
#> 9. ├─yardstick (local) `<clss_mtr>`(...)
#> 10. └─yardstick:::precision.data.frame(...)
#> 11. └─yardstick::class_metric_summarizer(...)
#> 12. └─yardstick:::yardstick_eval_select(...)
#> 13. └─tidyselect::eval_select(...)
#> 14. └─tidyselect:::eval_select_impl(...)
#> 15. ├─tidyselect:::with_subscript_errors(...)
#> 16. │ └─base::withCallingHandlers(...)
#> 17. └─tidyselect:::vars_select_eval(...)
#> 18. └─tidyselect:::check_empty(pos, allow_empty, call = error_call)
#> 19. └─cli::cli_abort("Must select at least one item.", call = call)
#> 20. └─rlang::abort(...)
yardstick::precision(Orange, Tree, age)
#> Error in `yardstick::precision()`:
#> ! `estimate` should be a factor, not a a double vector.
#> Backtrace:
#> ▆
#> 1. ├─yardstick::precision(Orange, Tree, age)
#> 2. └─yardstick:::precision.data.frame(Orange, Tree, age)
#> 3. └─yardstick::class_metric_summarizer(...)
#> 4. ├─rlang::inject(...)
#> 5. ├─base::withCallingHandlers(...)
#> 6. └─yardstick (local) fn(...)
#> 7. └─yardstick::check_class_metric(...)
#> 8. └─yardstick:::validate_factor_truth_factor_estimate(...)
#> 9. └─cli::cli_abort(...)
#> 10. └─rlang::abort(...)
yardstick::metric_set(yardstick::precision)(Orange, Tree, age)
#> Error in `metric_set()`:
#> ! Failed to compute `precision()`.
#> Caused by error:
#> ! Must select at least one item.
#> Backtrace:
#> ▆
#> 1. ├─(yardstick::metric_set(yardstick::precision))(Orange, Tree, age)
#> 2. │ └─base::mapply(...)
#> 3. │ └─yardstick (local) `<fn>`(dots[[1L]][[1L]], dots[[2L]][[1L]])
#> 4. │ ├─base::tryCatch(...)
#> 5. │ │ └─base (local) tryCatchList(expr, classes, parentenv, handlers)
#> 6. │ │ └─base (local) tryCatchOne(expr, names, parentenv, handlers[[1L]])
#> 7. │ │ └─base (local) doTryCatch(return(expr), name, parentenv, handler)
#> 8. │ └─rlang::eval_tidy(expr, data = data, env = env)
#> 9. ├─yardstick (local) `<clss_mtr>`(...)
#> 10. └─yardstick:::precision.data.frame(...)
#> 11. └─yardstick::class_metric_summarizer(...)
#> 12. └─yardstick:::yardstick_eval_select(...)
#> 13. └─tidyselect::eval_select(...)
#> 14. └─tidyselect:::eval_select_impl(...)
#> 15. ├─tidyselect:::with_subscript_errors(...)
#> 16. │ └─base::withCallingHandlers(...)
#> 17. └─tidyselect:::vars_select_eval(...)
#> 18. └─tidyselect:::check_empty(pos, allow_empty, call = error_call)
#> 19. └─cli::cli_abort("Must select at least one item.", call = call)
#> 20. └─rlang::abort(...)
Created on 2024-04-28 with reprex v2.0.2
Session info
sessioninfo::session_info()
#> ─ Session info ───────────────────────────────────────────────────────────────
#> setting value
#> version R version 4.3.0 (2023-04-21)
#> os macOS 14.4.1
#> system aarch64, darwin20
#> ui X11
#> language (EN)
#> collate en_US.UTF-8
#> ctype en_US.UTF-8
#> tz America/Chicago
#> date 2024-04-28
#> pandoc 3.1.12.3 @ /Applications/Positron.app/Contents/Resources/app/bin/ (via rmarkdown)
#>
#> ─ Packages ───────────────────────────────────────────────────────────────────
#> package * version date (UTC) lib source
#> class 7.3-22 2023-05-03 [1] CRAN (R 4.3.0)
#> cli 3.6.2.9000 2024-04-04 [1] Github (r-lib/cli@bcb5c78)
#> codetools 0.2-19 2023-02-01 [1] CRAN (R 4.3.0)
#> colorspace 2.1-0 2023-01-23 [1] CRAN (R 4.3.0)
#> data.table 1.14.8 2023-02-17 [1] CRAN (R 4.3.0)
#> dials 1.2.0 2023-04-03 [1] CRAN (R 4.3.0)
#> DiceDesign 1.9 2021-02-13 [1] CRAN (R 4.3.0)
#> digest 0.6.33 2023-07-07 [1] CRAN (R 4.3.0)
#> dplyr 1.1.4 2023-11-17 [1] CRAN (R 4.3.1)
#> evaluate 0.21 2023-05-05 [1] CRAN (R 4.3.0)
#> fansi 1.0.6 2023-12-08 [1] CRAN (R 4.3.1)
#> fastmap 1.1.1 2023-02-24 [1] CRAN (R 4.3.0)
#> foreach 1.5.2 2022-02-02 [1] CRAN (R 4.3.0)
#> fs 1.6.3 2023-07-20 [1] CRAN (R 4.3.0)
#> furrr 0.3.1 2022-08-15 [1] CRAN (R 4.3.0)
#> future 1.33.0 2023-07-01 [1] CRAN (R 4.3.0)
#> future.apply 1.11.0 2023-05-21 [1] CRAN (R 4.3.0)
#> generics 0.1.3 2022-07-05 [1] CRAN (R 4.3.0)
#> ggplot2 3.4.2 2023-04-03 [1] CRAN (R 4.3.0)
#> globals 0.16.2 2022-11-21 [1] CRAN (R 4.3.0)
#> glue 1.7.0 2024-01-09 [1] CRAN (R 4.3.1)
#> gower 1.0.1 2022-12-22 [1] CRAN (R 4.3.0)
#> GPfit 1.0-8 2019-02-08 [1] CRAN (R 4.3.0)
#> gtable 0.3.3 2023-03-21 [1] CRAN (R 4.3.0)
#> hardhat 1.3.1 2024-02-02 [1] CRAN (R 4.3.1)
#> htmltools 0.5.6 2023-08-10 [1] CRAN (R 4.3.0)
#> ipred 0.9-14 2023-03-09 [1] CRAN (R 4.3.0)
#> iterators 1.0.14 2022-02-05 [1] CRAN (R 4.3.0)
#> knitr 1.43 2023-05-25 [1] CRAN (R 4.3.0)
#> lattice 0.22-6 2024-03-20 [1] CRAN (R 4.3.1)
#> lava 1.7.2.1 2023-02-27 [1] CRAN (R 4.3.0)
#> lhs 1.1.6 2022-12-17 [1] CRAN (R 4.3.0)
#> lifecycle 1.0.4 2023-11-07 [1] CRAN (R 4.3.1)
#> listenv 0.9.0 2022-12-16 [1] CRAN (R 4.3.0)
#> lubridate 1.9.2 2023-02-10 [1] CRAN (R 4.3.0)
#> magrittr 2.0.3 2022-03-30 [1] CRAN (R 4.3.0)
#> MASS 7.3-60.0.1 2024-01-13 [1] CRAN (R 4.3.1)
#> Matrix 1.6-5 2024-01-11 [1] CRAN (R 4.3.1)
#> munsell 0.5.0 2018-06-12 [1] CRAN (R 4.3.0)
#> nnet 7.3-19 2023-05-03 [1] CRAN (R 4.3.0)
#> parallelly 1.36.0 2023-05-26 [1] CRAN (R 4.3.0)
#> parsnip 1.1.0 2023-04-12 [1] CRAN (R 4.3.0)
#> pillar 1.9.0 2023-03-22 [1] CRAN (R 4.3.0)
#> pkgconfig 2.0.3 2019-09-22 [1] CRAN (R 4.3.0)
#> prodlim 2023.03.31 2023-04-02 [1] CRAN (R 4.3.0)
#> purrr 1.0.2 2023-08-10 [1] CRAN (R 4.3.0)
#> R.cache 0.16.0 2022-07-21 [1] CRAN (R 4.3.0)
#> R.methodsS3 1.8.2 2022-06-13 [1] CRAN (R 4.3.0)
#> R.oo 1.25.0 2022-06-12 [1] CRAN (R 4.3.0)
#> R.utils 2.12.2 2022-11-11 [1] CRAN (R 4.3.0)
#> R6 2.5.1 2021-08-19 [1] CRAN (R 4.3.0)
#> Rcpp 1.0.12 2024-01-09 [1] CRAN (R 4.3.1)
#> recipes 1.0.6 2023-04-25 [1] CRAN (R 4.3.0)
#> reprex 2.0.2 2022-08-17 [1] CRAN (R 4.3.0)
#> rlang 1.1.3.9000 2024-04-04 [1] Github (r-lib/rlang@baedaca)
#> rmarkdown 2.24 2023-08-14 [1] CRAN (R 4.3.0)
#> rpart 4.1.19 2022-10-21 [1] CRAN (R 4.3.0)
#> rsample 1.1.1 2022-12-07 [1] CRAN (R 4.3.0)
#> scales 1.2.1 2022-08-20 [1] CRAN (R 4.3.0)
#> sessioninfo 1.2.2 2021-12-06 [1] CRAN (R 4.3.0)
#> styler 1.10.1 2023-06-05 [1] CRAN (R 4.3.0)
#> survival 3.5-5 2023-03-12 [1] CRAN (R 4.3.0)
#> tibble 3.2.1 2023-03-20 [1] CRAN (R 4.3.0)
#> tidyr 1.3.0 2023-01-24 [1] CRAN (R 4.3.0)
#> tidyselect 1.2.1 2024-03-11 [1] CRAN (R 4.3.1)
#> timechange 0.2.0 2023-01-11 [1] CRAN (R 4.3.0)
#> timeDate 4022.108 2023-01-07 [1] CRAN (R 4.3.0)
#> tune 1.1.1 2023-04-11 [1] CRAN (R 4.3.0)
#> utf8 1.2.4 2023-10-22 [1] CRAN (R 4.3.1)
#> vctrs 0.6.5 2023-12-01 [1] CRAN (R 4.3.1)
#> withr 3.0.0 2024-01-16 [1] CRAN (R 4.3.1)
#> workflows 1.1.3 2023-02-22 [1] CRAN (R 4.3.0)
#> xfun 0.40 2023-08-09 [1] CRAN (R 4.3.0)
#> yaml 2.3.7 2023-01-23 [1] CRAN (R 4.3.0)
#> yardstick 1.3.1.9000 2024-04-28 [1] Github (tidymodels/yardstick@be744a3)
#>
#> [1] /Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library
#>
#> ──────────────────────────────────────────────────────────────────────────────
Sorry, I'm realizing I misdiagnosed the issue here -- I think the real problem is that I didn't name the estimate argument (got too used to the signature for numeric metric sets!). I still the error message here could be more clear about what we need to do to select at least one item, though!
yardstick::metric_set(yardstick::precision)(Orange, age, estimate = Tree)
#> Error in `metric_set()`:
#> ! Failed to compute `precision()`.
#> Caused by error:
#> ! `truth` should be a factor, not a a double vector.
#> Backtrace:
#> ▆
#> 1. ├─(yardstick::metric_set(yardstick::precision))(Orange, age, estimate = Tree)
#> 2. │ └─base::mapply(...)
#> 3. │ └─yardstick (local) `<fn>`(dots[[1L]][[1L]], dots[[2L]][[1L]])
#> 4. │ ├─base::tryCatch(...)
#> 5. │ │ └─base (local) tryCatchList(expr, classes, parentenv, handlers)
#> 6. │ │ └─base (local) tryCatchOne(expr, names, parentenv, handlers[[1L]])
#> 7. │ │ └─base (local) doTryCatch(return(expr), name, parentenv, handler)
#> 8. │ └─rlang::eval_tidy(expr, data = data, env = env)
#> 9. ├─yardstick (local) `<clss_mtr>`(...)
#> 10. └─yardstick:::precision.data.frame(...)
#> 11. └─yardstick::class_metric_summarizer(...)
#> 12. ├─rlang::inject(...)
#> 13. ├─base::withCallingHandlers(...)
#> 14. └─yardstick (local) fn(...)
#> 15. └─yardstick::check_class_metric(...)
#> 16. └─yardstick:::validate_factor_truth_factor_estimate(...)
#> 17. └─cli::cli_abort(...)
#> 18. └─rlang::abort(...)
Created on 2024-04-28 with reprex v2.0.2