tabnet
tabnet copied to clipboard
mlverse
Trouble with learning when tune_grid() with CPU parallel processing
HI.
I want to used a tabnet model using tidymodels(parsnip) format.
and I try to train a model with parallel processing(doFuture package).
When I use the tune_grid() function, it returns the following error message :
Error in UseMethod("filter"): no applicable method for 'filter' applied to an object of class "NULL"
Here's the code:
################################################################################
#00-1. SYSTEM SETTING
#(1) Set System Options
options(scipen=100)
set.seed(seed=2022)
#(2) Import Packages
library(tidyverse)
library(tidymodels)
library(torch)
library(tabnet)
#>
#> Attaching package: 'tabnet'
#> The following object is masked from 'package:dials':
#>
#> momentum
library(finetune)
library(parallel)
library(doFuture)
#> Loading required package: foreach
#>
#> Attaching package: 'foreach'
#> The following objects are masked from 'package:purrr':
#>
#> accumulate, when
#> Loading required package: future
library(glue)
#(3) Set Parallel Process
THREAD_NUM <- detectCores(all.test=TRUE, logical=TRUE) - 2
plan(multisession, workers = THREAD_NUM)
registerDoFuture()
################################################################################
#01. READ TABLE
#(1) Read BOSTON Dataset
BOSTON_raw <- read_csv(file="a:/R_project/TIL_R/data/kaggle_Predict_Boston_Hosuing/boston_housing.csv")
#> Rows: 506 Columns: 14
#> ── Column specification ────────────────────────────────────────────────────────
#> Delimiter: ","
#> dbl (14): CRIM, ZN, INDUS, CHAS, NOX, RM, AGE, DIS, RAD, TAX, PTRATIO, B, LS...
#>
#> ℹ Use `spec()` to retrieve the full column specification for this data.
#> ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.
################################################################################
#02. PREPARE DATASET FOR MACHINE-LEARNING
#(1) Split Data
BOSTON_split <- initial_split(data=BOSTON_raw, prop=0.2)
BOSTON_train <- training(x=BOSTON_split)
BOSTON_test <- testing(x=BOSTON_split)
#(3) Define 5-Fold CV Object
BOSTON_train_cv <- vfold_cv(data=BOSTON_train, v=5)
#(4) Define Recipes
BOSTON_recipe <- recipe(formula=MEDV~., data=BOSTON_train) %>%
step_range(all_numeric_predictors()) %>%
step_zv(all_predictors())
################################################################################
#02. TRAIN AND PREDICT BY TABNET
#(1) Define Model
tabnet_model <- tabnet(
epochs = tune(),
decision_width = tune(),
num_steps = tune(),
penalty = tune()
) %>%
set_mode(mode="regression") %>%
set_engine(engine="torch")
#(2) Define Workflow
tabnet_wf <- workflow(
preprocessor = BOSTON_recipe,
spec = tabnet_model
)
#(3) Define Hyper-Parameters
tabnet_params <- tabnet_model %>%
extract_parameter_set_dials()
#(3) Set Grid
tabnet_grid <- tabnet_params %>%
grid_latin_hypercube(size=100)
#(4) Tune Hyper-Parameters
tabnet_tune <- tune_grid(
object = tabnet_wf,
param_info = tabnet_params,
grid = tabnet_grid,
resamples = BOSTON_train_cv,
metrics = metric_set(rmse),
control = control_grid(verbose=TRUE,
save_pred=TRUE,
save_workflow=TRUE,
parallel_over="everything")
)
#> x Fold1: internal: Error in UseMethod("filter"): no applicable method for 'filter' applied ...
#> x Fold1: internal: Error in UseMethod("filter"): no applicable method for 'filter' applied ...
#> x Fold1: internal: Error in UseMethod("filter"): no applicable method for 'filter' applied ...
#> x Fold1: internal: Error in UseMethod("filter"): no applicable method for 'filter' applied ...
#> x Fold1: internal: Error in UseMethod("filter"): no applicable method for 'filter' applied ...
#> x Fold1: internal: Error in UseMethod("filter"): no applicable method for 'filter' applied ...
#> x Fold1: internal: Error in UseMethod("filter"): no applicable method for 'filter' applied ...
#> x Fold1: internal: Error in UseMethod("filter"): no applicable method for 'filter' applied ...
#> x Fold1: internal: Error in UseMethod("filter"): no applicable method for 'filter' applied ...
#> x Fold1: internal: Error in UseMethod("filter"): no applicable method for 'filter' applied ...
#> x Fold1: internal: Error in UseMethod("filter"): no applicable method for 'filter' applied ...
#> x Fold1: internal: Error in UseMethod("filter"): no applicable method for 'filter' applied ...
#> x Fold1: internal: Error in UseMethod("filter"): no applicable method for 'filter' applied ...
#> x Fold1: internal: Error in UseMethod("filter"): no applicable method for 'filter' applied ...
#> x Fold1: internal: Error in UseMethod("filter"): no applicable method for 'filter' applied ...
#> x Fold1: internal: Error in UseMethod("filter"): no applicable method for 'filter' applied ...
#> x Fold1: internal: Error in UseMethod("filter"): no applicable method for 'filter' applied ...
#> x Fold1: internal: Error in UseMethod("filter"): no applicable method for 'filter' applied ...
#> x Fold1: internal: Error in UseMethod("filter"): no applicable method for 'filter' applied ...
#> x Fold1: internal: Error in UseMethod("filter"): no applicable method for 'filter' applied ...
#> x Fold1: internal: Error in UseMethod("filter"): no applicable method for 'filter' applied ...
#> x Fold1: internal: Error in UseMethod("filter"): no applicable method for 'filter' applied ...
#> x Fold1: internal: Error in UseMethod("filter"): no applicable method for 'filter' applied ...
#> x Fold1: internal: Error in UseMethod("filter"): no applicable method for 'filter' applied ...
#> x Fold1: internal: Error in UseMethod("filter"): no applicable method for 'filter' applied ...
#> x Fold1: internal: Error in UseMethod("filter"): no applicable method for 'filter' applied ...
#> x Fold1: internal: Error in UseMethod("filter"): no applicable method for 'filter' applied ...
#> x Fold1: internal: Error in UseMethod("filter"): no applicable method for 'filter' applied ...
#> x Fold1: internal: Error in UseMethod("filter"): no applicable method for 'filter' applied ...
#> x Fold1: internal: Error in UseMethod("filter"): no applicable method for 'filter' applied ...
#> x Fold1: internal: Error in UseMethod("filter"): no applicable method for 'filter' applied ...
#> x Fold1: internal: Error in UseMethod("filter"): no applicable method for 'filter' applied ...
#> x Fold1: internal: Error in UseMethod("filter"): no applicable method for 'filter' applied ...
#> x Fold1: internal: Error in UseMethod("filter"): no applicable method for 'filter' applied ...
#> x Fold1: internal: Error in UseMethod("filter"): no applicable method for 'filter' applied ...
#> x Fold1: internal: Error in UseMethod("filter"): no applicable method for 'filter' applied ...
#> x Fold1: internal: Error in UseMethod("filter"): no applicable method for 'filter' applied ...
#> x Fold1: internal: Error in UseMethod("filter"): no applicable method for 'filter' applied ...
#> x Fold1: internal: Error in UseMethod("filter"): no applicable method for 'filter' applied ...
#> x Fold1: internal: Error in UseMethod("filter"): no applicable method for 'filter' applied ...
#> x Fold1: internal: Error in UseMethod("filter"): no applicable method for 'filter' applied ...
#> x Fold1: internal: Error in UseMethod("filter"): no applicable method for 'filter' applied ...
#> x Fold1: internal: Error in UseMethod("filter"): no applicable method for 'filter' applied ...
#> x Fold1: internal: Error in UseMethod("filter"): no applicable method for 'filter' applied ...
#> x Fold1: internal: Error in UseMethod("filter"): no applicable method for 'filter' applied ...
#> x Fold1: internal: Error in UseMethod("filter"): no applicable method for 'filter' applied ...
#> x Fold1: internal: Error in UseMethod("filter"): no applicable method for 'filter' applied ...
#> x Fold1: internal: Error in UseMethod("filter"): no applicable method for 'filter' applied ...
#> x Fold1: internal: Error in UseMethod("filter"): no applicable method for 'filter' applied ...
#> x Fold1: internal: Error in UseMethod("filter"): no applicable method for 'filter' applied ...
#> x Fold1: internal: Error in UseMethod("filter"): no applicable method for 'filter' applied ...
#> x Fold1: internal: Error in UseMethod("filter"): no applicable method for 'filter' applied ...
#> x Fold1: internal: Error in UseMethod("filter"): no applicable method for 'filter' applied ...
#> x Fold1: internal: Error in UseMethod("filter"): no applicable method for 'filter' applied ...
#> x Fold1: internal: Error in UseMethod("filter"): no applicable method for 'filter' applied ...
#> x Fold1: internal: Error in UseMethod("filter"): no applicable method for 'filter' applied ...
#> x Fold1: internal: Error in UseMethod("filter"): no applicable method for 'filter' applied ...
#> x Fold1: internal: Error in UseMethod("filter"): no applicable method for 'filter' applied ...
#> x Fold1: internal: Error in UseMethod("filter"): no applicable method for 'filter' applied ...
#> x Fold1: internal: Error in UseMethod("filter"): no applicable method for 'filter' applied ...
#> x Fold1: internal: Error in UseMethod("filter"): no applicable method for 'filter' applied ...
#> x Fold1: internal: Error in UseMethod("filter"): no applicable method for 'filter' applied ...
#> x Fold1: internal: Error in UseMethod("filter"): no applicable method for 'filter' applied ...
#> x Fold1: internal: Error in UseMethod("filter"): no applicable method for 'filter' applied ...
#> x Fold1: internal: Error in UseMethod("filter"): no applicable method for 'filter' applied ...
#> x Fold1: internal: Error in UseMethod("filter"): no applicable method for 'filter' applied ...
#> x Fold1: internal: Error in UseMethod("filter"): no applicable method for 'filter' applied ...
#> x Fold1: internal: Error in UseMethod("filter"): no applicable method for 'filter' applied ...
#> x Fold1: internal: Error in UseMethod("filter"): no applicable method for 'filter' applied ...
#> x Fold1: internal: Error in UseMethod("filter"): no applicable method for 'filter' applied ...
#> x Fold1: internal: Error in UseMethod("filter"): no applicable method for 'filter' applied ...
#> x Fold1: internal: Error in UseMethod("filter"): no applicable method for 'filter' applied ...
#> x Fold1: internal: Error in UseMethod("filter"): no applicable method for 'filter' applied ...
#> x Fold1: internal: Error in UseMethod("filter"): no applicable method for 'filter' applied ...
#> x Fold1: internal: Error in UseMethod("filter"): no applicable method for 'filter' applied ...
#> x Fold1: internal: Error in UseMethod("filter"): no applicable method for 'filter' applied ...
#> x Fold1: internal: Error in UseMethod("filter"): no applicable method for 'filter' applied ...
#> x Fold1: internal: Error in UseMethod("filter"): no applicable method for 'filter' applied ...
#> x Fold1: internal: Error in UseMethod("filter"): no applicable method for 'filter' applied ...
#> x Fold1: internal: Error in UseMethod("filter"): no applicable method for 'filter' applied ...
#> x Fold1: internal: Error in UseMethod("filter"): no applicable method for 'filter' applied ...
#> x Fold1: internal: Error in UseMethod("filter"): no applicable method for 'filter' applied ...
#> x Fold1: internal: Error in UseMethod("filter"): no applicable method for 'filter' applied ...
#> x Fold1: internal: Error in UseMethod("filter"): no applicable method for 'filter' applied ...
#> x Fold1: internal: Error in UseMethod("filter"): no applicable method for 'filter' applied ...
#> x Fold1: internal: Error in UseMethod("filter"): no applicable method for 'filter' applied ...
#> x Fold1: internal: Error in UseMethod("filter"): no applicable method for 'filter' applied ...
#> x Fold1: internal: Error in UseMethod("filter"): no applicable method for 'filter' applied ...
#> x Fold1: internal: Error in UseMethod("filter"): no applicable method for 'filter' applied ...
#> x Fold1: internal: Error in UseMethod("filter"): no applicable method for 'filter' applied ...
#> x Fold1: internal: Error in UseMethod("filter"): no applicable method for 'filter' applied ...
#> x Fold1: internal: Error in UseMethod("filter"): no applicable method for 'filter' applied ...
#> x Fold1: internal: Error in UseMethod("filter"): no applicable method for 'filter' applied ...
#> x Fold1: internal: Error in UseMethod("filter"): no applicable method for 'filter' applied ...
#> x Fold1: internal: Error in UseMethod("filter"): no applicable method for 'filter' applied ...
#> x Fold1: internal: Error in UseMethod("filter"): no applicable method for 'filter' applied ...
#> x Fold1: internal: Error in UseMethod("filter"): no applicable method for 'filter' applied ...
#> x Fold1: internal: Error in UseMethod("filter"): no applicable method for 'filter' applied ...
#> x Fold1: internal: Error in UseMethod("filter"): no applicable method for 'filter' applied ...
#> x Fold1: internal: Error in UseMethod("filter"): no applicable method for 'filter' applied ...
#> x Fold2: internal: Error in UseMethod("filter"): no applicable method for 'filter' applied ...
#> x Fold2: internal: Error in UseMethod("filter"): no applicable method for 'filter' applied ...
#> x Fold2: internal: Error in UseMethod("filter"): no applicable method for 'filter' applied ...
#> x Fold2: internal: Error in UseMethod("filter"): no applicable method for 'filter' applied ...
#> x Fold2: internal: Error in UseMethod("filter"): no applicable method for 'filter' applied ...
#> ...
#> Warning: All models failed. Run `show_notes(.Last.tune.result)` for more
#> information.
Created on 2022-08-23 with reprex v2.0.2
and my session info is here
sessionInfo()
#> R version 4.2.1 (2022-06-23 ucrt)
#> Platform: x86_64-w64-mingw32/x64 (64-bit)
#> Running under: Windows 10 x64 (build 22000)
#>
#> Matrix products: default
#>
#> locale:
#> [1] LC_COLLATE=English_United States.utf8
#> [2] LC_CTYPE=English_United States.utf8
#> [3] LC_MONETARY=English_United States.utf8
#> [4] LC_NUMERIC=C
#> [5] LC_TIME=English_United States.utf8
#>
#> attached base packages:
#> [1] stats graphics grDevices utils datasets methods base
#>
#> loaded via a namespace (and not attached):
#> [1] withr_2.5.0 digest_0.6.29 lifecycle_1.0.1 magrittr_2.0.3
#> [5] reprex_2.0.2 evaluate_0.16 highr_0.9 stringi_1.7.8
#> [9] rlang_1.0.4 cli_3.3.0 rstudioapi_0.14 fs_1.5.2
#> [13] rmarkdown_2.15 tools_4.2.1 stringr_1.4.1 glue_1.6.2
#> [17] xfun_0.32 yaml_2.3.5 fastmap_1.1.0 compiler_4.2.1
#> [21] htmltools_0.5.3 knitr_1.39
Created on 2022-08-23 with reprex v2.0.2
Thank you.
I have run into this same issue, and have not yet found a solution outside of removing the parallel parts of my workflow. Would be interested in following along.
Hello
Some initial investigation may link the issue to {tune} not being robust to .NULL env problem :
A debug of tune:::fn_tune_grid_loop() returns
Browse[2]> results
[[1]]
<error/rlang_error>
Error in `rlang::env_bind()` at tune/R/logging.R:196:2:
! `.env` must be an environment, not NULL.
---
Backtrace:
1. parallel (local) workRSOCK()
26. base::eval(...)
27. base::eval(...)
29. base::lapply(...)
30. FUN(X[[i]], ...)
35. tune (local) fn_tune_grid_loop_iter_safely(...)
at tune/R/grid_code_paths.R:206:10
36. tune:::log_problems(notes, control, split, "internal", problems)
at tune/R/grid_code_paths.R:519:2
37. tune:::tune_catalog(...)
at tune/R/logging.R:314:6
38. rlang::env_bind(.env = tune_env$progress_env)
at tune/R/logging.R:196:2
[[2]]
<error/rlang_error>
Error in `rlang::env_bind()` at tune/R/logging.R:196:2:
! `.env` must be an environment, not NULL.
---
Backtrace:
1. parallel (local) workRSOCK()
26. base::eval(...)
27. base::eval(...)
29. base::lapply(...)
30. FUN(X[[i]], ...)
35. tune (local) fn_tune_grid_loop_iter_safely(...)
at tune/R/grid_code_paths.R:206:10
36. tune:::log_problems(notes, control, split, "internal", problems)
at tune/R/grid_code_paths.R:519:2
37. tune:::tune_catalog(...)
at tune/R/logging.R:314:6
38. rlang::env_bind(.env = tune_env$progress_env)
at tune/R/logging.R:196:2
as {h2o} has a special .env treatment in {tune}, maybe the same is needed for {torch}, as binding renvs for torch is certainly a bad idea...
Hello,
Sorry for this late update, but digging into this makes me definitively think it can't be supported. If I understqnd it well, the way {tune} runs parallelize compute load is through forking environements, which makes no sense with a torch backend (because you would never get enough memory to instantiate multiple torch instances on a GPU, or on even on a CPU ) so this should ideally be prevented.
Workaround for your example, you should run it sequentially with the allow_par = FALSE option, as {torch} is already using all your compute resources for each tuning iteration :
tabnet_tune <- tune:::tune_grid.workflow(
object = tabnet_wf,
param_info = tabnet_params,
grid = tabnet_grid,
resamples = BOSTON_train_cv,
metrics = metric_set(mae),
control = control_grid(verbose = TRUE,
save_pred = TRUE,
save_workflow = TRUE,
allow_par = FALSE)
)