Azure
Connecting to existing cluster fails to load imageName in code
Before submitting a bug please check the following:
- [yes] Start a new R session
- [yes] Check your credentials file
- [yes] Install the latest doAzureParallel package
- [yes] Submit a minimal, reproducible example
- [yes] run
sessionInfo()
If we connect to an existing cluster then jobs fail to run, likely due to missing "imageName" in code.
{"id":"1","commandLine":"/bin/bash -c "set -e; set -o pipefail; Rscript --no-save --no-environ --no-restore --no-site-file --verbose $AZ_BATCH_JOB_PREP_WORKING_DIR/worker.R 1 10 0 stop > $AZ_BATCH_TASK_ID.txt; wait"","userIdentity":{"autoUser":{"scope":"pool","elevationLevel":"admin"}},"environmentSettings"........removed code for privacy........{"filePattern":"../stdout.txt","destination":{"container":{"path":"stdout/1-........removed code for privacy........constraints":{"maxTaskRetryCount":3},"exitConditions":{"default":{"dependencyAction":"satisfy"}},"containerSettings":{"imageName":{},"containerRunOptions":"--rm"}}
Error in curl::curl_fetch_memory(url, handle = handle) : Failure when receiving data from the peer
But if we create a new cluster and then connect to the existing cluster, then the code runs fine and the verbose output shows "imageName":"rocker/tidyverse:3.6.3":
{"id":"1","commandLine":"/bin/bash -c "set -e; set -o pipefail; Rscript --no-save --no-environ --no-restore --no-site-file --verbose $AZ_BATCH_JOB_PREP_WORKING_DIR/worker.R 1 1 0 stop > $AZ_BATCH_TASK_ID.txt; wait"","userIdentity":{"autoUser":{"scope":"pool","elevationLevel":"admin"}},"environmentSettings"........removed code for privacy........{"filePattern":"../stdout.txt","destination":{"container":{"path":"stdout/1-........removed code for privacy........constraints":{"maxTaskRetryCount":3},"exitConditions":{"default":{"dependencyAction":"satisfy"}},"containerSettings":{"imageName":"rocker/tidyverse:3.6.3","containerRunOptions":"--rm"}}
Steps we follow to reproduce the issue:
cluster already exists with several idle nodes
#------------------# Load the doAzureParallel library library(doAzureParallel) #------------------# Logging on setVerbose(TRUE) setHttpTraffic(TRUE) #------------------# Set your credentials setCredentials("credentials.json") #------------------# Get existing cluster cluster <- getCluster("TestCluster_2020", verbose = TRUE) #------------------# Register the cluster as your parallel backend registerDoAzureParallel(cluster)
#------------------# Test simulation inputs mean_change = 1.001 volatility = 0.01 opening_price = 100
getClosingPrice <- function() { days <- 1825 # ~ 5 years movement <- rnorm(days, mean=mean_change, sd=volatility) path <- cumprod(c(opening_price, movement)) closingPrice <- path[days] return(closingPrice) }
#------------------# PARALLEL Test simulation
opt <- list(chunkSize = 10)
start_p <- Sys.time()
closingPrices_p <- foreach(i = 1:10, .combine='c', .options.azure = opt) %dopar% {
replicate(10, getClosingPrice())
}
end_p <- Sys.time()
hist(closingPrices_p)
difftime(end_p, start_p, unit = "min")
So the only way to run the code against an existing cluster is to create a "throw-away" cluster first and then use the existing cluster for execution:
#------------------# Create your cluster in Azure passing, it your cluster config file. #throw-away cluster# cluster <- makeCluster("cluster.json") #------------------# Get existing cluster cluster <- getCluster("TestCluster_2020", verbose = TRUE) #------------------# Register the cluster as your parallel backend registerDoAzureParallel(cluster)
#############################
sessionInfo() R version 3.6.3 (2020-02-29) Platform: x86_64-w64-mingw32/x64 (64-bit) Running under: Windows 10 x64 (build 19041)
Matrix products: default
locale: [1] LC_COLLATE=English_United Kingdom.1252 LC_CTYPE=English_United Kingdom.1252 LC_MONETARY=English_United Kingdom.1252 LC_NUMERIC=C LC_TIME=English_United Kingdom.1252
attached base packages: [1] stats graphics grDevices utils datasets methods base
other attached packages: [1] doAzureParallel_0.8.0 iterators_1.0.12 foreach_1.5.0
loaded via a namespace (and not attached):
[1] compiler_3.6.3 prettyunits_1.1.1 bitops_1.0-6 remotes_2.2.0 tools_3.6.3 testthat_2.3.2 digest_0.6.25 pkgbuild_1.1.0 pkgload_1.1.0 jsonlite_1.7.1 memoise_1.1.0 rlang_0.4.7 cli_2.0.2
[14] rstudioapi_0.11 curl_4.3 withr_2.3.0 httr_1.4.2 desc_1.2.0 fs_1.5.0 devtools_2.3.2 rprojroot_1.3-2 glue_1.4.2 R6_2.4.1 processx_3.4.4 fansi_0.4.1 sessioninfo_1.1.1
[27] callr_3.4.4 magrittr_1.5 backports_1.1.10 ps_1.3.4 codetools_0.2-16 ellipsis_0.3.1 usethis_1.6.3 assertthat_0.2.1 mime_0.9 rAzureBatch_0.7.0 RCurl_1.98-1.2 crayon_1.3.4 rjson_0.2.20
