finnts icon indicating copy to clipboard operation
finnts copied to clipboard

WISH: parallelly::availableCores() instead of parallel::detectCores()

Open HenrikBengtsson opened this issue 3 years ago • 1 comments

(disclaim: I'm the author) To minimize the risk of overusing CPUs on multi-tenant and multi-process machines, but also to work in more environments, please consider changing:

https://github.com/microsoft/finnts/blob/57207bf18b79949244017c4df8ec1296a9da0752/R/general_parallel.R#L6-L13

to

 get_cores <-function(num_cores){  
   min(c(parallelly::availableCores(omit = 1), num_cores))
 }

This also has the benefit of working on single-core hosts, where detectCores() == 1, e.g. free RStudio Cloud accounts, some k8s/container setups, etc. For more arguments, see https://parallelly.futureverse.org/#availablecores-vs-paralleldetectcores.

HenrikBengtsson avatar May 05 '22 18:05 HenrikBengtsson

Hey Henrik, thanks for reaching out. I'll look more into this and incorporate into package soon. Our main focus right now is to build out new parallel compute options like spark.

mitokic avatar May 06 '22 17:05 mitokic