scikit-learn-contrib
Too many threads spawed in DANCo().fit()
Hello, I am an HPC technichal consultant and one of our users is using your package. The function skdim.id.DANCo().fit(data) seems to be spawning a significant number of threads with no apparent way to control the number. In an HPC environment, this is an issue. Is this intended?
Hi, thanks for reporting this ! The function does not use any library for multithreading or multiprocessing. This likely comes from underlying libraries such as numpy which have implicit parallelism (see https://github.com/numpy/numpy/issues/11826)
Maybe adding this line at the beginning of the script would fix this ?
import os os.environ['OMP_NUM_THREADS']= "1")
Hi! The user already has the OMP_NUM_THREADS and MKL_NUM_THREADS to 1. Also, the number of threads created is in the hundreds, which leaves me to believe that is related to the dimensions or the number of timesteps. So there is something underlying this beyond the openblas/mkl threads
I see, then I am a bit confused what can be causing this and how to fix it. I will try to reproduce this
In the meantime maybe one of the solutions here could be a workaround: https://stackoverflow.com/questions/30791550/limit-number-of-threads-in-numpy
In particular answers by Josh Broomberg and by Amir