Tome icon indicating copy to clipboard operation
Tome copied to clipboard

OSError: [Errno 24] Too many open files

Open Hocnonsense opened this issue 3 years ago • 0 comments

First of all, thanks for your repository!

I'm trying to use this repository to estimate the temperature optima of my genomes. However, when I run it with 12 threads for about 2000 genomes, this software broken down after finished about 40 genomes. This is the error message:

/data/home/lenghao/software/miniconda3/envs/tome/lib/python3.6/site-packages/sklearn/base.py:253: UserWarning: Trying to unpickle estimator SVR from version 0.19.2 when using version 0.20.3. This might lead to breaking code or invalid results. Use at your own risk.
  UserWarning)
Traceback (most recent call last):
  File "/data/home/lenghao/software/miniconda3/envs/tome/bin/tome", line 33, in <module>
    sys.exit(load_entry_point('tome', 'console_scripts', 'tome')())
  File "/data/home/lenghao/software/Tome-master/tome/tome.py", line 106, in main
    else: predOGT.main(args)
  File "/data/home/lenghao/software/Tome-master/tome/core/predOGT.py", line 199, in main
    pred_ogt = predict(os.path.join(indir,name),model,means,stds,features,args.threads)
  File "/data/home/lenghao/software/Tome-master/tome/core/predOGT.py", line 170, in predict
    dimers_fq = get_dimer_frequency(fasta_file,p)
  File "/data/home/lenghao/software/Tome-master/tome/core/predOGT.py", line 156, in get_dimer_frequency
    dimers = count_dimer(fasta_file,p)
  File "/data/home/lenghao/software/Tome-master/tome/core/predOGT.py", line 151, in count_dimer
    results = Pool(num_cpus).map(do_count, seqs)
  File "/data/home/lenghao/software/miniconda3/envs/tome/lib/python3.6/multiprocessing/context.py", line 119, in Pool
    context=self.get_context())
  File "/data/home/lenghao/software/miniconda3/envs/tome/lib/python3.6/multiprocessing/pool.py", line 174, in __init__
    self._repopulate_pool()
  File "/data/home/lenghao/software/miniconda3/envs/tome/lib/python3.6/multiprocessing/pool.py", line 239, in _repopulate_pool
    w.start()
  File "/data/home/lenghao/software/miniconda3/envs/tome/lib/python3.6/multiprocessing/process.py", line 105, in start
    self._popen = self._Popen(self)
  File "/data/home/lenghao/software/miniconda3/envs/tome/lib/python3.6/multiprocessing/context.py", line 277, in _Popen
    return Popen(process_obj)
  File "/data/home/lenghao/software/miniconda3/envs/tome/lib/python3.6/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/data/home/lenghao/software/miniconda3/envs/tome/lib/python3.6/multiprocessing/popen_fork.py", line 65, in _launch
    parent_r, child_w = os.pipe()
OSError: [Errno 24] Too many open files
/data/home/lenghao/software/miniconda3/envs/tome/lib/python3.6/site-packages/sklearn/base.py:253: UserWarning: Trying to unpickle estimator SVR from version 0.19.2 when using version 0.20.3. This might lead to breaking code or invalid results. Use at your own risk.
  UserWarning)
Traceback (most recent call last):
  File "/data/home/lenghao/software/miniconda3/envs/tome/bin/tome", line 33, in <module>
    sys.exit(load_entry_point('tome', 'console_scripts', 'tome')())
  File "/data/home/lenghao/software/Tome-master/tome/tome.py", line 106, in main
    else: predOGT.main(args)
  File "/data/home/lenghao/software/Tome-master/tome/core/predOGT.py", line 199, in main
    pred_ogt = predict(os.path.join(indir,name),model,means,stds,features,args.threads)
  File "/data/home/lenghao/software/Tome-master/tome/core/predOGT.py", line 170, in predict
    dimers_fq = get_dimer_frequency(fasta_file,p)
  File "/data/home/lenghao/software/Tome-master/tome/core/predOGT.py", line 156, in get_dimer_frequency
    dimers = count_dimer(fasta_file,p)
  File "/data/home/lenghao/software/Tome-master/tome/core/predOGT.py", line 151, in count_dimer
    results = Pool(num_cpus).map(do_count, seqs)
  File "/data/home/lenghao/software/miniconda3/envs/tome/lib/python3.6/multiprocessing/context.py", line 119, in Pool
    context=self.get_context())
  File "/data/home/lenghao/software/miniconda3/envs/tome/lib/python3.6/multiprocessing/pool.py", line 174, in __init__
    self._repopulate_pool()
  File "/data/home/lenghao/software/miniconda3/envs/tome/lib/python3.6/multiprocessing/pool.py", line 239, in _repopulate_pool
    w.start()
  File "/data/home/lenghao/software/miniconda3/envs/tome/lib/python3.6/multiprocessing/process.py", line 105, in start
    self._popen = self._Popen(self)
  File "/data/home/lenghao/software/miniconda3/envs/tome/lib/python3.6/multiprocessing/context.py", line 277, in _Popen
    return Popen(process_obj)
  File "/data/home/lenghao/software/miniconda3/envs/tome/lib/python3.6/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/data/home/lenghao/software/miniconda3/envs/tome/lib/python3.6/multiprocessing/popen_fork.py", line 65, in _launch
    parent_r, child_w = os.pipe()
OSError: [Errno 24] Too many open files

This seems to be caused by opening too many files without closing it. Do you have any idea about this problem and to solve it? Thanks!

Hocnonsense avatar Jan 17 '23 03:01 Hocnonsense