multiprocessing connection error

Open YuanfengZhang opened this issue 6 months ago • 0 comments

CMD:

# cd to snakemake dir
cd /data/nanopore

THREADS="8"
EQA_ID="202401"
SAMPLE_ID="NP1_HF_1"

conda activate deepmod2
time python resources/DeepMod2/deepmod2 detect \
  --bam nanopore/bam_with_moves/${SAMPLE_ID}.bam \
  --input uploaded/20250719/${EQA_ID}_pod5_pass \
  --model transformer_r10.4.1_5khz_v5.0 \
  --file_type pod5 --seq_type dna --threads ${THREADS} \
  --ref ../custom_genomes/BL.fa \
  --output nanopore/deepmod2/ \
  --prefix {SAMPLE_ID}

ERROR:

2025-08-06 21:34:25.572372: Starting DeepMod2.
2025-08-06 21:34:25.860719: 
Command: python resources/DeepMod2/deepmod2 detect --bam nanopore/bam_with_moves/NP1_HF_1.bam --input uploaded/20250719/202401_pod5_pass --model transformer_r10.4.1_5khz_v5.0 --file_type pod5 --seq_type dna --threads 8 --ref ../custom_genomes/BL.fa --output nanopore/deepmod2/ --prefix {SAMPLE_ID}
/home/zyf/miniforge3/envs/deepmod2/lib/python3.13/site-packages/ont_fast5_api/compression_settings.py:1: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
  import pkg_resources
2025-08-06 21:34:28.674240: Starting Per Read Methylation Detection.
2025-08-06 21:34:30.391273: Getting motif positions from the reference.
2025-08-06 21:35:13.783261: Finished getting motif positions from the reference.
/home/zyf/miniforge3/envs/deepmod2/lib/python3.13/site-packages/ont_fast5_api/compression_settings.py:1: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
  import pkg_resources
2025-08-06 21:35:15.518431: Building BAM index.
/home/zyf/miniforge3/envs/deepmod2/lib/python3.13/site-packages/ont_fast5_api/compression_settings.py:1: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
  import pkg_resources
/home/zyf/miniforge3/envs/deepmod2/lib/python3.13/site-packages/ont_fast5_api/compression_settings.py:1: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
  import pkg_resources
CUDA test: True True
/home/zyf/miniforge3/envs/deepmod2/lib/python3.13/site-packages/ont_fast5_api/compression_settings.py:1: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
  import pkg_resources
CUDA test: True True
/home/zyf/miniforge3/envs/deepmod2/lib/python3.13/site-packages/ont_fast5_api/compression_settings.py:1: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
  import pkg_resources
CUDA test: True True
/home/zyf/miniforge3/envs/deepmod2/lib/python3.13/site-packages/ont_fast5_api/compression_settings.py:1: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
  import pkg_resources
CUDA test: True True
/home/zyf/miniforge3/envs/deepmod2/lib/python3.13/site-packages/ont_fast5_api/compression_settings.py:1: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
  import pkg_resources
CUDA test: True True
/home/zyf/miniforge3/envs/deepmod2/lib/python3.13/site-packages/ont_fast5_api/compression_settings.py:1: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
  import pkg_resources
CUDA test: True True
/home/zyf/miniforge3/envs/deepmod2/lib/python3.13/site-packages/ont_fast5_api/compression_settings.py:1: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
  import pkg_resources
CUDA test: True True
2025-08-06 21:43:58.631146: Finished building BAM index.
2025-08-06 21:49:07.353975: Number of reads processed: 10090
2025-08-06 21:54:01.712547: Number of reads processed: 20137
2025-08-06 22:00:15.612669: Number of reads processed: 30036
2025-08-06 22:06:56.162713: Number of reads processed: 40085
2025-08-06 22:11:58.202230: Number of reads processed: 50043
2025-08-06 22:19:16.906254: Number of reads processed: 60014
2025-08-06 22:25:41.123003: Number of reads processed: 70012
2025-08-06 22:31:10.760211: Number of reads processed: 80005
2025-08-06 22:36:31.096983: Number of reads processed: 90143
2025-08-06 22:42:14.753482: Number of reads processed: 100050
2025-08-06 22:48:03.413977: Number of reads processed: 110125
2025-08-06 22:53:57.734186: Number of reads processed: 120165
2025-08-06 22:58:59.334226: Number of reads processed: 130184
2025-08-06 23:04:29.837810: Number of reads processed: 140085
2025-08-06 23:11:12.808394: Number of reads processed: 150052
2025-08-06 23:17:50.178974: Number of reads processed: 160079
2025-08-06 23:23:22.460501: Number of reads processed: 170054
2025-08-06 23:28:40.029061: Number of reads processed: 180003
Process Process-14:
Process Process-12:
Process Process-13:
Process Process-18:
Process Process-15:
Process Process-16:
Process Process-17:
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
  File "/home/zyf/miniforge3/envs/deepmod2/lib/python3.13/multiprocessing/process.py", line 313, in _bootstrap
    self.run()
    ~~~~~~~~^^
  File "/home/zyf/miniforge3/envs/deepmod2/lib/python3.13/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/zyf/eqa/resources/DeepMod2/src/detect.py", line 508, in process
    chunk=signal_Q.get(block=False, timeout=10)
  File "<string>", line 2, in get
  File "/home/zyf/miniforge3/envs/deepmod2/lib/python3.13/multiprocessing/managers.py", line 831, in _callmethod
    kind, result = conn.recv()
                   ~~~~~~~~~^^
  File "/home/zyf/miniforge3/envs/deepmod2/lib/python3.13/multiprocessing/connection.py", line 250, in recv
    buf = self._recv_bytes()
  File "/home/zyf/miniforge3/envs/deepmod2/lib/python3.13/multiprocessing/connection.py", line 430, in _recv_bytes
    buf = self._recv(4)
  File "/home/zyf/miniforge3/envs/deepmod2/lib/python3.13/multiprocessing/connection.py", line 395, in _recv
    chunk = read(handle, remaining)
ConnectionResetError: [Errno 104] Connection reset by peer
  File "/home/zyf/miniforge3/envs/deepmod2/lib/python3.13/multiprocessing/process.py", line 313, in _bootstrap
    self.run()
    ~~~~~~~~^^
  File "/home/zyf/miniforge3/envs/deepmod2/lib/python3.13/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/zyf/eqa/resources/DeepMod2/src/detect.py", line 508, in process
    chunk=signal_Q.get(block=False, timeout=10)
  File "/home/zyf/miniforge3/envs/deepmod2/lib/python3.13/multiprocessing/process.py", line 313, in _bootstrap
    self.run()
    ~~~~~~~~^^
  File "<string>", line 2, in get
  File "/home/zyf/miniforge3/envs/deepmod2/lib/python3.13/multiprocessing/managers.py", line 831, in _callmethod
    kind, result = conn.recv()
                   ~~~~~~~~~^^
  File "/home/zyf/miniforge3/envs/deepmod2/lib/python3.13/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/zyf/miniforge3/envs/deepmod2/lib/python3.13/multiprocessing/connection.py", line 250, in recv
    buf = self._recv_bytes()
  File "/home/zyf/miniforge3/envs/deepmod2/lib/python3.13/multiprocessing/connection.py", line 430, in _recv_bytes
    buf = self._recv(4)
  File "/home/zyf/eqa/resources/DeepMod2/src/detect.py", line 503, in process
    if (signal_Q.empty() and input_event.is_set()):
        ~~~~~~~~~~~~~~^^
  File "/home/zyf/miniforge3/envs/deepmod2/lib/python3.13/multiprocessing/connection.py", line 399, in _recv
    raise EOFError
  File "<string>", line 2, in empty
  File "/home/zyf/miniforge3/envs/deepmod2/lib/python3.13/multiprocessing/managers.py", line 831, in _callmethod
    kind, result = conn.recv()
                   ~~~~~~~~~^^
  File "/home/zyf/miniforge3/envs/deepmod2/lib/python3.13/multiprocessing/connection.py", line 250, in recv
    buf = self._recv_bytes()
  File "/home/zyf/miniforge3/envs/deepmod2/lib/python3.13/multiprocessing/connection.py", line 430, in _recv_bytes
    buf = self._recv(4)
  File "/home/zyf/miniforge3/envs/deepmod2/lib/python3.13/multiprocessing/process.py", line 313, in _bootstrap
    self.run()
    ~~~~~~~~^^
  File "/home/zyf/miniforge3/envs/deepmod2/lib/python3.13/multiprocessing/connection.py", line 395, in _recv
    chunk = read(handle, remaining)
EOFError
  File "/home/zyf/miniforge3/envs/deepmod2/lib/python3.13/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/zyf/eqa/resources/DeepMod2/src/detect.py", line 503, in process
    if (signal_Q.empty() and input_event.is_set()):
        ~~~~~~~~~~~~~~^^
  File "<string>", line 2, in empty
ConnectionResetError: [Errno 104] Connection reset by peer
  File "/home/zyf/miniforge3/envs/deepmod2/lib/python3.13/multiprocessing/managers.py", line 831, in _callmethod
    kind, result = conn.recv()
                   ~~~~~~~~~^^
  File "/home/zyf/miniforge3/envs/deepmod2/lib/python3.13/multiprocessing/connection.py", line 250, in recv
    buf = self._recv_bytes()
  File "/home/zyf/miniforge3/envs/deepmod2/lib/python3.13/multiprocessing/connection.py", line 430, in _recv_bytes
    buf = self._recv(4)
  File "/home/zyf/miniforge3/envs/deepmod2/lib/python3.13/multiprocessing/connection.py", line 395, in _recv
    chunk = read(handle, remaining)
ConnectionResetError: [Errno 104] Connection reset by peer
  File "/home/zyf/miniforge3/envs/deepmod2/lib/python3.13/multiprocessing/process.py", line 313, in _bootstrap
    self.run()
    ~~~~~~~~^^
  File "/home/zyf/miniforge3/envs/deepmod2/lib/python3.13/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/zyf/eqa/resources/DeepMod2/src/detect.py", line 503, in process
    if (signal_Q.empty() and input_event.is_set()):
        ~~~~~~~~~~~~~~^^
  File "<string>", line 2, in empty
  File "/home/zyf/miniforge3/envs/deepmod2/lib/python3.13/multiprocessing/managers.py", line 831, in _callmethod
    kind, result = conn.recv()
                   ~~~~~~~~~^^
  File "/home/zyf/miniforge3/envs/deepmod2/lib/python3.13/multiprocessing/connection.py", line 250, in recv
    buf = self._recv_bytes()
  File "/home/zyf/miniforge3/envs/deepmod2/lib/python3.13/multiprocessing/connection.py", line 430, in _recv_bytes
    buf = self._recv(4)
  File "/home/zyf/miniforge3/envs/deepmod2/lib/python3.13/multiprocessing/connection.py", line 399, in _recv
    raise EOFError
EOFError
  File "/home/zyf/miniforge3/envs/deepmod2/lib/python3.13/multiprocessing/process.py", line 313, in _bootstrap
    self.run()
    ~~~~~~~~^^
  File "/home/zyf/miniforge3/envs/deepmod2/lib/python3.13/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/zyf/eqa/resources/DeepMod2/src/detect.py", line 503, in process
    if (signal_Q.empty() and input_event.is_set()):
                             ~~~~~~~~~~~~~~~~~~^^
  File "/home/zyf/miniforge3/envs/deepmod2/lib/python3.13/multiprocessing/managers.py", line 1103, in is_set
    return self._callmethod('is_set')
           ~~~~~~~~~~~~~~~~^^^^^^^^^^
  File "/home/zyf/miniforge3/envs/deepmod2/lib/python3.13/multiprocessing/managers.py", line 831, in _callmethod
    kind, result = conn.recv()
                   ~~~~~~~~~^^
  File "/home/zyf/miniforge3/envs/deepmod2/lib/python3.13/multiprocessing/connection.py", line 250, in recv
    buf = self._recv_bytes()
  File "/home/zyf/miniforge3/envs/deepmod2/lib/python3.13/multiprocessing/connection.py", line 430, in _recv_bytes
    buf = self._recv(4)
  File "/home/zyf/miniforge3/envs/deepmod2/lib/python3.13/multiprocessing/connection.py", line 399, in _recv
    raise EOFError
EOFError
  File "/home/zyf/miniforge3/envs/deepmod2/lib/python3.13/multiprocessing/process.py", line 313, in _bootstrap
    self.run()
    ~~~~~~~~^^
  File "/home/zyf/miniforge3/envs/deepmod2/lib/python3.13/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/zyf/eqa/resources/DeepMod2/src/detect.py", line 503, in process
    if (signal_Q.empty() and input_event.is_set()):
        ~~~~~~~~~~~~~~^^
  File "<string>", line 2, in empty
  File "/home/zyf/miniforge3/envs/deepmod2/lib/python3.13/multiprocessing/managers.py", line 831, in _callmethod
    kind, result = conn.recv()
                   ~~~~~~~~~^^
  File "/home/zyf/miniforge3/envs/deepmod2/lib/python3.13/multiprocessing/connection.py", line 250, in recv
    buf = self._recv_bytes()
  File "/home/zyf/miniforge3/envs/deepmod2/lib/python3.13/multiprocessing/connection.py", line 430, in _recv_bytes
    buf = self._recv(4)
  File "/home/zyf/miniforge3/envs/deepmod2/lib/python3.13/multiprocessing/connection.py", line 395, in _recv
    chunk = read(handle, remaining)
ConnectionResetError: [Errno 104] Connection reset by peer
Process Process-11:
Traceback (most recent call last):
  File "/home/zyf/miniforge3/envs/deepmod2/lib/python3.13/multiprocessing/process.py", line 313, in _bootstrap
    self.run()
    ~~~~~~~~^^
  File "/home/zyf/miniforge3/envs/deepmod2/lib/python3.13/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/zyf/eqa/resources/DeepMod2/src/detect.py", line 195, in get_cpg_output
    if methylation_event.is_set() and output_Q.empty():
       ~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/home/zyf/miniforge3/envs/deepmod2/lib/python3.13/multiprocessing/managers.py", line 1103, in is_set
    return self._callmethod('is_set')
           ~~~~~~~~~~~~~~~~^^^^^^^^^^
  File "/home/zyf/miniforge3/envs/deepmod2/lib/python3.13/multiprocessing/managers.py", line 831, in _callmethod
    kind, result = conn.recv()
                   ~~~~~~~~~^^
  File "/home/zyf/miniforge3/envs/deepmod2/lib/python3.13/multiprocessing/connection.py", line 250, in recv
    buf = self._recv_bytes()
  File "/home/zyf/miniforge3/envs/deepmod2/lib/python3.13/multiprocessing/connection.py", line 430, in _recv_bytes
    buf = self._recv(4)
  File "/home/zyf/miniforge3/envs/deepmod2/lib/python3.13/multiprocessing/connection.py", line 399, in _recv
    raise EOFError
EOFError
Process Process-10:
Traceback (most recent call last):
  File "/home/zyf/miniforge3/envs/deepmod2/lib/python3.13/multiprocessing/process.py", line 313, in _bootstrap
    self.run()
    ~~~~~~~~^^
  File "/home/zyf/miniforge3/envs/deepmod2/lib/python3.13/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/zyf/eqa/resources/DeepMod2/src/detect.py", line 766, in get_input
    signal_Q.put(chunk)
    ~~~~~~~~~~~~^^^^^^^
  File "<string>", line 2, in put
  File "/home/zyf/miniforge3/envs/deepmod2/lib/python3.13/multiprocessing/managers.py", line 831, in _callmethod
    kind, result = conn.recv()
                   ~~~~~~~~~^^
  File "/home/zyf/miniforge3/envs/deepmod2/lib/python3.13/multiprocessing/connection.py", line 250, in recv
    buf = self._recv_bytes()
  File "/home/zyf/miniforge3/envs/deepmod2/lib/python3.13/multiprocessing/connection.py", line 430, in _recv_bytes
    buf = self._recv(4)
  File "/home/zyf/miniforge3/envs/deepmod2/lib/python3.13/multiprocessing/connection.py", line 399, in _recv
    raise EOFError
EOFError
2025-08-06 23:33:10.868783: Reading inputs complete.
Traceback (most recent call last):
  File "/home/zyf/miniforge3/envs/deepmod2/lib/python3.13/multiprocessing/managers.py", line 823, in _callmethod
    conn = self._tls.connection
           ^^^^^^^^^^^^^^^^^^^^
AttributeError: 'ForkAwareLocal' object has no attribute 'connection'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/home/zyf/eqa/resources/DeepMod2/deepmod2", line 172, in <module>
    detect.call_manager(params)
    ~~~~~~~~~~~~~~~~~~~^^^^^^^^
  File "/home/zyf/eqa/resources/DeepMod2/src/detect.py", line 864, in call_manager
    methylation_event.set()
    ~~~~~~~~~~~~~~~~~~~~~^^
  File "/home/zyf/miniforge3/envs/deepmod2/lib/python3.13/multiprocessing/managers.py", line 1105, in set
    return self._callmethod('set')
           ~~~~~~~~~~~~~~~~^^^^^^^
  File "/home/zyf/miniforge3/envs/deepmod2/lib/python3.13/multiprocessing/managers.py", line 827, in _callmethod
    self._connect()
    ~~~~~~~~~~~~~^^
  File "/home/zyf/miniforge3/envs/deepmod2/lib/python3.13/multiprocessing/managers.py", line 814, in _connect
    conn = self._Client(self._token.address, authkey=self._authkey)
  File "/home/zyf/miniforge3/envs/deepmod2/lib/python3.13/multiprocessing/connection.py", line 519, in Client
    c = SocketClient(address)
  File "/home/zyf/miniforge3/envs/deepmod2/lib/python3.13/multiprocessing/connection.py", line 647, in SocketClient
    s.connect(address)
    ~~~~~~~~~^^^^^^^^^
ConnectionRefusedError: [Errno 111] Connection refused

Aug 07 '25 01:08 YuanfengZhang