AlphaFold Stuck at hhblits Step on Cluster Compute Node

Open Harryyang21 opened this issue 2 years ago • 0 comments

Issue Description: I am experiencing a problem running AlphaFold on a compute node in our cluster. The process consistently gets stuck at the hhblits step. However, when I run the same program directly on the login node, it proceeds without any issues. The issue arises specifically when submitting the job to the compute node - it hangs at "- 22:54:01.709 INFO: Prefiltering database" and does not progress further.

Additionally, system call tracing shows repeated occurrences of the following:

futex(0x2b5ff13dd634, FUTEX_WAIT_PRIVATE, 4294967295, NULL) = 0 futex(0x2b5ff13dd634, FUTEX_WAKE_PRIVATE, 1) = 0 ... During this time, there is no additional memory load, and the GPU does not appear to be computing, although the program itself seems to have a load.

22:15:22.605 INFO: Searching 32053680 column state sequences.

22:15:22.729 INFO: /tmp/yanghao2022/MSA_4508283962/seq.fasta is in A2M, A3M or FASTA format

22:15:22.730 INFO: Iteration 1

22:15:22.808 INFO: Prefiltering database

22:16:19.797 INFO: HMMs passed 1st prefilter (gapless profile-profile alignment) : 693794

22:16:25.614 INFO: HMMs passed 2nd prefilter (gapped profile-profile alignment) : 292

22:16:25.614 INFO: HMMs passed 2nd prefilter and not found in previous iterations : 292

22:16:25.614 INFO: Scoring 292 HMMs using HMM-HMM Viterbi alignment

22:16:26.110 INFO: Alternative alignment: 0

22:16:31.556 INFO: 292 alignments done

22:16:31.559 INFO: Alternative alignment: 1

22:16:31.623 INFO: 287 alignments done

22:16:31.624 INFO: Alternative alignment: 2

22:16:31.648 INFO: 20 alignments done

22:16:31.648 INFO: Alternative alignment: 3

22:16:31.679 INFO: 3 alignments done

22:16:31.984 INFO: Realigning 210 HMM-HMM alignments using Maximum Accuracy algorithm

22:16:33.013 INFO: 77 sequences belonging to 77 database HMMs found with an E-value < 0.001

22:16:33.013 INFO: Number of effective sequences of resulting query HMM: Neff = 5.92897

22:16:33.040 INFO: Iteration 2

Environment Description: _libgcc_mutex 0.1 main defaults _openmp_mutex 5.1 1_gnu defaults absl-py 0.13.0 pypi_0 pypi astunparse 1.6.3 pypi_0 pypi biopython 1.79 pypi_0 pypi ca-certificates 2023.12.12 h06a4308_0 defaults cachetools 5.3.2 pypi_0 pypi certifi 2023.11.17 pypi_0 pypi charset-normalizer 3.3.2 pypi_0 pypi chex 0.0.7 pypi_0 pypi click 8.1.7 pypi_0 pypi contextlib2 21.6.0 pypi_0 pypi cudatoolkit 11.3.1 h9edb442_10 conda-forge cudatoolkit-dev 11.3.1 py38h497a2fe_0 conda-forge cudnn 8.2.1.32 h86fa8c9_0 conda-forge dm-haiku 0.0.4 pypi_0 pypi dm-tree 0.1.6 pypi_0 pypi fftw 3.3.10 nompi_h77c792f_102 conda-forge flatbuffers 1.12 pypi_0 pypi gast 0.4.0 pypi_0 pypi google-auth 2.26.1 pypi_0 pypi google-auth-oauthlib 0.4.6 pypi_0 pypi google-pasta 0.2.0 pypi_0 pypi grpcio 1.34.1 pypi_0 pypi h5py 3.1.0 pypi_0 pypi hhsuite 3.3.0 py38pl5321h8ded8fe_5 bioconda hmmer 3.3.2 h87f3376_2 bioconda idna 3.6 pypi_0 pypi immutabledict 2.0.0 pypi_0 pypi importlib-metadata 7.0.1 pypi_0 pypi jax 0.2.14 pypi_0 pypi jaxlib 0.1.69+cuda111 pypi_0 pypi kalign2 2.04 hec16e2b_3 bioconda keras-nightly 2.5.0.dev2021032900 pypi_0 pypi keras-preprocessing 1.1.2 pypi_0 pypi libblas 3.9.0 15_linux64_openblas conda-forge libcblas 3.9.0 15_linux64_openblas conda-forge libedit 3.1.20230828 h5eee18b_0 defaults libffi 3.2.1 hf484d3e_1007 defaults libgcc-ng 11.2.0 h1234567_1 defaults libgfortran-ng 13.2.0 h69a702a_0 conda-forge libgfortran5 13.2.0 ha4646dd_0 conda-forge libgomp 11.2.0 h1234567_1 defaults liblapack 3.9.0 15_linux64_openblas conda-forge libnsl 2.0.0 h5eee18b_0 defaults libopenblas 0.3.20 pthreads_h78a6416_0 conda-forge libstdcxx-ng 11.2.0 h1234567_1 defaults markdown 3.5.1 pypi_0 pypi markupsafe 2.1.3 pypi_0 pypi ml-collections 0.1.0 pypi_0 pypi ncurses 6.4 h6a678d5_0 defaults numpy 1.19.5 pypi_0 pypi oauthlib 3.2.2 pypi_0 pypi ocl-icd 2.3.1 h7f98852_0 conda-forge ocl-icd-system 1.0.0 1 conda-forge openmm 7.5.1 py38ha082873_1 conda-forge openssl 1.1.1w h7f8727e_0 defaults opt-einsum 3.3.0 pypi_0 pypi pandas 1.3.4 pypi_0 pypi pdbfixer 1.7 pyhd3deb0d_0 conda-forge perl 5.32.1 0_h5eee18b_perl5 defaults pillow 10.2.0 pypi_0 pypi pip 23.3.2 pypi_0 pypi protobuf 3.20.3 pypi_0 pypi pyasn1 0.5.1 pypi_0 pypi pyasn1-modules 0.3.0 pypi_0 pypi python 3.8.0 h0371630_2 defaults python-dateutil 2.8.2 pypi_0 pypi python_abi 3.8 2_cp38 conda-forge pytz 2023.3.post1 pypi_0 pypi pyyaml 6.0.1 pypi_0 pypi readline 7.0 h7b6447c_5 defaults requests 2.31.0 pypi_0 pypi requests-oauthlib 1.3.1 pypi_0 pypi rsa 4.9 pypi_0 pypi scipy 1.7.0 pypi_0 pypi setuptools 68.2.2 py38h06a4308_0 defaults six 1.15.0 pypi_0 pypi sqlite 3.33.0 h62c20be_0 defaults svgwrite 1.4.3 pypi_0 pypi tabulate 0.9.0 pypi_0 pypi tensorboard 2.11.2 pypi_0 pypi tensorboard-data-server 0.6.1 pypi_0 pypi tensorboard-plugin-wit 1.8.1 pypi_0 pypi tensorflow 2.5.0 pypi_0 pypi tensorflow-cpu 2.5.0 pypi_0 pypi tensorflow-estimator 2.5.0 pypi_0 pypi termcolor 1.1.0 pypi_0 pypi tk 8.6.12 h1ccaba5_0 defaults toolz 0.12.0 pypi_0 pypi tree 0.2.4 pypi_0 pypi typing-extensions 3.7.4.3 pypi_0 pypi urllib3 2.1.0 pypi_0 pypi werkzeug 3.0.1 pypi_0 pypi wheel 0.41.2 py38h06a4308_0 defaults wrapt 1.12.1 pypi_0 pypi xz 5.4.5 h5eee18b_0 defaults zipp 3.17.0 pypi_0 pypi zlib 1.2.13 h5eee18b_0 defaults

AlphaFold Version: 2.3.2 Operating System and Version: CentOS Linux release 7.6.1810 (Core) Thank you very much in advance

Jan 07 '24 14:01 Harryyang21