DeepChainBio
a worker died or was killed while executing a task by an unexpected system error
I use 4 GPUs to calculate MSA embeddings, but each time the process terminated, the error was raise by ray, the error message is " a worker died or was killed while executing a task by an unexpected system error", the GPU process terminated one by one, I tried several times, I update ray with lastest version, the problem is same. How can I treat the problem? Thanks!
I tried again and just use CPU to calculate embeddings. and I found it still teminated. I check esm document and it is said that some problem with model esm_msa1_t12_100M_UR50S and recommend using esm_msa1b_t12_100M_UR50S, but I can't find where to modify the code to use esm_msa1b_t12_100M_UR50S, could somebody tell me? thanks.
I tried again and just use CPU to calculate embeddings. and I found it still teminated. I check esm document and it is said that some problem with model esm_msa1_t12_100M_UR50S and recommend using esm_msa1b_t12_100M_UR50S, but I can't find where to modify the code to use esm_msa1b_t12_100M_UR50S, could somebody tell me? thanks.
I use 4 GPUs to calculate MSA embeddings, but each time the process terminated, the error was raise by ray, the error message is " a worker died or was killed while executing a task by an unexpected system error", the GPU process terminated one by one, I tried several times, I update ray with lastest version, the problem is same. How can I treat the problem? Thanks!
I will check this, it seems there is an issue with Ray.