text2mesh icon indicating copy to clipboard operation
text2mesh copied to clipboard

CUDA error: CUBLAS_STATUS_INVALID_VALUE when calling `cublasGemmEx

Open qxzha opened this issue 2 years ago • 0 comments

when I run the sh script, I got an error:

./demo/run_alien_cobble.sh


Traceback (most recent call last): File "/home/zqx/text2mesh/main.py", line 499, in run_branched(args) File "/home/zqx/text2mesh/main.py", line 140, in run_branched encoded_text = clip_model.encode_text(prompt_token) File "/home/zqx/miniconda3/envs/text2mesh/lib/python3.9/site-packages/clip/model.py", line 348, in encode_text x = self.transformer(x) File "/home/zqx/miniconda3/envs/text2mesh/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl return forward_call(*input, **kwargs) File "/home/zqx/miniconda3/envs/text2mesh/lib/python3.9/site-packages/clip/model.py", line 203, in forward return self.resblocks(x) File "/home/zqx/miniconda3/envs/text2mesh/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl return forward_call(*input, **kwargs) File "/home/zqx/miniconda3/envs/text2mesh/lib/python3.9/site-packages/torch/nn/modules/container.py", line 139, in forward input = module(input) File "/home/zqx/miniconda3/envs/text2mesh/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl return forward_call(*input, **kwargs) File "/home/zqx/miniconda3/envs/text2mesh/lib/python3.9/site-packages/clip/model.py", line 190, in forward x = x + self.attention(self.ln_1(x)) File "/home/zqx/miniconda3/envs/text2mesh/lib/python3.9/site-packages/clip/model.py", line 187, in attention return self.attn(x, x, x, need_weights=False, attn_mask=self.attn_mask)[0] File "/home/zqx/miniconda3/envs/text2mesh/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl return forward_call(*input, **kwargs) File "/home/zqx/miniconda3/envs/text2mesh/lib/python3.9/site-packages/torch/nn/modules/activation.py", line 1153, in forward attn_output, attn_output_weights = F.multi_head_attention_forward( File "/home/zqx/miniconda3/envs/text2mesh/lib/python3.9/site-packages/torch/nn/functional.py", line 5066, in multi_head_attention_forward q, k, v = _in_projection_packed(query, key, value, in_proj_weight, in_proj_bias) File "/home/zqx/miniconda3/envs/text2mesh/lib/python3.9/site-packages/torch/nn/functional.py", line 4745, in _in_projection_packed return linear(q, w, b).chunk(3, dim=-1) RuntimeError: CUDA error: CUBLAS_STATUS_INVALID_VALUE when calling cublasGemmEx( handle, opa, opb, m, n, k, &falpha, a, CUDA_R_16F, lda, b, CUDA_R_16F, ldb, &fbeta, c, CUDA_R_16F, ldc, CUDA_R_32F, CUBLAS_GEMM_DFALT_TENSOR_OP)


Anyone knows how to fix it, please tell me.

qxzha avatar Jul 21 '23 05:07 qxzha