microsoft
Unknown Error
I am getting some mysterious error while running demo.py. +-----------------------------------------------------------------------------+ | NVIDIA-SMI 495.29.05 Driver Version: 495.29.05 CUDA Version: 11.5 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |===============================+======================+======================| | 0 NVIDIA RTX A4000 On | 00000000:17:00.0 Off | Off | | 68% 86C P2 129W / 140W | 12372MiB / 16117MiB | 54% Default | | | | N/A | +-------------------------------+----------------------+----------------------+ | 1 NVIDIA RTX A4000 On | 00000000:65:00.0 Off | Off | | 41% 49C P8 15W / 140W | 3MiB / 16108MiB | 0% Default | | | | N/A | +-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=============================================================================|
We have a gpu server and I am using device = 1. CUDA_VISIBLE_DEVICES=1 python demo.py and I am getting these errors
ERROR:tensorflow:==================================
Object was never used (type <class 'tensorflow.python.framework.ops.Operation'>):
<tf.Operation 'assert_greater/Assert/AssertGuard/Merge' type=Merge>
If you want to mark it as used call its "mark_used()" method.
It was originally created here:
File "demo.py", line 124, in
demo() File "demo.py", line 82, in demo
FaceReconstructor.Reconstruction_Block(coeff,opt) File "/mnt/NLPStorage/parikshat/Deep3DFaceReconstruction/face_decoder.py", line 63, in Reconstruction_Block
render_imgs,img_mask,img_mask_crop = self.Render_block(face_shape_t,norm_r,face_color,camera_scale,f_scale,self.facemodel,opt.batch_size,opt.is_train) File "/mnt/NLPStorage/parikshat/Deep3DFaceReconstruction/face_decoder.py", line 312, in Render_block
ambient_color = ambient_color) File "/mnt/NLPStorage/parikshat/Deep3DFaceReconstruction/renderer/mesh_renderer.py", line 364, in mesh_renderer
camera_up) File "/mnt/NLPStorage/parikshat/Deep3DFaceReconstruction/renderer/../renderer/camera_utils.py", line 89, in look_at
message='Camera matrix is degenerate because eye and center are close.') File "/home/ubuntu/anaconda3/envs/deep3d/lib/python3.6/site-packages/tensorflow/python/ops/check_ops.py", line 666, in assert_greater
return control_flow_ops.Assert(condition, data, summarize=summarize) File "/home/ubuntu/anaconda3/envs/deep3d/lib/python3.6/site-packages/tensorflow/python/util/tf_should_use.py", line 189, in wrapped
return _add_should_use_warning(fn(*args, **kwargs))
ERROR:tensorflow:==================================
Object was never used (type <class 'tensorflow.python.framework.ops.Operation'>):
<tf.Operation 'assert_greater_1/Assert/AssertGuard/Merge' type=Merge>
If you want to mark it as used call its "mark_used()" method.
It was originally created here:
File "demo.py", line 124, in
demo() File "demo.py", line 82, in demo
FaceReconstructor.Reconstruction_Block(coeff,opt) File "/mnt/NLPStorage/parikshat/Deep3DFaceReconstruction/face_decoder.py", line 63, in Reconstruction_Block
render_imgs,img_mask,img_mask_crop = self.Render_block(face_shape_t,norm_r,face_color,camera_scale,f_scale,self.facemodel,opt.batch_size,opt.is_train) File "/mnt/NLPStorage/parikshat/Deep3DFaceReconstruction/face_decoder.py", line 312, in Render_block
ambient_color = ambient_color) File "/mnt/NLPStorage/parikshat/Deep3DFaceReconstruction/renderer/mesh_renderer.py", line 364, in mesh_renderer
camera_up) File "/mnt/NLPStorage/parikshat/Deep3DFaceReconstruction/renderer/../renderer/camera_utils.py", line 97, in look_at
message='Camera matrix is degenerate because up and gaze are close or' File "/home/ubuntu/anaconda3/envs/deep3d/lib/python3.6/site-packages/tensorflow/python/ops/check_ops.py", line 666, in assert_greater
return control_flow_ops.Assert(condition, data, summarize=summarize) File "/home/ubuntu/anaconda3/envs/deep3d/lib/python3.6/site-packages/tensorflow/python/util/tf_should_use.py", line 189, in wrapped
return _add_should_use_warning(fn(*args, **kwargs))
2022-02-09 07:13:27.482419: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 AVX512F FMA
2022-02-09 07:13:27.607719: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 0 with properties:
name: NVIDIA RTX A4000 major: 8 minor: 6 memoryClockRate(GHz): 1.56
pciBusID: 0000:17:00.0
totalMemory: 15.74GiB freeMemory: 3.50GiB
2022-02-09 07:13:27.607743: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Adding visible gpu devices: 0
2022-02-09 07:16:30.596270: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device interconnect StreamExecutor with strength 1 edge matrix:
2022-02-09 07:16:30.596291: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988] 0
2022-02-09 07:16:30.596295: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0: N
2022-02-09 07:16:30.596389: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3203 MB memory) -> physical GPU (device: 0, name: NVIDIA RTX A4000, pci bus id: 0000:17:00.0, compute capability: 8.6)
reconstructing...
1
/mnt/NLPStorage/parikshat/Deep3DFaceReconstruction/preprocess_img.py:27: FutureWarning: rcond parameter will change to the default of machine precision times max(M, N) where M and N are the input matrix dimensions.
To use the future default and silence this warning we advise to pass rcond=None, to keep using the old, explicitly pass rcond=-1.
k,,,_ = np.linalg.lstsq(A,b)
/mnt/NLPStorage/parikshat/Deep3DFaceReconstruction/preprocess_img.py:74: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray
trans_params = np.array([w0,h0,102.0/s,t[0],t[1]])
2022-02-09 07:31:11.340602: E tensorflow/stream_executor/cuda/cuda_blas.cc:652] failed to run cuBLAS routine cublasSgemm_v2: CUBLAS_STATUS_EXECUTION_FAILED
Traceback (most recent call last):
File "/home/ubuntu/anaconda3/envs/deep3d/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1334, in _do_call
return fn(*args)
File "/home/ubuntu/anaconda3/envs/deep3d/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1319, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "/home/ubuntu/anaconda3/envs/deep3d/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1407, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.InternalError: Blas xGEMM launch failed : a.shape=[1,35709,3], b.shape=[1,3,3], m=35709, n=3, k=3
[[{{node MatMul_3}} = BatchMatMul[T=DT_FLOAT, adj_x=false, adj_y=false, _device="/job:localhost/replica:0/task:0/device:GPU:0"](sub, transpose)]]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "demo.py", line 124, in
Caused by op 'MatMul_3', defined at:
File "demo.py", line 124, in
InternalError (see above for traceback): Blas xGEMM launch failed : a.shape=[1,35709,3], b.shape=[1,3,3], m=35709, n=3, k=3 [[node MatMul_3 (defined at /mnt/NLPStorage/parikshat/Deep3DFaceReconstruction/face_decoder.py:258) = BatchMatMul[T=DT_FLOAT, adj_x=false, adj_y=false, _device="/job:localhost/replica:0/task:0/device:GPU:0"](sub, transpose)]]
I cannot understand what's wrong. Can somebody help me out. @YuDeng
