RuntimeError: CUDA error: an illegal memory access was encountered

[HengyiWang]

Hi Steven,

When I try to integrate your treefilterv2 into my network, it causes the RuntimeError: CUDA error: an illegal memory access was encountered:

File "TfTest.py", line 238, in main() File "TfTest.py", line 212, in main eval_loss, pred_l, attn_l = eval(ms, m4_all, r3e_all, r2e_all, Ms_all, scale_space, num_frames, device1) File "TfTest.py", line 117, in eval out, attn = model.forward(m4_in, r3e_in, r2e_in, mask, scale_space, n_scale=True) File "/homes/hw002/Test/testloader/Tree.py", line 269, in forward return self.nscale_forward(inputs, r3e, r2e, gts, scales) File "/homes/hw002/Test/testloader/Tree.py", line 185, in nscale_forward p, feats = self.Decoder(inputs[str(s)], r3e[str(s)], r2e[str(s)]) # .item() File "/homes/hw002/.conda/envs/pytorch36/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in call_impl result = self.forward(*input, **kwargs) File "/homes/hw002/Test/testloader/Tree.py", line 140, in forward m3 = self.tf2(m3, r2) File "/homes/hw002/.conda/envs/pytorch36/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl result = self.forward(*input, **kwargs) File "/homes/hw002/LearnableTreeFilterV2/cvpods/layers/tree_filter_v2.py", line 78, in forward feature = self.tree_filter_layer(feature, embed, tree, guide_embed, beta) File "/homes/hw002/.conda/envs/pytorch36/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl result = self.forward(*input, **kwargs) File "/homes/hw002/LearnableTreeFilterV2/cvpods/layers/tree_filter_core.py", line 235, in forward sorted_parent, sorted_child) File "/homes/hw002/LearnableTreeFilterV2/cvpods/layers/tree_filter_core.py", line 47, in forward _C.tree_filter_refine_forward(feature_in, edge_weight, self_weight, sorted_index, sorted_parent, sorted_child) RuntimeError: CUDA error: an illegal memory access was encountered

It is wired since this happens during my test time, if I use only 1 treefilter layer, my network will stop at frame 42 of the test sequence and if I use 2 treefilters, it will stop at the frame 21 of the test sequence.

Could you help me to find out the reason why this happens? Thanks a lot for your help, I wish you all the best!