Couldn't find union_boxes.conv.2.num_batches_tracked,union_boxes.conv.6.num_batches_tracked

[bibekyess]

Hello, Thanks for this awesome repo. While running ./scripts/eval_kern_sgdet.sh, I am getting the following error:

• [ ] We couldn't find union_boxes.conv.2.num_batches_tracked,union_boxes.conv.6.num_batches_tracked 0%| | 0/26446 [00:00<?, ?it/s]/home/riro/bibek_repo/KERN/dataloaders/blob.py:129: UserWarning: volatile was removed and now has no effect. Use with torch.no_grad(): instead. self.imgs = Variable(torch.stack(self.imgs, 0), volatile=self.volatile) /home/riro/bibek_repo/KERN/dataloaders/blob.py:120: UserWarning: volatile was removed and now has no effect. Use with torch.no_grad(): instead. return Variable(tensor(np.concatenate(datom, 0)), volatile=self.volatile), chunk_sizes THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch_1535491974311/work/aten/src/THC/THCGeneral.cpp line=663 error=8 : invalid device function 0%| | 0/26446 [00:00<?, ?it/s] Traceback (most recent call last): File "models/eval_rels.py", line 114, in val_batch(conf.num_gpus*val_b, batch, evaluator, evaluator_multiple_preds, evaluator_list, evaluator_multiple_preds_list) File "models/eval_rels.py", line 55, in val_batch det_res = detector[b] File "/home/riro/bibek_repo/KERN/lib/kern_model.py", line 423, in getitem return self(*batch[0]) File "/home/riro/anaconda3/envs/kern/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in call result = self.forward(*input, **kwargs) File "/home/riro/bibek_repo/KERN/lib/kern_model.py", line 355, in forward train_anchor_inds, return_fmap=True) File "/home/riro/anaconda3/envs/kern/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in call result = self.forward(*input, **kwargs) File "/home/riro/bibek_repo/KERN/lib/object_detector.py", line 293, in forward fmap = self.feature_map(x) File "/home/riro/bibek_repo/KERN/lib/object_detector.py", line 119, in feature_map return self.features(x) # Uncomment this for "stanford" setting in which it's frozen: .detach() File "/home/riro/anaconda3/envs/kern/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in call result = self.forward(*input, **kwargs) File "/home/riro/anaconda3/envs/kern/lib/python3.6/site-packages/torch/nn/modules/container.py", line 91, in forward input = module(input) File "/home/riro/anaconda3/envs/kern/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in call result = self.forward(*input, **kwargs) File "/home/riro/anaconda3/envs/kern/lib/python3.6/site-packages/torch/nn/modules/conv.py", line 301, in forward self.padding, self.dilation, self.groups) RuntimeError: CuDNN error: CUDNN_STATUS_EXECUTION_FAILED

Any help on how to solve this issue? Thank you!