DEIM “nan" problem in training custom dataset

Epoch: [18] [3500/4384] eta: 0:08:08 lr: 0.000003 loss: 31.5320 (30.7038) loss_vfl: 0.6807 (0.6671) loss_bbox: 0.1400 (0.1503) loss_giou: 0.4664 (0.4601) loss_fgl: 1.0752 (1.0601) loss_vfl_aux_0: 0.7646 (0.7457) loss_bbox_aux_0: 0.1571 (0.1619) loss_giou_aux_0: 0.5047 (0.4856) loss_fgl_aux_0: 1.1022 (1.0939) loss_ddf_aux_0: 0.0534 (0.0495) loss_vfl_aux_1: 0.7661 (0.7237) loss_bbox_aux_1: 0.1372 (0.1529) loss_giou_aux_1: 0.4784 (0.4662) loss_fgl_aux_1: 1.0835 (1.0638) loss_ddf_aux_1: 0.0085 (0.0080) loss_vfl_aux_2: 0.7612 (0.7056) loss_bbox_aux_2: 0.1392 (0.1508) loss_giou_aux_2: 0.4685 (0.4611) loss_fgl_aux_2: 1.0754 (1.0603) loss_ddf_aux_2: 0.0014 (0.0015) loss_vfl_aux_3: 0.7124 (0.6810) loss_bbox_aux_3: 0.1400 (0.1503) loss_giou_aux_3: 0.4668 (0.4602) loss_fgl_aux_3: 1.0765 (1.0602) loss_ddf_aux_3: 0.0002 (0.0003) loss_vfl_aux_4: 0.6865 (0.6691) loss_bbox_aux_4: 0.1401 (0.1503) loss_giou_aux_4: 0.4664 (0.4601) loss_fgl_aux_4: 1.0752 (1.0601) loss_ddf_aux_4: 0.0001 (0.0001) loss_vfl_pre: 0.7646 (0.7465) loss_bbox_pre: 0.1566 (0.1618) loss_giou_pre: 0.5061 (0.4844) loss_vfl_enc_0: 0.7573 (0.7581) loss_bbox_enc_0: 0.1973 (0.1956) loss_giou_enc_0: 0.5716 (0.5552) loss_vfl_dn_0: 0.5078 (0.4945) loss_bbox_dn_0: 0.2006 (0.2113) loss_giou_dn_0: 0.4881 (0.4648) loss_fgl_dn_0: 1.1960 (1.2022) loss_ddf_dn_0: 0.2418 (0.2392) loss_vfl_dn_1: 0.4163 (0.4028) loss_bbox_dn_1: 0.1490 (0.1491) loss_giou_dn_1: 0.3783 (0.3514) loss_fgl_dn_1: 1.0866 (1.0820) loss_ddf_dn_1: 0.0355 (0.0327) loss_vfl_dn_2: 0.4009 (0.3814) loss_bbox_dn_2: 0.1394 (0.1393) loss_giou_dn_2: 0.3552 (0.3362) loss_fgl_dn_2: 1.0661 (1.0647) loss_ddf_dn_2: 0.0066 (0.0064) loss_vfl_dn_3: 0.3857 (0.3712) loss_bbox_dn_3: 0.1330 (0.1367) loss_giou_dn_3: 0.3452 (0.3318) loss_fgl_dn_3: 1.0660 (1.0648) loss_ddf_dn_3: 0.0003 (0.0003) loss_vfl_dn_4: 0.3799 (0.3678) loss_bbox_dn_4: 0.1316 (0.1363) loss_giou_dn_4: 0.3448 (0.3314) loss_fgl_dn_4: 1.0670 (1.0650) loss_ddf_dn_4: 0.0000 (0.0000) loss_vfl_dn_5: 0.3794 (0.3678) loss_bbox_dn_5: 0.1316 (0.1363) loss_giou_dn_5: 0.3448 (0.3314) loss_fgl_dn_5: 1.0670 (1.0650) loss_vfl_dn_pre: 0.5098 (0.4947) loss_bbox_dn_pre: 0.2113 (0.2182) loss_giou_dn_pre: 0.4908 (0.4658) time: 0.4990 data: 0.0280 max mem: 20978 tensor([[[nan, nan, nan, nan], [nan, nan, nan, nan], [nan, nan, nan, nan], ..., [nan, nan, nan, nan], [nan, nan, nan, nan], [nan, nan, nan, nan]],

    [[nan, nan, nan, nan],
     [nan, nan, nan, nan],
     [nan, nan, nan, nan],
     ...,
     [nan, nan, nan, nan],
     [nan, nan, nan, nan],
     [nan, nan, nan, nan]],

    [[nan, nan, nan, nan],
     [nan, nan, nan, nan],
     [nan, nan, nan, nan],
     ...,
     [nan, nan, nan, nan],
     [nan, nan, nan, nan],
     [nan, nan, nan, nan]],

    ...,

    [[nan, nan, nan, nan],
     [nan, nan, nan, nan],
     [nan, nan, nan, nan],
     ...,
     [nan, nan, nan, nan],
     [nan, nan, nan, nan],
     [nan, nan, nan, nan]],

    [[nan, nan, nan, nan],
     [nan, nan, nan, nan],
     [nan, nan, nan, nan],
     ...,
     [nan, nan, nan, nan],
     [nan, nan, nan, nan],
     [nan, nan, nan, nan]],

    [[nan, nan, nan, nan],
     [nan, nan, nan, nan],
     [nan, nan, nan, nan],
     ...,
     [nan, nan, nan, nan],
     [nan, nan, nan, nan],
     [nan, nan, nan, nan]]], device='cuda:0', grad_fn=<SelectBackward0>)

Traceback (most recent call last): File "/home/GZH/net/DEIM/./train.py", line 86, in main(args) File "/home/GZH/net/DEIM/./train.py", line 56, in main solver.fit() File "/home/GZH/net/DEIM/engine/solver/det_solver.py", line 76, in fit train_stats = train_one_epoch( ^^^^^^^^^^^^^^^^ File "/home/GZH/net/DEIM/engine/solver/det_engine.py", line 65, in train_one_epoch loss_dict = criterion(outputs, targets, **metas) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/anaconda3/envs/dfine/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/anaconda3/envs/dfine/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/GZH/net/DEIM/engine/deim/deim_criterion.py", line 276, in forward indices = self.matcher(outputs_without_aux, targets)['indices'] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/anaconda3/envs/dfine/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/anaconda3/envs/dfine/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/anaconda3/envs/dfine/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/GZH/net/DEIM/engine/deim/matcher.py", line 101, in forward cost_giou = -generalized_box_iou(box_cxcywh_to_xyxy(out_bbox), box_cxcywh_to_xyxy(tgt_bbox)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/GZH/net/DEIM/engine/deim/box_ops.py", line 53, in generalized_box_iou assert (boxes1[:, 2:] >= boxes1[:, :2]).all() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AssertionError It comes up every time I'm halfway through training

Dec 27 '24 00:12 ohmyhair151263

Epoch: [18] [3500/4384] eta: 0:08:08 lr: 0.000003 loss: 31.5320 (30.7038) loss_vfl: 0.6807 (0.6671) loss_bbox: 0.1400 (0.1503) loss_giou: 0.4664 (0.4601) loss_fgl: 1.0752 (1.0601) loss_vfl_aux_0: 0.7646 (0.7457) loss_bbox_aux_0: 0.1571 (0.1619) loss_giou_aux_0: 0.5047 (0.4856) loss_fgl_aux_0: 1.1022 (1.0939) loss_ddf_aux_0: 0.0534 (0.0495) loss_vfl_aux_1: 0.7661 (0.7237) loss_bbox_aux_1: 0.1372 (0.1529) loss_giou_aux_1: 0.4784 (0.4662) loss_fgl_aux_1: 1.0835 (1.0638) loss_ddf_aux_1: 0.0085 (0.0080) loss_vfl_aux_2: 0.7612 (0.7056) loss_bbox_aux_2: 0.1392 (0.1508) loss_giou_aux_2: 0.4685 (0.4611) loss_fgl_aux_2: 1.0754 (1.0603) loss_ddf_aux_2: 0.0014 (0.0015) loss_vfl_aux_3: 0.7124 (0.6810) loss_bbox_aux_3: 0.1400 (0.1503) loss_giou_aux_3: 0.4668 (0.4602) loss_fgl_aux_3: 1.0765 (1.0602) loss_ddf_aux_3: 0.0002 (0.0003) loss_vfl_aux_4: 0.6865 (0.6691) loss_bbox_aux_4: 0.1401 (0.1503) loss_giou_aux_4: 0.4664 (0.4601) loss_fgl_aux_4: 1.0752 (1.0601) loss_ddf_aux_4: 0.0001 (0.0001) loss_vfl_pre: 0.7646 (0.7465) loss_bbox_pre: 0.1566 (0.1618) loss_giou_pre: 0.5061 (0.4844) loss_vfl_enc_0: 0.7573 (0.7581) loss_bbox_enc_0: 0.1973 (0.1956) loss_giou_enc_0: 0.5716 (0.5552) loss_vfl_dn_0: 0.5078 (0.4945) loss_bbox_dn_0: 0.2006 (0.2113) loss_giou_dn_0: 0.4881 (0.4648) loss_fgl_dn_0: 1.1960 (1.2022) loss_ddf_dn_0: 0.2418 (0.2392) loss_vfl_dn_1: 0.4163 (0.4028) loss_bbox_dn_1: 0.1490 (0.1491) loss_giou_dn_1: 0.3783 (0.3514) loss_fgl_dn_1: 1.0866 (1.0820) loss_ddf_dn_1: 0.0355 (0.0327) loss_vfl_dn_2: 0.4009 (0.3814) loss_bbox_dn_2: 0.1394 (0.1393) loss_giou_dn_2: 0.3552 (0.3362) loss_fgl_dn_2: 1.0661 (1.0647) loss_ddf_dn_2: 0.0066 (0.0064) loss_vfl_dn_3: 0.3857 (0.3712) loss_bbox_dn_3: 0.1330 (0.1367) loss_giou_dn_3: 0.3452 (0.3318) loss_fgl_dn_3: 1.0660 (1.0648) loss_ddf_dn_3: 0.0003 (0.0003) loss_vfl_dn_4: 0.3799 (0.3678) loss_bbox_dn_4: 0.1316 (0.1363) loss_giou_dn_4: 0.3448 (0.3314) loss_fgl_dn_4: 1.0670 (1.0650) loss_ddf_dn_4: 0.0000 (0.0000) loss_vfl_dn_5: 0.3794 (0.3678) loss_bbox_dn_5: 0.1316 (0.1363) loss_giou_dn_5: 0.3448 (0.3314) loss_fgl_dn_5: 1.0670 (1.0650) loss_vfl_dn_pre: 0.5098 (0.4947) loss_bbox_dn_pre: 0.2113 (0.2182) loss_giou_dn_pre: 0.4908 (0.4658) time: 0.4990 data: 0.0280 max mem: 20978 tensor([[[nan, nan, nan, nan], [nan, nan, nan, nan], [nan, nan, nan, nan], ..., [nan, nan, nan, nan], [nan, nan, nan, nan], [nan, nan, nan, nan]],
    [[nan, nan, nan, nan],
     [nan, nan, nan, nan],
     [nan, nan, nan, nan],
     ...,
     [nan, nan, nan, nan],
     [nan, nan, nan, nan],
     [nan, nan, nan, nan]],

    [[nan, nan, nan, nan],
     [nan, nan, nan, nan],
     [nan, nan, nan, nan],
     ...,
     [nan, nan, nan, nan],
     [nan, nan, nan, nan],
     [nan, nan, nan, nan]],

    ...,

    [[nan, nan, nan, nan],
     [nan, nan, nan, nan],
     [nan, nan, nan, nan],
     ...,
     [nan, nan, nan, nan],
     [nan, nan, nan, nan],
     [nan, nan, nan, nan]],

    [[nan, nan, nan, nan],
     [nan, nan, nan, nan],
     [nan, nan, nan, nan],
     ...,
     [nan, nan, nan, nan],
     [nan, nan, nan, nan],
     [nan, nan, nan, nan]],

    [[nan, nan, nan, nan],
     [nan, nan, nan, nan],
     [nan, nan, nan, nan],
     ...,
     [nan, nan, nan, nan],
     [nan, nan, nan, nan],
     [nan, nan, nan, nan]]], device='cuda:0', grad_fn=<SelectBackward0>)
Traceback (most recent call last): File "/home/GZH/net/DEIM/./train.py", line 86, in main(args) File "/home/GZH/net/DEIM/./train.py", line 56, in main solver.fit() File "/home/GZH/net/DEIM/engine/solver/det_solver.py", line 76, in fit train_stats = train_one_epoch( ^^^^^^^^^^^^^^^^ File "/home/GZH/net/DEIM/engine/solver/det_engine.py", line 65, in train_one_epoch loss_dict = criterion(outputs, targets, **metas) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/anaconda3/envs/dfine/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/anaconda3/envs/dfine/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/GZH/net/DEIM/engine/deim/deim_criterion.py", line 276, in forward indices = self.matcher(outputs_without_aux, targets)['indices'] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/anaconda3/envs/dfine/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/anaconda3/envs/dfine/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/anaconda3/envs/dfine/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/GZH/net/DEIM/engine/deim/matcher.py", line 101, in forward cost_giou = -generalized_box_iou(box_cxcywh_to_xyxy(out_bbox), box_cxcywh_to_xyxy(tgt_bbox)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/GZH/net/DEIM/engine/deim/box_ops.py", line 53, in generalized_box_iou assert (boxes1[:, 2:] >= boxes1[:, :2]).all() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AssertionError It comes up every time I'm halfway through training

Hi there, did you solve this issue?

Jan 09 '25 05:01 Xiaofei-Kevin-Yang

i have also encountered such a problem,maybe u can try to lower lr

Jan 16 '25 10:01 caiyang12

How has anyone solved this problem? In my case, there is no nan directly, but there is assert.

cost_giou = -generalized_box_iou(box_cxcywh_to_xyxy(out_bbox), box_cxcywh_to_xyxy(tgt_bbox))
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

    assert (boxes1[:, 2:] >= boxes1[:, :2]).all()
RuntimeError: CUDA error: device-side assert triggered

Mar 06 '25 15:03 MaksymTymkovych

How has anyone solved this problem? In my case, there is no nan directly, but there is assert.

cost_giou = -generalized_box_iou(box_cxcywh_to_xyxy(out_bbox), box_cxcywh_to_xyxy(tgt_bbox))
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

    assert (boxes1[:, 2:] >= boxes1[:, :2]).all()
RuntimeError: CUDA error: device-side assert triggered

I found my mistake. I used the wrong dataset, and therefore the wrong number of classes.

Mar 06 '25 16:03 MaksymTymkovych

How has anyone solved this problem? In my case, there is no nan directly, but there is assert.有人是如何解决这个问题的？在我的情况下，并没有直接的 nan，但是有 assert。
cost_giou = -generalized_box_iou(box_cxcywh_to_xyxy(out_bbox), box_cxcywh_to_xyxy(tgt_bbox))
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    assert (boxes1[:, 2:] >= boxes1[:, :2]).all()
RuntimeError: CUDA error: device-side assert triggered
I found my mistake. I used the wrong dataset, and therefore the wrong number of classes.我找到了我的错误。我使用了错误的数据集，因此类别的数量也错了。

Epoch: [18] [3500/4384] eta: 0:08:08 lr: 0.000003 loss: 31.5320 (30.7038) loss_vfl: 0.6807 (0.6671) loss_bbox: 0.1400 (0.1503) loss_giou: 0.4664 (0.4601) loss_fgl: 1.0752 (1.0601) loss_vfl_aux_0: 0.7646 (0.7457) loss_bbox_aux_0: 0.1571 (0.1619) loss_giou_aux_0: 0.5047 (0.4856) loss_fgl_aux_0: 1.1022 (1.0939) loss_ddf_aux_0: 0.0534 (0.0495) loss_vfl_aux_1: 0.7661 (0.7237) loss_bbox_aux_1: 0.1372 (0.1529) loss_giou_aux_1: 0.4784 (0.4662) loss_fgl_aux_1: 1.0835 (1.0638) loss_ddf_aux_1: 0.0085 (0.0080) loss_vfl_aux_2: 0.7612 (0.7056) loss_bbox_aux_2: 0.1392 (0.1508) loss_giou_aux_2: 0.4685 (0.4611) loss_fgl_aux_2: 1.0754 (1.0603) loss_ddf_aux_2: 0.0014 (0.0015) loss_vfl_aux_3: 0.7124 (0.6810) loss_bbox_aux_3: 0.1400 (0.1503) loss_giou_aux_3: 0.4668 (0.4602) loss_fgl_aux_3: 1.0765 (1.0602) loss_ddf_aux_3: 0.0002 (0.0003) loss_vfl_aux_4: 0.6865 (0.6691) loss_bbox_aux_4: 0.1401 (0.1503) loss_giou_aux_4: 0.4664 (0.4601) loss_fgl_aux_4: 1.0752 (1.0601) loss_ddf_aux_4: 0.0001 (0.0001) loss_vfl_pre: 0.7646 (0.7465) loss_bbox_pre: 0.1566 (0.1618) loss_giou_pre: 0.5061 (0.4844) loss_vfl_enc_0: 0.7573 (0.7581) loss_bbox_enc_0: 0.1973 (0.1956) loss_giou_enc_0: 0.5716 (0.5552) loss_vfl_dn_0: 0.5078 (0.4945) loss_bbox_dn_0: 0.2006 (0.2113) loss_giou_dn_0: 0.4881 (0.4648) loss_fgl_dn_0: 1.1960 (1.2022) loss_ddf_dn_0: 0.2418 (0.2392) loss_vfl_dn_1: 0.4163 (0.4028) loss_bbox_dn_1: 0.1490 (0.1491) loss_giou_dn_1: 0.3783 (0.3514) loss_fgl_dn_1: 1.0866 (1.0820) loss_ddf_dn_1: 0.0355 (0.0327) loss_vfl_dn_2: 0.4009 (0.3814) loss_bbox_dn_2: 0.1394 (0.1393) loss_giou_dn_2: 0.3552 (0.3362) loss_fgl_dn_2: 1.0661 (1.0647) loss_ddf_dn_2: 0.0066 (0.0064) loss_vfl_dn_3: 0.3857 (0.3712) loss_bbox_dn_3: 0.1330 (0.1367) loss_giou_dn_3: 0.3452 (0.3318) loss_fgl_dn_3: 1.0660 (1.0648) loss_ddf_dn_3: 0.0003 (0.0003) loss_vfl_dn_4: 0.3799 (0.3678) loss_bbox_dn_4: 0.1316 (0.1363) loss_giou_dn_4: 0.3448 (0.3314) loss_fgl_dn_4: 1.0670 (1.0650) loss_ddf_dn_4: 0.0000 (0.0000) loss_vfl_dn_5: 0.3794 (0.3678) loss_bbox_dn_5: 0.1316 (0.1363) loss_giou_dn_5: 0.3448 (0.3314) loss_fgl_dn_5: 1.0670 (1.0650) loss_vfl_dn_pre: 0.5098 (0.4947) loss_bbox_dn_pre: 0.2113 (0.2182) loss_giou_dn_pre: 0.4908 (0.4658) time: 0.4990 data: 0.0280 max mem: 20978 tensor([[[nan, nan, nan, nan], [nan, nan, nan, nan], [nan, nan, nan, nan], ..., [nan, nan, nan, nan], [nan, nan, nan, nan], [nan, nan, nan, nan]],
    [[nan, nan, nan, nan],
     [nan, nan, nan, nan],
     [nan, nan, nan, nan],
     ...,
     [nan, nan, nan, nan],
     [nan, nan, nan, nan],
     [nan, nan, nan, nan]],

    [[nan, nan, nan, nan],
     [nan, nan, nan, nan],
     [nan, nan, nan, nan],
     ...,
     [nan, nan, nan, nan],
     [nan, nan, nan, nan],
     [nan, nan, nan, nan]],

    ...,

    [[nan, nan, nan, nan],
     [nan, nan, nan, nan],
     [nan, nan, nan, nan],
     ...,
     [nan, nan, nan, nan],
     [nan, nan, nan, nan],
     [nan, nan, nan, nan]],

    [[nan, nan, nan, nan],
     [nan, nan, nan, nan],
     [nan, nan, nan, nan],
     ...,
     [nan, nan, nan, nan],
     [nan, nan, nan, nan],
     [nan, nan, nan, nan]],

    [[nan, nan, nan, nan],
     [nan, nan, nan, nan],
     [nan, nan, nan, nan],
     ...,
     [nan, nan, nan, nan],
     [nan, nan, nan, nan],
     [nan, nan, nan, nan]]], device='cuda:0', grad_fn=<SelectBackward0>)
Traceback (most recent call last): File "/home/GZH/net/DEIM/./train.py", line 86, in main(args) File "/home/GZH/net/DEIM/./train.py", line 56, in main solver.fit() File "/home/GZH/net/DEIM/engine/solver/det_solver.py", line 76, in fit train_stats = train_one_epoch( ^^^^^^^^^^^^^^^^ File "/home/GZH/net/DEIM/engine/solver/det_engine.py", line 65, in train_one_epoch loss_dict = criterion(outputs, targets, **metas) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/anaconda3/envs/dfine/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/anaconda3/envs/dfine/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/GZH/net/DEIM/engine/deim/deim_criterion.py", line 276, in forward indices = self.matcher(outputs_without_aux, targets)['indices'] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/anaconda3/envs/dfine/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/anaconda3/envs/dfine/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/anaconda3/envs/dfine/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/GZH/net/DEIM/engine/deim/matcher.py", line 101, in forward cost_giou = -generalized_box_iou(box_cxcywh_to_xyxy(out_bbox), box_cxcywh_to_xyxy(tgt_bbox)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/GZH/net/DEIM/engine/deim/box_ops.py", line 53, in generalized_box_iou assert (boxes1[:, 2:] >= boxes1[:, :2]).all() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AssertionError It comes up every time I'm halfway through training

I had this problem when using the model deim_hgnetv2_x_coco.yml

Mar 12 '25 14:03 ssgeb

How many classes are in the dataset, and which class is numbered (started from 0-index, 1-index)?

Mar 12 '25 14:03 MaksymTymkovych

I'm also experiencing this error. Here is some info:

The training seemed to go well until epoch 51, then this nan problem triggered
I checked my dataset annotations and yes, I admit the classes were 0-indexed
I fixed the indexing problem and also any kind of invalid bounding box coordinates problems
With the new 1-indexed annotations, I get a different error now, you can see it here below

Any suggestion is welcome.

------------------------------------- Calculate Flops Results -------------------------------------
Notations:
number of parameters (Params), number of multiply-accumulate operations(MACs),
number of floating-point operations (FLOPs), floating-point operations per second (FLOPS),
fwd FLOPs (model forward propagation FLOPs), bwd FLOPs (model backward propagation FLOPs),
default model backpropagation takes 2.00 times as much computation as forward propagation.

Total Training Params:                                                  61.64 M
fwd MACs:                                                               100.975 GMACs
fwd FLOPs:                                                              202.326 GFLOPS
fwd+bwd MACs:                                                           302.925 GMACs
fwd+bwd FLOPs:                                                          606.978 GFLOPS
---------------------------------------------------------------------------------------------------
{'Model FLOPs:202.326 GFLOPS   MACs:100.975 GMACs   Params:61636265'}
------------------------------------------Start training-------------------------------------------
     ## Using Self-defined Scheduler-flatcosine ##
[5e-06, 0.0005, 0.0005] [2.5e-06, 0.00025, 0.00025] 24012 2000 12006 3312
number of trainable parameters: 62557243
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [44,0,0], thread: [0,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [44,0,0], thread: [1,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [44,0,0], thread: [4,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [44,0,0], thread: [5,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [44,0,0], thread: [10,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [44,0,0], thread: [11,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [44,0,0], thread: [12,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [44,0,0], thread: [13,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [44,0,0], thread: [15,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [44,0,0], thread: [16,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [124,0,0], thread: [34,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [124,0,0], thread: [35,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [124,0,0], thread: [36,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [124,0,0], thread: [39,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [124,0,0], thread: [40,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [124,0,0], thread: [45,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [124,0,0], thread: [46,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [124,0,0], thread: [47,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [124,0,0], thread: [48,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [124,0,0], thread: [50,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [124,0,0], thread: [51,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [204,0,0], thread: [69,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [204,0,0], thread: [70,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [204,0,0], thread: [71,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [204,0,0], thread: [74,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [204,0,0], thread: [75,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [204,0,0], thread: [80,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [204,0,0], thread: [81,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [204,0,0], thread: [82,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [204,0,0], thread: [83,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [204,0,0], thread: [85,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [204,0,0], thread: [86,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [124,0,0], thread: [0,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [124,0,0], thread: [1,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [124,0,0], thread: [2,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [124,0,0], thread: [3,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [124,0,0], thread: [5,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [124,0,0], thread: [6,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [124,0,0], thread: [23,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [124,0,0], thread: [27,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [124,0,0], thread: [96,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [124,0,0], thread: [113,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [124,0,0], thread: [117,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [124,0,0], thread: [124,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [124,0,0], thread: [125,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [124,0,0], thread: [126,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [44,0,0], thread: [78,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [44,0,0], thread: [82,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [44,0,0], thread: [89,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [44,0,0], thread: [90,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [44,0,0], thread: [91,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [44,0,0], thread: [94,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [44,0,0], thread: [95,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [204,0,0], thread: [103,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [204,0,0], thread: [107,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [204,0,0], thread: [114,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [204,0,0], thread: [115,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [204,0,0], thread: [116,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [204,0,0], thread: [119,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [204,0,0], thread: [120,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [204,0,0], thread: [125,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [204,0,0], thread: [126,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [204,0,0], thread: [127,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [204,0,0], thread: [13,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [204,0,0], thread: [17,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [204,0,0], thread: [24,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [204,0,0], thread: [25,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [204,0,0], thread: [26,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [204,0,0], thread: [29,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [204,0,0], thread: [30,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [204,0,0], thread: [35,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [204,0,0], thread: [36,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [204,0,0], thread: [37,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [204,0,0], thread: [38,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [204,0,0], thread: [40,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [204,0,0], thread: [41,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [204,0,0], thread: [58,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [204,0,0], thread: [62,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [284,0,0], thread: [48,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [284,0,0], thread: [52,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [284,0,0], thread: [59,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [284,0,0], thread: [60,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [284,0,0], thread: [61,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [124,0,0], thread: [68,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [124,0,0], thread: [72,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [124,0,0], thread: [79,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [124,0,0], thread: [80,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [124,0,0], thread: [81,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [124,0,0], thread: [84,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [124,0,0], thread: [85,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [124,0,0], thread: [90,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [124,0,0], thread: [91,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [124,0,0], thread: [92,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [124,0,0], thread: [93,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [124,0,0], thread: [95,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [44,0,0], thread: [33,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [44,0,0], thread: [37,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [44,0,0], thread: [44,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [44,0,0], thread: [45,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [44,0,0], thread: [46,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [44,0,0], thread: [49,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [44,0,0], thread: [50,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [44,0,0], thread: [55,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [44,0,0], thread: [56,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [44,0,0], thread: [57,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [44,0,0], thread: [58,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [44,0,0], thread: [60,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [44,0,0], thread: [61,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [44,0,0], thread: [100,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [44,0,0], thread: [101,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [44,0,0], thread: [102,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [44,0,0], thread: [103,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [44,0,0], thread: [105,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [44,0,0], thread: [106,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [44,0,0], thread: [123,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [44,0,0], thread: [127,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [284,0,0], thread: [97,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [284,0,0], thread: [104,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [284,0,0], thread: [105,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [284,0,0], thread: [106,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [284,0,0], thread: [109,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [284,0,0], thread: [110,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [284,0,0], thread: [115,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [284,0,0], thread: [116,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [284,0,0], thread: [117,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [284,0,0], thread: [118,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [284,0,0], thread: [120,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [284,0,0], thread: [121,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [284,0,0], thread: [64,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [284,0,0], thread: [65,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [284,0,0], thread: [70,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [284,0,0], thread: [71,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [284,0,0], thread: [72,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [284,0,0], thread: [73,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [284,0,0], thread: [75,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [284,0,0], thread: [76,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [284,0,0], thread: [93,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [284,0,0], thread: [3,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [284,0,0], thread: [7,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [284,0,0], thread: [14,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [284,0,0], thread: [15,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [284,0,0], thread: [16,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [284,0,0], thread: [19,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [284,0,0], thread: [20,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [284,0,0], thread: [25,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [284,0,0], thread: [26,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [284,0,0], thread: [27,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [284,0,0], thread: [28,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [284,0,0], thread: [30,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [284,0,0], thread: [31,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [364,0,0], thread: [96,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [364,0,0], thread: [99,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [364,0,0], thread: [100,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [364,0,0], thread: [105,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [364,0,0], thread: [106,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [364,0,0], thread: [107,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [364,0,0], thread: [108,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [364,0,0], thread: [110,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [364,0,0], thread: [111,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [364,0,0], thread: [65,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [364,0,0], thread: [66,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [364,0,0], thread: [83,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [364,0,0], thread: [87,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [364,0,0], thread: [94,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [364,0,0], thread: [95,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [364,0,0], thread: [4,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [364,0,0], thread: [5,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [364,0,0], thread: [6,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [364,0,0], thread: [9,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [364,0,0], thread: [10,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [364,0,0], thread: [15,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [364,0,0], thread: [16,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [364,0,0], thread: [17,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [364,0,0], thread: [18,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [364,0,0], thread: [20,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [364,0,0], thread: [21,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [364,0,0], thread: [38,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [364,0,0], thread: [42,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [364,0,0], thread: [49,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [364,0,0], thread: [50,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [364,0,0], thread: [51,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [364,0,0], thread: [54,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [364,0,0], thread: [55,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [364,0,0], thread: [60,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [364,0,0], thread: [61,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [364,0,0], thread: [62,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [364,0,0], thread: [63,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [103,0,0], thread: [110,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [103,0,0], thread: [114,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [103,0,0], thread: [121,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [103,0,0], thread: [122,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [103,0,0], thread: [123,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [103,0,0], thread: [126,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [103,0,0], thread: [127,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [183,0,0], thread: [32,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [183,0,0], thread: [33,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [183,0,0], thread: [34,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [183,0,0], thread: [35,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [183,0,0], thread: [37,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [183,0,0], thread: [38,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [183,0,0], thread: [55,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [183,0,0], thread: [59,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [183,0,0], thread: [66,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [183,0,0], thread: [67,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [183,0,0], thread: [68,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [183,0,0], thread: [71,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [183,0,0], thread: [72,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [183,0,0], thread: [77,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [183,0,0], thread: [78,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [183,0,0], thread: [79,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [183,0,0], thread: [80,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [183,0,0], thread: [82,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [183,0,0], thread: [83,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [183,0,0], thread: [10,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [183,0,0], thread: [14,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [183,0,0], thread: [21,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [183,0,0], thread: [22,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [183,0,0], thread: [23,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [183,0,0], thread: [26,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [183,0,0], thread: [27,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [263,0,0], thread: [45,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [263,0,0], thread: [49,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [263,0,0], thread: [56,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [263,0,0], thread: [57,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [263,0,0], thread: [58,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [263,0,0], thread: [61,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [263,0,0], thread: [62,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [414,0,0], thread: [43,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [414,0,0], thread: [47,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [414,0,0], thread: [54,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [414,0,0], thread: [55,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [414,0,0], thread: [56,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [414,0,0], thread: [59,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:94: operator(): block: [414,0,0], thread: [60,0,0] Assertion `-sizes[i] <= index && index < sizes[i] && "index out of bounds"` failed.
[rank0]: Traceback (most recent call last):
[rank0]:   File "/home/project_name/DEIM/train.py", line 84, in <module>
[rank0]:     main(args)
[rank0]:   File "/home/project_name/DEIM/train.py", line 54, in main
[rank0]:     solver.fit()
[rank0]:   File "/home/project_name/DEIM/engine/solver/det_solver.py", line 76, in fit
[rank0]:     train_stats = train_one_epoch(
[rank0]:   File "/home/project_name/DEIM/engine/solver/det_engine.py", line 65, in train_one_epoch
[rank0]:     loss_dict = criterion(outputs, targets, **metas)
[rank0]:   File "/home/project_name/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
[rank0]:     return self._call_impl(*args, **kwargs)
[rank0]:   File "/home/project_name/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
[rank0]:     return forward_call(*args, **kwargs)
[rank0]:   File "/home/project_name/DEIM/engine/deim/deim_criterion.py", line 276, in forward
[rank0]:     indices = self.matcher(outputs_without_aux, targets)['indices']
[rank0]:   File "/home/project_name/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
[rank0]:     return self._call_impl(*args, **kwargs)
[rank0]:   File "/home/project_name/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
[rank0]:     return forward_call(*args, **kwargs)
[rank0]:   File "/home/project_name/venv/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
[rank0]:     return func(*args, **kwargs)
[rank0]:   File "/home/project_name/DEIM/engine/deim/matcher.py", line 98, in forward
[rank0]:     cost_bbox = torch.cdist(out_bbox, tgt_bbox, p=1)
[rank0]:   File "/home/project_name/venv/lib/python3.10/site-packages/torch/functional.py", line 1483, in cdist
[rank0]:     return _VF.cdist(x1, x2, p, None)  # type: ignore[attr-defined]
[rank0]: RuntimeError: CUDA error: device-side assert triggered
[rank0]: CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
[rank0]: For debugging consider passing CUDA_LAUNCH_BLOCKING=1
[rank0]: Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

Exception ignored in atexit callback: <function cleanup at 0x7f7c15f27b50>
Traceback (most recent call last):
  File "/home/project_name/DEIM/engine/misc/dist_utils.py", line 101, in cleanup
    torch.distributed.barrier()
  File "/home/project_name/venv/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 81, in wrapper
    return func(*args, **kwargs)
  File "/home/project_name/venv/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4551, in barrier
    work = group.barrier(opts=opts)
RuntimeError: CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

[rank0]:[W316 14:59:16.191523284 ProcessGroupNCCL.cpp:1496] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
terminate called after throwing an instance of 'c10::Error'
  what():  CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

Exception raised from c10_cuda_check_implementation at /pytorch/c10/cuda/CUDAException.cpp:43 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f7caeb971b6 in /home/project_name/venv/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&) + 0x64 (0x7f7caeb40a76 in /home/project_name/venv/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x118 (0x7f7d015a3918 in /home/project_name/venv/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)
frame #3: <unknown function> + 0x20d8e (0x7f7d01569d8e in /home/project_name/venv/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)
frame #4: <unknown function> + 0x22507 (0x7f7d0156b507 in /home/project_name/venv/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)
frame #5: <unknown function> + 0x2270f (0x7f7d0156b70f in /home/project_name/venv/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)
frame #6: <unknown function> + 0x6417b2 (0x7f7cf94d07b2 in /home/project_name/venv/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
frame #7: <unknown function> + 0x6f30f (0x7f7caeb7830f in /home/project_name/venv/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #8: c10::TensorImpl::~TensorImpl() + 0x21b (0x7f7caeb7133b in /home/project_name/venv/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #9: c10::TensorImpl::~TensorImpl() + 0x9 (0x7f7caeb714e9 in /home/project_name/venv/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #10: <unknown function> + 0x8fefb8 (0x7f7cf978dfb8 in /home/project_name/venv/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
frame #11: THPVariable_subclass_dealloc(_object*) + 0x2f6 (0x7f7cf978e306 in /home/project_name/venv/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
frame #12: <unknown function> + 0x169c71 (0x556826691c71 in /home/project_name/venv/bin/python3)
frame #13: <unknown function> + 0x169a6c (0x556826691a6c in /home/project_name/venv/bin/python3)
frame #14: <unknown function> + 0x169b57 (0x556826691b57 in /home/project_name/venv/bin/python3)
frame #15: <unknown function> + 0x1a2407 (0x5568266ca407 in /home/project_name/venv/bin/python3)
frame #16: <unknown function> + 0x169c71 (0x556826691c71 in /home/project_name/venv/bin/python3)
frame #17: <unknown function> + 0x169a6c (0x556826691a6c in /home/project_name/venv/bin/python3)
frame #18: <unknown function> + 0x169b93 (0x556826691b93 in /home/project_name/venv/bin/python3)
frame #19: <unknown function> + 0x1a2407 (0x5568266ca407 in /home/project_name/venv/bin/python3)
frame #20: <unknown function> + 0x169c71 (0x556826691c71 in /home/project_name/venv/bin/python3)
frame #21: <unknown function> + 0x169a6c (0x556826691a6c in /home/project_name/venv/bin/python3)
frame #22: <unknown function> + 0x169c71 (0x556826691c71 in /home/project_name/venv/bin/python3)
frame #23: <unknown function> + 0x169a6c (0x556826691a6c in /home/project_name/venv/bin/python3)
frame #24: <unknown function> + 0x168fc2 (0x556826690fc2 in /home/project_name/venv/bin/python3)
frame #25: <unknown function> + 0x188bfa (0x5568266b0bfa in /home/project_name/venv/bin/python3)
frame #26: PyDict_Clear + 0x13f (0x55682671d38f in /home/project_name/venv/bin/python3)
frame #27: <unknown function> + 0x27ac0a (0x5568267a2c0a in /home/project_name/venv/bin/python3)
frame #28: <unknown function> + 0x15d4cd (0x5568266854cd in /home/project_name/venv/bin/python3)
frame #29: <unknown function> + 0x2847a0 (0x5568267ac7a0 in /home/project_name/venv/bin/python3)
frame #30: Py_FinalizeEx + 0x148 (0x5568267a8b78 in /home/project_name/venv/bin/python3)
frame #31: Py_RunMain + 0x173 (0x55682679bd13 in /home/project_name/venv/bin/python3)
frame #32: Py_BytesMain + 0x2d (0x556826775e6d in /home/project_name/venv/bin/python3)
frame #33: <unknown function> + 0x29d90 (0x7f7d02409d90 in /lib/x86_64-linux-gnu/libc.so.6)
frame #34: __libc_start_main + 0x80 (0x7f7d02409e40 in /lib/x86_64-linux-gnu/libc.so.6)
frame #35: _start + 0x25 (0x556826775d65 in /home/project_name/venv/bin/python3)

Mar 16 '25 15:03 vittorio-prodomo

Ok I think I understood the problem, but please confirm from your side when you have the chance. The problem here is that the config files want you to specify the number of classes including the background class (a classic source of ambiguity when dealing with object detection models). Now everything makes sense, that's why my training seemed to work when I had put 3 classes but with 0-based indexing. Now that I have a 1-based indexing, and I am using 4=n+1 as number of classes, I guess everything is fine.

If this is true and your repo, code, and model work by including the background class, please mention it very clearly, maybe both as a comment in the config file and in your repo README file. It will prevent a lot of headaches from other people.

Keep up the good work!

Mar 16 '25 15:03 vittorio-prodomo

@sk1ddy I arrive at the same conclusion as you! I wrote a wrapper as a fork of this repo to make it easier and clearer for anyone interested to train DEIM models on a custom dataset.

I still recommend using the original DEIM repo if you're serious about training a SOTA model on a custom dataset. But if you're short of time and want to get started quickly, the wrapper should help.

Have a go - https://github.com/dnth/DEIMKit

Mar 17 '25 03:03 dnth

I also have the same problem when training customer datasets

Apr 14 '25 04:04 Sunburst7

Hi everyone, I've been following this thread and issue #72 as I was facing the exact same random NaN -> AssertionError crash halfway through training.

Like others have suggested, issues with the learning rate or the num_classes setting can cause instability. However, if you are specifically using Automatic Mixed Precision (--use-amp), there seems to be a more direct root cause.

The problem lies with the AdamW optimizer's default epsilon (eps=1e-8). This value is too small for the float16 data type used by AMP and can cause the denominator in the optimizer's update step to become zero, leading to a division-by-zero that produces NaN gradients and poisons the model's weights.

I was able to create a stable fix by setting a slightly larger epsilon (1e-7) in my .yml config file:

optimizer:
  type: AdamW
  # ... other params
  eps: 0.0000001

This prevents the numerical underflow and hopefully resolves the issue for AMP users.

This is a known interaction with Adam/AMP, and you can read the technical details in the official PyTorch repo here: pytorch/pytorch#26218.

I hope this provides a definitive solution for those of you using mixed precision!

Jul 31 '25 09:07 EwertzJN

大家好，我一直在关注这个线程和问题 #72，因为我在训练中途遇到了完全相同的随机 -> 崩溃。NaN``AssertionError

正如其他人所建议的那样，学习率或设置的问题可能会导致不稳定。但是，如果您专门使用自动混合精度（），则似乎有一个更直接的根本原因。num_classes``--use-amp

问题出在 AdamW 优化器的默认 epsilon （） 上。对于 AMP 使用的数据类型来说，此值太小，可能会导致优化器更新步骤中的分母变为零，从而导致除以零，从而产生梯度并影响模型的权重。float16``NaN

我能够通过在配置文件中设置稍大的 epsilon （）来创建稳定的修复程序：1e-7``.yml

optimizer: type: AdamW

... other params

eps: 0.0000001 这可以防止数字下溢，并有望解决 AMP 用户的问题。

这是与 Adam/AMP 的已知交互，您可以在此处阅读官方 PyTorch 存储库中的技术细节：pytorch/pytorch#26218。

我希望这能为那些使用混合精度的人提供一个明确的解决方案！

Setting the eps did not help me solve this problem; the NaN error still occurs in random batches.

Aug 12 '25 01:08 Unicorn123455678

大家好，我一直在关注这个线程和问题 #72，因为我在训练中途遇到了完全相同的随机 -> 崩溃。 NaNAssertionError 正如其他人所建议的那样，学习率或设置的问题可能会导致不稳定。但是，如果您专门使用自动混合精度（），则似乎有一个更直接的根本原因。 num_classes--use-amp 问题出在 AdamW 优化器的默认 epsilon （） 上。对于 AMP 使用的数据类型来说，此值太小，可能会导致优化器更新步骤中的分母变为零，从而导致除以零，从而产生梯度并影响模型的权重。 float16NaN 我能够通过在配置文件中设置稍大的 epsilon （）来创建稳定的修复程序： 1e-7.yml optimizer: type: AdamW

... other params

eps: 0.0000001 这可以防止数字下溢，并有望解决 AMP 用户的问题。这是与 Adam/AMP 的已知交互，您可以在此处阅读官方 PyTorch 存储库中的技术细节：pytorch/pytorch#26218。我希望这能为那些使用混合精度的人提供一个明确的解决方案！

Setting the eps did not help me solve this problem; the NaN error still occurs in random batches.

在关闭amp后，我的训练恢复了正常

Aug 12 '25 03:08 Unicorn123455678

Training instability with AMP is an occasional issue, but it's not a consistently reproducible problem. Currently, if resolving this is necessary, the preferred approach is to disable AMP.

Nov 01 '25 01:11 ShihuaHuang95

Training instability with AMP is an occasional issue, but it's not a consistently reproducible problem. Currently, if resolving this is necessary, the preferred approach is to disable AMP.

I found that this issue only occurs when using HGnetV2 as the backbone network. If we use ResNet50 instead, the problem does not arise.

Nov 24 '25 04:11 wuruoyu1997coder

Training instability with AMP is an occasional issue, but it's not a consistently reproducible problem. Currently, if resolving this is necessary, the preferred approach is to disable AMP.

I think the main issue is that the network architecture of HGnetV2 makes it unable to stably handle the numerical variations during the training process. By the way, the dataset I'm using is FLIR. In a single-modal setting, all backbone networks work fine, but when I switch to a multi-modal setting, problems occur with HGnetV2.

Nov 24 '25 04:11 wuruoyu1997coder