RuntimeError: CUDA error: device-side assert triggered

[kawaiiGTR]

Trying to run DETR on custom dataset. When executing the launch command:

• python tools/train_detector.py --epochs 20 --batch 2 --data data/aquarium.yaml --model detr_resnet50 --name detr_resnet50

The output is:

RuntimeError: CUDA error: device-side assert triggered

CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1. Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.

After some digging the error either points to class mismatch or problematic activation function. Either way it doesn't seem to work!! :(

Any advise on how to get this working? Cheers!

[sovit-123]

Hello @kawaiiGTR Can you please let me know how many classes you have and if you can provide the your custom dataset YAML file information here?

[kawaiiGTR]

I think there is a mismatch in Linear out between Input shape [2,100,92] and the output shape [2.100,114]

[sovit-123]

Were you able to solve it?

[kawaiiGTR]

I don't know where e.g. what file I need to edit to change the size of the input layer to match the output layer. Could you kindly advise? I have 114 classes not 92.. Cheers!

[sovit-123]

Is it possible for you to provide me link to the dataset? I will be able to debug it if I have the dataset.