Nvidia error after train begins

[godkaieethu]

thanks for your excellent work~but when I try to pretrain the model on my own dataset ,the train soon jammed and I can not run Nvidia smi even I kill the training.

Unable to determine the device handle for GPU 0000:81:00.0: Unknown Error