generate white images during training

Open NikLi66 opened this issue 1 year ago • 1 comments

Hi, I found the all of the saved reconstructed images are white images under the directory ./logs/ during training(shown below). Is there anyone can tell me the potential reason?

May 31 '24 02:05 NikLi66

Hi, I found the all of the saved reconstructed images are white images under the directory ./logs/ during training(shown below). Is there anyone can tell me the potential reason?

did you face any issue in training like below?

[rank0]: Parameter indices which did not receive grad for rank 0: 338 339 340 341 342 343 344 345 346 347 348 349 350 [rank0]: In addition, you can set the environment variable TORCH_DISTRIBUTED_DEBUG to either INFO or DETAIL to print out information about which particular parameters did not receive gradient on this rank as part of this error

Jun 12 '24 10:06 ashwani-ver