Training duration

Open Arksyd96 opened this issue 3 years ago • 1 comments

Hello,

I have created an implementation that is strongly influenced by yours, as I found it simple and well-implemented. However, I have a question regarding the training duration. I am currently training on BraTS MRIs with dimensions of 1x256x256, using a total of 500 images. While the results appear promising, the training process is taking an extended amount of time, approximately 3 days. Is this expected? It seems to differ from your example with the butterfly dataset.

Mar 31 '23 09:03 Arksyd96

Hey, Well, DDPM considers as a quite slow scheduler. Also I'm sure my scheduler implementation ain't the best in terms of optimization.
That's why most of diffusion algorithms use DDIM, PNDM or K-LSM schedulers. Current implementation takes about a minute on epoch on my setup (2080 Ti, 18GB on 6 batches, 224x224x3 image size). I think you can also tweak the number of diffusion steps and training process might be more efficient.

Apr 01 '23 14:04 mattroz