DenseCL icon indicating copy to clipboard operation
DenseCL copied to clipboard

training time

Open youngAt19 opened this issue 3 years ago • 4 comments

hi, thank you for sharing your codes. I try to train densecl on imagenet with 4*2080ti, but it takes about 13 days to finish training, which is much longer time than yours. Did you use imagenet in lmdb format so that training time is shorter?

youngAt19 avatar May 20 '22 06:05 youngAt19

I used 16 tesla T4 gpu amount 2 nodes, and it takes me about 4 days to train it. From my observation of gpu occupance, the data loading is not the bottleneck of training speed.

CoinCheung avatar May 20 '22 06:05 CoinCheung

Thanks for your reply. I thought it just takes you 53 hours to train the model from here

youngAt19 avatar May 20 '22 06:05 youngAt19

Thanks for your reply. I thought it just takes you 53 hours to train the model from here

youngAt19 avatar May 20 '22 06:05 youngAt19

This is mocov2 time from the original repo of mocov2.

Training denseCl is time-consuming, I recommand you to use the pretrained model if you are using resnet50/101 backbone. It will save you time.

CoinCheung avatar May 21 '22 02:05 CoinCheung