MaskFreeVIS icon indicating copy to clipboard operation
MaskFreeVIS copied to clipboard

OOM error

Open laybebe opened this issue 2 years ago • 5 comments

Thanks for your excellent work! What's the GPU memory of your Titan RTX, 24G? I have tried to train MaskFreeVIS with backbone ResNet50 and batch size 16 on 8x3090(24G), but it will result in an OOM error.

laybebe avatar Apr 22 '23 08:04 laybebe

which training script are you using?

lkeab avatar Apr 22 '23 08:04 lkeab

which training script are you using?

I used "MaskFreeVIS/mfvis_nococo/configs/youtubevis_2019/video_maskformer2_R50_bs16_8ep.yaml" and did not modify any parameter.

laybebe avatar Apr 22 '23 08:04 laybebe

In this case, you can reduce the SAMPLING_FRAME_NUM from 5 to 3, and modify the codes here accordingly (to make TK loss work in 3-frame tube). It will reduce memory a lot but the same time training will be slightly less stable.

lkeab avatar Apr 22 '23 09:04 lkeab

Thanks for your reply. In fact, I ran the code by reducing the batch size. The result (AP=41.7) was slightly lower than the result in the paper.

laybebe avatar Apr 22 '23 11:04 laybebe

Yeah, that's actually normal due to the weak supervision (no any mask usage) and the randomness in sampling frames of the dataloader.

lkeab avatar Apr 22 '23 11:04 lkeab