AdaptFormer icon indicating copy to clipboard operation
AdaptFormer copied to clipboard

Inconsistency in pretrained mae vitb16 weights

Open jafarinia opened this issue 2 years ago • 0 comments

Hi I have a question about your pretreined mae vitb16 weights. When I use "https://github.com/ShoufaChen/AdaptFormer/releases/download/v0.1/mae_pretrain_vit_b.pth" which is the one on your GitHub repo and train on Cifar100 for 1 epoch I generally get Acc@1 37.330 Acc@5 67.820 loss 2.768 (the command is python main_image.py --batch_size=128 --cls_token --epochs=1 --finetune=mae_pretrain_vit_b.pth --dist_eval --data_path=cifar100 --output_dir=output --num_workers=16 --drop_path=0.0 --blr=0.1 --dataset=cifar100 --ffn_adapt) But When I use "https://dl.fbaipublicfiles.com/mae/pretrain/mae_pretrain_vit_base.pth" which is the claimed weight on your paper's text and train on Cifar100 for 1 epoch I generally get Acc@1 5.660 Acc@5 21.350 loss 4.295. (the command is python main_image.py --batch_size=128 --cls_token --epochs=1 --finetune=mae_pretrain_vit_base.pth --dist_e val --data_path=cifar100 --output_dir=output --num_workers=16 --drop_path=0.0 --blr=0.1 --dataset=cifar100 --ffn_adapt) These results are consistent over multiple runs. My questions are, what is the difference between these two pretrained weights? And what causes this huge difference in results?

jafarinia avatar Dec 14 '23 10:12 jafarinia