FLYP Bad performance on ObjectNet

I run your code and got a similar result with the paper except ObjectNet. The top-1 acc on ObjectNet is only 54.7%. I found the images in my ObjectNet is 50273 but not 50000. I downloaded the ObjectNet from the official url https://www.dropbox.com/s/raw/cxeztdtm16nzvuw/objectnet-1.0.zip and unzip it. Do you have any idea why this happened?

Jun 07 '23 12:06 LixDemon

Hi, Thanks for pointing this out. I just checked my dataset, and it has 23.7k images only, probably some issue would have happened when I downloaded. I will update the paper with the correct numbers for ObjectNet once I redownload and re-evaluate. This issue would be their for all the baselines as well. So shouldn't affect any conclusions or ordering of various finetuning approaches. Thanks again for pointing this out.

Jun 07 '23 15:06 SachinG007

Really glad receive your reply! Another question I found is that when I evaluate on ObjectNet, not all images but only 18574 images of 113 classes are utilized. Is there something wrong with the code in ObjectNet?

Jun 11 '23 07:06 LixDemon

The code uses ImageNet data loader from folders (https://pytorch.org/vision/stable/generated/torchvision.datasets.ImageFolder.html). Can you check how many class folders are there in the local data path?

Jun 30 '23 23:06 SachinG007

There are 50273 images of 313 class folders in the local data path. Hope to see your revised results in your paper! Thanks!

Jul 03 '23 06:07 LixDemon

Hi, @SachinG007

I also have the same problem. If you already have any scores related to this, would you mind sharing them here? Thank you so much.

Jul 15 '23 03:07 BierOne

Hi @BierOne , I haven't yet got a chance to rerun the numbers. I can try though getting numbers by late next week.

Jul 15 '23 08:07 SachinG007

Glad to hear that! Thanks :)

Jul 15 '23 08:07 BierOne

Have you rerun the objectnet experiments yet? I got an acc of 55.4% for ZSL and 54.7% for FLYP. I can't match the performance of other methods (i.e. FT, LP, LP-FT), too.

Jul 29 '23 12:07 LixDemon

@LixDemon May I know if you could share your trained weight on ImageNet? It seems this repository does not provide it. Thank you!

Aug 02 '23 11:08 Simon4Yan

@LixDemon , I didn't yet get a chance to rerun the ImageNet experiments. I will get back on this. @Simon4Yan , I can upload the model once I rerun it.

Thanks

Aug 05 '23 01:08 SachinG007

Hi @LixDemon and @Simon4Yan , I am uploading a FLYP finetuned (on ImageNet) checkpoint of CLIP (ViT-B-16) here.

The updated FLYP and Zeroshot numbers for ObjectNet: Zeroshot: Evaluating on ObjectNet ObjectNet Top-1 accuracy: 0.5333

Finetuned: Evaluating on ObjectNet ObjectNet Top-1 accuracy: 0.5499

I will also update the paper once I get the numbers for the baselines as well. Thanks.

Aug 13 '23 20:08 SachinG007

Hi @SachinG007, Really glad to hear that! Can't wait to see the revised paper. Thanks for your detailed reply!

Aug 14 '23 01:08 LixDemon

@SachinG007 Thank you for sharing the model weights. Also, I am interested in the revised version. Kind Regards!

Aug 14 '23 01:08 Simon4Yan