some question about code

[Cccoooder]

When I tried to reproduce the code, I encountered the following problems. I hope I can get your help.

1. When I used the pre-training model you uploaded on GitHub, during the fine-tuning process, the training loss and test loss were always greater than 1, and the test-ACC remained about 0.55.
2. In trainval.py, there is a loop (for exp_dict in exp_list:) before trainval(), which causes the model to train until the end of the loop. I want to know what is the function of this loop?
3. During the pre-training and fine-tuning process, I noticed that the DataLoader generation was the same, with 64 classes for the training set, 16 for the validation set, and 20 for the test set. According to the model approach in your paper, pre-training should take 64 as the base class and divide it into training set and test set; The fine-tuning should use 20 as the Novel class and divide it into a training set and a test set during the fine-tuning process.Now I do not know how to divide the data set, which makes me very confused. I want to know whether I have a wrong understanding of the paper or the code.

[prlz77]

Hi @Cccoooder, thanks for the interest in our work :)

1. Could you provide me with details to reproduce that behavior?
2. By default our code runs a minimal hyper-parameter search, the for loop loops over different hyper-parameter configs.
3. We perform fine-tuning on the base classes only, could you please indicate what lines in the paper lead you to think we fine-tuning on 20 novel classes so that we can assess how to clear the confusion?

[Cccoooder]

Below are some details of the code I am reproducing. The accuracy and loss of the model have not changed since Epoch: 95, but the loss is relatively large and the highest accuracy is only 56 epoch current_lr_0 train_loss val_loss val_accuracy test_loss test_accuracy test_confidence 93 93 0.000010 1.025226 1.196003 0.557822 1.185151 0.564867 0.006464 94 94 0.000010 1.028784 1.202358 0.550467 1.191651 0.558707 0.006458 95 95 0.000001 1.060194 1.196753 0.558622 1.196398 0.558107 0.006696 96 96 0.000001 1.036160 1.197195 0.557267 1.187886 0.564120 0.006738 97 97 0.000001 1.033859 1.183196 0.562178 1.190399 0.561347 0.006869 Saved: ./logs/finetuning\f831e13d6e54b9e79a2684c8e212037d Saved Best: ./logs/finetuning\f831e13d6e54b9e79a2684c8e212037d
epoch current_lr_0 train_loss val_loss val_accuracy test_loss test_accuracy test_confidence 193 193 1.000000e-08 1.023047 1.194027 0.555933 1.185884 0.563320 0.006629 194 194 1.000000e-08 1.036564 1.197041 0.556867 1.188793 0.563773 0.006499 195 195 1.000000e-08 1.055097 1.199055 0.556289 1.190644 0.560160 0.006673 196 196 1.000000e-08 1.066227 1.192846 0.555200 1.188895 0.563027 0.006500 197 197 1.000000e-08 1.040845 1.193151 0.555600 1.196825 0.550600 0.006681 Saved: ./logs/finetuning\f831e13d6e54b9e79a2684c8e212037d epoch current_lr_0 train_loss val_loss val_accuracy test_loss test_accuracy test_confidence 194 194 1.000000e-08 1.036564 1.197041 0.556867 1.188793 0.563773 0.006499 195 195 1.000000e-08 1.055097 1.199055 0.556289 1.190644 0.560160 0.006673 196 196 1.000000e-08 1.066227 1.192846 0.555200 1.188895 0.563027 0.006500 197 197 1.000000e-08 1.040845 1.193151 0.555600 1.196825 0.550600 0.006681 198 198 1.000000e-08 1.051976 1.203936 0.547844 1.191300 0.556027 0.006801 Saved: ./logs/finetuning\f831e13d6e54b9e79a2684c8e212037d epoch current_lr_0 train_loss val_loss val_accuracy test_loss test_accuracy test_confidence 195 195 1.000000e-08 1.055097 1.199055 0.556289 1.190644 0.560160 0.006673 196 196 1.000000e-08 1.066227 1.192846 0.555200 1.188895 0.563027 0.006500 197 197 1.000000e-08 1.040845 1.193151 0.555600 1.196825 0.550600 0.006681 198 198 1.000000e-08 1.051976 1.203936 0.547844 1.191300 0.556027 0.006801 199 199 1.000000e-08 1.041681 1.192530 0.559400 1.191980 0.559573 0.006654

[prlz77]

Could you please share the contents of the exp_dict.json in ./logs/finetuning\f831e13d6e54b9e79a2684c8e212037d ? Also the python command you used to run this experiment.