F1=0.01 with vgg16_gcan-afat-u_imcpt_100_stage1.yaml

[Lin-Yijie]

Dear authors, I have conducted experiments using the AFAT-GCAN model on the IMCPT-100 dataset but encountered random matching results. In contrast, directly running GCAN achieves an F1 score of approximately 66.7.

Below are the details and results:

STATISTIC_STEP: 100
RANDOM_SEED: 321
FP16: False
GCAN:
  FEATURE_CHANNEL: 512
  NODE_FEATURE_DIM: 1024
  NODE_HIDDEN_SIZE: [1024]
  SK_ITER_NUM: 20
  SK_EPSILON: 1e-10
  SK_TAU: 0.05
  CROSS_ITER: False
  CROSS_ITER_NUM: 1
AFA:
  UNIV_SIZE: 100
  K_FACTOR: 50.0
  REG_HIDDEN_FEAT: 8
  REGRESSION: False
  HEAD_NUM: 8
  KQV_DIM: 16
  FF_HIDDEN_DIM: 16
  MS_HIDDEN_DIM: 8
  MS_LAYER1_INIT: 10
  MS_LAYER2_INIT: 10
  MEAN_K: True
  K_GNN_LAYER: 2
  TN_NEURONS: 16
  AFAU: True
IMC_PT_SparseGM:
  ROOT_DIR_NPZ: data/IMC-PT-SparseGM/annotations_100
  MAX_KPT_NUM: 100
Start training...
model on device: cuda:0
----------
lr = 2.00e-03, 2.00e-05

Final obtrain

lr = 3.13e-05, 3.13e-07
Epoch 14   Iteration 100  2.20sample/s Loss=7.2406   Ks_Loss=0.0000   Ks_Error=0.0000  
Epoch 14   Iteration 200  2.15sample/s Loss=6.5199   Ks_Loss=0.0000   Ks_Error=0.0000  
Epoch 14   Iteration 300  2.21sample/s Loss=6.7213   Ks_Loss=0.0000   Ks_Error=0.0000  
Epoch 14   Iteration 400  2.26sample/s Loss=5.9973   Ks_Loss=0.0000   Ks_Error=0.0000  
Epoch 14   Iteration 500  2.28sample/s Loss=6.1740   Ks_Loss=0.0000   Ks_Error=0.0000  
Epoch 14   Iteration 600  2.29sample/s Loss=6.3601   Ks_Loss=0.0000   Ks_Error=0.0000  
Epoch 14   Iteration 700  2.12sample/s Loss=6.3862   Ks_Loss=0.0000   Ks_Error=0.0000  
Epoch 14   Iteration 800  2.14sample/s Loss=6.2037   Ks_Loss=0.0000   Ks_Error=0.0000  
Epoch 14   Iteration 900  2.15sample/s Loss=7.3311   Ks_Loss=0.0000   Ks_Error=0.0000  
Epoch 14   Iteration 1000 2.27sample/s Loss=6.1954   Ks_Loss=0.0000   Ks_Error=0.0000  
Epoch 14   Loss: 6.5130

Start evaluation...
In class reichstag: Ks_Loss=0.0000   Ks_Error=0.0000  
In class sacre_coeur: Ks_Loss=0.0000   Ks_Error=0.0000  
In class st_peters_square: Ks_Loss=0.0000   Ks_Error=0.0000  
Matching accuracy
reichstag: p = 0.0113±0.0176, r = 0.0113±0.0176, f1 = 0.0113±0.0176, cvg = 0.3604
sacre_coeur: p = 0.0145±0.0369, r = 0.0145±0.0369, f1 = 0.0145±0.0369, cvg = 0.0014
st_peters_square: p = 0.0149±0.0211, r = 0.0149±0.0211, f1 = 0.0149±0.0211, cvg = 0.0003
average accuracy: p = 0.0136±0.0252, r = 0.0136±0.0252, f1 = 0.0136±0.0252
gt perm mat cache deleted
Evaluation complete in 26m 49s

[ziao-guo]

It seems that some unstable random factors during training caused the model to crash. You can check the training logs, and there should be a sudden spike in the loss at a certain epoch.

To resolve this issue, you can adjust the random seed and learning rate appropriately. Additionally, when the model crashes, you should terminate the current training and resume training from an earlier checkpoint.

[rogerwwww]

I am closing this issue because it is no longer active. Please feel free to re-open it if the issue still exists!