Loss not decreasing on default config settings

[codefreakSubham]

@rowanz Hi, I am trying to train the model from scratch, but am not able to reproduce the actual results. Specifically the loss is not decreasing in each epoch. I ran it for 20 epochs and the results are below. Anyone faced such an issue or know the possible reason for this? Any kind of suggestions will be of great help. Thank you.

TRAIN EPOCH 0: loss 1.356284 crl 0.144345 accuracy 0.311996 sec_per_batch 1.702358 hr_per_epoch 1.048369 dtype: float64

Val epoch 0 has acc 0.249 and loss 1.386 Best validation performance so far. Copying weights to 'saves/flagship_rationale/best.th'.

TRAIN EPOCH 1: loss 1.386393 crl 0.089470 accuracy 0.249471 sec_per_batch 2.008696 hr_per_epoch 1.237022 dtype: float64

Val epoch 1 has acc 0.249 and loss 1.386

TRAIN EPOCH 2: loss 1.386381 crl 0.075422 accuracy 0.251220 sec_per_batch 1.946174 hr_per_epoch 1.198519 dtype: float64

Epoch 2: reducing learning rate of group 0 to 1.0000e-04. Val epoch 2 has acc 0.249 and loss 1.386

TRAIN EPOCH 3: loss 1.386379 crl 0.050537 accuracy 0.248640 sec_per_batch 1.870728 hr_per_epoch 1.152057 dtype: float64

Val epoch 3 has acc 0.249 and loss 1.386

TRAIN EPOCH 4: loss 1.386330 crl 0.042339 accuracy 0.250779 sec_per_batch 2.006369 hr_per_epoch 1.235589 dtype: float64

Val epoch 4 has acc 0.249 and loss 1.386

TRAIN EPOCH 5: loss 1.386332 crl 0.037035 accuracy 0.250581 sec_per_batch 1.735174 hr_per_epoch 1.068578 dtype: float64

Val epoch 5 has acc 0.249 and loss 1.386

TRAIN EPOCH 6: loss 1.386333 crl 0.032566 accuracy 0.249394 sec_per_batch 2.384569 hr_per_epoch 1.468497 dtype: float64

Epoch 6: reducing learning rate of group 0 to 5.0000e-05. Val epoch 6 has acc 0.249 and loss 1.386

TRAIN EPOCH 7: loss 1.386345 crl 0.020694 accuracy 0.247829 sec_per_batch 2.088539 hr_per_epoch 1.286192 dtype: float64

Val epoch 7 has acc 0.249 and loss 1.386

TRAIN EPOCH 8: loss 1.386309 crl 0.017643 accuracy 0.251004 sec_per_batch 1.965981 hr_per_epoch 1.210717 dtype: float64

Val epoch 8 has acc 0.249 and loss 1.386

TRAIN EPOCH 9: loss 1.386299 crl 0.015537 accuracy 0.251415 sec_per_batch 1.872479 hr_per_epoch 1.153135 dtype: float64

Val epoch 9 has acc 0.249 and loss 1.386

TRAIN EPOCH 10: loss 1.386302 crl 0.014494 accuracy 0.251420 sec_per_batch 1.644809 hr_per_epoch 1.012928 dtype: float64

Epoch 10: reducing learning rate of group 0 to 2.5000e-05. Val epoch 10 has acc 0.249 and loss 1.386

TRAIN EPOCH 11: loss 1.386306 crl 0.009551 accuracy 0.252025 sec_per_batch 1.408009 hr_per_epoch 0.867099 dtype: float64

Val epoch 11 has acc 0.249 and loss 1.386

TRAIN EPOCH 12: loss 1.386314 crl 0.007876 accuracy 0.250382 sec_per_batch 1.419217 hr_per_epoch 0.874001 dtype: float64

Val epoch 12 has acc 0.249 and loss 1.386

TRAIN EPOCH 13: loss 1.386337 crl 0.007333 accuracy 0.248957 sec_per_batch 1.800047 hr_per_epoch 1.108529 dtype: float64

Val epoch 13 has acc 0.249 and loss 1.386

TRAIN EPOCH 14: loss 1.386308 crl 0.006972 accuracy 0.251202 sec_per_batch 1.691500 hr_per_epoch 1.041682 dtype: float64

Epoch 14: reducing learning rate of group 0 to 1.2500e-05. Val epoch 14 has acc 0.249 and loss 1.386

TRAIN EPOCH 15: loss 1.386294 crl 0.004941 accuracy 0.250033 sec_per_batch 1.976553 hr_per_epoch 1.217227 dtype: float64

Val epoch 15 has acc 0.249 and loss 1.386

TRAIN EPOCH 16: loss 1.386299 crl 0.004361 accuracy 0.250594 sec_per_batch 2.385966 hr_per_epoch 1.469357 dtype: float64

Val epoch 16 has acc 0.249 and loss 1.386

TRAIN EPOCH 17: loss 1.386329 crl 0.004206 accuracy 0.249658 sec_per_batch 2.463118 hr_per_epoch 1.516870 dtype: float64

Val epoch 17 has acc 0.249 and loss 1.386

TRAIN EPOCH 18: loss 1.386311 crl 0.003819 accuracy 0.249090 sec_per_batch 2.041939 hr_per_epoch 1.257494 dtype: float64

Epoch 18: reducing learning rate of group 0 to 6.2500e-06. Val epoch 18 has acc 0.249 and loss 1.386

TRAIN EPOCH 19: loss 1.386334 crl 0.003092 accuracy 0.249248 sec_per_batch 1.784414 hr_per_epoch 1.098902 dtype: float64

Val epoch 19 has acc 0.249 and loss 1.386