Model predicting wrong classes for custom dataset

[lalitr994]

Hi @NielsRogge I am using custom dataset for Fine_tuning_LayoutLMv2ForSequenceClassification_on_RVL_CDIP.ipynb notebook. for 500,600 examples training accuracy is 97 % and inference by the trained model is also correct. but when I train the model for around 6K examples. model is trained with 95-98 % trained accuracy but it's predicting the same class.

here are my dataset and params values. total examples 6800 total classes 5 (two classes examples are around 2.5K and 2.5K and rest are less in number) epoch 20 learning rate 0.000005

I tried with different LR's and epochs but result is same for most of them.