Fine-tune Segnet Pascal with VGG architecture

[bhadresh74]

I was wondering if I can fine-tune the VGG architecture (Segnet_train.prototxt in this tutorial) with pre-trained segnet_pascal.caffemodel. I gave it a try and my per class accuracy is gradually decreasing after each iteration. But, over all accuracy is increasing which is good. But I would prefer that my both class has overall good accuracy rather the one doing good and one doing bad. Any help would be appreciated.