Best model architecture + Backbone for accuracy ?

[sudo-vinnie]

What is the best combination model architecture - backbone for accuracy?

I trained two networks 1.- Lightweight OpenPose + ResNet18 2.- OpenPose + ResNet50

After performing the evaluation here are the results:

I got better performance with resnet18 and with fewer iterations. I added a couple of example images. I also noticed the images used for evaluation are inconsistent. but regardless of the variation, 7-11 images is a very small sample for evaluation.

ResNet 18 ( 100 000 iterations)

Evluating on total 11 images... Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets= 20 ] = 0.188
Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets= 20 ] = 0.587
Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets= 20 ] = 0.104   Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets= 20 ] = 0.182   Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets= 20 ] = 0.281   Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 20 ] = 0.265   Average Recall     (AR) @[ IoU=0.50      | area=   all | maxDets= 20 ] = 0.650   Average Recall     (AR) @[ IoU=0.75      | area=   all | maxDets= 20 ] = 0.200   Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets= 20 ] = 0.180   Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets= 20 ] = 0.520

ResNet 50 ( 1 000 000 iterations)

Evluating on total 7 images Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets= 20 ] = 0.007   Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets= 20 ] = 0.059   Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets= 20 ] = 0.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets= 20 ] = 0.027   Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets= 20 ] = 0.033   Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 20 ] = 0.029   Average Recall     (AR) @[ IoU=0.50      | area=   all | maxDets= 20 ] = 0.176   Average Recall     (AR) @[ IoU=0.75      | area=   all | maxDets= 20 ] = 0.000   Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets= 20 ] = 0.025   Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets= 20 ] = 0.100

[Gyx-One]

Hello @sudo-vinnie

1. All the model evaluation metric reported in this repo is over the whole MSCOCO or MPII dataset, so maybe you forget to give the function evaluate the parameter called total_eval_num which gives instruction on how many images are going to be evaluated. I'll have a check on the default setting. 2.The Openpose + Vgg19(default backbone) is the largest model with the best accuracy. 3.Tipycally to train a model needs 400000+ iterations, so you may need to wait more iterations for model to converge to a better optim. (The larger the model is, often the more iterations it needs to converge well) Thanks for using our library! :)

[sudo-vinnie]

@Gyx-One Thaks for the answer I tried training the network with Vgg19 as a backbone, but I get the following error when evaluating the network (​This does not happen with the ResNet18):

Config:

# >>> set model name is necessary to distinguish models (neccesarry)
Config.set_model_name("Human_OP_VGG")
# >>> set model architecture using Config.MODEL enum class (neccesarry)
Config.set_model_type(Config.MODEL.Openpose)
# >>> set model backbone using Config.BACKBONE enum class (not neccessary, each model has its default backbone)
Config.set_model_backbone(Config.BACKBONE.Vgg19)
# >>> set dataset to use (neccesarry)
Config.set_dataset_type(Config.DATA.MSCOCO)
# >>> set training type (not neccesary, default is single training, can use parallel training)
Config.set_train_type(Config.TRAIN.Single_train)
# >>> congratulations!, the simplest configuration is done, it's time to assemble the model and training pipeline

---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-9-d69ae292f048> in <module>
----> 1 evalu(model,dataset)

~/Kyutech/hyperpose-2.1.0_Human/hyperpose/Model/openpose/eval.py in evaluate(model, dataset, config, vis_num, total_eval_num)
    135 
    136     result_dic={"annotations":pd_anns}
--> 137     dataset.official_eval(result_dic,vis_dir)

~/Kyutech/hyperpose-2.1.0_Human/hyperpose/Dataset/mscoco_dataset/dataset.py in official_eval(self, pd_json, eval_dir)
    231         print(f"evluating on total {len(image_ids)} images...")
    232         gt_coco=COCO(gt_json_path)
--> 233         pd_coco=gt_coco.loadRes(pd_json_path)
    234 
    235         '''

~/anaconda3/envs/hyperpose/lib/python3.7/site-packages/pycocotools/coco.py in loadRes(self, resFile)
    326         assert set(annsImgIds) == (set(annsImgIds) & set(self.getImgIds())), \
    327                'Results do not correspond to current coco set'
--> 328         if 'caption' in anns[0]:
    329             imgIds = set([img['id'] for img in res.dataset['images']]) & set([ann['image_id'] for ann in anns])
    330             res.dataset['images'] = [img for img in res.dataset['images'] if img['id'] in imgIds]

IndexError: list index out of range