OpenPCDet icon indicating copy to clipboard operation
OpenPCDet copied to clipboard
verified publisher icon open-mmlab

AP drop for pointpillars on kitti dataset

Open MLbeginner2 opened this issue 3 years ago • 1 comments

I used the pre-trained model of pointpillars to continue training on kitti, but the AP on the validation set dropped by 6-7.Why is this? 2022-07-05 17:11:15,104 INFO Start logging 2022-07-05 17:11:15,104 INFO CUDA_VISIBLE_DEVICES=ALL 2022-07-05 17:11:15,104 INFO cfg_file cfgs/kitti_models/pointpillar_test.yaml 2022-07-05 17:11:15,104 INFO batch_size 1 2022-07-05 17:11:15,104 INFO epochs 20 2022-07-05 17:11:15,104 INFO workers 4 2022-07-05 17:11:15,104 INFO extra_tag default 2022-07-05 17:11:15,104 INFO ckpt None 2022-07-05 17:11:15,104 INFO pretrained_model /data/DL/tracking/OpenPCDet-Track/checkpoints/pointpillar_7728.pth 2022-07-05 17:11:15,104 INFO launcher none 2022-07-05 17:11:15,104 INFO tcp_port 18888 2022-07-05 17:11:15,105 INFO sync_bn False 2022-07-05 17:11:15,105 INFO fix_random_seed False 2022-07-05 17:11:15,105 INFO ckpt_save_interval 1 2022-07-05 17:11:15,105 INFO local_rank 0 2022-07-05 17:11:15,105 INFO max_ckpt_save_num 5 2022-07-05 17:11:15,105 INFO merge_all_iters_to_one_epoch False 2022-07-05 17:11:15,105 INFO set_cfgs None 2022-07-05 17:11:15,105 INFO max_waiting_mins 0 2022-07-05 17:11:15,105 INFO start_epoch 0 2022-07-05 17:11:15,105 INFO num_epochs_to_eval 5 2022-07-05 17:11:15,105 INFO save_to_file False 2022-07-05 17:11:15,105 INFO cfg.ROOT_DIR: /data/DL/3D_Detect/IA-SSD-main 2022-07-05 17:11:15,105 INFO cfg.LOCAL_RANK: 0 2022-07-05 17:11:15,105 INFO cfg.CLASS_NAMES: ['Car', 'Pedestrian', 'Cyclist'] 2022-07-05 17:11:15,105 INFO
cfg.DATA_CONFIG = edict() 2022-07-05 17:11:15,105 INFO cfg.DATA_CONFIG.DATASET: KittiDataset 2022-07-05 17:11:15,105 INFO cfg.DATA_CONFIG.DATA_PATH: ../data/kitti 2022-07-05 17:11:15,105 INFO cfg.DATA_CONFIG.POINT_CLOUD_RANGE: [0, -39.68, -3, 69.12, 39.68, 1] 2022-07-05 17:11:15,106 INFO
cfg.DATA_CONFIG.DATA_SPLIT = edict() 2022-07-05 17:11:15,106 INFO cfg.DATA_CONFIG.DATA_SPLIT.train: train 2022-07-05 17:11:15,106 INFO cfg.DATA_CONFIG.DATA_SPLIT.test: val 2022-07-05 17:11:15,106 INFO
cfg.DATA_CONFIG.INFO_PATH = edict() 2022-07-05 17:11:15,106 INFO cfg.DATA_CONFIG.INFO_PATH.train: ['kitti_infos_train.pkl'] 2022-07-05 17:11:15,106 INFO cfg.DATA_CONFIG.INFO_PATH.test: ['kitti_infos_val.pkl'] 2022-07-05 17:11:15,106 INFO cfg.DATA_CONFIG.GET_ITEM_LIST: ['points'] 2022-07-05 17:11:15,106 INFO cfg.DATA_CONFIG.FOV_POINTS_ONLY: True 2022-07-05 17:11:15,106 INFO
cfg.DATA_CONFIG.DATA_AUGMENTOR = edict() 2022-07-05 17:11:15,106 INFO cfg.DATA_CONFIG.DATA_AUGMENTOR.DISABLE_AUG_LIST: ['placeholder'] 2022-07-05 17:11:15,106 INFO cfg.DATA_CONFIG.DATA_AUGMENTOR.AUG_CONFIG_LIST: [{'NAME': 'gt_sampling', 'USE_ROAD_PLANE': False, 'DB_INFO_PATH': ['kitti_dbinfos_train.pkl'], 'PREPARE': {'filter_by_min_points': ['Car:5', 'Pedestrian:5', 'Cyclist:5'], 'filter_by_difficulty': [-1]}, 'SAMPLE_GROUPS': ['Car:15', 'Pedestrian:15', 'Cyclist:15'], 'NUM_POINT_FEATURES': 4, 'DATABASE_WITH_FAKELIDAR': False, 'REMOVE_EXTRA_WIDTH': [0.0, 0.0, 0.0], 'LIMIT_WHOLE_SCENE': False}, {'NAME': 'random_world_flip', 'ALONG_AXIS_LIST': ['x']}, {'NAME': 'random_world_rotation', 'WORLD_ROT_ANGLE': [-0.78539816, 0.78539816]}, {'NAME': 'random_world_scaling', 'WORLD_SCALE_RANGE': [0.95, 1.05]}] 2022-07-05 17:11:15,106 INFO
cfg.DATA_CONFIG.POINT_FEATURE_ENCODING = edict() 2022-07-05 17:11:15,106 INFO cfg.DATA_CONFIG.POINT_FEATURE_ENCODING.encoding_type: absolute_coordinates_encoding 2022-07-05 17:11:15,106 INFO cfg.DATA_CONFIG.POINT_FEATURE_ENCODING.used_feature_list: ['x', 'y', 'z', 'intensity'] 2022-07-05 17:11:15,106 INFO cfg.DATA_CONFIG.POINT_FEATURE_ENCODING.src_feature_list: ['x', 'y', 'z', 'intensity'] 2022-07-05 17:11:15,106 INFO cfg.DATA_CONFIG.DATA_PROCESSOR: [{'NAME': 'mask_points_and_boxes_outside_range', 'REMOVE_OUTSIDE_BOXES': True}, {'NAME': 'shuffle_points', 'SHUFFLE_ENABLED': {'train': True, 'test': False}}, {'NAME': 'transform_points_to_voxels', 'VOXEL_SIZE': [0.16, 0.16, 4], 'MAX_POINTS_PER_VOXEL': 32, 'MAX_NUMBER_OF_VOXELS': {'train': 16000, 'test': 40000}}] 2022-07-05 17:11:15,107 INFO cfg.DATA_CONFIG.BASE_CONFIG: cfgs/dataset_configs/kitti_dataset.yaml 2022-07-05 17:11:15,107 INFO
cfg.MODEL = edict() 2022-07-05 17:11:15,107 INFO cfg.MODEL.NAME: PointPillar 2022-07-05 17:11:15,107 INFO
cfg.MODEL.VFE = edict() 2022-07-05 17:11:15,107 INFO cfg.MODEL.VFE.NAME: PillarVFE 2022-07-05 17:11:15,107 INFO cfg.MODEL.VFE.WITH_DISTANCE: False 2022-07-05 17:11:15,107 INFO cfg.MODEL.VFE.USE_ABSLOTE_XYZ: True 2022-07-05 17:11:15,107 INFO cfg.MODEL.VFE.USE_NORM: True 2022-07-05 17:11:15,107 INFO cfg.MODEL.VFE.NUM_FILTERS: [64] 2022-07-05 17:11:15,107 INFO
cfg.MODEL.MAP_TO_BEV = edict() 2022-07-05 17:11:15,107 INFO cfg.MODEL.MAP_TO_BEV.NAME: PointPillarScatter 2022-07-05 17:11:15,107 INFO cfg.MODEL.MAP_TO_BEV.NUM_BEV_FEATURES: 64 2022-07-05 17:11:15,107 INFO
cfg.MODEL.BACKBONE_2D = edict() 2022-07-05 17:11:15,107 INFO cfg.MODEL.BACKBONE_2D.NAME: BaseBEVBackbone 2022-07-05 17:11:15,107 INFO cfg.MODEL.BACKBONE_2D.LAYER_NUMS: [3, 5, 5] 2022-07-05 17:11:15,107 INFO cfg.MODEL.BACKBONE_2D.LAYER_STRIDES: [2, 2, 2] 2022-07-05 17:11:15,107 INFO cfg.MODEL.BACKBONE_2D.NUM_FILTERS: [64, 128, 256] 2022-07-05 17:11:15,107 INFO cfg.MODEL.BACKBONE_2D.UPSAMPLE_STRIDES: [1, 2, 4] 2022-07-05 17:11:15,107 INFO cfg.MODEL.BACKBONE_2D.NUM_UPSAMPLE_FILTERS: [128, 128, 128] 2022-07-05 17:11:15,107 INFO
cfg.MODEL.DENSE_HEAD = edict() 2022-07-05 17:11:15,107 INFO cfg.MODEL.DENSE_HEAD.NAME: AnchorHeadSingle 2022-07-05 17:11:15,107 INFO cfg.MODEL.DENSE_HEAD.CLASS_AGNOSTIC: False 2022-07-05 17:11:15,107 INFO cfg.MODEL.DENSE_HEAD.USE_DIRECTION_CLASSIFIER: True 2022-07-05 17:11:15,107 INFO cfg.MODEL.DENSE_HEAD.DIR_OFFSET: 0.78539 2022-07-05 17:11:15,107 INFO cfg.MODEL.DENSE_HEAD.DIR_LIMIT_OFFSET: 0.0 2022-07-05 17:11:15,107 INFO cfg.MODEL.DENSE_HEAD.NUM_DIR_BINS: 2 2022-07-05 17:11:15,107 INFO cfg.MODEL.DENSE_HEAD.ANCHOR_GENERATOR_CONFIG: [{'class_name': 'Car', 'anchor_sizes': [[3.9, 1.6, 1.56]], 'anchor_rotations': [0, 1.57], 'anchor_bottom_heights': [-1.78], 'align_center': False, 'feature_map_stride': 2, 'matched_threshold': 0.6, 'unmatched_threshold': 0.45}, {'class_name': 'Pedestrian', 'anchor_sizes': [[0.8, 0.6, 1.73]], 'anchor_rotations': [0, 1.57], 'anchor_bottom_heights': [-0.6], 'align_center': False, 'feature_map_stride': 2, 'matched_threshold': 0.5, 'unmatched_threshold': 0.35}, {'class_name': 'Cyclist', 'anchor_sizes': [[1.76, 0.6, 1.73]], 'anchor_rotations': [0, 1.57], 'anchor_bottom_heights': [-0.6], 'align_center': False, 'feature_map_stride': 2, 'matched_threshold': 0.5, 'unmatched_threshold': 0.35}] 2022-07-05 17:11:15,108 INFO
cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG = edict() 2022-07-05 17:11:15,108 INFO cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG.NAME: AxisAlignedTargetAssigner 2022-07-05 17:11:15,108 INFO cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG.POS_FRACTION: -1.0 2022-07-05 17:11:15,108 INFO cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG.SAMPLE_SIZE: 512 2022-07-05 17:11:15,108 INFO cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG.NORM_BY_NUM_EXAMPLES: False 2022-07-05 17:11:15,108 INFO cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG.MATCH_HEIGHT: False 2022-07-05 17:11:15,108 INFO cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG.BOX_CODER: ResidualCoder 2022-07-05 17:11:15,108 INFO
cfg.MODEL.DENSE_HEAD.LOSS_CONFIG = edict() 2022-07-05 17:11:15,108 INFO
cfg.MODEL.DENSE_HEAD.LOSS_CONFIG.LOSS_WEIGHTS = edict() 2022-07-05 17:11:15,108 INFO cfg.MODEL.DENSE_HEAD.LOSS_CONFIG.LOSS_WEIGHTS.cls_weight: 1.0 2022-07-05 17:11:15,108 INFO cfg.MODEL.DENSE_HEAD.LOSS_CONFIG.LOSS_WEIGHTS.loc_weight: 2.0 2022-07-05 17:11:15,108 INFO cfg.MODEL.DENSE_HEAD.LOSS_CONFIG.LOSS_WEIGHTS.dir_weight: 0.2 2022-07-05 17:11:15,108 INFO cfg.MODEL.DENSE_HEAD.LOSS_CONFIG.LOSS_WEIGHTS.code_weights: [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] 2022-07-05 17:11:15,108 INFO
cfg.MODEL.POST_PROCESSING = edict() 2022-07-05 17:11:15,108 INFO cfg.MODEL.POST_PROCESSING.RECALL_THRESH_LIST: [0.3, 0.5, 0.7] 2022-07-05 17:11:15,108 INFO cfg.MODEL.POST_PROCESSING.SCORE_THRESH: 0.1 2022-07-05 17:11:15,108 INFO cfg.MODEL.POST_PROCESSING.OUTPUT_RAW_SCORE: False 2022-07-05 17:11:15,109 INFO cfg.MODEL.POST_PROCESSING.EVAL_METRIC: kitti 2022-07-05 17:11:15,109 INFO
cfg.MODEL.POST_PROCESSING.NMS_CONFIG = edict() 2022-07-05 17:11:15,109 INFO cfg.MODEL.POST_PROCESSING.NMS_CONFIG.MULTI_CLASSES_NMS: False 2022-07-05 17:11:15,109 INFO cfg.MODEL.POST_PROCESSING.NMS_CONFIG.NMS_TYPE: nms_gpu 2022-07-05 17:11:15,109 INFO cfg.MODEL.POST_PROCESSING.NMS_CONFIG.NMS_THRESH: 0.01 2022-07-05 17:11:15,109 INFO cfg.MODEL.POST_PROCESSING.NMS_CONFIG.NMS_PRE_MAXSIZE: 4096 2022-07-05 17:11:15,109 INFO cfg.MODEL.POST_PROCESSING.NMS_CONFIG.NMS_POST_MAXSIZE: 500 2022-07-05 17:11:15,109 INFO
cfg.OPTIMIZATION = edict() 2022-07-05 17:11:15,109 INFO cfg.OPTIMIZATION.BATCH_SIZE_PER_GPU: 4 2022-07-05 17:11:15,109 INFO cfg.OPTIMIZATION.NUM_EPOCHS: 80 2022-07-05 17:11:15,109 INFO cfg.OPTIMIZATION.OPTIMIZER: adam_onecycle 2022-07-05 17:11:15,109 INFO cfg.OPTIMIZATION.LR: 0.003 2022-07-05 17:11:15,109 INFO cfg.OPTIMIZATION.WEIGHT_DECAY: 0.01 2022-07-05 17:11:15,109 INFO cfg.OPTIMIZATION.MOMENTUM: 0.9 2022-07-05 17:11:15,109 INFO cfg.OPTIMIZATION.MOMS: [0.95, 0.85] 2022-07-05 17:11:15,109 INFO cfg.OPTIMIZATION.PCT_START: 0.4 2022-07-05 17:11:15,109 INFO cfg.OPTIMIZATION.DIV_FACTOR: 10 2022-07-05 17:11:15,109 INFO cfg.OPTIMIZATION.DECAY_STEP_LIST: [35, 45] 2022-07-05 17:11:15,109 INFO cfg.OPTIMIZATION.LR_DECAY: 0.1 2022-07-05 17:11:15,109 INFO cfg.OPTIMIZATION.LR_CLIP: 1e-07 2022-07-05 17:11:15,109 INFO cfg.OPTIMIZATION.LR_WARMUP: False 2022-07-05 17:11:15,109 INFO cfg.OPTIMIZATION.WARMUP_EPOCH: 1 2022-07-05 17:11:15,109 INFO cfg.OPTIMIZATION.GRAD_NORM_CLIP: 10 2022-07-05 17:11:15,109 INFO cfg.TAG: pointpillar_test 2022-07-05 17:11:15,109 INFO cfg.EXP_GROUP_PATH: kitti_models 2022-07-05 17:11:15,213 INFO Database filter by min points Car: 14357 => 13532 2022-07-05 17:11:15,214 INFO Database filter by min points Pedestrian: 2207 => 2168 2022-07-05 17:11:15,214 INFO Database filter by min points Cyclist: 734 => 705 2022-07-05 17:11:15,230 INFO Database filter by difficulty Car: 13532 => 10759 2022-07-05 17:11:15,232 INFO Database filter by difficulty Pedestrian: 2168 => 2075 2022-07-05 17:11:15,233 INFO Database filter by difficulty Cyclist: 705 => 581 2022-07-05 17:11:15,240 INFO Loading KITTI dataset 2022-07-05 17:11:15,320 INFO Total samples for KITTI dataset: 3712 2022-07-05 17:11:17,594 INFO ==> Loading parameters from checkpoint /data/DL/tracking/OpenPCDet-Track/checkpoints/pointpillar_7728.pth to GPU 2022-07-05 17:11:17,609 INFO ==> Done (loaded 127/127) 2022-07-05 17:11:17,610 INFO PointPillar( (vfe): PillarVFE( (pfn_layers): ModuleList( (0): PFNLayer( (linear): Linear(in_features=10, out_features=64, bias=False) (norm): BatchNorm1d(64, eps=0.001, momentum=0.01, affine=True, track_running_stats=True) ) ) ) (backbone_3d): None (map_to_bev_module): PointPillarScatter() (pfe): None (backbone_2d): BaseBEVBackbone( (blocks): ModuleList( (0): Sequential( (0): ZeroPad2d(padding=(1, 1, 1, 1), value=0.0) (1): Conv2d(64, 64, kernel_size=(3, 3), stride=(2, 2), bias=False) (2): BatchNorm2d(64, eps=0.001, momentum=0.01, affine=True, track_running_stats=True) (3): ReLU() (4): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (5): BatchNorm2d(64, eps=0.001, momentum=0.01, affine=True, track_running_stats=True) (6): ReLU() (7): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (8): BatchNorm2d(64, eps=0.001, momentum=0.01, affine=True, track_running_stats=True) (9): ReLU() (10): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (11): BatchNorm2d(64, eps=0.001, momentum=0.01, affine=True, track_running_stats=True) (12): ReLU() ) (1): Sequential( (0): ZeroPad2d(padding=(1, 1, 1, 1), value=0.0) (1): Conv2d(64, 128, kernel_size=(3, 3), stride=(2, 2), bias=False) (2): BatchNorm2d(128, eps=0.001, momentum=0.01, affine=True, track_running_stats=True) (3): ReLU() (4): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (5): BatchNorm2d(128, eps=0.001, momentum=0.01, affine=True, track_running_stats=True) (6): ReLU() (7): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (8): BatchNorm2d(128, eps=0.001, momentum=0.01, affine=True, track_running_stats=True) (9): ReLU() (10): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (11): BatchNorm2d(128, eps=0.001, momentum=0.01, affine=True, track_running_stats=True) (12): ReLU() (13): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (14): BatchNorm2d(128, eps=0.001, momentum=0.01, affine=True, track_running_stats=True) (15): ReLU() (16): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (17): BatchNorm2d(128, eps=0.001, momentum=0.01, affine=True, track_running_stats=True) (18): ReLU() ) (2): Sequential( (0): ZeroPad2d(padding=(1, 1, 1, 1), value=0.0) (1): Conv2d(128, 256, kernel_size=(3, 3), stride=(2, 2), bias=False) (2): BatchNorm2d(256, eps=0.001, momentum=0.01, affine=True, track_running_stats=True) (3): ReLU() (4): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (5): BatchNorm2d(256, eps=0.001, momentum=0.01, affine=True, track_running_stats=True) (6): ReLU() (7): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (8): BatchNorm2d(256, eps=0.001, momentum=0.01, affine=True, track_running_stats=True) (9): ReLU() (10): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (11): BatchNorm2d(256, eps=0.001, momentum=0.01, affine=True, track_running_stats=True) (12): ReLU() (13): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (14): BatchNorm2d(256, eps=0.001, momentum=0.01, affine=True, track_running_stats=True) (15): ReLU() (16): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (17): BatchNorm2d(256, eps=0.001, momentum=0.01, affine=True, track_running_stats=True) (18): ReLU() ) ) (deblocks): ModuleList( (0): Sequential( (0): ConvTranspose2d(64, 128, kernel_size=(1, 1), stride=(1, 1), bias=False) (1): BatchNorm2d(128, eps=0.001, momentum=0.01, affine=True, track_running_stats=True) (2): ReLU() ) (1): Sequential( (0): ConvTranspose2d(128, 128, kernel_size=(2, 2), stride=(2, 2), bias=False) (1): BatchNorm2d(128, eps=0.001, momentum=0.01, affine=True, track_running_stats=True) (2): ReLU() ) (2): Sequential( (0): ConvTranspose2d(256, 128, kernel_size=(4, 4), stride=(4, 4), bias=False) (1): BatchNorm2d(128, eps=0.001, momentum=0.01, affine=True, track_running_stats=True) (2): ReLU() ) ) ) (dense_head): AnchorHeadSingle( (cls_loss_func): SigmoidFocalClassificationLoss() (reg_loss_func): WeightedSmoothL1Loss() (dir_loss_func): WeightedCrossEntropyLoss() (conv_cls): Conv2d(384, 18, kernel_size=(1, 1), stride=(1, 1)) (conv_box): Conv2d(384, 42, kernel_size=(1, 1), stride=(1, 1)) (conv_dir_cls): Conv2d(384, 12, kernel_size=(1, 1), stride=(1, 1)) ) (point_head): None (roi_head): None ) 2022-07-05 17:11:17,611 INFO Start training kitti_models/pointpillar_test(default) 2022-07-05 19:58:07,948 INFO End training kitti_models/pointpillar_test(default)

2022-07-05 19:58:07,973 INFO Start evaluation kitti_models/pointpillar_test(default) 2022-07-05 19:58:07,976 INFO Loading KITTI dataset 2022-07-05 19:58:08,061 INFO Total samples for KITTI dataset: 3769 2022-07-05 19:58:08,063 INFO ==> Loading parameters from checkpoint /data/DL/3D_Detect/IA-SSD-main/output/kitti_models/pointpillar_test/default/ckpt/checkpoint_epoch_16.pth to GPU 2022-07-05 19:58:08,099 INFO ==> Checkpoint trained from version: pcdet+0.5.2+0000000 2022-07-05 19:58:08,102 INFO ==> Done (loaded 127/127) 2022-07-05 19:58:08,103 INFO *************** EPOCH 16 EVALUATION ***************** 2022-07-05 20:00:54,139 INFO *************** Performance of EPOCH 16 ***************** 2022-07-05 20:00:54,140 INFO Generate label finished(sec_per_example: 0.0441 second). 2022-07-05 20:00:54,140 INFO recall_roi_0.3: 0.000000 2022-07-05 20:00:54,140 INFO recall_rcnn_0.3: 0.925960 2022-07-05 20:00:54,141 INFO recall_roi_0.5: 0.000000 2022-07-05 20:00:54,141 INFO recall_rcnn_0.5: 0.858526 2022-07-05 20:00:54,142 INFO recall_roi_0.7: 0.000000 2022-07-05 20:00:54,142 INFO recall_rcnn_0.7: 0.578654 2022-07-05 20:00:54,144 INFO Average predicted number of objects(3769 samples): 16.401 2022-07-05 20:01:15,080 INFO Car [email protected], 0.70, 0.70: bbox AP:90.5750, 88.3857, 86.0865 bev AP:89.3396, 85.0189, 83.1125 3d AP:81.5228, 70.9720, 66.9649 aos AP:90.55, 88.08, 85.63 Car [email protected], 0.70, 0.70: bbox AP:95.3271, 89.1974, 88.1293 bev AP:91.5223, 86.8896, 84.2669 3d AP:83.1988, 71.7330, 67.3356 aos AP:95.30, 88.90, 87.65 Car [email protected], 0.50, 0.50: bbox AP:90.5750, 88.3857, 86.0865 bev AP:90.7142, 89.6465, 88.9790 3d AP:90.7027, 89.4347, 88.6086 aos AP:90.55, 88.08, 85.63 Car [email protected], 0.50, 0.50: bbox AP:95.3271, 89.1974, 88.1293 bev AP:95.4448, 94.0844, 91.6915 3d AP:95.5418, 93.3780, 91.1845 aos AP:95.30, 88.90, 87.65 Pedestrian [email protected], 0.50, 0.50: bbox AP:62.1489, 57.4330, 54.7125 bev AP:58.7746, 53.0083, 49.1152 3d AP:54.1386, 48.0799, 43.7748 aos AP:47.90, 44.45, 42.29 Pedestrian [email protected], 0.50, 0.50: bbox AP:62.0908, 57.0187, 54.2108 bev AP:58.5449, 51.8038, 47.6346 3d AP:52.5406, 46.0169, 41.5474 aos AP:45.95, 42.12, 40.02 Pedestrian [email protected], 0.25, 0.25: bbox AP:62.1489, 57.4330, 54.7125 bev AP:69.5415, 65.7106, 62.3450 3d AP:69.2756, 64.6791, 62.0074 aos AP:47.90, 44.45, 42.29 Pedestrian [email protected], 0.25, 0.25: bbox AP:62.0908, 57.0187, 54.2108 bev AP:70.8415, 65.7049, 62.5043 3d AP:70.4680, 65.0275, 61.7655 aos AP:45.95, 42.12, 40.02 Cyclist [email protected], 0.50, 0.50: bbox AP:85.2944, 73.5966, 70.2669 bev AP:80.8355, 64.8972, 61.2995 3d AP:77.6805, 60.1552, 55.6732 aos AP:82.94, 70.06, 66.54 Cyclist [email protected], 0.50, 0.50: bbox AP:87.9844, 74.7345, 70.7053 bev AP:82.7146, 65.4508, 61.1836 3d AP:78.4769, 59.3127, 55.3605 aos AP:85.43, 70.74, 66.63 Cyclist [email protected], 0.25, 0.25: bbox AP:85.2944, 73.5966, 70.2669 bev AP:84.4721, 71.4156, 68.0897 3d AP:84.4721, 71.4096, 68.0369 aos AP:82.94, 70.06, 66.54 Cyclist [email protected], 0.25, 0.25: bbox AP:87.9844, 74.7345, 70.7053 bev AP:87.6999, 72.7384, 68.8026 3d AP:87.6965, 72.7326, 68.7343 aos AP:85.43, 70.74, 66.63

2022-07-05 20:01:15,086 INFO Result is save to /data/DL/3D_Detect/IA-SSD-main/output/kitti_models/pointpillar_test/default/eval/eval_with_train/epoch_16/val 2022-07-05 20:01:15,086 INFO Evaluation done.* 2022-07-05 20:01:15,107 INFO Epoch 16 has been evaluated 2022-07-05 20:01:15,107 INFO ==> Loading parameters from checkpoint /data/DL/3D_Detect/IA-SSD-main/output/kitti_models/pointpillar_test/default/ckpt/checkpoint_epoch_17.pth to GPU 2022-07-05 20:01:15,148 INFO ==> Checkpoint trained from version: pcdet+0.5.2+0000000 2022-07-05 20:01:15,152 INFO ==> Done (loaded 127/127) 2022-07-05 20:01:15,154 INFO *************** EPOCH 17 EVALUATION ***************** 2022-07-05 20:04:00,353 INFO *************** Performance of EPOCH 17 ***************** 2022-07-05 20:04:00,354 INFO Generate label finished(sec_per_example: 0.0438 second). 2022-07-05 20:04:00,354 INFO recall_roi_0.3: 0.000000 2022-07-05 20:04:00,354 INFO recall_rcnn_0.3: 0.926814 2022-07-05 20:04:00,354 INFO recall_roi_0.5: 0.000000 2022-07-05 20:04:00,355 INFO recall_rcnn_0.5: 0.861488 2022-07-05 20:04:00,355 INFO recall_roi_0.7: 0.000000 2022-07-05 20:04:00,355 INFO recall_rcnn_0.7: 0.597392 2022-07-05 20:04:00,358 INFO Average predicted number of objects(3769 samples): 16.236 2022-07-05 20:04:13,783 INFO Car [email protected], 0.70, 0.70: bbox AP:90.5785, 88.8793, 87.6688 bev AP:89.6335, 86.1599, 83.1422 3d AP:81.0658, 71.0352, 67.2043 aos AP:90.56, 88.61, 87.27 Car [email protected], 0.70, 0.70: bbox AP:95.3315, 91.0710, 88.6800 bev AP:91.9145, 87.4162, 84.9052 3d AP:83.4161, 72.1408, 69.3407 aos AP:95.31, 90.79, 88.28 Car [email protected], 0.50, 0.50: bbox AP:90.5785, 88.8793, 87.6688 bev AP:90.7079, 89.9143, 89.3040 3d AP:90.7079, 89.7474, 88.9866 aos AP:90.56, 88.61, 87.27 Car [email protected], 0.50, 0.50: bbox AP:95.3315, 91.0710, 88.6800 bev AP:95.5784, 94.4501, 93.7532 3d AP:95.5344, 93.9736, 91.5970 aos AP:95.31, 90.79, 88.28 Pedestrian [email protected], 0.50, 0.50: bbox AP:63.6979, 59.3309, 56.4585 bev AP:60.7070, 54.7192, 51.0262 3d AP:55.8270, 49.4805, 45.6247 aos AP:46.50, 42.49, 40.02 Pedestrian [email protected], 0.50, 0.50: bbox AP:63.8552, 59.0659, 55.9743 bev AP:60.7116, 53.7687, 49.6443 3d AP:55.1246, 47.9774, 43.4279 aos AP:45.79, 41.56, 39.21 Pedestrian [email protected], 0.25, 0.25: bbox AP:63.6979, 59.3309, 56.4585 bev AP:71.4658, 67.2748, 63.6635 3d AP:71.4290, 66.4444, 63.3790 aos AP:46.50, 42.49, 40.02 Pedestrian [email protected], 0.25, 0.25: bbox AP:63.8552, 59.0659, 55.9743 bev AP:72.3396, 67.4012, 64.0901 3d AP:72.3000, 66.9662, 63.5221 aos AP:45.79, 41.56, 39.21 Cyclist [email protected], 0.50, 0.50: bbox AP:85.7287, 76.4986, 72.3026 bev AP:83.8341, 68.6811, 65.1311 3d AP:80.0190, 62.2262, 58.2070 aos AP:84.81, 74.10, 69.85 Cyclist [email protected], 0.50, 0.50: bbox AP:89.4769, 77.1309, 73.2237 bev AP:87.0907, 70.0608, 65.4785 3d AP:80.3351, 61.9782, 57.6415 aos AP:88.36, 74.64, 70.55 Cyclist [email protected], 0.25, 0.25: bbox AP:85.7287, 76.4986, 72.3026 bev AP:87.8492, 74.8887, 70.3792 3d AP:87.8440, 74.8380, 70.3661 aos AP:84.81, 74.10, 69.85 Cyclist [email protected], 0.25, 0.25: bbox AP:89.4769, 77.1309, 73.2237 bev AP:89.8614, 75.5202, 71.1347 3d AP:89.8588, 75.4845, 71.0164 aos AP:88.36, 74.64, 70.55

2022-07-05 20:04:13,783 INFO Result is save to /data/DL/3D_Detect/IA-SSD-main/output/kitti_models/pointpillar_test/default/eval/eval_with_train/epoch_17/val 2022-07-05 20:04:13,783 INFO Evaluation done.* 2022-07-05 20:04:13,805 INFO Epoch 17 has been evaluated 2022-07-05 20:04:13,805 INFO ==> Loading parameters from checkpoint /data/DL/3D_Detect/IA-SSD-main/output/kitti_models/pointpillar_test/default/ckpt/checkpoint_epoch_18.pth to GPU 2022-07-05 20:04:13,844 INFO ==> Checkpoint trained from version: pcdet+0.5.2+0000000 2022-07-05 20:04:13,848 INFO ==> Done (loaded 127/127) 2022-07-05 20:04:13,850 INFO *************** EPOCH 18 EVALUATION ***************** 2022-07-05 20:06:58,857 INFO *************** Performance of EPOCH 18 ***************** 2022-07-05 20:06:58,857 INFO Generate label finished(sec_per_example: 0.0438 second). 2022-07-05 20:06:58,858 INFO recall_roi_0.3: 0.000000 2022-07-05 20:06:58,858 INFO recall_rcnn_0.3: 0.932225 2022-07-05 20:06:58,858 INFO recall_roi_0.5: 0.000000 2022-07-05 20:06:58,858 INFO recall_rcnn_0.5: 0.867810 2022-07-05 20:06:58,858 INFO recall_roi_0.7: 0.000000 2022-07-05 20:06:58,858 INFO recall_rcnn_0.7: 0.604852 2022-07-05 20:06:58,860 INFO Average predicted number of objects(3769 samples): 20.460 2022-07-05 20:07:13,313 INFO Car [email protected], 0.70, 0.70: bbox AP:90.5744, 88.5590, 87.2596 bev AP:89.4479, 84.4622, 83.1055 3d AP:81.7456, 72.2882, 67.6867 aos AP:90.56, 88.33, 86.91 Car [email protected], 0.70, 0.70: bbox AP:95.4368, 89.2235, 88.4704 bev AP:91.9770, 86.8301, 84.8105 3d AP:83.6588, 73.5420, 69.8167 aos AP:95.42, 88.99, 88.12 Car [email protected], 0.50, 0.50: bbox AP:90.5744, 88.5590, 87.2596 bev AP:90.7035, 89.7046, 89.0777 3d AP:90.7035, 89.5184, 88.8153 aos AP:90.56, 88.33, 86.91 Car [email protected], 0.50, 0.50: bbox AP:95.4368, 89.2235, 88.4704 bev AP:95.7004, 93.9564, 93.2297 3d AP:95.6537, 92.9416, 91.3200 aos AP:95.42, 88.99, 88.12 Pedestrian [email protected], 0.50, 0.50: bbox AP:62.2947, 57.8193, 55.8970 bev AP:59.8191, 53.7009, 50.0723 3d AP:55.3460, 48.7542, 45.0069 aos AP:44.89, 41.12, 39.37 Pedestrian [email protected], 0.50, 0.50: bbox AP:62.2131, 57.5512, 54.9309 bev AP:59.4325, 52.6783, 48.5984 3d AP:54.3884, 47.3121, 42.7932 aos AP:44.04, 40.39, 38.47 Pedestrian [email protected], 0.25, 0.25: bbox AP:62.2947, 57.8193, 55.8970 bev AP:68.9931, 65.1805, 62.3831 3d AP:68.9215, 64.7851, 62.0722 aos AP:44.89, 41.12, 39.37 Pedestrian [email protected], 0.25, 0.25: bbox AP:62.2131, 57.5512, 54.9309 bev AP:70.0013, 65.5867, 62.4833 3d AP:69.9280, 65.0688, 62.0148 aos AP:44.04, 40.39, 38.47 Cyclist [email protected], 0.50, 0.50: bbox AP:85.2178, 74.3177, 70.5701 bev AP:82.6834, 67.2174, 62.8096 3d AP:79.3809, 61.1168, 57.2711 aos AP:83.56, 71.60, 67.81 Cyclist [email protected], 0.50, 0.50: bbox AP:88.7532, 75.6581, 71.3898 bev AP:85.4843, 67.6186, 63.1074 3d AP:80.2613, 60.9532, 56.9327 aos AP:86.96, 72.61, 68.19 Cyclist [email protected], 0.25, 0.25: bbox AP:85.2178, 74.3177, 70.5701 bev AP:85.0907, 72.4793, 69.4119 3d AP:85.0907, 72.2337, 68.6059 aos AP:83.56, 71.60, 67.81 Cyclist [email protected], 0.25, 0.25: bbox AP:88.7532, 75.6581, 71.3898 bev AP:88.4911, 73.5748, 69.5934 3d AP:88.4861, 73.4360, 69.2510 aos AP:86.96, 72.61, 68.19

2022-07-05 20:07:13,314 INFO Result is save to /data/DL/3D_Detect/IA-SSD-main/output/kitti_models/pointpillar_test/default/eval/eval_with_train/epoch_18/val 2022-07-05 20:07:13,314 INFO Evaluation done.* 2022-07-05 20:07:13,336 INFO Epoch 18 has been evaluated 2022-07-05 20:07:13,336 INFO ==> Loading parameters from checkpoint /data/DL/3D_Detect/IA-SSD-main/output/kitti_models/pointpillar_test/default/ckpt/checkpoint_epoch_19.pth to GPU 2022-07-05 20:07:13,377 INFO ==> Checkpoint trained from version: pcdet+0.5.2+0000000 2022-07-05 20:07:13,381 INFO ==> Done (loaded 127/127) 2022-07-05 20:07:13,382 INFO *************** EPOCH 19 EVALUATION ***************** 2022-07-05 20:09:58,593 INFO *************** Performance of EPOCH 19 ***************** 2022-07-05 20:09:58,593 INFO Generate label finished(sec_per_example: 0.0438 second). 2022-07-05 20:09:58,594 INFO recall_roi_0.3: 0.000000 2022-07-05 20:09:58,594 INFO recall_rcnn_0.3: 0.931769 2022-07-05 20:09:58,594 INFO recall_roi_0.5: 0.000000 2022-07-05 20:09:58,594 INFO recall_rcnn_0.5: 0.866784 2022-07-05 20:09:58,594 INFO recall_roi_0.7: 0.000000 2022-07-05 20:09:58,594 INFO recall_rcnn_0.7: 0.608156 2022-07-05 20:09:58,596 INFO Average predicted number of objects(3769 samples): 18.737 2022-07-05 20:10:12,294 INFO Car [email protected], 0.70, 0.70: bbox AP:90.6179, 88.7373, 87.3052 bev AP:89.5666, 85.3960, 83.2504 3d AP:82.8452, 74.1180, 71.2733 aos AP:90.60, 88.47, 86.86 Car [email protected], 0.70, 0.70: bbox AP:95.6401, 89.3999, 88.5563 bev AP:92.1471, 87.2365, 84.9539 3d AP:84.6978, 74.5439, 71.1697 aos AP:95.61, 89.14, 88.12 Car [email protected], 0.50, 0.50: bbox AP:90.6179, 88.7373, 87.3052 bev AP:90.7513, 89.8628, 89.1450 3d AP:90.7479, 89.6432, 88.8636 aos AP:90.60, 88.47, 86.86 Car [email protected], 0.50, 0.50: bbox AP:95.6401, 89.3999, 88.5563 bev AP:95.9663, 94.2266, 93.2518 3d AP:95.9216, 92.1401, 91.4740 aos AP:95.61, 89.14, 88.12 Pedestrian [email protected], 0.50, 0.50: bbox AP:62.6270, 57.4009, 54.9493 bev AP:59.2799, 52.9963, 49.1072 3d AP:54.4392, 47.5878, 43.3693 aos AP:46.37, 41.91, 40.07 Pedestrian [email protected], 0.50, 0.50: bbox AP:62.4103, 57.0318, 54.2652 bev AP:59.0257, 51.7396, 47.7057 3d AP:53.1719, 45.6963, 40.9132 aos AP:45.52, 40.83, 38.68 Pedestrian [email protected], 0.25, 0.25: bbox AP:62.6270, 57.4009, 54.9493 bev AP:69.5815, 64.9131, 61.7862 3d AP:69.5559, 64.7573, 61.4401 aos AP:46.37, 41.91, 40.07 Pedestrian [email protected], 0.25, 0.25: bbox AP:62.4103, 57.0318, 54.2652 bev AP:70.2714, 65.0817, 61.9881 3d AP:70.2412, 64.8066, 61.5829 aos AP:45.52, 40.83, 38.68 Cyclist [email protected], 0.50, 0.50: bbox AP:85.3952, 73.6672, 69.7396 bev AP:83.3018, 66.9391, 62.3290 3d AP:80.1782, 61.1657, 57.9421 aos AP:83.98, 70.92, 66.84 Cyclist [email protected], 0.50, 0.50: bbox AP:89.3106, 74.9441, 70.8156 bev AP:86.6693, 66.9877, 62.5633 3d AP:80.9615, 61.1776, 57.3112 aos AP:87.69, 71.83, 67.59 Cyclist [email protected], 0.25, 0.25: bbox AP:85.3952, 73.6672, 69.7396 bev AP:85.1261, 72.5620, 68.8019 3d AP:85.1261, 72.3222, 68.7696 aos AP:83.98, 70.92, 66.84 Cyclist [email protected], 0.25, 0.25: bbox AP:89.3106, 74.9441, 70.8156 bev AP:89.4130, 73.4385, 69.3648 3d AP:89.4114, 73.3245, 69.2916 aos AP:87.69, 71.83, 67.59

2022-07-05 20:10:12,298 INFO Result is save to /data/DL/3D_Detect/IA-SSD-main/output/kitti_models/pointpillar_test/default/eval/eval_with_train/epoch_19/val 2022-07-05 20:10:12,298 INFO Evaluation done.* 2022-07-05 20:10:12,321 INFO Epoch 19 has been evaluated 2022-07-05 20:10:12,321 INFO ==> Loading parameters from checkpoint /data/DL/3D_Detect/IA-SSD-main/output/kitti_models/pointpillar_test/default/ckpt/checkpoint_epoch_20.pth to GPU 2022-07-05 20:10:12,367 INFO ==> Checkpoint trained from version: pcdet+0.5.2+0000000 2022-07-05 20:10:12,371 INFO ==> Done (loaded 127/127) 2022-07-05 20:10:12,373 INFO *************** EPOCH 20 EVALUATION ***************** 2022-07-05 20:12:57,972 INFO *************** Performance of EPOCH 20 ***************** 2022-07-05 20:12:57,972 INFO Generate label finished(sec_per_example: 0.0439 second). 2022-07-05 20:12:57,972 INFO recall_roi_0.3: 0.000000 2022-07-05 20:12:57,972 INFO recall_rcnn_0.3: 0.931142 2022-07-05 20:12:57,972 INFO recall_roi_0.5: 0.000000 2022-07-05 20:12:57,972 INFO recall_rcnn_0.5: 0.865759 2022-07-05 20:12:57,972 INFO recall_roi_0.7: 0.000000 2022-07-05 20:12:57,972 INFO recall_rcnn_0.7: 0.607529 2022-07-05 20:12:57,975 INFO Average predicted number of objects(3769 samples): 18.543 2022-07-05 20:13:11,697 INFO Car [email protected], 0.70, 0.70: bbox AP:90.6512, 88.8421, 87.5406 bev AP:89.6233, 85.9728, 83.3138 3d AP:83.1098, 74.3727, 71.4245 aos AP:90.63, 88.58, 87.14 Car [email protected], 0.70, 0.70: bbox AP:95.7790, 89.5114, 88.6559 bev AP:92.2252, 87.4551, 85.0330 3d AP:85.0417, 74.7527, 71.8429 aos AP:95.76, 89.27, 88.26 Car [email protected], 0.50, 0.50: bbox AP:90.6512, 88.8421, 87.5406 bev AP:90.7660, 89.8675, 89.2209 3d AP:90.7660, 89.6894, 88.9294 aos AP:90.63, 88.58, 87.14 Car [email protected], 0.50, 0.50: bbox AP:95.7790, 89.5114, 88.6559 bev AP:96.0746, 94.3300, 93.3327 3d AP:96.0346, 93.2606, 91.5472 aos AP:95.76, 89.27, 88.26 Pedestrian [email protected], 0.50, 0.50: bbox AP:62.1122, 56.9283, 54.7295 bev AP:58.7824, 52.3828, 48.7614 3d AP:54.5900, 47.5613, 43.5874 aos AP:46.95, 42.57, 40.66 Pedestrian [email protected], 0.50, 0.50: bbox AP:62.0461, 56.7338, 53.9252 bev AP:58.3611, 51.3068, 47.0826 3d AP:53.3105, 45.8904, 41.1278 aos AP:46.27, 41.65, 39.26 Pedestrian [email protected], 0.25, 0.25: bbox AP:62.1122, 56.9283, 54.7295 bev AP:69.5566, 64.8114, 61.8084 3d AP:69.5056, 63.9941, 61.3358 aos AP:46.95, 42.57, 40.66 Pedestrian [email protected], 0.25, 0.25: bbox AP:62.0461, 56.7338, 53.9252 bev AP:70.4623, 64.9483, 61.8657 3d AP:70.2710, 64.4790, 61.1392 aos AP:46.27, 41.65, 39.26 Cyclist [email protected], 0.50, 0.50: bbox AP:85.6064, 74.1578, 70.1859 bev AP:82.8039, 68.0793, 63.9360 3d AP:79.3475, 60.9698, 58.0594 aos AP:84.17, 71.43, 67.34 Cyclist [email protected], 0.50, 0.50: bbox AP:89.4778, 75.6134, 71.3811 bev AP:85.2932, 68.3946, 63.8085 3d AP:80.0532, 61.2597, 57.1948 aos AP:87.80, 72.61, 68.10 Cyclist [email protected], 0.25, 0.25: bbox AP:85.6064, 74.1578, 70.1859 bev AP:85.5461, 73.5042, 69.9655 3d AP:85.5461, 72.9736, 68.9360 aos AP:84.17, 71.43, 67.34 Cyclist [email protected], 0.25, 0.25: bbox AP:89.4778, 75.6134, 71.3811 bev AP:89.4346, 74.2956, 70.4806 3d AP:89.4318, 74.0075, 69.9389 aos AP:87.80, 72.61, 68.10

2022-07-05 20:13:11,697 INFO Result is save to /data/DL/3D_Detect/IA-SSD-main/output/kitti_models/pointpillar_test/default/eval/eval_with_train/epoch_20/val 2022-07-05 20:13:11,697 INFO Evaluation done.* 2022-07-05 20:13:11,723 INFO Epoch 20 has been evaluated 2022-07-05 20:13:41,755 INFO End evaluation kitti_models/pointpillar_test(default)

MLbeginner2 avatar Jul 08 '22 00:07 MLbeginner2

Probably it's because your initial learning rate is too high (cfg.OPTIMIZATION.LR: 0.003) and the supervision signals are too strong. This might lead to you exiting the minima reached by the initial training and land in a worse one.

You could try to start with a smaller LR and test if things improve.

aldipiroli avatar Jul 20 '22 09:07 aldipiroli

This issue is stale because it has been open for 30 days with no activity.

github-actions[bot] avatar Aug 20 '22 02:08 github-actions[bot]

This issue was closed because it has been inactive for 14 days since being marked as stale.

github-actions[bot] avatar Sep 03 '22 02:09 github-actions[bot]