S3DIS Evalution Problem: AP=0 with pretrained model

[BlueBell93]

Hello, thank you for your great work!

I followed the steps in the README to evaluate the pretrained model on S3DIS (Area 5), but the results show almost all AP values as 0.

I downloaded the pretrained model and ran the following command python3 tools/test.py configs/s3dis/isbnet_s3dis_area5.yaml dataset/s3dis/head_s3dis_area5.pth. Here is the console output:

INFO - Load state dict from dataset/s3dis/head_s3dis_area5.pth
INFO - unexpected key in source state_dict: criterion.empty_weight
INFO - Load test dataset: 68 scans
INFO - Infer scene 1/68
INFO - Infer scene 11/68
INFO - Infer scene 21/68
INFO - Infer scene 31/68
INFO - Infer scene 41/68
INFO - Infer scene 51/68
INFO - Infer scene 61/68
INFO - Evaluate instance segmentation

################################################################
what           :      AP  AP_50%  AP_25%      AR  RC_50%  RC_25%
################################################################
ceiling        :   0.000   0.000   0.000   0.000   0.000   0.000
floor          :   0.097   0.167   0.390   0.163   0.265   0.515
wall           :   0.000   0.000   0.001   0.000   0.000   0.015
beam           :   0.000   0.000   0.000   0.000   0.000   0.000
column         :   0.000   0.000   0.000   0.000   0.000   0.000
window         :   0.000   0.000   0.000   0.000   0.000   0.000
door           :   0.000   0.000   0.000   0.000   0.000   0.000
chair          :   0.000   0.000   0.000   0.000   0.000   0.000
table          :   0.000   0.000   0.000   0.000   0.000   0.000
bookcase       :   0.000   0.000   0.000   0.000   0.000   0.000
sofa           :   0.000   0.000   0.000   0.000   0.000   0.000
board          :   0.000   0.000   0.000   0.000   0.000   0.000
clutter        :   0.000   0.000   0.000   0.000   0.000   0.000
----------------------------------------------------------------
average        :   0.007   0.013   0.030   0.013   0.020   0.041
################################################################

INFO - Evaluate instance segmentation by S3DIS metrics
ISBNet/isbnet/evaluation/s3dis_eval.py:186: RuntimeWarning: invalid value encountered in double_scalars
  prec = tp / (tp + fp)

################################################
what           :   MUCov   MWCov    Prec     Rec
################################################
ceiling        :   0.000   0.000   0.000   0.000
floor          :   0.343   0.343   0.265   0.265
wall           :   0.000   0.000   0.000   0.000
beam           :   0.000   0.000   0.000   0.000
column         :   0.000   0.000     nan   0.000
window         :   0.000   0.000   0.000   0.000
door           :   0.000   0.000     nan   0.000
chair          :   0.001   0.001   0.000   0.000
table          :   0.000   0.000     nan   0.000
bookcase       :   0.000   0.000   0.000   0.000
sofa           :   0.032   0.041   0.000   0.000
board          :   0.000   0.000     nan   0.000
clutter        :   0.001   0.001   0.000   0.000
################################################

mMUCov: 0.029013030115538803
mMWCov: 0.02975100740393172
mPrecision: 0.029411764705882353
mRecall: 0.020361990950226245

Would appreciate any insights on why the AP values are 0. Thanks!

[LIP773]

The download link for the checkpoint of the S3DIS dataset is broken. Could you please send me the checkpoint file you downloaded? Thank you very much.