Command to generate RegionPLC captions

[junha-l]

Hello,

Thank you for your time and for sharing your work!

I'm interested in reproducing the generated caption dataset provided in here. However, I couldn't find specific instructions on how to generate this dataset.

Could you please provide the exact command(s) or script used to produce the generated captions? Especially, the one used for scannet - spconv_clip_base15 experiment, i.e. caption_detic-template_and_kosmos_125k_iou0.2.json and scannet_caption_idx_detic-template_and_kosmos_125k_iou0.2.pkl?
This would be extremely helpful for reproducing your results and understanding the generation process.

[jihanyang]

Hi, the guideline for generating regional captions in RegionPLC.

• Step 1: Generate captions. You can just check https://github.com/CVMI-Lab/PLA/blob/648b7d78ad5c526a0b116969b449c13a1b3dfb03/tools/process_tools/generate_caption.py#L932. You should generate kosmos2 and detic_crop_caption in this case.

• Step 2: Generate caption index. This is to build the mapping between captions and points in 3D. Please check this file: https://github.com/CVMI-Lab/PLA/blob/regionplc/tools/process_tools/generate_caption_idx.py.

• Step 3: SFusion. You can run this file to merge different captions with SFusion: https://github.com/CVMI-Lab/PLA/blob/648b7d78ad5c526a0b116969b449c13a1b3dfb03/tools/process_tools/concate_captions_with_iou.py#L106-L123

[junha-l]

Thanks for the prompt and thorough instruction!

[junha-l]

Could you also share the arguments for generate_caption_idx.py for ScanNet dataset? I've checked the script but there are only examples for nuscenes dataset