nerfstudio icon indicating copy to clipboard operation
nerfstudio copied to clipboard

Add depth supervision to the semantic-nerfw model

Open yimingzhou1 opened this issue 2 years ago • 0 comments

This PR integrates depth supervision to the existing semantic-nerfw model.

  • Added a include_depth flag to the semantic-nerfw model
  • Added (optional) depth information to SemanticDataset
  • Added semantic_filenames to the nerfstudio_dataparser
  • Updated configurations for semantic-nerfw in method_configs.py

Note: The sitcom3d_dataparser is removed in method_configs.py to pass the check. Looks like it is more unified to have the semantics info included in the nerfstudio_dataparser just like depth info.

Trained the model on Replica Dataset with 2 views of a single scene using both semantics and depth supervision, the results of rendered rgb, depth and semantics of two other adjacent views in the eval set are shown below (left: groundtruth, right: rendered result)

view1 (RGB) Screenshot 2023-03-20 at 1 45 02 PM

view1 (depth) Screenshot 2023-03-20 at 1 42 57 PM

view1 (semantics) Screenshot 2023-03-20 at 1 45 12 PM

view2 (RGB) Screenshot 2023-03-20 at 3 51 27 PM

view2 (depth) Screenshot 2023-03-20 at 3 51 42 PM

view2 (semantics) Screenshot 2023-03-20 at 3 51 56 PM

yimingzhou1 avatar Mar 20 '23 22:03 yimingzhou1