Saurabh Nair

Update dataloader to support 360 scenes

1

360 scenes weren't working without this change. After changing the ray-sampling bounds, it seems to work fine

LPIPS metric computation

1

Hi Author! The eval script doesn't seem to compute the LPIPS metric. The function [here](https://github.com/google/hypernerf/blob/main/eval.py#L69) seems to compute the SSIM instead.

Example script with dr435i crashes on launch

10

`./Examples/Stereo-Inertial/stereo_inertial_realsense_D435i Vocabulary/ORBvoc.txt ./Examples/Stereo-Inertial/RealSense_D435i.yaml` crashes with the following error in the end: ``` terminate called after throwing an instance of 'rs2::error' what(): Couldn't resolve requests Aborted (core dumped) ```

Hi @DerrickWang005, did you guys try an experiment with the VIT as the image backbone instead of the Resnet. CLIP says the VIT backbone worked better for the contrastive part....

Question about the paper / training

2

Hi Authors, Maybe I missed this while reading the paper: How did you tackle the dataset imbalance problem for each mode? For e.g. you'll have a lot more Image-Text pairs...

Why not just have the direction field with the origin, and remove the `viewdirs` field in [Rays](https://github.com/sxyu/plenoctree/blob/master/nerf_sh/nerf/utils.py#L53)? The direction field should always contain the normalized direction (unit vector) anyway. If...

Should [this](https://github.com/sxyu/plenoctree/blob/master/nerf_sh/nerf/datasets.py#L298) variable be separate from the training view poses?

Moving the foreground grid's location / increasing initial voxel size

1

Hi Authors, I tried out the models with our own data, but the top of the foreground objects seemed to have been clipped and sent to the background layers. Could...

Hi authors, I just read your work with Lisa++. Pretty impressive how the conversational dataset improved the segmentation metrics. Are you guys planning on releasing it sometime?

In the forward pass and the evaluation function, you'll seem to be indexing from the first token and then adding the 255 falses. https://github.com/dvlab-research/LISA/blob/main/model/LISA.py#L187 What's the logic behind this? Shouldn't...