Error when trying to train a model with masks

[nlml]

Describe the bug When training a nerfacto model on data whose transforms.json contains mask_paths (i.e., I have corresponding masks for all images), I get an error that ends in the following:

  File "/home/liam-schoneveld/nerfstudio/nerfstudio/data/pixel_samplers.py", line 98, in collate_image_dataset_batch
    c, y, x = (i.flatten() for i in torch.split(indices, 1, dim=-1))
ValueError: too many values to unpack (expected 3)

To Reproduce ns-train nerfacto --data <path-to-data-with-masks-in-transforms-json> --vis viewer

I believe the fix here is to change this line https://github.com/nerfstudio-project/nerfstudio/blob/99ab8bd669a590efcbce0ff27727b16e0403d71d/nerfstudio/data/pixel_samplers.py#L98

from

c, y, x = (i.flatten() for i in torch.split(indices, 1, dim=-1))

to

c, y, x = (i.flatten() for i in torch.split(indices[..., :3], 1, dim=-1))

but I'm not 100% sure

[sagesimhon]

Had the same issue when using mask jpg's in my dataset. The bug was that I was using 3 RGB channels in the mask image, instead of 1 channel for a grayscale image. See https://github.com/nerfstudio-project/nerfstudio/issues/1208. According to the requirements specified, the masks must have 1 channel of black or white pixels.

Here is the code I used to convert to grayscale. Also, make sure that your black pixels represent areas to ignore- I had to invert my images:

from PIL import Image, ImageChops

path = '/path/to/images'
filenames = [f'mask_{i}' for i in range(1, 5)]
for f in filenames:
    name = f'{path}/{f}.jpg'
    out_name = f'{path}/{f}_gs.jpg'
    # img = ImageChops.invert(Image.open(name).convert('L')) # Uncomment if you need to invert 
    img.show()
    img.save(out_name)