Error while training on custom dataset

[peach6]

Hi,

Great work. I am trying to train the model on my custom dataset but I am stuck on this error.

File "train_PBAFN_stage1_fs.py", line 106, in concat = torch.cat([preserve_mask.cuda(),densepose,pose.cuda()],1) RuntimeError: CUDA error: device-side assert triggered terminate called without an active exception

[tengolagan]

hi, how did you manage to get the npy files for densepose ?

[ds-nat-j]

@tengolagan If you can get the jpg, .npy is simply another form, you can use np.asarray(img) ref: https://www.geeksforgeeks.org/how-to-convert-images-to-numpy-array/

[ds-nat-j]

@peach6 I think it's something to do with the image size not sure though. I'm able to train on the VITON_traindata dataset which has images of size 192,256. I'm trying to do the same for 768,1024 size images, I've changed the hardcoded values (wherever 256 * 192 was present) in train_PFAFN_stage1_fs.py, data/base_dataset.py, data/aligned_dataset.py but I'm still getting this error.

@SenHe Can we have some support for different image sizes? 192,256 is too low a resolution.

[dnyandeepc16]

@ds-nat-j @peach6 Hi, are you able to resolve this error while training on a custom dataset? If yes, could you please tell me what changes have you made? I was trying to train for 1024x768 resolution. Please let me know! @SenHe could you please resolve this issue?

[ds-nat-j]

@ds-nat-j @peach6 Hi, are you able to resolve this error while training on a custom dataset? If yes, could you please tell me what changes have you made? I was trying to train for 1024x768 resolution. Please let me know! @SenHe could you please resolve this issue?

Nope, I gave up on this implementation (and also on the vton space for a while) but maybe try GP-VTON They release a pre-trained model recently that accepts higher dimension, I also think they follow a PF-AFN type approach like this paper. Might be worth a shot.

[dnyandeepc16]

@ds-nat-j @peach6 Hi, are you able to resolve this error while training on a custom dataset? If yes, could you please tell me what changes have you made? I was trying to train for 1024x768 resolution. Please let me know! @SenHe could you please resolve this issue?

Nope, I gave up on this implementation (and also on the vton space for a while) but maybe try GP-VTON They release a pre-trained model recently that accepts higher dimension, I also think they follow a PF-AFN type approach like this paper. Might be worth a shot.

GP-VTON hasn't yet released the full inference code. Could you please tell me how can I train the Flow-Style model for training on images of size 384x512 or higher dim? If you have modified code for this or a similar higher resolution please share it. I'm really stuck at this point. And want to finish this project ASAP! A little help from your end would be great!

[KevinLiu0716]

@dnyandeepc16 Hi, have you manage to solve the error while training on larger size(like 384x512) dataset? Thank you