sdfstudio icon indicating copy to clipboard operation
sdfstudio copied to clipboard

Training stops at random points with no errors

Open davidceka opened this issue 2 years ago • 0 comments

Hi! I've just downloaded the given datasets (dtu and the replica one) and i'm started the training on them. Problem is, after a short while (usually before 1% training progress) the training freezes and shows no more progress, but the terminal page is always there (see image) image

it's been stuck like this for half an hour now. I've also noticed that the GPU usage is around 90% while it progresses, and suddenly drops to 0% at the time it stops progressing.

I don't think it's a memory issue since i haven't even swapped and ram usage is around 80% (16 GB total+16 swap)

VRAM usage is also fine, around 60% (8 GB total)

Can anyone help on this matter?

Also, training with nerfstudio works absolutely fine (if it can help)

davidceka avatar Oct 28 '23 13:10 davidceka