cuda out of memory

[Freeverc]

Is there any parameter can reduce cuda memory , please? "cuda out of memory" for 400+ images on 24G cuda device 3090

[yuedajiong]

My GPU is OK: 12G Beggar's，700*1200 size, 240 images.

1. reduce image size
2. reduce gauss-points (important, BUT maybe your images are very complex.)

[cubantonystark]

if the error is directly related to pytorch reserving GPU memory, you can try adding:

os.environ["PYTORCH_NO_CUDA_MEMORY_CACHING"]= "1"

and clearing the cache with

torch.cuda.empty_cache()

Hope that helps

[JuliusQv]

suagr_model will load the dataset to the GPU by default, which will consume a lot of GPU memory.

So, you can try the fllowing steps:

1. modified this code, image_height=image_height, image_width=image_width, data_device='cpu')
2. move the image to cuda during training process, modified this code gt_image = nerfmodel.get_gt_image(camera_indices=camera_indices).cuda()
3. same action in coarse_density.py and refined.py

By use this method, 2000+images on my 24GB device 4090 is okay.

Hope this help.

[cubantonystark]

Gentlemen,

After implementing this suggestion, I get the following error now:

scales 0.005 quaternions 0.001 Densifier initialized. Using loss function: l1+dssim Traceback (most recent call last): File "/home/reynel/SuGaR/train.py", line 118, in coarse_sugar_path = coarse_training_with_sdf_regularization(coarse_args) File "/home/reynel/SuGaR/sugar_trainers/coarse_sdf.py", line 526, in coarse_training_with_sdf_regularization loss = loss_fn(pred_rgb, gt_rgb) File "/home/reynel/SuGaR/sugar_trainers/coarse_sdf.py", line 447, in loss_fn return (1.0 - dssim_factor) * l1_loss(pred_rgb, gt_rgb) + dssim_factor * (1.0 - ssim(pred_rgb, gt_rgb)) File "/home/reynel/SuGaR/sugar_utils/loss_utils.py", line 18, in l1_loss return torch.abs((network_output - gt)).mean() RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!

Any help will be greatly appreciated.

From: @.> Sent: Thursday, December 21, 2023 9:09 PM To: @.> Cc: Cuban Tony @.>; @.> Subject: Re: [Anttwo/SuGaR] cuda out of memory (Issue #40)

suagr_model will load the dataset to the GPU by default, which will consume a lot of GPU memory.

So, you can try the fllowing steps:

1. modified this codehttps://github.com/Anttwo/SuGaR/blob/72b9baa942593a8d3d7ef205288788821cedb1e8/sugar_scene/cameras.py#L100C65-L100C65, image_height=image_height, image_width=image_width, data_device='cpu')
2. move the image to cuda during training process, modified this codehttps://github.com/Anttwo/SuGaR/blob/72b9baa942593a8d3d7ef205288788821cedb1e8/sugar_trainers/coarse_sdf.py#L521C81-L521C81 gt_image = nerfmodel.get_gt_image(camera_indices=camera_indices).cuda()
3. same action in coarse_density.py and refined.py

By use this method, 2000+images on my 24GB device 4090 is okay.

Hope this help.

— Reply to this email directly, view it on GitHubhttps://github.com/Anttwo/SuGaR/issues/40#issuecomment-1867142488, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ABN7U2J2UQ5452HS5555OMLYKTTWJAVCNFSM6AAAAABA6C5JMCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNRXGE2DENBYHA. You are receiving this because you commented.Message ID: @.***>

[cubantonystark]

Please disregard previous email, I had mistyped the suggestion and formatted the parentheses in …camera_indices).cuda() as …camera_indices.cuda())

My mistake. I’m rerunning my datasets again now. Will report back with results. Anttwo, please let me know if you would like me to share results of my dataset with you to showcase quality.

Sent from Mailhttps://go.microsoft.com/fwlink/?LinkId=550986 for Windows

From: @.> Sent: Thursday, December 21, 2023 9:09 PM To: @.> Cc: Cuban Tony @.>; @.> Subject: Re: [Anttwo/SuGaR] cuda out of memory (Issue #40)

suagr_model will load the dataset to the GPU by default, which will consume a lot of GPU memory.

So, you can try the fllowing steps:

1. modified this codehttps://github.com/Anttwo/SuGaR/blob/72b9baa942593a8d3d7ef205288788821cedb1e8/sugar_scene/cameras.py#L100C65-L100C65, image_height=image_height, image_width=image_width, data_device='cpu')
2. move the image to cuda during training process, modified this codehttps://github.com/Anttwo/SuGaR/blob/72b9baa942593a8d3d7ef205288788821cedb1e8/sugar_trainers/coarse_sdf.py#L521C81-L521C81 gt_image = nerfmodel.get_gt_image(camera_indices=camera_indices).cuda()
3. same action in coarse_density.py and refined.py

By use this method, 2000+images on my 24GB device 4090 is okay.

Hope this help.

— Reply to this email directly, view it on GitHubhttps://github.com/Anttwo/SuGaR/issues/40#issuecomment-1867142488, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ABN7U2J2UQ5452HS5555OMLYKTTWJAVCNFSM6AAAAABA6C5JMCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNRXGE2DENBYHA. You are receiving this because you commented.Message ID: @.***>

[cubantonystark]

Julius, Are you reducing image size bwfore processing?  Got OOM again.  My dataset is only 26 images at 1900x1423.Best regards,Reynel RodriguezOn Dec 22, 2023 01:40, Reynel Rodriguez @.***> wrote:

Please disregard previous email, I had mistyped the suggestion and formatted the parentheses in …camera_indices).cuda() as …camera_indices.cuda())   My mistake. I’m rerunning my datasets again now. Will report back with results. Anttwo, please let me know if you would like me to share results of my dataset with you to showcase quality.   Sent from Mail for Windows  

From: julius Sent: Thursday, December 21, 2023 9:09 PM To: Anttwo/SuGaR Cc: Cuban Tony Stark; Comment Subject: Re: [Anttwo/SuGaR] cuda out of memory (Issue #40)

  suagr_model will load the dataset to the GPU by default, which will consume a lot of GPU memory. So, you can try the fllowing steps:

modified this code, image_height=image_height, image_width=image_width, data_device='cpu') move the image to cuda during training process, modified this code gt_image = nerfmodel.get_gt_image(camera_indices=camera_indices).cuda() same action in coarse_density.py and refined.py By use this method, 2000+images on my 24GB device 4090 is okay. Hope this help. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.***>  

[JuliusQv]

Julius, Are you reducing image size bwfore processing?  Got OOM again.  My dataset is only 26 images at 1900x1423.Best regards,Reynel RodriguezOn Dec 22, 2023 01:40, Reynel Rodriguez @.> wrote: Please disregard previous email, I had mistyped the suggestion and formatted the parentheses in …camera_indices).cuda() as …camera_indices.cuda())   My mistake. I’m rerunning my datasets again now. Will report back with results. Anttwo, please let me know if you would like me to share results of my dataset with you to showcase quality.   Sent from Mail for Windows   From: julius Sent: Thursday, December 21, 2023 9:09 PM To: Anttwo/SuGaR Cc: Cuban Tony Stark; Comment Subject: Re: [Anttwo/SuGaR] cuda out of memory (Issue #40)   suagr_model will load the dataset to the GPU by default, which will consume a lot of GPU memory. So, you can try the fllowing steps: modified this code, image_height=image_height, image_width=image_width, data_device='cpu') move the image to cuda during training process, modified this code gt_image = nerfmodel.get_gt_image(camera_indices=camera_indices).cuda() same action in coarse_density.py and refined.py By use this method, 2000+images on my 24GB device 4090 is okay. Hope this help. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.>

No，I did not reduce the images size.
What GPU are you using and how much memory? And which step does OOM happen? In my experiments, I found that the final texture mesh extraction stage requires a large amount of GPU memory.

[cubantonystark]

Thats the step where im sruck and i get the OOM. Best regards,Reynel RodriguezOn Dec 22, 2023 04:16, julius @.***> wrote:

Julius, Are you reducing image size bwfore processing?  Got OOM again.  My dataset is only 26 images at 1900x1423.Best regards,Reynel RodriguezOn Dec 22, 2023 01:40, Reynel Rodriguez @.> wrote: Please disregard previous email, I had mistyped the suggestion and formatted the parentheses in …camera_indices).cuda() as …camera_indices.cuda())   My mistake. I’m rerunning my datasets again now. Will report back with results. Anttwo, please let me know if you would like me to share results of my dataset with you to showcase quality.   Sent from Mail for Windows   From: julius Sent: Thursday, December 21, 2023 9:09 PM To: Anttwo/SuGaR Cc: Cuban Tony Stark; Comment Subject: Re: [Anttwo/SuGaR] cuda out of memory (Issue #40)   suagr_model will load the dataset to the GPU by default, which will consume a lot of GPU memory. So, you can try the fllowing steps: modified this code, image_height=image_height, image_width=image_width, data_device='cpu') move the image to cuda during training process, modified this code gt_image = nerfmodel.get_gt_image(camera_indices=camera_indices).cuda() same action in coarse_density.py and refined.py By use this method, 2000+images on my 24GB device 4090 is okay. Hope this help. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.>

No，I did not reduce the images size. What GPU are you using and how much memory? And which step does OOM happen? In my experiments, I found that the final texture mesh extraction stage requires a large amount of GPU memory.

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.Message ID: @.***>

[cubantonystark]

I am using an RTX 4090 with 16Gb. On an Alienware M18 with 64 Gb RAM. I Also ran it on an Asus Razer Blade 16 with 32 Gb RAM and RTX 4090 and got OOM at that stage as well. Best regards,Reynel RodriguezOn Dec 22, 2023 04:16, julius @.***> wrote:

Julius, Are you reducing image size bwfore processing?  Got OOM again.  My dataset is only 26 images at 1900x1423.Best regards,Reynel RodriguezOn Dec 22, 2023 01:40, Reynel Rodriguez @.> wrote: Please disregard previous email, I had mistyped the suggestion and formatted the parentheses in …camera_indices).cuda() as …camera_indices.cuda())   My mistake. I’m rerunning my datasets again now. Will report back with results. Anttwo, please let me know if you would like me to share results of my dataset with you to showcase quality.   Sent from Mail for Windows   From: julius Sent: Thursday, December 21, 2023 9:09 PM To: Anttwo/SuGaR Cc: Cuban Tony Stark; Comment Subject: Re: [Anttwo/SuGaR] cuda out of memory (Issue #40)   suagr_model will load the dataset to the GPU by default, which will consume a lot of GPU memory. So, you can try the fllowing steps: modified this code, image_height=image_height, image_width=image_width, data_device='cpu') move the image to cuda during training process, modified this code gt_image = nerfmodel.get_gt_image(camera_indices=camera_indices).cuda() same action in coarse_density.py and refined.py By use this method, 2000+images on my 24GB device 4090 is okay. Hope this help. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.>

No，I did not reduce the images size. What GPU are you using and how much memory? And which step does OOM happen? In my experiments, I found that the final texture mesh extraction stage requires a large amount of GPU memory.

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.Message ID: @.***>

[X-WN]

suagr_model will load the dataset to the GPU by default, which will consume a lot of GPU memory.

So, you can try the fllowing steps:

1. modified this code, image_height=image_height, image_width=image_width, data_device='cpu')
2. move the image to cuda during training process, modified this code gt_image = nerfmodel.get_gt_image(camera_indices=camera_indices).cuda()
3. same action in coarse_density.py and refined.py

By use this method, 2000+images on my 24GB device 4090 is okay.

Hope this help.

It's very useful, thanks!!!

[MagicJoeXZ]

suagr_model will load the dataset to the GPU by default, which will consume a lot of GPU memory.

So, you can try the fllowing steps:

modified this code, image_height=image_height, image_width=image_width, data_device='cpu') move the image to cuda during training process, modified this code gt_image = nerfmodel.get_gt_image(camera_indices=camera_indices).cuda() same action in coarse_density.py and refined.py By use this method, 2000+images on my 24GB device 4090 is okay.

Hope this help.

Verified this works well!

Besides the suggestion above, i also change device type here from cuda to cpu