Very high RAM usage ~73 GB

[naisanzaa]

Describe the bug

Darktable would shoot up to high levels of RAM usage out of nowhere.

Steps to reproduce

1. Unknown

Expected behavior

No response

Logfile | Screenshot | Screencast

Commit

No response

Where did you install darktable from?

darktable.org

darktable version

4.4.2

What OS are you using?

Mac

What is the version of your OS?

14.0 (23A344)

Describe your system?

Macbook AIr M1 16GB

Are you using OpenCL GPU in darktable?

None

If yes, what is the GPU card and driver?

No response

Please provide additional context if applicable. You can attach files too, but might need to rename to .txt or .zip

No response

[pehar1]

No information what you did before you ran into this issue, no information how to reproduce, no information on whether this issue has occurred more than once, nearly no information about your system (GPU ?, OpenCL ?), no information about the image(s) you edited, no debug log. I'm afraid it will be very difficult to get help this way... Please provide as much information as possible to narrow down the issue.

[ralfbrown]

Several of the processing modules can take upwards of 200 bytes per pixel in the image, depending on settings. The Surface Blur module with (ironically) small values on the sliders is the worst, but any of the algorithms using wavelet decomposition are also memory hungry (Diffuse/Sharpen, Filmic's highlight reconstruction, Contrast Equalizer, Denoise(profiled), Retouch with multiple scales, and more).

You can trade off slower processing to save memory by reducing the "darktable resources" setting on the "Processing" tab in darktable's preferences. My guess is it's currently set at "unlimited".

[naisanzaa]

No information what you did before you ran into this issue, no information how to reproduce, no information on whether this issue has occurred more than once, nearly no information about your system (GPU ?, OpenCL ?), no information about the image(s) you edited, no debug log. I'm afraid it will be very difficult to get help this way...

Please provide as much information as possible to narrow down the issue.

Trying to replicate this, since it's still Unknown what is causing this.

So I can update this ticket with some better logs of some sort.

I pretty much only had the app open in the background, not using it.

And then all of a sudden, I'm given a warning that I am out of memory.

[naisanzaa]

Several of the processing modules can take upwards of 200 bytes per pixel in the image, depending on settings. The Surface Blur module with (ironically) small values on the sliders is the worst, but any of the algorithms using wavelet decomposition are also memory hungry (Diffuse/Sharpen, Filmic's highlight reconstruction, Contrast Equalizer, Denoise(profiled), Retouch with multiple scales, and more).

You can trade off slower processing to save memory by reducing the "darktable resources" setting on the "Processing" tab in darktable's preferences. My guess is it's currently set at "unlimited".

Processing settings are all default.

[pehar1]

Trying to replicate

Maybe it can help if you monitor the RAM consumption with the help of a task manager. Or running dt from the console with a debug option to watch if something weird is happening while the application is running in background. By the way: I have been using the current developer version and/or the stable version for years and constantly monitor the RAM consumption and the processor load (CPU and GPU) to quickly become aware of any problems of this kind. I have never observed such massive RAM consumption while dt is idle.

[naisanzaa]

Trying to replicate

Maybe it can help if you monitor the RAM consumption with the help of a task manager. Or running dt from the console with a debug option to watch if something weird is happening while the application is running in background. By the way: I have been using the current developer version and/or the stable version for years and constantly monitor the RAM consumption and the processor load (CPU and GPU) to quickly become aware of any problems of this kind. I have never observed such massive RAM consumption while dt is idle.

Exactly, which is why I have so little information to help replicate and troubleshoot this.

But screenshot proves that it happens at least.

What's the flag for verbose? --debug?

[ptilopteri]

darktable --help

[pehar1]

Or https://darktable-org.github.io/dtdocs/en/special-topics/program-invocation/darktable/

[naisanzaa]

/Applications/darktable.app/Contents/MacOS/darktable -d memory

example:

eric@DORABOOK ~ % /Applications/darktable.app/Contents/MacOS/darktable -d memory 
[dt_detect_cpu_features] Not implemented for this architecture.
[dt_detect_cpu_features] Please contribute a patch.
     0.0003 [dt_init] SSE2 is unavailable, some functions will be noticeably slower.
     0.0005 [memory] at startup
     0.0005 [memory] max address space (vmpeak):         unknown
            [memory] cur address space (vmsize):    408675744 kB
            [memory] max used memory   (vmhwm ):         unknown
            [memory] cur used memory   (vmrss ):        22976 kB

(process:43842): GLib-GObject-CRITICAL **: 09:49:54.843: g_object_set: assertion 'G_IS_OBJECT (object)' failed
[dt_detect_cpu_features] Not implemented for this architecture.
[dt_detect_cpu_features] Please contribute a patch.
     0.6779 [dt_get_sysresource_level] switched to 1 as `default'
     0.6779   total mem:       16384MB
     0.6779   mipmap cache:    2048MB
     0.6779   available mem:   8192MB
     0.6779   singlebuff:      128MB
     0.6779   OpenCL tune mem: OFF
     0.6779   OpenCL pinned:   OFF

(darktable:43842): GLib-GObject-CRITICAL **: 09:49:55.720: invalid cast from 'GtkMenuBar' to 'GtkWindow'

(darktable:43842): Gtk-CRITICAL **: 09:49:55.720: gtk_window_add_accel_group: assertion 'GTK_IS_WINDOW (window)' failed
     3.4903 [memory] after successful startup
     3.4904 [memory] max address space (vmpeak):         unknown
            [memory] cur address space (vmsize):    409576640 kB
            [memory] max used memory   (vmhwm ):         unknown
            [memory] cur used memory   (vmrss ):       168160 kB
2023-11-05 09:49:59.386 darktable[43842:20474607] WARNING: Secure coding is not enabled for restorable state! Enable secure coding by implementing NSApplicationDelegate.applicationSupportsSecureRestorableState: and returning YES.

I'll play around.

[MStraeten]

have seens that exceeding memory consumption recently with current master. usual behaviour: zooming out or panning results temporarily in up to triple memory consumption (574MB just displaying - 1,5 GB during zoom or panning around) but that increase up to >200GB not reproduced with debugger yet ...

[naisanzaa]

have seens that exceeding memory consumption recently with current master.

usual behaviour: zooming out or panning results temporarily in up to triple memory consumption (574MB just displaying - 1,5 GB during zoom or panning around)

but that increase up to >200GB not reproduced with debugger yet ...

What debugger flags are you using when running darktable?

I'll run with the same ones

[zisoft]

Maybe related: #15488 Do you have background mipmap generation activated?

[naisanzaa]

Maybe related: #15488 Do you have background mipmap generation activated?

if it's not on by default, then I don't have that activated

[jenshannoschwalm]

have seens that exceeding memory consumption recently with current master. usual behaviour: zooming out or panning results temporarily in up to triple memory consumption (574MB just displaying - 1,5 GB during zoom or panning around)

Have not seen that yet - and i often run dt with logs. Would be very interested in such a log with -d memory -d pipe active.

[github-actions[bot]]

This issue has been marked as stale due to inactivity for the last 60 days. It will be automatically closed in 300 days if no update occurs. Please check if the master branch has fixed it and report again or close the issue.

[jenshannoschwalm]

Closing as at least one problem has been fixed and we have no further data.