ControlNet crashes with high-resolution reference images when using Canny preprocessor

[rmatif]

ControlNet crashes when processing high-resolution reference images with the Canny preprocessor enabled. The issue only occurs with large images while smaller resolutions (like 512x512) work fine

./build/bin/sd -m models/realisticVisionV60B1_v51HyperVAE.safetensors --control-net models/control_v11p_sd15_canny.pth --control-image input_test.png --canny -W 640 -H 512 -v -p "c
ute cat, lego style" --type q8_0
Option: 
    n_threads:         2
    mode:              txt2img
    model_path:        models/realisticVisionV60B1_v51HyperVAE.safetensors
    wtype:             q8_0
    clip_l_path:       
    clip_g_path:       
    t5xxl_path:        
    diffusion_model_path:   
    vae_path:          
    taesd_path:        
    esrgan_path:       
    controlnet_path:   models/control_v11p_sd15_canny.pth
    embeddings_path:   
    stacked_id_embeddings_path:   
    input_id_images_path:   
    style ratio:       20.00
    normalize input image :  false
    output_path:       output.png
    init_img:          
    mask_img:          
    control_image:     input_test.png
    clip on cpu:       false
    controlnet cpu:    false
    vae decoder on cpu:false
    diffusion flash attention:false
    strength(control): 0.90
    prompt:            cute cat, lego style
    negative_prompt:   
    min_cfg:           1.00
    cfg_scale:         7.00
    slg_scale:         0.00
    guidance:          3.50
    eta:               0.00
    clip_skip:         -1
    width:             640
    height:            512
    sample_method:     euler_a
    schedule:          default
    sample_steps:      20
    strength(img2img): 0.75
    rng:               cuda
    seed:              42
    batch_count:       1
    vae_tiling:        false
    upscale_repeats:   1
System Info: 
    SSE3 = 1
    AVX = 1
    AVX2 = 1
    AVX512 = 1
    AVX512_VBMI = 1
    AVX512_VNNI = 1
    FMA = 1
    NEON = 0
    ARM_FMA = 0
    F16C = 1
    FP16_VA = 0
    WASM_SIMD = 0
    VSX = 0
[DEBUG] stable-diffusion.cpp:188  - Using CPU backend
[INFO ] stable-diffusion.cpp:197  - loading model from 'models/realisticVisionV60B1_v51HyperVAE.safetensors'
[INFO ] model.cpp:908  - load models/realisticVisionV60B1_v51HyperVAE.safetensors using safetensors format
[DEBUG] model.cpp:979  - init from 'models/realisticVisionV60B1_v51HyperVAE.safetensors'
[INFO ] stable-diffusion.cpp:244  - Version: SD 1.x 
[INFO ] stable-diffusion.cpp:277  - Weight type:                 q8_0
[INFO ] stable-diffusion.cpp:278  - Conditioner weight type:     q8_0
[INFO ] stable-diffusion.cpp:279  - Diffusion model weight type: q8_0
[INFO ] stable-diffusion.cpp:280  - VAE weight type:             q8_0
[DEBUG] stable-diffusion.cpp:282  - ggml tensor size = 400 bytes
[DEBUG] clip.hpp:171  - vocab size: 49408
[DEBUG] clip.hpp:182  -  trigger word img already in vocab
[DEBUG] ggml_extend.hpp:1174 - clip params backend buffer size =  231.50 MB(RAM) (196 tensors)
[DEBUG] ggml_extend.hpp:1174 - unet params backend buffer size =  1400.64 MB(RAM) (686 tensors)
[DEBUG] ggml_extend.hpp:1174 - vae params backend buffer size =  94.47 MB(RAM) (140 tensors)
[DEBUG] stable-diffusion.cpp:419  - loading weights
[DEBUG] model.cpp:1727 - loading tensors from models/realisticVisionV60B1_v51HyperVAE.safetensors
  |==================================================| 1130/1130 - 125.00it/s
[INFO ] control.hpp:434  - loading control net from 'models/control_v11p_sd15_canny.pth'
[DEBUG] ggml_extend.hpp:1174 - control_net params backend buffer size =  937.60 MB(RAM) (340 tensors)
[INFO ] model.cpp:911  - load models/control_v11p_sd15_canny.pth using checkpoint format
[DEBUG] model.cpp:1445 - init from 'models/control_v11p_sd15_canny.pth'
ZIP 0, name = control_v11p_sd15_canny/data.pkl, dir = control_v11p_sd15_canny/ 
[DEBUG] model.cpp:1727 - loading tensors from models/control_v11p_sd15_canny.pth
  |==================================================| 340/340 - 0.00it/s
[INFO ] control.hpp:453  - control net model loaded
[INFO ] stable-diffusion.cpp:503  - total params memory size = 2664.21MB (VRAM 937.60MB, RAM 1726.61MB): clip 231.50MB(RAM), unet 1400.64MB(RAM), vae 94.47MB(RAM), controlnet 937.60MB(VRAM), pmid 0.00MB(RAM)
[INFO ] stable-diffusion.cpp:522  - loading model from 'models/realisticVisionV60B1_v51HyperVAE.safetensors' completed, taking 15.86s
[INFO ] stable-diffusion.cpp:556  - running in eps-prediction mode
[DEBUG] stable-diffusion.cpp:600  - finished loaded file
ggml_new_object: not enough space in the context's memory pool (needed 10489856, available 10485760)
/workspaces/stable-diffusion.cpp/ggml/src/ggml.c:1597: GGML_ASSERT(obj_new) failed
Could not attach to process.  If your uid matches the uid of the target
process, check the setting of /proc/sys/kernel/yama/ptrace_scope, or try
again as the root user.  For more details, see /etc/sysctl.d/10-ptrace.conf
ptrace: Operation not permitted.
No stack.
The program is not being run.
Aborted (core dumped)

[chinshou]

check this issue https://github.com/leejet/stable-diffusion.cpp/issues/253

[rmatif]

@chinshou Thanks for pointing that out, I missed the issue. I've also done the same, which is increasing manually the hardcoded value for max memory allocation, but it doesn't feel like a proper solution, more like a workaround. That's why I opened the issue

[leejet]

Fixed by https://github.com/leejet/stable-diffusion.cpp/pull/861.