Server Service Incident Report

Open iznogood opened this issue 9 months ago • 1 comments

Expected Behavior

We are currently facing two distinct issues:

1 - SSL and Proxy Configuration: When enabling SSL in conjunction with a proxy, we encounter 404 errors related to file requests at the API level—specifically during workflow execution. These errors do not occur when SSL is disabled. We have verified that the proxy configuration on our end is correct.

Sample of files : http://ai-portal.domain.net:8189/api/userdata/workflows%2Fxx%20SD15%20-%20Fond%20produit.json ok http://10.0.10.45:8189/api/userdata/workflows%2Fxx%20SD15%20-%20Fond%20produit.json ok https://comfyui.ai-portal.domain.net/api/userdata/workflows%2Fxx%20SD15%20-%20Fond%20produit.json KO

We use Xamp to share the proxies

2 - ComfyUI Stability with Flux1 Models: The server experiences crashes either overnight or during the day following usage of the Flux1 models within ComfyUI. Notably, if the ComfyUI application remains active but is not being used, the server remains stable and does not crash.

We would appreciate your assistance in identifying and resolving these issues.

Actual Behavior

Steps to Reproduce

To reproduce the issue, we configured ComfyUI using the --server-remote option, specifying the server's IP address and a designated port. Attempts to enable SSL directly within ComfyUI cause the application to crash. As a workaround, we are handling SSL termination at the proxy level instead.

Debug Logs

## ComfyUI-Manager: installing dependencies done.
[2025-04-25 17:18:50.729] ** ComfyUI startup time: 2025-04-25 17:18:50.729
[2025-04-25 17:18:50.729] ** Platform: Windows
[2025-04-25 17:18:50.729] ** Python version: 3.12.8 (main, Jan 14 2025, 22:49:36) [MSC v.1942 64 bit (AMD64)]
[2025-04-25 17:18:50.729] ** Python executable: D:\Echange102\.venv\Scripts\python.exe
[2025-04-25 17:18:50.729] ** ComfyUI Path: C:\Users\comfyui\AppData\Local\Programs\@comfyorgcomfyui-electron\resources\ComfyUI
[2025-04-25 17:18:50.729] ** ComfyUI Base Folder Path: C:\Users\comfyui\AppData\Local\Programs\@comfyorgcomfyui-electron\resources\ComfyUI
[2025-04-25 17:18:50.729] ** User directory: D:\Echange102\user
[2025-04-25 17:18:50.729] ** ComfyUI-Manager config path: D:\Echange102\user\default\ComfyUI-Manager\config.ini
[2025-04-25 17:18:50.745] ** Log path: D:\Echange102\user\comfyui.log
[ComfyUI-Manager] Skipped fixing the 'comfyui-frontend-package' dependency because the ComfyUI is outdated.
[2025-04-25 17:18:51.831] [START] Security scan
[2025-04-25 17:18:52.826] [DONE] Security scan
[2025-04-25 17:18:52.826] [ComfyUI-Manager] Logging failed: [WinError 32] Le processus ne peut pas accéder au fichier car ce fichier est utilisé par un autre processus: 'D:\\Echange102\\user\\comfyui_81.log' -> 'D:\\Echange102\\user\\comfyui_81.prev.log'
[2025-04-25 17:18:52.826] ## ComfyUI-Manager: installing dependencies done.
[2025-04-25 17:18:52.826] ** ComfyUI startup time: 2025-04-25 17:18:52.826
[2025-04-25 17:18:52.826] ** Platform: Windows
[2025-04-25 17:18:52.826] ** Python version: 3.12.8 (main, Jan 14 2025, 22:49:36) [MSC v.1942 64 bit (AMD64)]
[2025-04-25 17:18:52.826] ** Python executable: D:\Echange102\.venv\Scripts\python.exe
[2025-04-25 17:18:52.826] ** ComfyUI Path: C:\Users\comfyui\AppData\Local\Programs\@comfyorgcomfyui-electron\resources\ComfyUI
[2025-04-25 17:18:52.826] ** ComfyUI Base Folder Path: C:\Users\comfyui\AppData\Local\Programs\@comfyorgcomfyui-electron\resources\ComfyUI
[2025-04-25 17:18:52.826] ** User directory: D:\Echange102\user
[2025-04-25 17:18:52.826] ** ComfyUI-Manager config path: D:\Echange102\user\default\ComfyUI-Manager\config.ini
[2025-04-25 17:18:52.826] ** Log path: D:\Echange102\user\comfyui.log
[2025-04-25 17:18:53.913] [ComfyUI-Manager] Skipped fixing the 'comfyui-frontend-package' dependency because the ComfyUI is outdated.
[2025-04-25 17:18:53.917] 
Prestartup times for custom nodes:
[2025-04-25 17:18:53.917]    2.1 seconds: C:\Users\comfyui\AppData\Local\Programs\@comfyorgcomfyui-electron\resources\ComfyUI\custom_nodes\ComfyUI-Manager
[2025-04-25 17:18:53.917] 
[2025-04-25 17:18:54.942] Checkpoint files will always be loaded safely.
[2025-04-25 17:18:55.073] Total VRAM 49140 MB, total RAM 130572 MB
[2025-04-25 17:18:55.073] pytorch version: 2.6.0+cu126
[2025-04-25 17:18:55.073] Set vram state to: NORMAL_VRAM
[2025-04-25 17:18:55.073] Disabling smart memory management
[2025-04-25 17:18:55.073] Device: cuda:0 NVIDIA RTX 6000 Ada Generation : cudaMallocAsync
[2025-04-25 17:18:55.831] Using pytorch attention
[2025-04-25 17:18:56.571] Python version: 3.12.8 (main, Jan 14 2025, 22:49:36) [MSC v.1942 64 bit (AMD64)]
[2025-04-25 17:18:56.571] ComfyUI version: 0.3.29
[2025-04-25 17:18:56.591] [Prompt Server] web root: C:\Users\comfyui\AppData\Local\Programs\@comfyorgcomfyui-electron\resources\ComfyUI\web_custom_versions\desktop_app
[2025-04-25 17:18:57.129] Note: NumExpr detected 48 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 16.
[2025-04-25 17:18:57.129] NumExpr defaulting to 16 threads.
[2025-04-25 17:18:57.228] [Crystools [0;32mINFO[0m] Crystools version: 1.22.1
[2025-04-25 17:18:57.255] [Crystools [0;32mINFO[0m] CPU: AMD Ryzen Threadripper PRO 7965WX 24-Cores - Arch: AMD64 - OS: Windows 11
[2025-04-25 17:18:57.264] [Crystools [0;32mINFO[0m] Pynvml (Nvidia) initialized.
[2025-04-25 17:18:57.264] [Crystools [0;32mINFO[0m] GPU/s:
[2025-04-25 17:18:57.279] [Crystools [0;32mINFO[0m] 0) NVIDIA RTX 6000 Ada Generation
[2025-04-25 17:18:57.279] [Crystools [0;32mINFO[0m] NVIDIA Driver: 572.61
[2025-04-25 17:18:57.296] ### Loading: ComfyUI-Manager (V3.31.8)
[2025-04-25 17:18:57.296] [ComfyUI-Manager] network_mode: public
[2025-04-25 17:18:57.296] ### ComfyUI Revision: UNKNOWN (The currently installed ComfyUI is not a Git repository)
[2025-04-25 17:18:57.327] [ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/alter-list.json
[2025-04-25 17:18:57.354] [ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/github-stats.json
[2025-04-25 17:18:57.374] [ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/extension-node-map.json
[2025-04-25 17:18:57.388] [ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/custom-node-list.json
[2025-04-25 17:18:57.404] [ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/model-list.json
[2025-04-25 17:18:57.981] ------------------------------------------
[2025-04-25 17:18:57.981] [34mComfyroll Studio v1.76 : [92m 175 Nodes Loaded[0m
[2025-04-25 17:18:57.981] ------------------------------------------
[2025-04-25 17:18:57.981] ** For changes, please see patch notes at https://github.com/Suzie1/ComfyUI_Comfyroll_CustomNodes/blob/main/Patch_Notes.md
[2025-04-25 17:18:57.981] ** For help, please see the wiki at https://github.com/Suzie1/ComfyUI_Comfyroll_CustomNodes/wiki
[2025-04-25 17:18:57.981] ------------------------------------------
[2025-04-25 17:18:57.992] 
Import times for custom nodes:
[2025-04-25 17:18:57.992]    0.0 seconds: C:\Users\comfyui\AppData\Local\Programs\@comfyorgcomfyui-electron\resources\ComfyUI\custom_nodes\websocket_image_save.py
[2025-04-25 17:18:57.992]    0.0 seconds: C:\Users\comfyui\AppData\Local\Programs\@comfyorgcomfyui-electron\resources\ComfyUI\custom_nodes\ComfyUI-Manager
[2025-04-25 17:18:57.992]    0.0 seconds: D:\Echange102\custom_nodes\comfyui-custom-scripts
[2025-04-25 17:18:57.992]    0.0 seconds: D:\Echange102\custom_nodes\ComfyUI-Manager
[2025-04-25 17:18:57.992]    0.0 seconds: D:\Echange102\custom_nodes\comfyui_essentials
[2025-04-25 17:18:57.992]    0.2 seconds: D:\Echange102\custom_nodes\ComfyUI-Crystools
[2025-04-25 17:18:57.992]    0.3 seconds: D:\Echange102\custom_nodes\ComfyUI_Comfyroll_CustomNodes
[2025-04-25 17:18:57.992]    0.4 seconds: D:\Echange102\custom_nodes\comfyui-ollama
[2025-04-25 17:18:57.992] 
[2025-04-25 17:18:58.010] Starting server

[2025-04-25 17:18:58.012] To see the GUI go to: https://comfyui.ai-portal.domain.net:81
[2025-04-25 17:19:00.354] FETCH ComfyRegistry Data: 5/82
[2025-04-25 17:19:03.647] FETCH ComfyRegistry Data: 10/82
[2025-04-25 17:19:06.958] FETCH ComfyRegistry Data: 15/82
[2025-04-25 17:19:10.167] FETCH ComfyRegistry Data: 20/82
[2025-04-25 17:19:13.386] FETCH ComfyRegistry Data: 25/82

Other

VHOST

 
<VirtualHost *:443>

    ServerName comfyui.ai-portal.domain.net
 
    SSLEngine on

    SSLProtocol all -SSLv3

    SSLCertificateFile "C:/Users/comfyui/certif/_.ai-portal.domain.net.crt"

    SSLCertificateKeyFile "C:/Users/comfyui/certif/wilcard.ai-portal.domain.net.key"

    SSLCertificateChainFile "C:/Users/comfyui/certif/GandiCert.pem"
 
 
	ProxyPreserveHost On

    ProxyPass "/" "http://10.0.10.45:8189/"

    ProxyPassReverse "/" "http://10.0.10.45:8189/"

    # Proxy WebSockets (corrigé)

    RewriteEngine On

    RewriteCond %{HTTP:Upgrade} =websocket [NC]

    RewriteRule /(.*) ws://10.0.10.45:8189/$1 [P,L]
 
    ProxyPass "/ws/" "ws://comfyui.ai-portal.domain.net:8189/"

    ProxyPassReverse "/ws/" "ws://comfyui.ai-portal.domain.net:8189/"
 
    <Location />

        Require all granted
</Location>
 
</VirtualHost>

Apr 29 '25 10:04 iznogood

You change the path with one of the version and the the API is not cofigured to point to the correct path by now in /user/default/workflows .... you need to update it or to patch like this :

#!/bin/bash

echo "=== Création directe de la structure userdata ComfyUI ==="

# Stop ComfyUI
echo "Arrêt du service ComfyUI..."
sudo systemctl stop comfyui

# wich is active
COMFYUI_PATH=$(grep "WorkingDirectory" /etc/systemd/system/comfyui.service | cut -d'=' -f2 2>/dev/null)
if [ -z "$COMFYUI_PATH" ]; then
    echo "Service non trouvé, utilisation du chemin par défaut..."
    COMFYUI_PATH="/home/comfyui/comfy/ComfyUI"
fi

echo "Installation ComfyUI détectée : $COMFYUI_PATH"

# who is the user
if [[ "$COMFYUI_PATH" == "/home/comfyui/comfy/ComfyUI" ]]; then
    COMFY_USER="comfyui"
    COMFY_GROUP="comfyui"
else
    COMFY_USER="www-data"
    COMFY_GROUP="www-data"
fi

echo "user/Group : $COMFY_USER:$COMFY_GROUP"

# create full structure of userdata 
echo "Création de la structure userdata complète..."

# main Structure
sudo mkdir -p "$COMFYUI_PATH/user/default/workflows"
sudo mkdir -p "$COMFYUI_PATH/web/userdata"

# Create workflows directories in web/userdata (CRUCIAL pour l'API)
sudo mkdir -p "$COMFYUI_PATH/web/userdata/workflows"

# create user.css si not present
if [ ! -f "$COMFYUI_PATH/web/user.css" ]; then
    sudo touch "$COMFYUI_PATH/web/user.css"
fi

# Create template  file comfy.templates.json in web/userdata
sudo tee "$COMFYUI_PATH/web/userdata/comfy.templates.json" > /dev/null << 'EOF'
{
  "templates": []
}
EOF

# Create file comfy.templates.json in user/default
sudo tee "$COMFYUI_PATH/user/default/comfy.templates.json" > /dev/null << 'EOF'
{
  "templates": []
}
EOF

# Create a workflow in  default in user/default/workflows
sudo tee "$COMFYUI_PATH/user/default/workflows/Unsaved Workflow.json" > /dev/null << 'EOF'
{
  "last_node_id": 9,
  "last_link_id": 9,
  "nodes": [
    {
      "id": 7,
      "type": "CLIPTextEncode",
      "pos": [413, 389],
      "size": {"0": 425.27801513671875, "1": 180.6060791015625},
      "flags": {},
      "order": 3,
      "mode": 0,
      "inputs": [{"name": "clip", "type": "CLIP", "link": 5}],
      "outputs": [{"name": "CONDITIONING", "type": "CONDITIONING", "links": [6], "slot_index": 0}],
      "properties": {"Node name for S&R": "CLIPTextEncode"},
      "widgets_values": ["beautiful scenery nature glass bottle landscape, purple galaxy bottle,"]
    }
  ],
  "links": [],
  "groups": [],
  "config": {},
  "extra": {},
  "version": 0.4
}
EOF

# Create file in web/userdata/workflows too
sudo cp "$COMFYUI_PATH/user/default/workflows/Unsaved Workflow.json" "$COMFYUI_PATH/web/userdata/workflows/"

# Apply chmod
echo "Application des permissions..."
sudo chown -R $COMFY_USER:$COMFY_GROUP "$COMFYUI_PATH/user"
sudo chown -R $COMFY_USER:$COMFY_GROUP "$COMFYUI_PATH/web"

# Chmod
sudo chmod -R 755 "$COMFYUI_PATH/user"
sudo chmod -R 755 "$COMFYUI_PATH/web"
sudo chmod -R 775 "$COMFYUI_PATH/user/default/workflows"
sudo chmod -R 775 "$COMFYUI_PATH/web/userdata"

# Check
echo ""
echo "=== Vérification de la structure créée ==="
echo "Structure user/default :"
ls -la "$COMFYUI_PATH/user/default/"
echo ""
echo "Structure user/default/workflows :"
ls -la "$COMFYUI_PATH/user/default/workflows/"
echo ""
echo "Structure web :"
ls -la "$COMFYUI_PATH/web/"
echo ""
echo "Structure web/userdata :"
ls -la "$COMFYUI_PATH/web/userdata/"

# Restart comfyui
echo ""
echo "Redémarrage du service ComfyUI..."
sudo systemctl start comfyui

# Attendre le démarrage
echo "Attente du démarrage (15 secondes)..."
sleep 15

# Tests API
echo ""
echo "=== Tests de l'API userdata ==="

echo "Test 1 - API userdata de base :"
curl -s -w "HTTP %{http_code}\n" http://127.0.0.1:8188/api/userdata -o /dev/null

echo "Test 2 - Répertoire workflows :"
curl -s -w "HTTP %{http_code}\n" "http://127.0.0.1:8188/api/userdata?dir=workflows" -o /dev/null

echo "Test 3 - Templates :"
curl -s -w "HTTP %{http_code}\n" http://127.0.0.1:8188/api/userdata/comfy.templates.json -o /dev/null

echo "Test 4 - Workflow spécifique :"
curl -s -w "HTTP %{http_code}\n" "http://127.0.0.1:8188/api/userdata/workflows/Unsaved%20Workflow.json" -o /dev/null

echo ""
echo "Test via HTTPS :"
curl -s -w "HTTPS %{http_code}\n" "https://comfyui.domaint/api/userdata?dir=workflows" -o /dev/null

echo ""
echo "=== Check final ==="
echo "Status du service :"
sudo systemctl status comfyui --no-pager -l | head -10

echo ""
echo "=== Instructions ==="
echo "Si les tests montrent 200 OK, l'API fonctionne !"
echo "Si 404 persiste :"
echo "1. Vérifiez les logs : sudo journalctl -u comfyui -f"
echo "2. Vérifiez que ComfyUI démarre : curl http://127.0.0.1:8188"
echo "3. Vérifiez les permissions des dossiers ci-dessus"`

Jun 19 '25 14:06 iznogood