Feature: Add dedicated Dockerfile.worker and queue-mode docker-compose setup

[Nek-11]

Problem:

Running Flowise workers effectively in various environments, especially scaled deployments or platforms like Azure App Service, presents several challenges:

1. Lack of Official Worker Dockerfile: There is no official, dedicated Dockerfile specifically designed for running Flowise worker instances.
2. Deployment Issues: Attempting to run the standard flowiseai/flowise image as a worker on platforms like Azure App Service by overriding the startup command (e.g., to flowise worker) often fails. This is due to conflicts with the main image's existing ENTRYPOINT or CMD, which is tailored for flowise start. This results in unreliable worker startup and makes debugging difficult.
3. Complex Local Queue-Mode Setup: Setting up a local environment to test Flowise in queue mode (main app + worker + Redis) requires manual configuration and lacks a simple, integrated Docker Compose solution.

This work is related to #4345.

Solution:

This PR introduces two main components to address these problems and standardize worker deployment:

1. New Dedicated Worker Dockerfile (docker/worker/Dockerfile.worker):

   • A new Dockerfile.worker has been created, specifically for building Flowise worker instances.
   • Its primary command is flowise worker.
   • It incorporates a simple Node.js/Express script (docker/worker/healthcheck/healthcheck.js) which runs as a background "keep-alive" service within the worker container. This is particularly beneficial for environments like Azure App Service, which might terminate containers not perceived as actively serving HTTP traffic. This internal healthcheck ensures the container remains running and the worker process is active.

2. New Comprehensive Docker Compose for Queue Mode (docker/docker-compose.queue.yml):

   • A new docker-compose.queue.yml file is provided to offer a minimal, all-in-one setup for running and testing Flowise in queue mode locally.
   • This compose file defines and orchestrates three services:
     • flowise: The main Flowise application (built from docker/Dockerfile).
     • flowise-worker: The worker instance (built using the new docker/worker/Dockerfile.worker).
     • redis: The Redis message queue.
   • Usage: docker-compose -f docker/docker-compose.queue.yml up --build (run from the project root).

Proposal for Official Docker Hub Image:

To ensure users have access to a consistently updated and tested image, I propose that an official Docker image be built from docker/worker/Dockerfile.worker and published to Docker Hub (e.g., as flowiseai/flowise-worker). This would align with the main flowiseai/flowise image and make deploying worker nodes significantly easier.

Testing:

• The docker/docker-compose.queue.yml setup has been successfully tested locally, confirming that all services (redis, flowise-main, flowise-worker) start correctly and Flowise operates in queue mode as expected.
• The docker/worker/Dockerfile.worker, with its dedicated flowise worker entrypoint and internal healthcheck/keep-alive script, has been tested both locally and on Azure App Service (with a custom image published to Docker Hub).

Closes:

#4345

[Zeouterlimits]

It defo be great if the worker apps had a healthcheck endpoint like the main apps, to be able to ensure workers rollout in a healthy way and are killed if they are unhealthy (we run flowise in k8s).

[Nek-11]

Hi @HenryHengZJ

Any news on this PR? I forked the repo, published this on Docker Hub and am currently using it in production smoothly. However it would be nice to have it integrated in the main repo so it can benefit from all updates and such :)

[HenryHengZJ]

Hi @HenryHengZJ

Any news on this PR? I forked the repo, published this on Docker Hub and am currently using it in production smoothly. However it would be nice to have it integrated in the main repo so it can benefit from all updates and such :)

hey we were facing some weird issue for worker in v3.0.1, here's the updated PR (https://github.com/FlowiseAI/Flowise/pull/4643) that extends from your work here, thank you!

closing for now