ditto icon indicating copy to clipboard operation
ditto copied to clipboard
verified publisher icon eclipse-ditto

Eclipse Ditto on Docker Swarm | nginx: [emerg] host not found in upstream "gateway" in /etc/nginx/nginx.conf:44 | [error] 33#33: *2 connect() failed (111: Connection refused) while connecting to upstream

Open aissam-en opened this issue 1 year ago • 4 comments

Challenges Running Eclipse Ditto on Docker Swarm

I'm trying to run Eclipse Ditto on Docker Swarm, initially, I was able to successfully run Ditto using the provided docker-compose.yml file in Docker (not Swarm mode) without modifying the nginx.conf file.

I made some adjustments to the docker-compose.yml file to work in Docker Swarm :

  • Commented out restart: always;
  • Commented out healthcheck section;
  • Commented out volumes for the swagger-ui service, since volumes are not be suitable for Swarm, I will configure it later;
  • Replaced volumes for the nginx service with configs, to mount nginx files;

I'm encountering an issue where the nginx service is not recongizing gateway service, here is its logs:

docker service logs -f my_stack_nginx
my_stack_nginx.1.64bhclqugkeq@docker-desktop    | /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
my_stack_nginx.1.64bhclqugkeq@docker-desktop    | /docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
my_stack_nginx.1.64bhclqugkeq@docker-desktop    | /docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
my_stack_nginx.1.64bhclqugkeq@docker-desktop    | 10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf
my_stack_nginx.1.64bhclqugkeq@docker-desktop    | 10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
my_stack_nginx.1.64bhclqugkeq@docker-desktop    | /docker-entrypoint.sh: Sourcing /docker-entrypoint.d/15-local-resolvers.envsh
my_stack_nginx.1.64bhclqugkeq@docker-desktop    | /docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
my_stack_nginx.1.64bhclqugkeq@docker-desktop    | /docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
my_stack_nginx.1.64bhclqugkeq@docker-desktop    | /docker-entrypoint.sh: Configuration complete; ready for start up
my_stack_nginx.1.rtextyzpzpsl@docker-desktop    | /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
my_stack_nginx.1.rtextyzpzpsl@docker-desktop    | /docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
my_stack_nginx.1.rtextyzpzpsl@docker-desktop    | /docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
my_stack_nginx.1.rtextyzpzpsl@docker-desktop    | 10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf
my_stack_nginx.1.rtextyzpzpsl@docker-desktop    | 10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
my_stack_nginx.1.rtextyzpzpsl@docker-desktop    | /docker-entrypoint.sh: Sourcing /docker-entrypoint.d/15-local-resolvers.envsh
my_stack_nginx.1.rtextyzpzpsl@docker-desktop    | /docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
my_stack_nginx.1.rtextyzpzpsl@docker-desktop    | /docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
my_stack_nginx.1.rtextyzpzpsl@docker-desktop    | /docker-entrypoint.sh: Configuration complete; ready for start up
my_stack_nginx.1.rtextyzpzpsl@docker-desktop    | 2024/06/18 15:44:39 [emerg] 1#1: host not found in upstream "gateway" in /etc/nginx/nginx.conf:44
my_stack_nginx.1.rtextyzpzpsl@docker-desktop    | nginx: [emerg] host not found in upstream "gateway" in /etc/nginx/nginx.conf:44

the line 44 from nginx.conf file :

39     ...
40     # connections api using devops user configured in Ditto
41    location /api/2/connections {
42      include nginx-cors.conf;
43
44      proxy_pass                    http://gateway:8080;
45      proxy_http_version            1.1;
46      proxy_set_header              Host                $http_host;
47      proxy_set_header              X-Real-IP           $remote_addr;
48      proxy_set_header              X-Forwarded-For     $proxy_add_x_forwarded_for;
49      proxy_set_header              X-Forwarded-User    $remote_user;
50
51      proxy_set_header Connection  '';
52      chunked_transfer_encoding    off;
53      proxy_buffering              off;
54      proxy_cache                  off;
55    }
56    ...

and when I send request from Postman, ex : GET http://localhost:8080/api/2/things/ with ditto:ditto in Authorization, I get this as output :

<html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx</center>
</body>
</html>

The nginx logs :

docker service logs -f my_stack_nginx
my_stack_nginx.1.64bhclqugkeq@docker-desktop    | /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
my_stack_nginx.1.64bhclqugkeq@docker-desktop    | /docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
my_stack_nginx.1.64bhclqugkeq@docker-desktop    | /docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
my_stack_nginx.1.64bhclqugkeq@docker-desktop    | 10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf
my_stack_nginx.1.64bhclqugkeq@docker-desktop    | 10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
my_stack_nginx.1.64bhclqugkeq@docker-desktop    | /docker-entrypoint.sh: Sourcing /docker-entrypoint.d/15-local-resolvers.envsh
my_stack_nginx.1.64bhclqugkeq@docker-desktop    | /docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
my_stack_nginx.1.64bhclqugkeq@docker-desktop    | /docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
my_stack_nginx.1.64bhclqugkeq@docker-desktop    | /docker-entrypoint.sh: Configuration complete; ready for start up
my_stack_nginx.1.rtextyzpzpsl@docker-desktop    | /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
my_stack_nginx.1.rtextyzpzpsl@docker-desktop    | /docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
my_stack_nginx.1.rtextyzpzpsl@docker-desktop    | /docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
my_stack_nginx.1.rtextyzpzpsl@docker-desktop    | 10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf
my_stack_nginx.1.rtextyzpzpsl@docker-desktop    | 10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
my_stack_nginx.1.rtextyzpzpsl@docker-desktop    | /docker-entrypoint.sh: Sourcing /docker-entrypoint.d/15-local-resolvers.envsh
my_stack_nginx.1.rtextyzpzpsl@docker-desktop    | /docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
my_stack_nginx.1.rtextyzpzpsl@docker-desktop    | /docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
my_stack_nginx.1.rtextyzpzpsl@docker-desktop    | /docker-entrypoint.sh: Configuration complete; ready for start up
my_stack_nginx.1.rtextyzpzpsl@docker-desktop    | 2024/06/18 15:44:39 [emerg] 1#1: host not found in upstream "gateway" in /etc/nginx/nginx.conf:44
my_stack_nginx.1.rtextyzpzpsl@docker-desktop    | nginx: [emerg] host not found in upstream "gateway" in /etc/nginx/nginx.conf:44
my_stack_nginx.1.64bhclqugkeq@docker-desktop    | 2024/06/18 15:47:55 [error] 33#33: *2 connect() failed (111: Connection refused) while connecting to upstream, client: 10.0.0.2, server: localhost, request: "GET /api/2/things/ HTTP/1.1", upstream: "http://10.0.1.13:8080/api/2/things/", host: "localhost:8080"
my_stack_nginx.1.64bhclqugkeq@docker-desktop    | 10.0.0.2 - ditto [18/Jun/2024:15:47:55 +0000] "GET /api/2/things/ HTTP/1.1" 502 150 "-" "PostmanRuntime/7.39.0"

Additional Information:

  • Windows 10, 8GB RAM and CPU : Name : Intel(R) Core(TM) i5-9400H CPU @ 2.50GHz Number Of Cores : 4 Number Of Logical Processors : 8
  • Docker : Client: Version: 26.1.1 Context: default Debug Mode: false Plugins: buildx: Docker Buildx (Docker Inc.) Version: v0.14.0-desktop.1 Path: C:\Program Files\Docker\cli-plugins\docker-buildx.exe compose: Docker Compose (Docker Inc.) Version: v2.27.0-desktop.2 Path: C:\Program Files\Docker\cli-plugins\docker-compose.exe debug: Get a shell into any image or container (Docker Inc.) Version: 0.0.29 Path: C:\Program Files\Docker\cli-plugins\docker-debug.exe dev: Docker Dev Environments (Docker Inc.) Version: v0.1.2 Path: C:\Program Files\Docker\cli-plugins\docker-dev.exe extension: Manages Docker extensions (Docker Inc.) Version: v0.2.23 Path: C:\Program Files\Docker\cli-plugins\docker-extension.exe feedback: Provide feedback, right in your terminal! (Docker Inc.) Version: v1.0.4 Path: C:\Program Files\Docker\cli-plugins\docker-feedback.exe init: Creates Docker-related starter files for your project (Docker Inc.) Version: v1.1.0 Path: C:\Program Files\Docker\cli-plugins\docker-init.exe sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.) Version: 0.6.0 Path: C:\Program Files\Docker\cli-plugins\docker-sbom.exe scout: Docker Scout (Docker Inc.) Version: v1.8.0 Path: C:\Program Files\Docker\cli-plugins\docker-scout.exe Server: Containers: 34 Running: 9 Paused: 0 Stopped: 25 Images: 53 Server Version: 26.1.1 Storage Driver: overlay2 Backing Filesystem: extfs Supports d_type: true Using metacopy: false Native Overlay Diff: true userxattr: false Logging Driver: json-file Cgroup Driver: cgroupfs Cgroup Version: 1 Plugins: Volume: local Network: bridge host ipvlan macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog Swarm: active NodeID: mahw8hl8i23zzwm7v2i78vaq7 Is Manager: true ClusterID: ijzjixaizfhrhv754v8ir4fdy Managers: 1 Nodes: 1 Data Path Port: 4789 Orchestration: Task History Retention Limit: 5 Raft: Snapshot Interval: 10000 Number of Old Snapshots to Retain: 0 Heartbeat Tick: 1 Election Tick: 10 Dispatcher: Heartbeat Period: 5 seconds CA Configuration: Expiry Duration: 3 months Force Rotate: 0 Autolock Managers: false Root Rotation In Progress: false Node Address: 192.168.65.3 Manager Addresses: 192.168.65.3:2377 Runtimes: io.containerd.runc.v2 runc Default Runtime: runc Init Binary: docker-init containerd version: e377cd56a71523140ca6ae87e30244719194a521 runc version: v1.1.12-0-g51d5e94 init version: de40ad0 Security Options: seccomp Profile: unconfined Kernel Version: 5.15.153.1-microsoft-standard-WSL2 Operating System: Docker Desktop OSType: linux Architecture: x86_64 CPUs: 8 Total Memory: 3.721GiB Name: docker-desktop ID: ec241856-9f66-4d6c-908f-3ff0184e56b5 Docker Root Dir: /var/lib/docker Debug Mode: false HTTP Proxy: http.docker.internal:3128 HTTPS Proxy: http.docker.internal:3128 No Proxy: hubproxy.docker.internal Labels: com.docker.desktop.address=npipe://\.\pipe\docker_cli Experimental: false Insecure Registries: hubproxy.docker.internal:5555 127.0.0.0/8 Live Restore Enabled: false

The docker-compose.yml file : https://pastebin.com/nbKcXjDc

The services are runnig :

the services are runnig : 
docker service ls
ID             NAME                     MODE         REPLICAS   IMAGE                                PORTS
idgug4xs4u6g   my_stack_connectivity    replicated   1/1        eclipse/ditto-connectivity:latest
16uijjs8vpeb   my_stack_ditto-ui        replicated   1/1        eclipse/ditto-ui:latest
5pd7diz1uz1i   my_stack_gateway         replicated   1/1        eclipse/ditto-gateway:latest         *:8081->8080/tcp
jkkfj363z20o   my_stack_mongodb         replicated   1/1        mongo:6.0                            *:27017->27017/tcp
mc58254lk2h2   my_stack_nginx           replicated   1/1        nginx:1.26-alpine                    *:8080->80/tcp
kn5evgmkpvdw   my_stack_policies        replicated   1/1        eclipse/ditto-policies:latest
t4agxrku1991   my_stack_swagger-ui      replicated   1/1        swaggerapi/swagger-ui:v5.9.1
v42cz47b90qu   my_stack_things          replicated   1/1        eclipse/ditto-things:latest
xyoa5siftouk   my_stack_things-search   replicated   1/1        eclipse/ditto-things-search:latest

Inspect ditto-net-swarm network :

> docker network ls

NETWORK ID     NAME                               DRIVER    SCOPE
e661fefa2fba   bridge                             bridge    local
afa0a7369fed   docker_default                     bridge    local
636762e55a0e   host                               host      local
ipd43285nlar   ingress                            overlay   swarm
miy4fnbluu70   my_stack_ditto-net-swarm           overlay   swarm
4934f4344622   none                               null      local

>docker network inspect my_stack_ditto-net-swarm

[
    {
        "Name": "my_stack_ditto-net-swarm",
        "Id": "miy4fnbluu70bdc30qcqpyqy9",
        "Created": "2024-06-18T16:46:24.45599018Z",
        "Scope": "swarm",
        "Driver": "overlay",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "10.0.1.0/24",
                    "Gateway": "10.0.1.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "22f5721bd207a012953cf4f194abf7ce472ef6b18a6d167df73c4efae26a2ff9": {
                "Name": "my_stack_ditto-ui.1.jcnhcxrkxu9kl8v1a7dwjk4e1",
                "EndpointID": "008daec7000072bb81e7fef018f022a53a86b27a124aa32392c584b3ab479166",
                "MacAddress": "02:42:0a:00:01:06",
                "IPv4Address": "10.0.1.6/24",
                "IPv6Address": ""
            },
            "469a013c5ef0ab8c914dbd0aceeecbd522eba314a55648abd0512c0fb844e6c7": {
                "Name": "my_stack_things-search.1.x030dszx0c2kdwpptlmfsybbu",
                "EndpointID": "eb6fb56aa6d6a5b650e057473cb4576faab98aea3407c53564cae38f3de9b16d",
                "MacAddress": "02:42:0a:00:01:0c",
                "IPv4Address": "10.0.1.12/24",
                "IPv6Address": ""
            },
            "4bba3cf66c010d629b6fd3ecd7467b9f9a85442363e276d18901e6559247d029": {
                "Name": "my_stack_gateway.1.8s3y9sezounm8c0l0sfnp8fcb",
                "EndpointID": "62f02710d5f9bf5b861a86c0a348141607167125c10a977d5c1766ca807b8cc7",
                "MacAddress": "02:42:0a:00:01:03",
                "IPv4Address": "10.0.1.3/24",
                "IPv6Address": ""
            },
            "67b18c17bc245bd547a49c36119f4404d0a15e4d674b681b8bdeb6273aaae364": {
                "Name": "my_stack_policies.1.rbgb8unxtoqqg2l0rhw026ppq",
                "EndpointID": "96a54ba12d049aebbd4d65caa888676ec009a535fa5bdd713a2d9ab873c9ebd2",
                "MacAddress": "02:42:0a:00:01:0f",
                "IPv4Address": "10.0.1.15/24",
                "IPv6Address": ""
            },
            "92fe4111b89f0e5a890f877d2d69e33bd0be8ce4765a28a1b8bf7e30811d3c65": {
                "Name": "my_stack_swagger-ui.1.xbtbyt3gd63wff5hrph6naj4o",
                "EndpointID": "0265305aa5b16b532e304b8f5557e95996e983bfad04bac7e90949c3459c287b",
                "MacAddress": "02:42:0a:00:01:0a",
                "IPv4Address": "10.0.1.10/24",
                "IPv6Address": ""
            },
            "a46e189a9c4ea4be0bc3cb49213af4d608bdc08995f2f4078d3d287e35333810": {
                "Name": "my_stack_mongodb.1.fsatefqvrmwsyo8yw8tp6qgf6",
                "EndpointID": "1ca0f13f71fc59027ca925be64c3b11c9a13ce9dc1fca56c77f22ef9f1d4913c",
                "MacAddress": "02:42:0a:00:01:13",
                "IPv4Address": "10.0.1.19/24",
                "IPv6Address": ""
            },
            "abaf31aa0a26d89630dfc90c0e90fd38fb9ea3dab141855f7820e0d5ed75ad3e": {
                "Name": "my_stack_things.1.396rkdghvfvhoxaohkprmyy58",
                "EndpointID": "80c8e0ff38da65c9df9db924325743a49311d6e7211d395e0f62b38bd527b7c8",
                "MacAddress": "02:42:0a:00:01:15",
                "IPv4Address": "10.0.1.21/24",
                "IPv6Address": ""
            },
            "ed73ab9cc3e99314f1e2809fc41eda5c7f700b8f59d129a912cc05659ac73ffa": {
                "Name": "my_stack_connectivity.1.njn438mpqzyhgg16r67mrriar",
                "EndpointID": "4436d51cf549f07aaf777afa9bf8e310e593ad7cde3df9d4f0395f684a78a1f0",
                "MacAddress": "02:42:0a:00:01:11",
                "IPv4Address": "10.0.1.17/24",
                "IPv6Address": ""
            },
            "efe84e1eec4843608fb5d950a88b0e63dc81247f2420b75fd4b47293610a74b7": {
                "Name": "my_stack_nginx.1.e5c48q82xwb9z8ebapm20pri0",
                "EndpointID": "3bf4e453701f08a584d7fed4e24a6953130553cd5741ebd58dac5934d810d71a",
                "MacAddress": "02:42:0a:00:01:0d",
                "IPv4Address": "10.0.1.13/24",
                "IPv6Address": ""
            },
            "lb-my_stack_ditto-net-swarm": {
                "Name": "my_stack_ditto-net-swarm-endpoint",
                "EndpointID": "ed13713847ca2649c46e0e4ffe5736485a2d96150c9abf596779a3a053d6ccbe",
                "MacAddress": "02:42:0a:00:01:04",
                "IPv4Address": "10.0.1.4/24",
                "IPv6Address": ""
            }
        },
        "Options": {
            "com.docker.network.driver.overlay.vxlanid_list": "4097"
        },
        "Labels": {
            "com.docker.stack.namespace": "my_stack"
        },
        "Peers": [
            {
                "Name": "c4f6c67a05a0",
                "IP": "192.168.65.3"
            }
        ]
    }
]

The logs of my_stack_gateway : https://pastebin.com/LJrQX5Hw

aissam-en avatar Jun 18 '24 17:06 aissam-en

Hi. I assume that Ditto will not run on docker swarm out of the box. I used it 7 years ago or so and back then it required a special DNS based service discovery of the Ditto cluster nodes.

Why would you use docker swarm when just locally starting it on a windows machine?

thjaeckle avatar Jun 19 '24 18:06 thjaeckle

So, there is no chance to run Ditto on docker swarm.

And regarding why I am running Ditto in Docker Swarm locally : I ran Ditto locally with regular Docker during development without issues. after finishing development, I tried deploying it on a production server with Docker Swarm, and I faced that issue, so, to troubleshoot it, I set Ditto up locally in docker swarm.

aissam-en avatar Jun 19 '24 21:06 aissam-en

So, there is no chance to run Ditto on docker swarm.

Of course there is. I just can't provide you the exact steps how to do it.
In a former company where I worked, we ran it in Docker swarm.

thjaeckle avatar Jun 20 '24 06:06 thjaeckle

I see in your docker-compose.yml you have BIND_HOSTNAME=0.0.0.0 You can try to have the specific service name. For example: BIND_HOSTNAME=policies

This way the pekko-dns discovery method may actually be used correctly:

https://pekko.apache.org/docs/pekko/current/discovery/index.html#discovery-method-dns

kalinkostashki avatar Jun 20 '24 07:06 kalinkostashki