kitchen docker is unable to ssh into docker when using static ip in the run_options: --net network --ip "192.168.1.1"

[claudio-viola]

Context: I want to be able to use kitchen docker for multi node testing through static ip within a network set by me on the different docker containers / kitchen suites

OS: Mac

kitchen.yml with run_options: --net mynetwork --ip 192.168.33.32

Test Kitchen version 1.10.2 docker version Client: Version: 1.11.2 API version: 1.23 Go version: go1.5.4 Git commit: b9f10c9 Built: Wed Jun 1 21:20:08 2016 OS/Arch: darwin/amd64

Waiting for SSH service on 'Docker Daemon IP:port' retrying in 3 seconds Waiting for SSH service on 'Docker Daemon IP:port', retrying in 3 seconds Waiting for SSH service on 'Docker Daemon IP:port', retrying in 3 seconds

Can anyone help? Which setting should I give kitchen so that he knows how to login? Am I doing something wrong?

P.S. Removing run_options kitchen is able to login and converge successfully

Thank you all

[ytsarev]

I faced the similar problem, you can use https://github.com/test-kitchen/kitchen-docker/pull/203 to solve it

[claudio-viola]

thank you so much @ytsarev . I will give it a try . I was thinking to do something similar but I thought it was something wrong with me. now I know its not Wouldyou have a working example kitchen.yml you could share that works with your PR?

• docker static ip setting
• docker network settings
• ssh_host ip

Many thanks

[claudio-viola]

hi @ytsarev i used your changes however I get Waiting for SSH service on 192.168.33.32:32782, retrying in 3 seconds

the port 32782 is not apparently the container port in use

i have changed the port manually to 22 i then get asked for password on my kitchen username 'kitchen kitchen' fails to login manually

[ytsarev]

@claudio-viola the only thing you can control with ssh_host is the host to connect to. The network connectivity will not be solved automatically. And apparently your are trying to connect from the host to a container private network which is not going to work out of the box, iirc

[claudio-viola]

yes @ytsarev that's exactly what i want to do. Any clue how it is possible to achieve this? With vagrant it is quite easy and kitchen vagrant is able to connect to the private network. Trying to achieve the same with kitchen-docker

[manojry]

If you are running basic centos you docker file should look like this FROM centos:6 RUN yum clean all RUN yum install -y sudo openssh-server openssh-clients which curl htop RUN ssh-keygen -t rsa -f /etc/ssh/ssh_host_rsa_key RUN ssh-keygen -t dsa -f /etc/ssh/ssh_host_dsa_key RUN mkdir -p /var/run/sshd RUN useradd -d /home/<%= @username %> -m -s /bin/bash <%= @username %> RUN echo <%= "#{@username}:#{@password}" %> | chpasswd RUN echo '<%= @username %> ALL=(ALL) NOPASSWD:ALL' >> /etc/sudoers RUN mkdir -p /home/<%= @username %>/.ssh RUN chown -R <%= @username %> /home/<%= @username %>/.ssh RUN chmod 0700 /home/<%= @username %>/.ssh RUN touch /home/<%= @username %>/.ssh/authorized_keys RUN chown <%= @username %> /home/<%= @username %>/.ssh/authorized_keys RUN chmod 0600 /home/<%= @username %>/.ssh/authorized_keys RUN curl -L https://www.chef.io/chef/install.sh | bash RUN echo '<%= IO.read(@public_key).strip %>' >> /home/<%= @username %>/.ssh/authorized_keys

then try kitchen create and kitchen converge

[manojry]

FROM centos:6 RUN yum clean all RUN yum install -y sudo openssh-server openssh-clients which curl htop RUN ssh-keygen -t rsa -f /etc/ssh/ssh_host_rsa_key RUN ssh-keygen -t dsa -f /etc/ssh/ssh_host_dsa_key RUN mkdir -p /var/run/sshd RUN useradd -d /home/<%= @username %> -m -s /bin/bash <%= @username %> RUN echo <%= "#{@username}:#{@password}" %> | chpasswd RUN echo '<%= @username %> ALL=(ALL) NOPASSWD:ALL' >> /etc/sudoers RUN mkdir -p /home/<%= @username %>/.ssh RUN chown -R <%= @username %> /home/<%= @username %>/.ssh RUN chmod 0700 /home/<%= @username %>/.ssh RUN touch /home/<%= @username %>/.ssh/authorized_keys RUN chown <%= @username %> /home/<%= @username %>/.ssh/authorized_keys RUN chmod 0600 /home/<%= @username %>/.ssh/authorized_keys RUN curl -L https://www.chef.io/chef/install.sh | bash RUN echo '<%= IO.read(@public_key).strip %>' >> /home/<%= @username %>/.ssh/authorized_keys

[renaudhager-ef]

Hi guys, I'm facing the same issue. I think it's related to the command used by kitchen to fetch the IP of the container. If I correctly understand, kitchen use docker inspect --format '{{ .NetworkSettings.IPAddress }} https://github.com/test-kitchen/kitchen-docker/blob/8743b4561645aea7ef66f67deafc9711e438bd7f/lib/kitchen/driver/docker.rb#L412

The problem is, when you use a network different than the default to run your container, this field is empty.

As you can see in the following exemple:

"NetworkSettings": {
            "Bridge": "",
            "SandboxID": "fca84b497cc9975708e0eeebcf2998cfd52a812565bf95965e9c1302fd396898",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {},
            "SandboxKey": "/var/run/docker/netns/fca84b497cc9",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "",
            "Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "MacAddress": "",
            "Networks": {
                "va475an4bu3nv5lbmd7r4zemc9prla5v": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": [
                        "Unit test",
                        "b2897c85bf64"
                    ],
                    "NetworkID": "34cb275a0826995fcfc5cce9653eeda581b402dd8d6c41cfc2842ebaee1b09a5",
                    "EndpointID": "fe36c8e44d8ba7d463cad104746d824f71a374303fa48da29f2fb256458a0e09",
                    "Gateway": "172.19.0.1",
                    "IPAddress": "172.19.0.2",
                    "IPPrefixLen": 16,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "02:42:ac:13:00:02",
                    "DriverOpts": null
                }
            }
        }

To get the IP you can do something like this:

docker -H unix:///var/run/docker.sock inspect --format '{{ .NetworkSettings.Networks.va475an4bu3nv5lbmd7r4zemc9prla5v.IPAddress }}' b2897c85bf64
172.19.0.2

But it's not easy because you need to pass the name of the network on the filter. One quick and a bit dirty way is to do something like this:

docker -H unix:///var/run/docker.sock inspect  b2897c85bf64 | grep IPAddress | tail -n 1| cut -d "\"" -f 4
172.19.0.2

But it's not very elegant. A solution could be to do it with 2 steps:

• 1 get the network name.
• 2 get the IP.

[an0nz]

@renaudhager-ef I think is on to the right track to a solution.

According to Docker Inspect Documentaiton The new method for getting an instances IP address now uses the below command $ docker inspect --format='{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' $INSTANCE_ID

The issue with this is when containers have multiple networks you get back all IP's in a single string as its using a range loop. eg: 172.17.0.2172.18.0.2

If only a single IP is needed the following format only returns the first IP it comes across under networks, which when not specifying any additional networks will get the bridge IP $ docker inspect --format '{{$done:=false}}{{range.NetworkSettings.Networks}}{{if not $done}}{{.IPAddress}}{{$done = true}}{{end}}{{end}}' $INSTANCE_ID

Propose changing https://github.com/test-kitchen/kitchen-docker/blob/41e80fed3a7cc86323e19c16a5a340cebf7e5848/lib/kitchen/docker/helpers/container_helper.rb#L137 To cmd = "inspect --format '{{$done:=false}}{{range.NetworkSettings.Networks}}{{if not $done}}{{.IPAddress}}{{$done = true}}{{end}}{{end}}'"

I have tested this change on a number of our build tests, it successfully retrieves the IP for containers on the default bridge network and for those with custom networks defined using run_options.

@tas50 - I see you previously submitted a similar fix in https://github.com/test-kitchen/kitchen-docker/pull/335, is it worthwhile someone putting in a merge request to solve this finally and if so who can do it?

@claudio-viola - If you wish to use this you need to modify the ruby file with the change For 2.9 the it is line 412 in /usr/lib/ruby/gems/2.7.0/gems/kitchen-docker-2.9.0/lib/kitchen/driver/docker.rb For 2.10 it is line 137 in /usr/lib/ruby/gems//gems/kitchen-docker-/lib/kitchen/docker/helpers/container_helper.rb

My build jobs use sed to replace the line in the appropriate file depending on the version For 2.9.0 sed -i 's/cmd = "inspect --format '\''{{ \.NetworkSettings\.IPAddress }}'\''"/cmd = "inspect --format '\''{{\$done:=false}}{{range\.NetworkSettings\.Networks}}{{if not \$done}}{{\.IPAddress}}{{\$done = true}}{{end}}{{end}}'\''"/g' /usr/lib/ruby/gems/2.7.0/gems/kitchen-docker-2.9.0/lib/kitchen/driver/docker.rb

For 2.10.0 sed -i 's/cmd = "inspect --format '\''{{ \.NetworkSettings\.IPAddress }}'\''"/cmd = "inspect --format '\''{{\$done:=false}}{{range\.NetworkSettings\.Networks}}{{if not \$done}}{{\.IPAddress}}{{\$done = true}}{{end}}{{end}}'\''"/g' /usr/lib/ruby/gems/2.7.0/gems/kitchen-docker-2.10.0/lib/kitchen/docker/helpers/container_helper.rb