docker
Docker uses wrong DNS server
Description Suddenly, docker build and docker run started to use the DNS server provided by my WiFi's DHCP (192.168.1.1) instead of the DNS server provided by my VPN connection (192.168.94.1), leading to buid & runtime issues. On my machine, everything works well, as it uses the correct DNS server (VPN).
Steps to reproduce the issue:
➜ ~ nslookup my.site.custom
Server: 127.0.0.53
Address: 127.0.0.53#53
Non-authoritative answer:
Name: my.site.custom
Address: 192.168.94.108
➜ ~ docker run busybox nslookup my.site.custom
Server: 192.168.1.1
Address: 192.168.1.1:53
Non-authoritative answer:
*** Can't find my.site.custom: No answer
Describe the results you expected:
I expect the DNS resolution to work the same way inside a docker container than on the host.
Output of docker version:
➜ ~ docker version
Client: Docker Engine - Community
Version: 19.03.12
API version: 1.40
Go version: go1.13.10
Git commit: 48a66213fe
Built: Mon Jun 22 15:45:44 2020
OS/Arch: linux/amd64
Experimental: false
Server: Docker Engine - Community
Engine:
Version: 19.03.12
API version: 1.40 (minimum version 1.12)
Go version: go1.13.10
Git commit: 48a66213fe
Built: Mon Jun 22 15:44:15 2020
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.2.13
GitCommit: 7ad184331fa3e55e52b890ea95e65ba581ae3429
runc:
Version: 1.0.0-rc10
GitCommit: dc9208a3303feef5b3839f4323d9beb36df0a9dd
docker-init:
Version: 0.18.0
GitCommit: fec3683
Output of docker info:
Client:
Debug Mode: false
Server:
Containers: 79
Running: 3
Paused: 0
Stopped: 76
Images: 402
Server Version: 19.03.12
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 7ad184331fa3e55e52b890ea95e65ba581ae3429
runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
init version: fec3683
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 5.4.0-45-generic
Operating System: Ubuntu 20.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 15.3GiB
Name: nico-laptop
ID: LMZR:H2CD:L26I:3CI5:YW6F:SYC6:F5OP:5QBQ:RHVX:TQK4:TTS5:M7SF
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: No swap limit support
Additional environment details (AWS, VirtualBox, physical, etc.): Lenovo laptop OS: Ubuntu 20.04 (5.4.0-45-generic)
My resolv.conf file (empy because of systemd-resolve)
➜ ~ cat /etc/resolv.conf
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
# DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
# 127.0.0.53 is the systemd-resolved stub resolver.
# run "systemd-resolve --status" to see details about the actual nameservers.
nameserver 127.0.0.53
search home
systemd-resolve status:
➜ ~ systemd-resolve --status
Global
LLMNR setting: no
MulticastDNS setting: no
DNSOverTLS setting: no
DNSSEC setting: no
DNSSEC supported: no
DNS Domain: home
DNSSEC NTA: 10.in-addr.arpa
16.172.in-addr.arpa
168.192.in-addr.arpa
17.172.in-addr.arpa
18.172.in-addr.arpa
19.172.in-addr.arpa
20.172.in-addr.arpa
21.172.in-addr.arpa
22.172.in-addr.arpa
23.172.in-addr.arpa
24.172.in-addr.arpa
25.172.in-addr.arpa
26.172.in-addr.arpa
27.172.in-addr.arpa
28.172.in-addr.arpa
29.172.in-addr.arpa
30.172.in-addr.arpa
31.172.in-addr.arpa
corp
d.f.ip6.arpa
home
internal
intranet
lan
local
private
test
Link 187 (veth9d78356)
Current Scopes: none
DefaultRoute setting: no
LLMNR setting: yes
MulticastDNS setting: no
DNSOverTLS setting: no
DNSSEC setting: no
DNSSEC supported: no
Link 185 (vethc70837a)
Current Scopes: none
DefaultRoute setting: no
LLMNR setting: yes
MulticastDNS setting: no
DNSOverTLS setting: no
DNSSEC setting: no
DNSSEC supported: no
Link 183 (veth7766a10)
Current Scopes: none
DefaultRoute setting: no
LLMNR setting: yes
MulticastDNS setting: no
DNSOverTLS setting: no
DNSSEC setting: no
DNSSEC supported: no
Link 181 (br-10860fc429a6)
Current Scopes: none
DefaultRoute setting: no
LLMNR setting: yes
MulticastDNS setting: no
DNSOverTLS setting: no
DNSSEC setting: no
DNSSEC supported: no
Link 180 (br-dd31e3eb53c6)
Current Scopes: none
DefaultRoute setting: no
LLMNR setting: yes
MulticastDNS setting: no
DNSOverTLS setting: no
DNSSEC setting: no
DNSSEC supported: no
Link 179 (br-9dea358c8ee5)
Current Scopes: none
DefaultRoute setting: no
LLMNR setting: yes
MulticastDNS setting: no
DNSOverTLS setting: no
DNSSEC setting: no
DNSSEC supported: no
Link 178 (docker0)
Current Scopes: none
DefaultRoute setting: no
LLMNR setting: yes
MulticastDNS setting: no
DNSOverTLS setting: no
DNSSEC setting: no
DNSSEC supported: no
Link 177 (br-2fb348246195)
Current Scopes: none
DefaultRoute setting: no
LLMNR setting: yes
MulticastDNS setting: no
DNSOverTLS setting: no
DNSSEC setting: no
DNSSEC supported: no
Link 18 (tun0)
Current Scopes: DNS
DefaultRoute setting: yes
LLMNR setting: yes
MulticastDNS setting: no
DNSOverTLS setting: no
DNSSEC setting: no
DNSSEC supported: no
Current DNS Server: 192.168.94.1
DNS Servers: 192.168.94.1
DNS Domain: ~.
local.enterprise
Link 4 (enx4ce173428067)
Current Scopes: none
DefaultRoute setting: no
LLMNR setting: yes
MulticastDNS setting: no
DNSOverTLS setting: no
DNSSEC setting: no
DNSSEC supported: no
Link 3 (wlp0s20f3)
Current Scopes: DNS
DefaultRoute setting: yes
LLMNR setting: yes
MulticastDNS setting: no
DNSOverTLS setting: no
DNSSEC setting: no
DNSSEC supported: no
Current DNS Server: 192.168.1.1
DNS Servers: 192.168.1.1
DNS Domain: home
Link 2 (enp0s31f6)
Current Scopes: none
DefaultRoute setting: no
LLMNR setting: yes
MulticastDNS setting: no
DNSOverTLS setting: no
DNSSEC setting: no
DNSSEC supported: no
What I tried so far:
- Reboot
- Uninstall/Purge/Reinstall of Docker
- Flush iptables nat rules, delete docker0 & docker br-xxx bridge
I had the same issue today, even with a slightly different setup but I guess it's for the same reason: Docker relies on the hosts's /etc/resolv.conf. If your server is running systemd-resolved, then it will read /run/systemd/resolve/resolv.conf instead.
So check your appropriate */resolv.conf if the DNS servers are in the right order – if they are not, it might be easier to set the correct dns entries in your Docker daemon.json. There is no way to configure the order of nameserver entries in the generated by systemd-resolved (I am aware of).
Additional side-note: Docker will parse the resolv.conf and remove all localhost (127.* & ::1) DNS entries from it, because they cannot be resolved correctly from containers. (source) This might happen if you are running e.g. dnsmasq locally. If there are no more non-local nameservers left, Docker will use Google's DNS servers instead for containers (8.8.8.8 and 8.8.4.4).
I was also having this issue.
First i discovered that by setting --network=host on docker build all comands like curl and wget started resolving my VPN addresses. Because by default docker uses the network=bridge, used for communication between containers. By choosing network=host the container has access to the network of the host.
Then i tried creating the daemon.json as @HenningCash described. It worked! Just add your DNS ip to daemon.json like this:
{ "dns": [ "dns_ip"] }
Hope it helps other people and it resolves this issue.
