Consolidate dns resolution requests per docker cli command into one
Description
Occasionally I will get docker pull auth failure when the docker registry is behind the consul cluster. From the docker pull implementation here, it will issue multiple dns resolution request in different steps of the CLI. Since the docker registry is behind consul dns interface, it could be possible that the resolved hosts for all the above dns requests are different, leading to the authentication failure. The packet dump on the registry side proves my theory.
It would be better to consolidate all the dns requests into one during the initial stage of the request, so that the docker cli will be supporting the DNS based LB better. Logically, we could also work around in the dns recursor side to have the DNS response ttl to be longer, but I think less dns request will still be better to increase the robustness of the commands.
Steps to reproduce the issue:
- Add docker registry behind consul (or any other DNS based LB solution)
- Issue docker pull for an upstream docker image.
Describe the results you received: user@host$ docker pull <registery_url>:<registry_port>/kubernetesui/dashboard:v2.0.3 v2.0.3: Pulling from kubernetesui/dashboard unauthorized: authentication required
Describe the results you expected: v2.0.3: Pulling from kubernetesui/dashboard Digest: sha256:45ef224759bc50c84445f233fffae4aa3bdaec705cb5ee4bfe36d183b270b45d Status: Image is up to date for nexus.sjc1.corp.pony.ai:5000/kubernetesui/dashboard:v2.0.3
Additional information you deem important (e.g. issue happens only occasionally):
Output of docker version:
Client: Docker Engine - Community
Version: 19.03.8
API version: 1.40
Go version: go1.12.17
Git commit: afacb8b7f0
Built: Wed Mar 11 01:25:46 2020
OS/Arch: linux/amd64
Experimental: false
Server: Docker Engine - Community
Engine:
Version: 19.03.8
API version: 1.40 (minimum version 1.12)
Go version: go1.12.17
Git commit: afacb8b7f0
Built: Wed Mar 11 01:24:19 2020
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.2.13
GitCommit: 7ad184331fa3e55e52b890ea95e65ba581ae3429
nvidia:
Version: 1.0.0-rc10
GitCommit: dc9208a3303feef5b3839f4323d9beb36df0a9dd
docker-init:
Version: 0.18.0
GitCommit: fec3683
Output of docker info:
Client:
Debug Mode: false
Server:
Containers: 9
Running: 2
Paused: 0
Stopped: 7
Images: 154
Server Version: 19.03.8
Storage Driver: overlay2
Backing Filesystem: <unknown>
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: systemd
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: nvidia runc
Default Runtime: nvidia
Init Binary: docker-init
containerd version: 7ad184331fa3e55e52b890ea95e65ba581ae3429
runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
init version: fec3683
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 5.4.0-70-generic
Operating System: Ubuntu 18.04.5 LTS
OSType: linux
Architecture: x86_64
CPUs: 12
Total Memory: 31.03GiB
Name: jianan-pony-laptop
ID: YGC7:XQE6:6CEZ:IYFW:QX4R:53S7:7NB3:GLMR:KNIJ:VQDT:W6LA:MGCO
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: true
WARNING: No swap limit support
Additional environment details (AWS, VirtualBox, physical, etc.): physical laptop