cli icon indicating copy to clipboard operation
cli copied to clipboard

Consolidate dns resolution requests per docker cli command into one

Open lordofire opened this issue 3 years ago • 0 comments

Description

Occasionally I will get docker pull auth failure when the docker registry is behind the consul cluster. From the docker pull implementation here, it will issue multiple dns resolution request in different steps of the CLI. Since the docker registry is behind consul dns interface, it could be possible that the resolved hosts for all the above dns requests are different, leading to the authentication failure. The packet dump on the registry side proves my theory.

It would be better to consolidate all the dns requests into one during the initial stage of the request, so that the docker cli will be supporting the DNS based LB better. Logically, we could also work around in the dns recursor side to have the DNS response ttl to be longer, but I think less dns request will still be better to increase the robustness of the commands.

Steps to reproduce the issue:

  1. Add docker registry behind consul (or any other DNS based LB solution)
  2. Issue docker pull for an upstream docker image.

Describe the results you received: user@host$ docker pull <registery_url>:<registry_port>/kubernetesui/dashboard:v2.0.3 v2.0.3: Pulling from kubernetesui/dashboard unauthorized: authentication required

Describe the results you expected: v2.0.3: Pulling from kubernetesui/dashboard Digest: sha256:45ef224759bc50c84445f233fffae4aa3bdaec705cb5ee4bfe36d183b270b45d Status: Image is up to date for nexus.sjc1.corp.pony.ai:5000/kubernetesui/dashboard:v2.0.3

Additional information you deem important (e.g. issue happens only occasionally):

Output of docker version:

Client: Docker Engine - Community
 Version:           19.03.8
 API version:       1.40
 Go version:        go1.12.17
 Git commit:        afacb8b7f0
 Built:             Wed Mar 11 01:25:46 2020
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.8
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.17
  Git commit:       afacb8b7f0
  Built:            Wed Mar 11 01:24:19 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.2.13
  GitCommit:        7ad184331fa3e55e52b890ea95e65ba581ae3429
 nvidia:
  Version:          1.0.0-rc10
  GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

Output of docker info:

Client:
 Debug Mode: false

Server:
 Containers: 9
  Running: 2
  Paused: 0
  Stopped: 7
 Images: 154
 Server Version: 19.03.8
 Storage Driver: overlay2
  Backing Filesystem: <unknown>
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: systemd
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: nvidia runc
 Default Runtime: nvidia
 Init Binary: docker-init
 containerd version: 7ad184331fa3e55e52b890ea95e65ba581ae3429
 runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
 init version: fec3683
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 5.4.0-70-generic
 Operating System: Ubuntu 18.04.5 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 12
 Total Memory: 31.03GiB
 Name: jianan-pony-laptop
 ID: YGC7:XQE6:6CEZ:IYFW:QX4R:53S7:7NB3:GLMR:KNIJ:VQDT:W6LA:MGCO
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: true

WARNING: No swap limit support

Additional environment details (AWS, VirtualBox, physical, etc.): physical laptop

lordofire avatar Jun 14 '22 06:06 lordofire