Can't push to own registry which has a self-signed certificate

[devzsolt]

Description

I have a self-hosted registry provided by harbor which works well with docker and docker-compose. The registry is only accessible through HTTPS and the certificate is self-signed. For this to make work I needed to put the ca.crt into /etc/docker/certs.d/my.domain:customport/. docker login was successful. Seems like docker-app doesn't consider the saved ca.crt but rather fails pushing.

$ cat metadata.yml | grep namespace
namespace: my.domain:customport/projectname

$ ls -al /etc/docker/certs.d/my.domain\:customport/ca.crt 
-rwxr-xr-x 1 root root 1972 Nov  2 23:23 /etc/docker/certs.d/my.domain:customport/ca.crt

$ docker-app push
Error: Get https://my.domain:customport/v2/: x509: certificate signed by unknown authority

This is the exact same error with docker before putting the ca.crt in place.

Steps to reproduce the issue:

1. install a self-hosted registry with TLS enabled having a self-signed certificate
2. put the ca.crt into /etc/docker/certs.d/your.domain:customport/
3. set the namespace so that it will use your registry instead of Docker Hub
4. try push

Describe the results you received:

Error: Get https://my.domain:customport/v2/: x509: certificate signed by unknown authority

Describe the results you expected:

A successful push to the registry.

Additional information you deem important (e.g. issue happens only occasionally):

It happens always.

Output of docker version:

Client:
 Version:           18.06.1-ce
 API version:       1.38
 Go version:        go1.10.3
 Git commit:        e68fc7a
 Built:             Tue Aug 21 17:24:51 2018
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          18.06.1-ce
  API version:      1.38 (minimum version 1.12)
  Go version:       go1.10.3
  Git commit:       e68fc7a
  Built:            Tue Aug 21 17:23:15 2018
  OS/Arch:          linux/amd64
  Experimental:     false

Output of docker-app version:

Version:      v0.6.0
Git commit:   9f9c6680
Built:        Thu Oct  4 13:30:33 2018
OS/Arch:      linux/amd64
Experimental: off
Renderers:    none

Output of docker info:

WARNING: No swap limit support
Containers: 14
 Running: 12
 Paused: 0
 Stopped: 2
Images: 20
Server Version: 18.06.1-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: active
 NodeID: niout9091kaxngunf7qco0uo9
 Is Manager: true
 ClusterID: x9umkic6692awwc381qgg4kn1
 Managers: 1
 Nodes: 1
 Orchestration:
  Task History Retention Limit: 5
 Raft:
  Snapshot Interval: 10000
  Number of Old Snapshots to Retain: 0
  Heartbeat Tick: 1
  Election Tick: 10
 Dispatcher:
  Heartbeat Period: 5 seconds
 CA Configuration:
  Expiry Duration: 3 months
  Force Rotate: 0
 Autolock Managers: false
 Root Rotation In Progress: false
 Node Address: *deleted*
 Manager Addresses:
  *deleted*:2377
Runtimes: runc
Default Runtime: runc
WARNING: No swap limit support
Init Binary: docker-init
containerd version: 468a545b9edcd5932818eb9de8e72413e616e86e
runc version: 69663f0bd4b60df09991c08812a60108003fa340
init version: fec3683
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.15.0-38-generic
Operating System: Ubuntu 18.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 1.947GiB
Name: *deleted*
ID: MTSL:CLWJ:JV3I:V5XZ:MYQE:3CTL:EP7B:UWZ6:FPQY:LNGN:RWDU:YGXP
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

Additional environment details (AWS, VirtualBox, physical, etc.):

Running on DigitalOcean but I believe it doesn't matter.

[devzsolt]

Looks like adding the cert to /usr/local/share/ca-certificates/ and updating resolves the problem. So it feels like a minor bug for not supporting certs in /etc/docker/certs.d

[simonferquel]

The push/pull story is being reworked as part as moving to the CNAB runtime. I have no idea if/when it will fix the issue, but that is the reason we did not report back sooner on this. Sorry!

[vce-xx]

Looks like adding the cert to /usr/local/share/ca-certificates/ and updating resolves the problem. So it feels like a minor bug for not supporting certs in /etc/docker/certs.d

Would this work with Docker Desktop for Mac ?

[doanac]

This is still broken:

$ ./bin/docker-app -D app pull hub.foundries.io/andy-corp/skiparchs@sha256:882cee7b2978dd6d79f68cf0f1c683c644c96d558ade759171cad57269bd7d29
DEBU[0000] insecure registries: []                      
DEBU[0000] Pulling CNAB Bundle hub.foundries.io/andy-corp/skiparchs@sha256:882cee7b2978dd6d79f68cf0f1c683c644c96d558ade759171cad57269bd7d29 
DEBU[0000] Getting OCI Index Descriptor                 
hub.foundries.io/andy-corp/skiparchs@sha256:882cee7b2978dd6d79f68cf0f1c683c644c96d558ade759171cad57269bd7d29: failed to resolve bundle manifest "hub.foundries.io/andy-corp/skiparchs@sha256:882cee7b2978dd6d79f68cf0f1c683c644c96d558ade759171cad57269bd7d29": failed to do request: Head https://hub.foundries.io/v2/andy-corp/skiparchs/manifests/sha256:882cee7b2978dd6d79f68cf0f1c683c644c96d558ade759171cad57269bd7d29: x509: certificate signed by unknown authority

I can get it further along with a hack like: partial-fix.diff.txt. Its roughly copied from github.com/docker/docker/registry/registry.go. However, I don't see a sane way of handling this in docker-app. The containerd code path for this seems to want the TLS config ahead of time before the code knows what host it needs to connect to.

[doanac]

I've found a slightly better way, but its still probably something that's not palatable to the upstream:

https://github.com/doanac/app/commit/f26d65516dfa77766f46eb9129cd8f3e424a4826