google
Pushes to gitlab fail with SSL error
from https://github.com/bazelbuild/rules_docker/issues/820:
When attempting to run a docker push through a container_push rule to a registry.gitlab.com target, I am recieving the following error: registry.gitlab.com/gauntletwizard/bazel-go:{BUILD_EMBED_LABEL} was resolved to registry.gitlab.com/gauntletwizard/bazel-go:foo F0430 20:27:05.873595 4586 fast_pusher_.py:194] Error publishing registry.gitlab.com/gauntletwizard/bazel-go:foo: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:727)
Users report they have made no change to their dependencies. Error is coming from containerregistry libraries so posting here. My suspicion is something changed in registry.gitlab.com that makes it so that containerregistry is no longer being able to validate certificates.
@nlopezgi Is there a way to set PYTHONHTTPSVERIFY property in bazel, such that it trickles down to containerregistry? (found this thread to turn off cert verification in python: http://blog.pengyifan.com/how-to-fix-python-ssl-certificate_verify_failed/)
Also is it possible to figure out for which URL cert verification is failing? The SSL cert for https://registry.gitlab.com seems to be fine - current cert is valid from March 23rd, 2019 - May 22nd, 2020.
The issue is coming from httplib2. You can reproduce the issue using the following python code:
import httplib2
h = httplib2.Http()
h.request('https://registry.gitlab.com')
And the issue can be resolved by adding the root certificate to the Http definition.
import httplib2
h = httplib2.Http('./USERTrustRSACertificationAuthority.crt')
h.request('https://registry.gitlab.com')
The root certificates used by httplib2 are coming from the cacerts.txt file. (https://github.com/httplib2/httplib2/blob/master/python2/httplib2/cacerts.txt)
registry.gitlab.com probably switched the root CA last week and that has triggered the problem.
I don't see any option in the containerregistry registry code to specify a CACerts file. I see three open PRs related to configuring CA Certs.
- https://github.com/google/containerregistry/pull/52
- https://github.com/google/containerregistry/pull/134
- https://github.com/google/containerregistry/pull/89
However, they have been open for a while. Is there any workaround for this problem that others have used?
Until support is available in httplib2, use an http_archive patch: https://gist.github.com/arcticwaters/d3310d2b6e362a277501615c6ff07878. This patch works for gitlab.
Containerregistry won't register that repository if one is already defined so it has to go at the top of your WORKSPACE. Downside: you'll have to remember to update when containerregistry updates httplib2.
I came up with my own hack to get it to work. But the patch approach seems better. Thanks for sharing!
