source-controller Flux fails to login to HTTP OCI registry where helm CLI logs in

Describe the bug

opening an issue per @souleb 's request. Slack discussion here

Excerpts:

Hello trying to confirm whether this is a bug: I have deployed helm's OCI registry from (described here https://helm.sh/docs/topics/registries/) into my kind k8s cluster. I have configured it to have a single authorized user: username foo, password bar The deployment is exposed as a service like this:

$ kubectl get service registry-app-svc
NAME               TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)    AGE
registry-app-svc   ClusterIP   10.96.71.93   <none>        5000/TCP   19m

i.e. accessible within the cluster as registry-app-svc.default.svc.cluster.local:5000/helm-charts

sure enough, I can open a new shell into the cluster and do this:

$ curl -i -XGET "http://registry-app-svc.default.svc.cluster.local:5000/v2/"
HTTP/1.1 401 Unauthorized
Content-Type: application/json; charset=utf-8
Docker-Distribution-Api-Version: registry/2.0
Www-Authenticate: Basic realm="localhost"
X-Content-Type-Options: nosniff
Date: Thu, 30 Jun 2022 07:24:03 GMT
Content-Length: 87

{"errors":[{"code":"UNAUTHORIZED","message":"authentication required","detail":null}]}

$ curl -i -XGET "http://registry-app-svc.default.svc.cluster.local:5000/v2/" -u foo:bar
HTTP/1.1 200 OK
Content-Length: 2
Content-Type: application/json; charset=utf-8
Docker-Distribution-Api-Version: registry/2.0
X-Content-Type-Options: nosniff
Date: Thu, 30 Jun 2022 07:24:14 GMT

{}

This is all as expected. Now, if I port-forward 5000 from the cluster to my localhost (client) like this $ kubectl port-forward --namespace default service/registry-app-svc 5000:5000 I can also do

$ helm registry login localhost:5000 -u foo -p bar
WARNING: Using --password via the CLI is insecure. Use --password-stdin.
INFO[0000] Error logging in to endpoint, trying next endpoint  error="Get \"https://localhost:5000/v2/\": http: server gave HTTP response to HTTPS client"
Login Succeeded

So far so good. Notice the INFO message above, I think it is key here. I can also use other helm registry commands such as:

$ helm push charts/podinfo-6.0.0.tgz oci://localhost:5000/helm-charts
Pushed: localhost:5000/helm-charts/podinfo:6.0.0
Digest: sha256:4f157078b10fd81f09d72efe301d90721ea49e036b960cd45ffaa7f6bbd9e5e8
$ helm show all oci://localhost:5000/helm-charts/podinfo | head -9
apiVersion: v1
appVersion: 6.0.0
description: Podinfo Helm chart for Kubernetes
home: https://github.com/stefanprodan/podinfo
kubeVersion: '>=1.19.0-0'
maintainers:
- email: [email protected]
  name: stefanprodan
name: podinfo

Now if I create a secret like this:

$ flux create secret helm local-auth-12345 \
        --namespace=default \
        --username=foo \
        --password=bar

followed by

$ flux create source helm podinfo \
    --url=oci://registry-app-svc.default.svc.cluster.local:5000/helm-charts \
    --namespace=default \
    --secret-ref=local-auth-12345

I get an error:

$ flux create source helm podinfo \
>     --url=oci://registry-app-svc.default.svc.cluster.local:5000/helm-charts \
>     --namespace=default \
>     --secret-ref=local-auth-12345
✚ generating HelmRepository source
► applying HelmRepository source
:heavy_check_mark: source updated
◎ waiting for HelmRepository source reconciliation
✗ failed to log into registry 'oci://registry-app-svc.default.svc.cluster.local:5000/helm-charts': Get "https://registry-app-svc.default.svc.cluster.local:5000/v2/": http: server gave HTTP response to HTTPS client

and the HelmRepository CRD is stuck in a failed state:

$ kubectl get helmrepository podinfo
NAME      URL                                                                 AGE   READY   STATUS
podinfo   oci://registry-app-svc.default.svc.cluster.local:5000/helm-charts   25m   False   failed to log into registry 'oci://registry-app-svc.default.svc.cluster.local:5000/helm-charts': Get "https://registry-app-svc.default.svc.cluster.local:5000/v2/": http: server gave HTTP response to HTTPS client

It seems to me that flux source controller tries to communicate via HTTPs, fails and gives up. Compare that to helm client which also gets an error but then tries plain HTTP and it works

I can provide much more detailed repro steps if needed. Thanks in advance for consideration Greg

my environment:

$ helm version
version.BuildInfo{Version:"v3.9.0", GitCommit:"7ceeda6c585217a19a1131663d8cd1f7d641b2a7", GitTreeState:"clean", GoVersion:"go1.18.2"}

$ flux version
flux: v0.31.1
helm-controller: v0.22.1
image-automation-controller: v0.23.4
image-reflector-controller: v0.19.2
kustomize-controller: v0.26.1
notification-controller: v0.24.0
source-controller: v0.25.8

... As an end user I like the fact that helm cli does not require me to know what's behind the OCI registry. helm client allows me to work with both registeries that require HTTPs (ghcr.io) and require HTTP (this sample registry image from docker). I don't need to know. Period. Are you sure you want fluxcd to work differently than helm CLI? That seems an odd choice to me - I would think fluxcd would want to be consistent with helm CLI and not burden the end user with this. Just a thought

Steps to reproduce

see above

Expected behavior

see above

Kubernetes version / Distro / Cloud provider

kind

Flux version

see above

Git provider

see above

Container Registry provider

see above

Additional context

see above

Maintenance Acknowledgement

[X] I am aware of Flux v1's maintenance status

Code of Conduct

[X] I agree to follow this project's Code of Conduct

Jun 30 '22 09:06 gfichtenholt

A comment after looking at the code.

The difference between when the helm cli is used vs source-controller:

the cli tries 2 endpoints https then http.
the source-controller tries only one endpoint, the https

In order to try both endpoints, insecure is set. Do you provide that flag to the cli?

Flux does not support the insecure option yet. I will add a separate issue to track that enhancement.

edit: issue is #807

Jun 30 '22 09:06 souleb

Also this:

	// Localhost is by default considered as an insecure registry
	// This is a stop-gap for people who are running a private registry on localhost (especially on Boot2docker).
	//
	// TODO: should we deprecate this once it is easier for people to set up a TLS registry or change
	// daemon flags on boot2docker?
	registries = append(registries, "127.0.0.0/8")

That explains why it tried both endpoints. It has to do with the port-forward to localhost.

Jun 30 '22 10:06 souleb

A comment after looking at the code.

The difference between when the helm cli is used vs source-controller:
* the cli tries 2 endpoints `https` then `http`.

* the source-controller tries only one endpoint, the `https`
In order to try both endpoints, insecure is set. Do you provide that flag to the cli?

I did not. Commands were issues exactly as I've stated above. Its a copy and paste from my terminal

Flux does not support the insecure option yet. I will add a separate issue to track that enhancement.

edit: issue is #807

👍 That's another long standing issue with flux I've had. Thanks for filing it, saves me work

Jun 30 '22 15:06 gfichtenholt