zalando
support https endpoints in Kubernetes clusters
Consider supporting https backends in routes generated for Kubernetes ingress.
Questions:
- how should the ingress spec indicate when a service expects to receive the requests via https?
- Skipper is not a TLS proxy. It can listen on https, but it decrypts the traffic before proxying it to its backends. Can this be misleading if someone expects end-to-end encryption until the actual service?
I don't understand, what do you want to achieve? What does "https" for Kubernetes backends mean?
@hjacobs I tried to rephrase the content of the issue. What I meant was that a service in the cluster listens accepts requests via https instead of http.
So you mean to use sni to identify the target or how do you think it should work? Can you elaborate more on how a user would use this feature?
I don't know how we could identify these targets.
The use case I can imagine is that a user wants to run a service only over HTTPS even inside the cluster, but also wants to make it accessible for the world outside the cluster, so wants to create an ingress for it.
This use case is covered by service type loadbalancer, use plain tcp. If you want to support it with ingress we would need to do the route decision based on SNI. We have to understand how to get the data from the transport to make this decision and we need to have a lower layer predicate which isn’t based on http.Request . It would be nice to have it and can enable a lot of use cases that you are not aware of. For example manual access to databases.
@szuecs @aryszka I don't think you are talking about the same thing :)
As I understand @aryszka (correct me if I'm wrong) you want to support the case where you have an application configured like this:
apiVersion: apps/v1
kind: Deployment
metadata:
name: myapp
labels:
application: myapp
spec:
selector:
matchLabels:
application: myapp
template:
metadata:
labels:
application: myapp
spec:
containers:
- name: myapp
image: myapp:latest
ports:
- containerPort: 443
---
apiVersion: v1
kind: Service
metadata:
labels:
application: "myapp"
name: "myapp"
spec:
selector:
application: "myapp"
type: ClusterIP
ports:
- port: 443
protocol: TCP
targetPort: 443
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: "myapp"
spec:
rules:
- host: "myapp.example.org"
http:
paths:
- backend:
serviceName: "myapp"
servicePort: 443
And the skipper endpoints should be configured to be https://10.2.1.1 or something like that?
The biggest challenge for supporting this IMO, is how do you handle certificates? you would need to tell skipper which CA to use and make sure the pods have the pod IP in their certificate. This could be simplified a bit by only using the service endpoint for HTTPS backends as you can share the predictable service name for all the pods. The downside is you would loose the load balancing feature of skipper.
@szuecs I think what you are talking about should not be done in skipper. It's an HTTP router by design, so doing routing based on TLS is out of scope IMO.
@mikkeloscar true but it is a very similar problem. In the end you have to do SNI anyways. How else you can choose the right certificate to the client for different hostnames? If we think it is a good idea than why should we limit to http? In the backend calls it’s easy as you wrote you have to have a dynamic ca that provisions a cert per pod ip (vault can do this) and you have to add the vault ca cert to skipper’s configured CAs, this could be easily done by the alpine image or a configmap that could probably overwrite the capath from the operating system or we use a flag.
@mikkeloscar correct, that's what i meant. I myself am not sure at the moment if we want this, just wanted to raise the topic for consideration/discussion.
Than the problem can be solved with an annotation to switch a backend call to https. The problem with ingress annotation is that it should belong to the backend or mixed http and https backends will not work.
Maybe better the service should have the annotation and hopefully the endpoint will also get the annotation propagated by the service, such that we know in the route creation that the backend should be http or https.
Another thing to be done is that we need to be able to read/update multiple certs,keys from files for example kubernetes secrets. This would enable the use case to terminate https in skipper and rencrypt to backend with https if the backend service has a cert matching their ip.
FYI: https://github.com/zalando/skipper/pull/1606 makes it possible to use https backend protocol also in routegroups similar to how it works in ingress with the same annotation zalando.org/skipper-backend-protocol.