Allow members to advertise using IPs

[jamiehannaford]

Is it possible for etcd members to advertise themselves via their IPs rather than DNS hostnames? Since they're on hostNetwork, I assume they will advertise their host IP.

The reason this is needed is that we're trying to integrate etcd-operator into kubeadm, but we do not have a working DNS server on bootstrap (because there is no CNI yet). The user sets up CNI after the control plane is working. We also want kube-apiserver on each node to hit its own etcd pod via localhost, so we really have no need to use services in our use case.

@hongchaodeng @xiang90 WDYT?

/cc @luxas

[jamiehannaford]

I'm suggesting making this configurable, with a new field in EtcdCluster called AdvertiseWithHostnames=true or something.

[jamiehannaford]

Looks like this was already considered in https://github.com/coreos/etcd-operator/pull/122.

It seems that using the operator without a working DNS server is possible, just requires a bit of hacking because the host IP is only known after the pod is scheduled. @xiang90 what would be needed to make that work?

I'm wondering if this line could be updated to conditionally use --listen-peer-urls=$(POD_IP), and then add this to the pod's env:

- name: POD_IP
  valueFrom:
    fieldRef:
      fieldPath: status.podIP

[hongchaodeng]

Hi @jamiehannaford

It's possible to use Pod IP. Actually we initially use pod IP in self hosted etcd to avoid depending on kubedns. The issue arises when adding TLS certs because IPs are dynamic.

With using of k8s csr endpoint we can switch to use pod IP: https://github.com/coreos/etcd-operator/issues/1465

[jamiehannaford]

@hongchaodeng Oh, cool! I think we use CSRs in kubeadm too with auto-approve so that might work really well. So I'm imagining how this would work...

1. We'd need to update addOneSelfHostedMember to find the node with no etcd pod on it. Then pass that node's IP into NewSelfHostedEtcdPod which can generate the CSR and make the pod command.

2. Add new fields to the spec struct for dynamic TLS. I'm thinking the RBAC user or group name that the operator will use to send the CSRs.

3. Use status.podIP if the user wants to use pod IPs instead of DNS.

Did I miss anything?

[hongchaodeng]

We'd need to update addOneSelfHostedMember to find the node with no etcd pod on it.

Scheduler does this for us via anti-affinity.

Then pass that node's IP into NewSelfHostedEtcdPod which can generate the CSR and make the pod command.

Correct. We will use init-container.

Use status.podIP if the user wants to use pod IPs instead of DNS.

DNS would be get ridden of. Instead of adding member in etcd operator, there will be an init container that does the add member command.

[jamiehannaford]

@hongchaodeng

Correct. We will use init-container.

SGTM. Will the init container also create the CSR, or will the operator handle that?

DNS would be get ridden of. Instead of adding member in etcd operator, there will be an init container that does the add member command.

So after init containers are added, only Node IPs are supported? If so, we will also need to add a new field to Member, right? Because m.Name is currently used to find its address

Also, do you plan to work on this soon? We'd like to have operator in for 1.9. I can help out to add this, but want to synchronise with you first.

[hongchaodeng]

Will the init container also create the CSR, or will the operator handle that?

The init container. Pod IP can't be known until a Pod is created.

So after init containers are added, only Node IPs are supported? If so, we will also need to add a new field to Member, right? Because m.Name is currently used to find its address

We can think of this implementation later. We need to implement the dynamic TLS feature first.

[jamiehannaford]

@hongchaodeng Based on my comment, I don't think auto-approvals for CSRs can work for now. Instead, is it possible to get host IPs working with static TLS for 1.9? It will work if the cluster has its own CA right?

[hongchaodeng]

Hi @jamiehannaford

Using host IPs doesn't satisfy our use case because our nodes on which etcd pods are running have dynamic IPs.

Also see https://github.com/coreos/etcd-operator/issues/1465#issuecomment-342322365

[jamiehannaford]

@hongchaodeng kubeadm definitely has a use case of using host IPs. Can we implement this as a first step? How do you expect to solve dynamic host IPs?

[hongchaodeng]

How do you expect to solve dynamic host IPs?

Each etcd pod generates CSR and gets a cert for itself.

[jamiehannaford]

@hongchaodeng Why a CSR and not generate certs from a custom CA file for the etcd cluster? It solves the exact same problem and is a lot simpler

[hongchaodeng]

Why a CSR and not generate certs from a custom CA file for the etcd cluster? It solves the exact same problem and is a lot simpler

Sorry. Dismissed this in email flood. I have no opinion on which is better. Any solution that solves user issues.

[hongchaodeng]

Hi @jamiehannaford Sorry to revert https://github.com/coreos/etcd-operator/pull/1662. Could you add a flag/field in spec to switch that?

[zmlcc]

Hi @hongchaodeng Why to revert #1662 ? Any detailed description？

[jamiehannaford]

@zmlcc Because it breaks compatibility and we need a new spec field. I've been on vacation so didn't have time to do this. I will try to get round to it soon.

[yzargari]

Is there any plan on developing the option to advertise the client IP? Has anyone found a workaround? I'm toying with the idea of exporting the DNS entries (using external-dns) to a VPC private DNS zone (r53)...