Whitelist domains on top of VERIFY_CERTIFICATES validation

[jackjia-ibm]

Is your feature request related to a problem? Please describe.

Comparing to NONSTRICT_VERIFY_CERTIFICATES, VERIFY_CERTIFICATES will also validate domain name against the Common Name and Subject Alt Name defined in the certificate. We do suggest the user to enable VERIFY_CERTIFICATES for security purpose.

In Kubernetes deployment, in order to validate the components connection, domains like *.zowe.pod.cluster.local and *.zowe.svc.cluster.local will be used. For example, APICATALOG may register itself from 10-1-2-3.zowe.pod.cluster.local to DISCOVERY, and use gateway-service.zowe.svc.cluster.local to access gateway internally.

Adding *.zowe.pod.cluster.local and *.zowe.svc.cluster.local to Zowe generated certificate is complicated but doable, but it will be very hard to convince customers who use external certificate to request certificates with those wildcard domains.

Describe the solution you'd like

To make certificate setup easier with Kubernetes deployment, we would like to have ability to white list domains to bypass this check. And these domains should be customizable for these reasons:

• zowe inside the domain is Kubernetes namespace, and it's customizable by each customers.
• cluster.local is the cluster name. It's default value for most of clusters but it's customizable.

The new whitelist configuration can be empty for default setting. It will be enabled only for APIML running in Kubernetes. We can document how to enable this and add it to migration script.

Additional context

Please also evaluate if there are security concerns related to this change. Thanks.

[jackjia-ibm]

After discussion, we will want to think more on this topic and this won't be blocker for Kubernetes MVP.

[jandadav]

On a technical level, we could do this. The feature could be indeed enabled as you state through configuration that can be disabled by default.

We can do this in our services as they might be called directly (GW + DS) but perhaps this has to be done in additional services that are supposed to trust calls coming from Gateway equipped with such certificate? Not all services might need to do that though, but for example consumers of our x509 authentication scheme might want to. Another measure that can be considered is to decouple Gateway server certificate from the Gateway's proxy client certificate to cater for Kubernetes needs. They don't necessarily have to be the same.

Regarding security, the vulnerability I see is if somebody can manipulate the host resolution and get a call with the host that seems like the Kubernetes host, then the host validation mechanism might be bypassed. But we still have the signing mechanism which should be a good guarantee still. In opposite case we are dealing with leaked CA which is a bigger problem than hostname verification imo.

@jackjia-ibm Let us know how you intend to proceed! I will leave the issue in clarification for the time being

[jackjia-ibm]

Thanks David. I agree we can leave this item open for now until people complains about the setup. Currently our thoughts/solution are:

• if the user is using Zowe generated CA and certificate, we will generate a new certificate with *.zowe.pod.cluster.local, *.discovery-service.zowe.svc.cluster.local and *.gateway-service.zowe.svc.cluster.local into SAN. This work has been done.
• if the user is using external certificate, we just ask the user to get an external certificate with those domains in SAN. The original reason for this request is I worry user may not be able to get a certificate like this from their IT department.

My understanding is this works same as whitelisting on Gateway, basically just trust those domains from certificate level. So think even if we implement this, it doesn't increase the risk. We may need this feature if the user is in this situation:

• don't want to turn off strict VERIFY_CERTIFICATES
• cannot get external certificate with those domains in SAN
• don't want to use Zowe generated certificate or CA, even we configure it only used within the cluster and use external certificate for gateway server certificate.

So far I think neither of the solution will fully solve your concerns of manipulate the host resolution and get a call with the host that seems like the Kubernetes host and leaked CA. I hope these concerns will rely on the Kubernetes admin to keep the cluster secure:

• the cluster shouldn't take/accept any external connections other than the services/ports we defined to expose (currently only gateway port as must, and discovery port as optional)
• the cluster should be secure and no rogue pod will be started and manipulate dns server inside the cluster.
• the cluster should keep Secret safe, that's where we store CA, certificates and private keys.

Hope the user agree these cluster admin works are out of scope of Zowe setup.

[jandadav]

As per our discussion from 10/5/2021 parking this issue, to when we see demand from customers for this feature.