ControlPlane TLS not possible if SCv2 controller is clusterwide

[Kolajik]

Describe the bug

TLS connection in Control Plane is not possible when using clusterwide controller in Seldon Core v2. The user would have to deploy the same set of Certificates/Secrets/Issuers in all namespaces where SCv2 runtime is deployed.

These secrets/certificates would have to be the same in all of the namespaces:

• seldon-controlplane-server
• seldon-controlplane-client

as they are needed in Deployment of SCv2 controller as well as SeldonConfig (scheduler, pipeline-gateway and model-gateway, hodometer) and with ServerConfig.

To reproduce

1. Install clusterwide SCv2 into your cluster in namespace ns-controller a. helm install seldon-v2 k8s/helm-charts/seldon-core-v2-setup/ -n ns-controller --set security.envoy.protocol=SSL --set security.controlplane.protocol=SSL --set controller.clusterwide=true
2. Install SCv2 certs in namespace ns-controller a. helm install seldon-v2-certs k8s/helm-charts/seldon-core-v2-certs/ -n ns-controller
3. Deploy SCv2 runtime in namespace ns-runtime a. helm install seldon-v2-runtime seldon-charts/seldon-core-v2-runtime --namespace ns-runtime

Expected behaviour

SCv2 runtime will deploy as expected in ns-runtime namespace.

Actual behaviour

Most of the components in ns-runtime namespace will fail to run because they won't be able to find all necessary secrets. Envoy proxy won't be able to load the certificates.