After installing pixie index-server and vzmgr-server are in crashloopbackoff

[trevorlinton]

Describe the bug After installing pixie the vzmgr-server pod and index-server pod both are in a crashloopbackoff state.

To Reproduce Steps to reproduce the behavior:

1. Install pixie on an AWS KOPS based service with the following customizations:
2. vzconn_service.yaml - add AWS internal load balancer annotations to service
3. proxy_service.yaml - add AWS internal load balancer annotations to service
4. Delete the kubectl delete secrets/cloud-proxy-tls-certs -n plc secret and replace it with a real TLS certificate from letsencrypt.

Expected behavior All pods functioning and running

Logs Please attach the logs by running the following command:

% kubectl logs -n plc indexer-server-59fbcbb64f-2rkrr 
time="2021-07-26T15:26:44Z" level=info msg="Starting service" func=px.dev/pixie/src/shared/services.SetupService file="src/shared/services/service_flags.go:71" service=indexer-service version=0.0.0-dev+Modified.0000000.19700101000000.0
time="2021-07-26T15:26:44Z" level=info msg="Loading HTTP TLS certs" func=px.dev/pixie/src/shared/services.GetGRPCClientDialOpts file="src/shared/services/service_flags.go:156" tlsCA=/certs/ca.crt tlsCertFile=/certs/client.crt tlsKeyFile=/certs/client.key
time="2021-07-26T15:26:44Z" level=info msg="[core] parsed scheme: \"kubernetes\"" func=google.golang.org/grpc/internal/grpclog.InfoDepth file="external/org_golang_google_grpc/internal/grpclog/grpclog.go:37" system=system

% kubectl logs -vzmgr-server-6cdcf68fd9-c2twm -n plc
time="2021-07-26T15:20:53Z" level=info msg="Starting service" func=px.dev/pixie/src/shared/services.SetupService file="src/shared/services/service_flags.go:71" service=vzmgr-service version=0.0.0-dev+Modified.0000000.19700101000000.0
time="2021-07-26T15:20:53Z" level=info msg="Loading HTTP TLS certs" func=px.dev/pixie/src/shared/services.GetGRPCClientDialOpts file="src/shared/services/service_flags.go:156" tlsCA=/certs/ca.crt tlsCertFile=/certs/client.crt tlsKeyFile=/certs/client.key
time="2021-07-26T15:20:53Z" level=info msg="[core] parsed scheme: \"kubernetes\"" func=google.golang.org/grpc/internal/grpclog.InfoDepth file="external/org_golang_google_grpc/internal/grpclog/grpclog.go:37" system=system
time="2021-07-26T15:20:53Z" level=info msg="Setting up database" func=px.dev/pixie/src/shared/services/pg.MustCreateDefaultPostgresDB file="src/shared/services/pg/pg.go:65" dbURI="postgres://pl:[email protected]:5432/pl?sslmode=disable"
time="2021-07-26T15:20:53Z" level=info msg="[core] ccResolverWrapper: sending new addresses to cc: [{100.108.65.251:51900  <nil> 0 <nil>}]" func=google.golang.org/grpc/internal/grpclog.InfoDepth file="external/org_golang_google_grpc/internal/grpclog/grpclog.go:37" system=system
time="2021-07-26T15:20:53Z" level=info msg="[core] ClientConn switching balancer to \"round_robin\"" func=google.golang.org/grpc/internal/grpclog.InfoDepth file="external/org_golang_google_grpc/internal/grpclog/grpclog.go:37" system=system
time="2021-07-26T15:20:53Z" level=info msg="[core] Channel switches to new LB policy \"round_robin\"" func=google.golang.org/grpc/internal/grpclog.InfoDepth file="external/org_golang_google_grpc/internal/grpclog/grpclog.go:37" system=system
time="2021-07-26T15:20:53Z" level=info msg="[core] Subchannel Connectivity change to CONNECTING" func=google.golang.org/grpc/internal/grpclog.InfoDepth file="external/org_golang_google_grpc/internal/grpclog/grpclog.go:37" system=system
time="2021-07-26T15:20:53Z" level=info msg="[core] Subchannel picks a new address \"100.108.65.251:51900\" to connect" func=google.golang.org/grpc/internal/grpclog.InfoDepth file="external/org_golang_google_grpc/internal/grpclog/grpclog.go:37" system=system
time="2021-07-26T15:20:53Z" level=info msg="[core] Channel Connectivity change to CONNECTING" func=google.golang.org/grpc/internal/grpclog.InfoDepth file="external/org_golang_google_grpc/internal/grpclog/grpclog.go:37" system=system
time="2021-07-26T15:20:53Z" level=info msg="[core] Subchannel Connectivity change to READY" func=google.golang.org/grpc/internal/grpclog.InfoDepth file="external/org_golang_google_grpc/internal/grpclog/grpclog.go:37" system=system
time="2021-07-26T15:20:53Z" level=info msg="[roundrobin] roundrobinPicker: newPicker called with info: {map[0xc000410660:{{100.108.65.251:51900  <nil> 0 <nil>}}]}" func=google.golang.org/grpc/internal/grpclog.InfoDepth file="external/org_golang_google_grpc/internal/grpclog/grpclog.go:37" system=system
time="2021-07-26T15:20:53Z" level=info msg="[core] Channel Connectivity change to READY" func=google.golang.org/grpc/internal/grpclog.InfoDepth file="external/org_golang_google_grpc/internal/grpclog/grpclog.go:37" system=system
time="2021-07-26T15:20:53Z" level=info msg="Connected to Postgres" func=px.dev/pixie/src/shared/services/pg.MustConnectDefaultPostgresDB file="src/shared/services/pg/pg.go:87"
time="2021-07-26T15:20:53Z" level=info msg="Connected to NATS" func=px.dev/pixie/src/shared/services/msgbus.MustConnectNATS file="src/shared/services/msgbus/nats.go:59" URL=pl-nats
time="2021-07-26T15:20:53Z" level=info msg="Connected to STAN" func=px.dev/pixie/src/shared/services/msgbus.MustConnectSTAN file="src/shared/services/msgbus/stan.go:42" ClusterID=pl-stan
time="2021-07-26T15:20:53Z" level=info msg="Loading HTTP TLS certs" func=px.dev/pixie/src/shared/services.GetGRPCClientDialOpts file="src/shared/services/service_flags.go:156" tlsCA=/certs/ca.crt tlsCertFile=/certs/client.crt tlsKeyFile=/certs/client.key
time="2021-07-26T15:20:53Z" level=info msg="[core] parsed scheme: \"kubernetes\"" func=google.golang.org/grpc/internal/grpclog.InfoDepth file="external/org_golang_google_grpc/internal/grpclog/grpclog.go:37" system=system
time="2021-07-26T15:20:53Z" level=info msg="[core] ccResolverWrapper: sending new addresses to cc: [{100.96.147.176:50750  <nil> 0 <nil>}]" func=google.golang.org/grpc/internal/grpclog.InfoDepth file="external/org_golang_google_grpc/internal/grpclog/grpclog.go:37" system=system
time="2021-07-26T15:20:53Z" level=info msg="[core] ClientConn switching balancer to \"round_robin\"" func=google.golang.org/grpc/internal/grpclog.InfoDepth file="external/org_golang_google_grpc/internal/grpclog/grpclog.go:37" system=system
time="2021-07-26T15:20:53Z" level=info msg="[core] Channel switches to new LB policy \"round_robin\"" func=google.golang.org/grpc/internal/grpclog.InfoDepth file="external/org_golang_google_grpc/internal/grpclog/grpclog.go:37" system=system
time="2021-07-26T15:20:53Z" level=info msg="[core] Subchannel Connectivity change to CONNECTING" func=google.golang.org/grpc/internal/grpclog.InfoDepth file="external/org_golang_google_grpc/internal/grpclog/grpclog.go:37" system=system
time="2021-07-26T15:20:53Z" level=info msg="[core] Channel Connectivity change to CONNECTING" func=google.golang.org/grpc/internal/grpclog.InfoDepth file="external/org_golang_google_grpc/internal/grpclog/grpclog.go:37" system=system
time="2021-07-26T15:20:53Z" level=info msg="[core] Subchannel picks a new address \"100.96.147.176:50750\" to connect" func=google.golang.org/grpc/internal/grpclog.InfoDepth file="external/org_golang_google_grpc/internal/grpclog/grpclog.go:37" system=system
time="2021-07-26T15:20:53Z" level=info msg="[core] Subchannel Connectivity change to READY" func=google.golang.org/grpc/internal/grpclog.InfoDepth file="external/org_golang_google_grpc/internal/grpclog/grpclog.go:37" system=system
time="2021-07-26T15:20:53Z" level=info msg="[roundrobin] roundrobinPicker: newPicker called with info: {map[0xc000032800:{{100.96.147.176:50750  <nil> 0 <nil>}}]}" func=google.golang.org/grpc/internal/grpclog.InfoDepth file="external/org_golang_google_grpc/internal/grpclog/grpclog.go:37" system=system
time="2021-07-26T15:20:53Z" level=info msg="[core] Channel Connectivity change to READY" func=google.golang.org/grpc/internal/grpclog.InfoDepth file="external/org_golang_google_grpc/internal/grpclog/grpclog.go:37" system=system
time="2021-07-26T15:20:53Z" level=fatal msg="Could not start vizier updater" func=main.main file="src/cloud/vzmgr/vzmgr_server.go:135" error="Could not find Vizier artifact"

App information (please complete the following information):

• Pixie version - gcr.io/pixie-oss/pixie-prod/cloud/api_server_image@sha256:7614a80d9aedaae80e9ffaaeacfe973cc1a56be5ef85d94442395391bc4e902f
• K8s cluster version - 1.18

Additional context

• Added internal load balancer annotations to service to prevent public load balancers
• Added real certificates rather than using auto-generated ones.