pixie
pixie copied to clipboard
pixie-io
provisioning failed because vizier with specified UID is already active
Describe the bug An aborted install attempt has left installation impossible. A clear and concise description of what the bug is.
To Reproduce Clean up as best as I can
px delete
kubectl delete crd catalogsources.operators.coreos.com clusterserviceversions.operators.coreos.com installplans.operators.coreos.com operatorconditions.operators.coreos.com operatorgroups.operators.coreos.com operators.operators.coreos.com subscriptions.operators.coreos.com viziers.px.dev
k delete ns olm
Reinstall
➜ px deploy
Pixie CLI
Running Cluster Checks:
✔ Kernel version > 4.14.0
✔ Cluster type is supported
✔ K8s version > 1.16.0
✔ Kubectl > 1.10.0 is present
✔ User can create namespace
✕ Cluster type is in list of known supported types ERR: Cluster type is not in list of known supported cluster types. Please see: https://docs.px.dev/installing-pixie/requirements/
Some cluster checks failed. Pixie may not work properly on your cluster. Continue with deploy? (y/n) [y] :
Installing Vizier version: 0.12.2
Generating YAMLs for Pixie
Deploying Pixie to the following cluster: tpkube
Is the cluster correct? (y/n) [y] :
Found 36 nodes
✔ Installing OLM CRDs
✔ Deploying OLM
✔ Deploying Pixie OLM Namespace
✔ Installing Vizier CRD
✔ Deploying OLM Catalog
✔ Deploying OLM Subscription
✔ Creating namespace
✔ Deploying Vizier
⠸ Waiting for Cloud Connector to come online
FATA[0396] Timed out waiting for cluster ID assignment
It took the operator a long time to get ready. Eventually, the pods were all running in px-operator. When the operator is ready it tries to initialize pixie in the pl namespace. This fails. The pod status is
➜ k get po
NAME READY STATUS RESTARTS AGE
kelvin-699b4c9b9b-blkdd 0/1 Init:0/2 0 4m19s
pl-nats-0 1/1 Running 0 4m34s
vizier-cloud-connector-6f9fc774d4-mvxgc 0/1 CrashLoopBackOff 5 (30s ago) 4m19s
vizier-metadata-0 1/1 Running 2 (3m46s ago) 4m19s
vizier-pem-26gz9 0/1 Init:0/1 0 4m24s
vizier-pem-2ncsl 0/1 Init:0/1 0 4m24s
vizier-pem-4s2mw 0/1 Init:0/1 0 4m24s
...
vizier-pem-wfq2h 0/1 Init:0/1 0 4m24s
vizier-pem-wmmwb 0/1 Pending 0 4m24s
vizier-pem-zx8vh 0/1 Init:0/1 0 4m24s
vizier-query-broker-55967b7dc4-q674f 0/1 Init:0/2 0 4m19s
The last log line before vizier-cloud-connector restarts is
➜ k logs -p vizier-cloud-connector-6f9fc774d4-mvxgc|tail -n1
time="2022-09-13T21:21:45Z" level=fatal msg="Failed to register vizier deployment. If deploying via Helm or Manifest, please check your deployment key. Otherwise, ensure you have deployed with an authorized account." error="rpc error: code = ResourceExhausted desc = provisioning failed because vizier with specified UID is already active"
Expected behavior
I expect px deploy to complete successfully.
Logs Please attach the logs by running the following command:
❯ px collect-logs
Pixie CLI
WARN[0001] Failed to log pod: kelvin-699b4c9b9b-blkdd error="container \"app\" in pod \"kelvin-699b4c9b9b-blkdd\" is waiting to start: PodInitializing"
WARN[0001] Failed to log pod: vizier-pem-26gz9 error="container \"pem\" in pod \"vizier-pem-26gz9\" is waiting to start: PodInitializing"
WARN[0001] Failed to log pod: vizier-pem-2ncsl error="container \"pem\" in pod \"vizier-pem-2ncsl\" is waiting to start: PodInitializing"
WARN[0001] Failed to log pod: vizier-pem-4s2mw error="container \"pem\" in pod \"vizier-pem-4s2mw\" is waiting to start: PodInitializing"
WARN[0002] Failed to log pod: vizier-pem-59hss error="container \"pem\" in pod \"vizier-pem-59hss\" is waiting to start: PodInitializing"
WARN[0002] Failed to log pod: vizier-pem-5hwrm error="container \"pem\" in pod \"vizier-pem-5hwrm\" is waiting to start: PodInitializing"
WARN[0003] Failed to log pod: vizier-pem-622vz error="container \"pem\" in pod \"vizier-pem-622vz\" is waiting to start: PodInitializing"
WARN[0003] Failed to log pod: vizier-pem-6vjc6 error="container \"pem\" in pod \"vizier-pem-6vjc6\" is waiting to start: PodInitializing"
WARN[0003] Failed to log pod: vizier-pem-6xgqn error="container \"pem\" in pod \"vizier-pem-6xgqn\" is waiting to start: PodInitializing"
WARN[0004] Failed to log pod: vizier-pem-74k54 error="container \"pem\" in pod \"vizier-pem-74k54\" is waiting to start: PodInitializing"
WARN[0004] Failed to log pod: vizier-pem-7fb7m error="container \"pem\" in pod \"vizier-pem-7fb7m\" is waiting to start: PodInitializing"
WARN[0005] Failed to log pod: vizier-pem-7vq85 error="container \"pem\" in pod \"vizier-pem-7vq85\" is waiting to start: PodInitializing"
WARN[0005] Failed to log pod: vizier-pem-7xnsh error="container \"pem\" in pod \"vizier-pem-7xnsh\" is waiting to start: PodInitializing"
WARN[0005] Failed to log pod: vizier-pem-8mfjk error="container \"pem\" in pod \"vizier-pem-8mfjk\" is waiting to start: PodInitializing"
WARN[0006] Failed to log pod: vizier-pem-9596k error="container \"pem\" in pod \"vizier-pem-9596k\" is waiting to start: PodInitializing"
WARN[0006] Failed to log pod: vizier-pem-9rzd6 error="container \"pem\" in pod \"vizier-pem-9rzd6\" is waiting to start: PodInitializing"
WARN[0007] Failed to log pod: vizier-pem-c2wwz error="container \"pem\" in pod \"vizier-pem-c2wwz\" is waiting to start: PodInitializing"
WARN[0007] Failed to log pod: vizier-pem-dlt95 error="container \"pem\" in pod \"vizier-pem-dlt95\" is waiting to start: PodInitializing"
WARN[0007] Failed to log pod: vizier-pem-f247b error="container \"pem\" in pod \"vizier-pem-f247b\" is waiting to start: PodInitializing"
WARN[0008] Failed to log pod: vizier-pem-fcfhd error="container \"pem\" in pod \"vizier-pem-fcfhd\" is waiting to start: PodInitializing"
WARN[0008] Failed to log pod: vizier-pem-fchg8 error="container \"pem\" in pod \"vizier-pem-fchg8\" is waiting to start: PodInitializing"
WARN[0009] Failed to log pod: vizier-pem-gw52n error="container \"pem\" in pod \"vizier-pem-gw52n\" is waiting to start: PodInitializing"
WARN[0009] Failed to log pod: vizier-pem-h6hbd error="container \"pem\" in pod \"vizier-pem-h6hbd\" is waiting to start: PodInitializing"
WARN[0009] Failed to log pod: vizier-pem-hk982 error="container \"pem\" in pod \"vizier-pem-hk982\" is waiting to start: PodInitializing"
WARN[0010] Failed to log pod: vizier-pem-kvn9l error="container \"pem\" in pod \"vizier-pem-kvn9l\" is waiting to start: PodInitializing"
WARN[0010] Failed to log pod: vizier-pem-m54fw error="container \"pem\" in pod \"vizier-pem-m54fw\" is waiting to start: PodInitializing"
WARN[0011] Failed to log pod: vizier-pem-mbtdd error="container \"pem\" in pod \"vizier-pem-mbtdd\" is waiting to start: PodInitializing"
WARN[0011] Failed to log pod: vizier-pem-mfh89 error="container \"pem\" in pod \"vizier-pem-mfh89\" is waiting to start: PodInitializing"
WARN[0011] Failed to log pod: vizier-pem-mng6b error="container \"pem\" in pod \"vizier-pem-mng6b\" is waiting to start: PodInitializing"
WARN[0012] Failed to log pod: vizier-pem-nvmfz error="container \"pem\" in pod \"vizier-pem-nvmfz\" is waiting to start: PodInitializing"
WARN[0012] Failed to log pod: vizier-pem-qgflt error="container \"pem\" in pod \"vizier-pem-qgflt\" is waiting to start: PodInitializing"
WARN[0013] Failed to log pod: vizier-pem-qqq6w error="container \"pem\" in pod \"vizier-pem-qqq6w\" is waiting to start: PodInitializing"
WARN[0013] Failed to log pod: vizier-pem-rqrl6 error="container \"pem\" in pod \"vizier-pem-rqrl6\" is waiting to start: PodInitializing"
WARN[0013] Failed to log pod: vizier-pem-rvlfh error="container \"pem\" in pod \"vizier-pem-rvlfh\" is waiting to start: PodInitializing"
WARN[0014] Failed to log pod: vizier-pem-s4xtr error="container \"pem\" in pod \"vizier-pem-s4xtr\" is waiting to start: PodInitializing"
WARN[0014] Failed to log pod: vizier-pem-sps4m error="container \"pem\" in pod \"vizier-pem-sps4m\" is waiting to start: PodInitializing"
WARN[0015] Failed to log pod: vizier-pem-t4c9x error="container \"pem\" in pod \"vizier-pem-t4c9x\" is waiting to start: PodInitializing"
WARN[0015] Failed to log pod: vizier-pem-t642c error="container \"pem\" in pod \"vizier-pem-t642c\" is waiting to start: PodInitializing"
WARN[0015] Failed to log pod: vizier-pem-thrh5 error="container \"pem\" in pod \"vizier-pem-thrh5\" is waiting to start: PodInitializing"
WARN[0016] Failed to log pod: vizier-pem-vdpsb error="container \"pem\" in pod \"vizier-pem-vdpsb\" is waiting to start: PodInitializing"
WARN[0016] Failed to log pod: vizier-pem-wfq2h error="container \"pem\" in pod \"vizier-pem-wfq2h\" is waiting to start: PodInitializing"
WARN[0017] Failed to log pod: vizier-pem-wmmwb error="container \"pem\" in pod \"vizier-pem-wmmwb\" is waiting to start: PodInitializing"
WARN[0017] Failed to log pod: vizier-pem-zx8vh error="container \"pem\" in pod \"vizier-pem-zx8vh\" is waiting to start: PodInitializing"
WARN[0017] Failed to log pod: vizier-query-broker-55967b7dc4-q674f error="container \"app\" in pod \"vizier-query-broker-55967b7dc4-q674f\" is waiting to start: PodInitializing"
Logs written to pixie_logs_20220913152349.zip
App information (please complete the following information):
- Pixie version - 0.7.17+Distribution.1f8cdfb.20220815151559.3.jenkins
- K8s cluster version - v1.23.10
- Node Kernel version - 5.15.0-46-generic
- Browser version - NA
