pixie
pixie copied to clipboard
pixie-io
Setup Pixie locally with Colima fails to initialise ElasticSearch
Hello team, First, I'd like to say that I'm very excited to try out Pixie
I was trying to install locally on a local Kubernetes cluster on top of Colima, is it supported? I was not sure but I tried anyway as this is how the local development happens at my company
I'm running on a MacOS M3 Max with Sonoma 14.4
Colima version
colima version 0.6.8
git commit: 9b0809d0ed9ad3ff1e57c405f27324e6298ca04f
runtime: docker
arch: aarch64
client: v25.0.0
server: v24.0.7
Then I have the first issue on the Install Pixie Cloud section
A few pods in CrashLoopBackoff
postgres-56f89f7d58-vlmns 1/1 Running 1 (49m ago) 75m
artifact-tracker-server-76ddc4d6bc-5d4dp 1/1 Running 1 (49m ago) 75m
config-manager-server-6d4767f49b-w8ddd 1/1 Running 1 (49m ago) 75m
hydra-7c757587bf-pxrh8 2/2 Running 2 (49m ago) 75m
profile-server-7bdbb6b487-dmf96 1/1 Running 2 (49m ago) 75m
cron-script-server-6545944d64-tmgjp 1/1 Running 2 (49m ago) 75m
plugin-server-6fc5474d7b-kr2rw 1/1 Running 2 (49m ago) 75m
auth-server-54cd5cd974-rptf4 1/1 Running 2 (49m ago) 75m
kratos-5fd6cd6c7d-ltbmr 2/2 Running 2 (49m ago) 75m
metrics-server-6dc964b448-prsfm 1/1 Running 2 (49m ago) 75m
project-manager-server-5d798ff9f6-sm2m5 1/1 Running 2 (49m ago) 75m
pl-nats-2 1/1 Running 1 (49m ago) 75m
pl-nats-1 1/1 Running 1 (49m ago) 75m
pl-nats-0 1/1 Running 1 (49m ago) 75m
vzmgr-server-6db898b648-dj8zl 1/1 Running 3 (48m ago) 75m
vzconn-server-6499968767-4zln8 1/1 Running 3 (48m ago) 75m
scriptmgr-server-5b468f58ff-xlrfk 1/1 Running 3 (48m ago) 75m
pl-elastic-es-data-0 0/1 CrashLoopBackOff 23 (2m35s ago) 75m
pl-elastic-es-master-0 0/1 CrashLoopBackOff 11 (2m9s ago) 33m
indexer-server-58f7846bb7-5hzj9 0/1 CrashLoopBackOff 25 (97s ago) 75m
api-server-5ccbc5b66c-gs78n 0/1 CrashLoopBackOff 26 (50s ago) 75m
cloud-proxy-7897b497cb-d6mdk 1/2 CrashLoopBackOff 32 (33s ago) 68m
pl-elastic-es-master-1 0/1 Running 12 (5m6s ago) 36m
plugin-db-updater-job-bd6mf 0/1 PodInitializing 0 2s
It seems to be a cascade issue due to the elastic search pods crashing Logs from the elastic search:
sysctl vm.max_map_count = 262144
Stream closed EOF for plc/pl-elastic-es-master-0 (sysctl)
elasticsearch Exception in thread "main" java.io.IOException: Cannot run program "/usr/share/elasticsearch/jdk/bin/java": error=0, Failed to exec spawn helper.
elastic-internal-init-filesystem Starting init script
elastic-internal-init-filesystem Linking /mnt/elastic-internal/xpack-file-realm/users to /usr/share/elasticsearch/config/users
elastic-internal-init-filesystem Linking /mnt/elastic-internal/xpack-file-realm/roles.yml to /usr/share/elasticsearch/config/roles.yml
elastic-internal-init-filesystem Linking /mnt/elastic-internal/xpack-file-realm/users_roles to /usr/share/elasticsearch/config/users_roles
elastic-internal-init-filesystem Linking /mnt/elastic-internal/elasticsearch-config/elasticsearch.yml to /usr/share/elasticsearch/config/elasticsearch.yml
elasticsearch at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1128)
elasticsearch at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1071)
elasticsearch at org.elasticsearch.tools.launchers.JvmErgonomics.flagsFinal(JvmErgonomics.java:111)
elasticsearch at org.elasticsearch.tools.launchers.JvmErgonomics.finalJvmOptions(JvmErgonomics.java:88)
elasticsearch at org.elasticsearch.tools.launchers.JvmErgonomics.choose(JvmErgonomics.java:59)
elasticsearch at org.elasticsearch.tools.launchers.JvmOptionsParser.main(JvmOptionsParser.java:95)
elasticsearch Caused by: java.io.IOException: error=0, Failed to exec spawn helper.
elasticsearch at java.base/java.lang.ProcessImpl.forkAndExec(Native Method)
elasticsearch at java.base/java.lang.ProcessImpl.<init>(ProcessImpl.java:319)
elasticsearch at java.base/java.lang.ProcessImpl.start(ProcessImpl.java:250)
elasticsearch at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1107)
elasticsearch ... 5 more
elastic-internal-init-filesystem Linking /mnt/elastic-internal/unicast-hosts/unicast_hosts.txt to /usr/share/elasticsearch/config/unicast_hosts.txt
elastic-internal-init-filesystem File linking duration: 1 sec.
elastic-internal-init-filesystem Copying /usr/share/elasticsearch/config/* to /mnt/elastic-internal/elasticsearch-config-local/
elastic-internal-init-filesystem '/usr/share/elasticsearch/config/elasticsearch.yml' -> '/mnt/elastic-internal/elasticsearch-config-local/elasticsearch.yml'
elastic-internal-init-filesystem '/usr/share/elasticsearch/config/http-certs' -> '/mnt/elastic-internal/elasticsearch-config-local/http-certs'
elastic-internal-init-filesystem '/usr/share/elasticsearch/config/http-certs/ca.crt' -> '/mnt/elastic-internal/elasticsearch-config-local/http-certs/ca.crt'
elastic-internal-init-filesystem '/usr/share/elasticsearch/config/http-certs/tls.key' -> '/mnt/elastic-internal/elasticsearch-config-local/http-certs/tls.key'
elastic-internal-init-filesystem '/usr/share/elasticsearch/config/http-certs/tls.crt' -> '/mnt/elastic-internal/elasticsearch-config-local/http-certs/tls.crt'
elastic-internal-init-filesystem '/usr/share/elasticsearch/config/http-certs/..data' -> '/mnt/elastic-internal/elasticsearch-config-local/http-certs/..data'
elastic-internal-init-filesystem '/usr/share/elasticsearch/config/http-certs/..2024_03_15_18_24_41.3325295662' -> '/mnt/elastic-internal/elasticsearch-config-local/http-certs/..2024_03_15_18_24_41.3325295662'
elastic-internal-init-filesystem '/usr/share/elasticsearch/config/http-certs/..2024_03_15_18_24_41.3325295662/ca.crt' -> '/mnt/elastic-internal/elasticsearch-config-local/http-certs/..2024_03_15_18_24_41.3325295662/ca.crt'
elastic-internal-init-filesystem '/usr/share/elasticsearch/config/http-certs/..2024_03_15_18_24_41.3325295662/tls.key' -> '/mnt/elastic-internal/elasticsearch-config-local/http-certs/..2024_03_15_18_24_41.3325295662/tls.key'
elastic-internal-init-filesystem '/usr/share/elasticsearch/config/http-certs/..2024_03_15_18_24_41.3325295662/tls.crt' -> '/mnt/elastic-internal/elasticsearch-config-local/http-certs/..2024_03_15_18_24_41.3325295662/tls.crt'
elastic-internal-init-filesystem '/usr/share/elasticsearch/config/jvm.options' -> '/mnt/elastic-internal/elasticsearch-config-local/jvm.options'
elastic-internal-init-filesystem '/usr/share/elasticsearch/config/log4j2.properties' -> '/mnt/elastic-internal/elasticsearch-config-local/log4j2.properties'
elastic-internal-init-filesystem '/usr/share/elasticsearch/config/role_mapping.yml' -> '/mnt/elastic-internal/elasticsearch-config-local/role_mapping.yml'
elastic-internal-init-filesystem '/usr/share/elasticsearch/config/roles.yml' -> '/mnt/elastic-internal/elasticsearch-config-local/roles.yml'
elastic-internal-init-filesystem '/usr/share/elasticsearch/config/transport-remote-certs' -> '/mnt/elastic-internal/elasticsearch-config-local/transport-remote-certs'
elastic-internal-init-filesystem '/usr/share/elasticsearch/config/transport-remote-certs/ca.crt' -> '/mnt/elastic-internal/elasticsearch-config-local/transport-remote-certs/ca.crt'
elastic-internal-init-filesystem '/usr/share/elasticsearch/config/transport-remote-certs/..data' -> '/mnt/elastic-internal/elasticsearch-config-local/transport-remote-certs/..data'
elastic-internal-init-filesystem '/usr/share/elasticsearch/config/transport-remote-certs/..2024_03_15_18_24_41.3095262462' -> '/mnt/elastic-internal/elasticsearch-config-local/transport-remote-certs/..2024_03_15_18_24_41.3095262462'
elastic-internal-init-filesystem '/usr/share/elasticsearch/config/transport-remote-certs/..2024_03_15_18_24_41.3095262462/ca.crt' -> '/mnt/elastic-internal/elasticsearch-config-local/transport-remote-certs/..2024_03_15_18_24_41.3095262462/ca.crt'
elastic-internal-init-filesystem '/usr/share/elasticsearch/config/unicast_hosts.txt' -> '/mnt/elastic-internal/elasticsearch-config-local/unicast_hosts.txt'
elastic-internal-init-filesystem '/usr/share/elasticsearch/config/users' -> '/mnt/elastic-internal/elasticsearch-config-local/users'
elastic-internal-init-filesystem '/usr/share/elasticsearch/config/users_roles' -> '/mnt/elastic-internal/elasticsearch-config-local/users_roles'
elastic-internal-init-filesystem Empty dir /usr/share/elasticsearch/plugins
elastic-internal-init-filesystem Copying /usr/share/elasticsearch/bin/* to /mnt/elastic-internal/elasticsearch-bin-local/
elastic-internal-init-filesystem '/usr/share/elasticsearch/bin/elasticsearch' -> '/mnt/elastic-internal/elasticsearch-bin-local/elasticsearch'
elastic-internal-init-filesystem '/usr/share/elasticsearch/bin/elasticsearch-certgen' -> '/mnt/elastic-internal/elasticsearch-bin-local/elasticsearch-certgen'
elastic-internal-init-filesystem '/usr/share/elasticsearch/bin/elasticsearch-certutil' -> '/mnt/elastic-internal/elasticsearch-bin-local/elasticsearch-certutil'
elastic-internal-init-filesystem '/usr/share/elasticsearch/bin/elasticsearch-cli' -> '/mnt/elastic-internal/elasticsearch-bin-local/elasticsearch-cli'
elastic-internal-init-filesystem '/usr/share/elasticsearch/bin/elasticsearch-croneval' -> '/mnt/elastic-internal/elasticsearch-bin-local/elasticsearch-croneval'
elastic-internal-init-filesystem '/usr/share/elasticsearch/bin/elasticsearch-env' -> '/mnt/elastic-internal/elasticsearch-bin-local/elasticsearch-env'
elastic-internal-init-filesystem '/usr/share/elasticsearch/bin/elasticsearch-env-from-file' -> '/mnt/elastic-internal/elasticsearch-bin-local/elasticsearch-env-from-file'
elastic-internal-init-filesystem '/usr/share/elasticsearch/bin/elasticsearch-keystore' -> '/mnt/elastic-internal/elasticsearch-bin-local/elasticsearch-keystore'
elastic-internal-init-filesystem '/usr/share/elasticsearch/bin/elasticsearch-migrate' -> '/mnt/elastic-internal/elasticsearch-bin-local/elasticsearch-migrate'
elastic-internal-init-filesystem '/usr/share/elasticsearch/bin/elasticsearch-node' -> '/mnt/elastic-internal/elasticsearch-bin-local/elasticsearch-node'
elastic-internal-init-filesystem '/usr/share/elasticsearch/bin/elasticsearch-plugin' -> '/mnt/elastic-internal/elasticsearch-bin-local/elasticsearch-plugin'
elastic-internal-init-filesystem '/usr/share/elasticsearch/bin/elasticsearch-saml-metadata' -> '/mnt/elastic-internal/elasticsearch-bin-local/elasticsearch-saml-metadata'
elastic-internal-init-filesystem '/usr/share/elasticsearch/bin/elasticsearch-setup-passwords' -> '/mnt/elastic-internal/elasticsearch-bin-local/elasticsearch-setup-passwords'
elastic-internal-init-filesystem '/usr/share/elasticsearch/bin/elasticsearch-shard' -> '/mnt/elastic-internal/elasticsearch-bin-local/elasticsearch-shard'
elastic-internal-init-filesystem '/usr/share/elasticsearch/bin/elasticsearch-sql-cli' -> '/mnt/elastic-internal/elasticsearch-bin-local/elasticsearch-sql-cli'
elastic-internal-init-filesystem '/usr/share/elasticsearch/bin/elasticsearch-sql-cli-7.6.0.jar' -> '/mnt/elastic-internal/elasticsearch-bin-local/elasticsearch-sql-cli-7.6.0.jar'
elastic-internal-init-filesystem '/usr/share/elasticsearch/bin/elasticsearch-syskeygen' -> '/mnt/elastic-internal/elasticsearch-bin-local/elasticsearch-syskeygen'
elastic-internal-init-filesystem '/usr/share/elasticsearch/bin/elasticsearch-users' -> '/mnt/elastic-internal/elasticsearch-bin-local/elasticsearch-users'
elastic-internal-init-filesystem '/usr/share/elasticsearch/bin/x-pack-env' -> '/mnt/elastic-internal/elasticsearch-bin-local/x-pack-env'
elastic-internal-init-filesystem '/usr/share/elasticsearch/bin/x-pack-security-env' -> '/mnt/elastic-internal/elasticsearch-bin-local/x-pack-security-env'
elastic-internal-init-filesystem '/usr/share/elasticsearch/bin/x-pack-watcher-env' -> '/mnt/elastic-internal/elasticsearch-bin-local/x-pack-watcher-env'
elastic-internal-init-filesystem Files copy duration: 1 sec.
elastic-internal-init-filesystem chowning /usr/share/elasticsearch/data to elasticsearch:elasticsearch
elastic-internal-init-filesystem ownership of '/usr/share/elasticsearch/data' retained as elasticsearch:elasticsearch
elastic-internal-init-filesystem chowning /usr/share/elasticsearch/logs to elasticsearch:elasticsearch
elastic-internal-init-filesystem changed ownership of '/usr/share/elasticsearch/logs' from root:root to elasticsearch:elasticsearch
elastic-internal-init-filesystem chown duration: 1 sec.
elastic-internal-init-filesystem waiting for the transport certificates (/mnt/elastic-internal/transport-certificates/pl-elastic-es-master-0.tls.key)
elastic-internal-init-filesystem wait duration: 0 sec.
elastic-internal-init-filesystem Linking /usr/share/elasticsearch/config/transport-certs/pl-elastic-es-master-0.tls.crt to /mnt/elastic-internal/elasticsearch-config-local/node-transport-cert/transport.tls.crt
elastic-internal-init-filesystem Linking /usr/share/elasticsearch/config/transport-certs/pl-elastic-es-master-0.tls.crt to /mnt/elastic-internal/elasticsearch-config-local/node-transport-cert/transport.tls.crt
elastic-internal-init-filesystem Certs linking duration: 1 sec.
elastic-internal-init-filesystem Init script successful
elastic-internal-init-filesystem Script duration: 4 sec.
Stream closed EOF for plc/pl-elastic-es-master-0 (elastic-internal-init-filesystem)
Stream closed EOF for plc/pl-elastic-es-master-0 (elasticsearch)
I tried to continue the setup, but then I get other errors, I believe this one should be fixed first For example, running dev_dns_updater gets stuck after typing the password
Any idea what could be causing ElasticSearch failure? What I found googling was related to not working on Apple Silicion, but it seems that since then it was fixed: https://stackoverflow.com/questions/65962810/m1-mac-issue-bringing-up-elasticsearch-cannot-run-jdk-bin-java
The version of Elasticsearch in the stackoverflow you linked is 7.10.2. Pixie currently uses 7.6. It's possible this is an issue with elasticsearch that got fixed between 7.6 and 7.10.2.
You could try changing this line https://github.com/pixie-io/pixie/blob/cf88e332875094fdf4cc423f582c23c1fe957729/k8s/cloud_deps/base/elastic/cluster/elastic_cluster.yaml#L8 to use the image suggested in the stackoverflow post.
@JamesMBartlett If I change the image to the one from the StackOverflow answer I get
chroot: failed to set supplemental groups: Operation not permitted
Not sure if it might be because the one from the SO answer is a public image, while the Pixie one is a patched image
Ok, I managed to make it work removing the securityContext capabilities restrictions, which is not ideal, but it should work for local tests, if you know a Pixie patched image that I can use instead it would help
allowPrivilegeEscalation: false │ • (×) Install SWE app
- capabilities: │ • (×) Check Active OrderUp
- add: │ • (×) List deployments in execution (spin)
- - SYS_CHROOT │ • (×) Grab latest build in Teran
- - SETUID │
- drop: │ • (×) OrderUp: Python Client
- - ALL