Self Hosted Air gapped production Pixie Cloud deployment: no job created

[Pk-sketch]

Describe the bug Followed the self hosted Air gapped production instruction (https://docs.px.dev/installing-pixie/install-guides/self-hosted-pixie/production-readiness/) and switched to https://docs.px.dev/installing-pixie/install-guides/self-hosted-pixie/airgap-pixie/ per instruction on Production Readiness page. The login page is visible however, when I click on login, I receive 400 Error.

To Reproduce Steps to reproduce the behavior:

1. follow the self hosted Air gapped production instruction
2. Verify all pods are up and running (there is no job create-admin-job or create-hydra-client-job created)
3. Click on login bottom on the custom domain page Authentication using Kratos / Hydra
4. Receiving HTTP Error 400 after clicking Login instead of login user/pass page

#kubectl get pod NAME READY STATUS RESTARTS AGE api-server-875984977-glwww 1/1 Running 0 4d13h artifact-tracker-server-857d8f47c9-dj4g8 1/1 Running 0 4d13h auth-server-7c78b47f94-jg5bz 1/1 Running 0 4d13h cloud-proxy-5f5c59bfbf-d9vc2 2/2 Running 0 4d13h config-manager-server-5dd98bbb76-wmlz9 1/1 Running 0 4d13h cron-script-server-54f758b6fd-q6jvh 1/1 Running 0 4d13h hydra-767f5cbd8-fn4z5 2/2 Running 0 4d13h indexer-server-74c9f97d49-gzxms 1/1 Running 0 4d13h kratos-795566bd8f-4rlxq 2/2 Running 0 4d13h metrics-server-696f7f8446-58cjx 1/1 Running 0 4d13h pl-elastic-es-data-0 1/1 Running 0 5d19h pl-elastic-es-master-0 1/1 Running 0 5d19h pl-elastic-es-master-1 1/1 Running 0 5d19h pl-nats-0 1/1 Running 0 5d19h pl-nats-1 1/1 Running 0 5d19h pl-nats-2 1/1 Running 0 5d19h plugin-server-7d76679674-l725t 1/1 Running 0 4d13h postgres-6877f879-fz5wn 1/1 Running 0 5d19h profile-server-5c974b5775-q5r6c 1/1 Running 0 4d13h project-manager-server-799f8c5b6d-fzk46 1/1 Running 0 4d13h scriptmgr-server-76f8c4b54c-dc5kq 1/1 Running 0 4d13h vzconn-server-5f76f4dcdc-7bptf 1/1 Running 0 4d13h vzmgr-server-6d4bccd56d-86blq 1/1 Running 0 4d13h

Expected behavior Viewing user/pass page after click on Login button

Screenshots

Logs log.tar.gz

** Yaml ** Uploading pixie_cloud.tar.gz…

App information (please complete the following information):

• Pixie version latest
• K8s cluster version v1.29.7
• Node Kernel version 4.18.0-553.8.1.el8_10.x86_64
• Browser version Chrome latest

Additional context in addition, in https://docs.px.dev/installing-pixie/install-guides/airgap-pixie/#deploy-pixie-cloud-authentication-using-kratos-hydra is written to set the admin credentials should update k8s/cloud/base/ory_auth/kratos/kratos_deployment.yaml while no where in the document this yaml is called.

[Mr-Artemus]

Hi, I remember something similar happening. Make sure to use the version 0.1.7 on all images instead of latest (latest is older than 0.1.7)

[YoniC0de]

Same issue for me, @Pk-sketch , did you manage to fix it?

[Pk-sketch]

The recommendation from Mr-Artemus resolved the issue. Thanks to him I was able to bring up Pixie in Air gaped env.