openshift
kube-apiserver readiness check: stat /var/lib/microshift/resources/kubeadmin/kubeconfig: no such file or directory
NOTE: we suspect this is because initialization is half way done sometimes, and on next boot the generation of files is not completed
What happened:
Unable to start microshift via systemd. It panics with the following error:
I0311 17:14:22.716602 1 manager.go:112] Starting kube-apiserver
E0311 17:14:22.716772 1 kube-apiserver.go:225] kube-apiserver readiness check: stat /var/lib/microshift/resources/kubeadmin/kubeconfig: no such file or directory
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x30c257a]
goroutine 306 [running]:
k8s.io/client-go/kubernetes.NewForConfig(0x0, 0xc00100efa8, 0x0, 0x0)
/opt/app-root/src/github.com/redhat-et/microshift/vendor/k8s.io/client-go/kubernetes/clientset.go:395 +0x3a
github.com/openshift/microshift/pkg/controllers.(*KubeAPIServer).Run.func1(0xc000728c60, 0xc0017bc180, 0xc001614000)
/opt/app-root/src/github.com/redhat-et/microshift/pkg/controllers/kube-apiserver.go:229 +0xa5
created by github.com/openshift/microshift/pkg/controllers.(*KubeAPIServer).Run
/opt/app-root/src/github.com/redhat-et/microshift/pkg/controllers/kube-apiserver.go:222 +0xc8
What you expected to happen:
Expected microshift to start successfully.
How to reproduce it (as minimally and precisely as possible):
- Follow the steps in the Getting Started docs. Failure occurs on this step:
sudo systemctl enable microshift --now
Anything else we need to know?:
A few of these warnings early in start process:
WARNING: The same type, major and minor should not be used for multiple devices.
It's unclear if these are related to the issue.
Environment:
- Microshift version (use
microshift version):quay.io/microshift/microshift:latestwith digestsha256:efea2a587594be185554ea399f532e4d0b6b483ea9b4842c161098b554f9e54f - Hardware configuration:
- OS (e.g:
cat /etc/os-release): Fedora 35 - Kernel (e.g.
uname -a):Linux localhost.localdomain 5.16.12-200.fc35.x86_64 #1 SMP PREEMPT Wed Mar 2 19:06:17 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux - Others:
Relevant Logs
Mar 11 12:14:20 localhost.localdomain systemd[1]: Starting MicroShift Containerized...
Mar 11 12:14:20 localhost.localdomain podman[26716]: WARNING: The same type, major and minor should not be used for multiple devices.
Mar 11 12:14:20 localhost.localdomain podman[26716]: WARNING: The same type, major and minor should not be used for multiple devices.
Mar 11 12:14:20 localhost.localdomain podman[26716]: WARNING: The same type, major and minor should not be used for multiple devices.
Mar 11 12:14:20 localhost.localdomain podman[26716]: WARNING: The same type, major and minor should not be used for multiple devices.
Mar 11 12:14:20 localhost.localdomain podman[26716]: WARNING: The same type, major and minor should not be used for multiple devices.
Mar 11 12:14:20 localhost.localdomain podman[26716]: WARNING: The same type, major and minor should not be used for multiple devices.
Mar 11 12:14:20 localhost.localdomain podman[26716]: WARNING: The same type, major and minor should not be used for multiple devices.
Mar 11 12:14:20 localhost.localdomain podman[26716]: WARNING: The same type, major and minor should not be used for multiple devices.
Mar 11 12:14:20 localhost.localdomain podman[26716]: WARNING: The same type, major and minor should not be used for multiple devices.
Mar 11 12:14:20 localhost.localdomain podman[26716]: WARNING: The same type, major and minor should not be used for multiple devices.
Mar 11 12:14:20 localhost.localdomain podman[26716]:
Mar 11 12:14:20 localhost.localdomain podman[26716]: 2022-03-11 12:14:20.750306211 -0500 EST m=+0.073820434 container create ea62f7801a7cc9044e17d0e2a5f86365119b15710c95f2020f2abfb7aa159f4d (image=quay.io/microshift/microshift:latest, name=microshift, io.containers.autoupdate=registry, io.buildah.version=1.23.1, io.openshift.expose-services=, build-date=2021-10-26T13:03:06.167238, io.k8s.display-name=Red Hat Universal Base Image 8 Minimal, com.redhat.build-host=cpt-1006.osbs.prod.upshift.rdu2.redhat.com, PODMAN_SYSTEMD_UNIT=microshift.service, architecture=x86_64, com.redhat.component=ubi8-minimal-container, vcs-ref=7256039d3c371a38cf13906dcf5688c19700c73b, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8-minimal/images/8.4-212, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.tags=minimal rhel8, version=8.4, vcs-type=git, name=ubi8-minimal, release=212, description=The Universal Base Image Minimal is a stripped down image that uses microdnf as a package manager. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Red Hat, Inc., summary=Provides the latest release of the minimal Red Hat Universal Base Image 8., vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image Minimal is a stripped down image that uses microdnf as a package manager. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public)
Mar 11 12:14:20 localhost.localdomain oci-umount[26772]: umounthook <debug>: prestart container_id:ea62f7801a7c rootfs:/var/lib/containers/storage/overlay/1263437b45ea09594199bc5b502a2cc2611e06699150471d4ab181db0f6c507b/merged
Mar 11 12:14:20 localhost.localdomain oci-umount[26772]: umounthook <warning>: ea62f7801a7c: path /var/lib/containers/storage from /usr/share/oci-umount/oci-umount.d/crio-umount.conf already exists
Mar 11 12:14:20 localhost.localdomain podman[26716]: 2022-03-11 12:14:20.799207333 -0500 EST m=+0.122721541 container init ea62f7801a7cc9044e17d0e2a5f86365119b15710c95f2020f2abfb7aa159f4d (image=quay.io/microshift/microshift:latest, name=microshift, com.redhat.build-host=cpt-1006.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, com.redhat.component=ubi8-minimal-container, name=ubi8-minimal, vendor=Red Hat, Inc., release=212, io.containers.autoupdate=registry, description=The Universal Base Image Minimal is a stripped down image that uses microdnf as a package manager. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=7256039d3c371a38cf13906dcf5688c19700c73b, distribution-scope=public, io.openshift.tags=minimal rhel8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.k8s.display-name=Red Hat Universal Base Image 8 Minimal, summary=Provides the latest release of the minimal Red Hat Universal Base Image 8., maintainer=Red Hat, Inc., io.k8s.description=The Universal Base Image Minimal is a stripped down image that uses microdnf as a package manager. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8-minimal/images/8.4-212, PODMAN_SYSTEMD_UNIT=microshift.service, io.openshift.expose-services=, vcs-type=git, build-date=2021-10-26T13:03:06.167238, io.buildah.version=1.23.1, version=8.4)
Mar 11 12:14:20 localhost.localdomain podman[26716]: 2022-03-11 12:14:20.720222708 -0500 EST m=+0.043736935 image pull quay.io/microshift/microshift:latest
Mar 11 12:14:20 localhost.localdomain podman[26716]: 2022-03-11 12:14:20.808228662 -0500 EST m=+0.131742908 container start ea62f7801a7cc9044e17d0e2a5f86365119b15710c95f2020f2abfb7aa159f4d (image=quay.io/microshift/microshift:latest, name=microshift, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, description=The Universal Base Image Minimal is a stripped down image that uses microdnf as a package manager. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, vcs-type=git, release=212, io.buildah.version=1.23.1, com.redhat.build-host=cpt-1006.osbs.prod.upshift.rdu2.redhat.com, PODMAN_SYSTEMD_UNIT=microshift.service, distribution-scope=public, build-date=2021-10-26T13:03:06.167238, maintainer=Red Hat, Inc., io.openshift.tags=minimal rhel8, io.k8s.description=The Universal Base Image Minimal is a stripped down image that uses microdnf as a package manager. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=Red Hat Universal Base Image 8 Minimal, io.containers.autoupdate=registry, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8-minimal/images/8.4-212, com.redhat.component=ubi8-minimal-container, name=ubi8-minimal, vcs-ref=7256039d3c371a38cf13906dcf5688c19700c73b, vendor=Red Hat, Inc., architecture=x86_64, version=8.4, summary=Provides the latest release of the minimal Red Hat Universal Base Image 8.)
Mar 11 12:14:20 localhost.localdomain podman[26716]: ea62f7801a7cc9044e17d0e2a5f86365119b15710c95f2020f2abfb7aa159f4d
Mar 11 12:14:20 localhost.localdomain microshift[26768]: I0311 17:14:20.909574 1 openshift-apiserver.go:88] starting openshift-apiserver 192.168.86.38, args: [--config=/var/lib/microshift/resources/openshift-apiserver/config/config.yaml]
Mar 11 12:14:20 localhost.localdomain microshift[26768]: I0311 17:14:20.912846 1 run.go:117] Starting Microshift
Mar 11 12:14:20 localhost.localdomain microshift[26768]: I0311 17:14:20.912903 1 run.go:122] Started service-manager
Mar 11 12:14:20 localhost.localdomain microshift[26768]: I0311 17:14:20.912963 1 manager.go:112] Starting etcd
Mar 11 12:14:20 localhost.localdomain microshift[26768]: {"level":"info","ts":"2022-03-11T17:14:20.913Z","caller":"embed/etcd.go:125","msg":"configuring peer listeners","listen-peer-urls":["https://192.168.86.38:2380"]}
Mar 11 12:14:20 localhost.localdomain microshift[26768]: {"level":"info","ts":"2022-03-11T17:14:20.913Z","caller":"embed/etcd.go:477","msg":"starting with peer TLS","tls-info":"cert = /var/lib/microshift/certs/etcd/etcd-peer.crt, key = /var/lib/microshift/certs/etcd/etcd-peer.key, trusted-ca = /var/lib/microshift/certs/ca-bundle/ca-bundle.crt, client-cert-auth = false, crl-file = ","cipher-suites":["TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384","TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384","TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256","TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256","TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305","TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305"]}
Mar 11 12:14:20 localhost.localdomain microshift[26768]: {"level":"info","ts":"2022-03-11T17:14:20.913Z","caller":"embed/etcd.go:135","msg":"configuring client listeners","listen-client-urls":["https://127.0.0.1:2379","https://192.168.86.38:2379"]}
Mar 11 12:14:20 localhost.localdomain microshift[26768]: {"level":"info","ts":"2022-03-11T17:14:20.913Z","caller":"embed/etcd.go:311","msg":"starting an etcd server","etcd-version":"3.4.14","git-sha":"Not provided (use ./build instead of go build)","go-version":"go1.16.12","go-os":"linux","go-arch":"amd64","max-cpu-set":8,"max-cpu-available":8,"member-initialized":true,"name":"localhost.localdomain","data-dir":"/var/lib/microshift/etcd","wal-dir":"","wal-dir-dedicated":"","member-dir":"/var/lib/microshift/etcd/member","force-new-cluster":false,"heartbeat-interval":"100ms","election-timeout":"1s","initial-election-tick-advance":true,"snapshot-count":100000,"snapshot-catchup-entries":5000,"initial-advertise-peer-urls":["https://192.168.86.38:2380"],"listen-peer-urls":["https://192.168.86.38:2380"],"advertise-client-urls":["https://192.168.86.38:2379"],"listen-client-urls":["https://127.0.0.1:2379","https://192.168.86.38:2379"],"listen-metrics-urls":["https://127.0.0.1:2381"],"cors":["*"],"host-whitelist":["*"],"initial-cluster":"","initial-cluster-state":"new","initial-cluster-token":"","quota-size-bytes":2147483648,"pre-vote":false,"initial-corrupt-check":false,"corrupt-check-time-interval":"0s","auto-compaction-mode":"","auto-compaction-retention":"0s","auto-compaction-interval":"0s","discovery-url":"","discovery-proxy":""}
Mar 11 12:14:20 localhost.localdomain microshift[26768]: {"level":"info","ts":"2022-03-11T17:14:20.913Z","caller":"etcdserver/backend.go:80","msg":"opened backend db","path":"/var/lib/microshift/etcd/member/snap/db","took":"94.193µs"}
Mar 11 12:14:20 localhost.localdomain microshift[26768]: {"level":"info","ts":"2022-03-11T17:14:20.914Z","caller":"etcdserver/raft.go:536","msg":"restarting local member","cluster-id":"1fbf54d7fd7fe23a","local-member-id":"129e664cf22c2231","commit-index":162}
Mar 11 12:14:20 localhost.localdomain microshift[26768]: {"level":"info","ts":"2022-03-11T17:14:20.914Z","caller":"raft/raft.go:1530","msg":"129e664cf22c2231 switched to configuration voters=()"}
Mar 11 12:14:20 localhost.localdomain microshift[26768]: {"level":"info","ts":"2022-03-11T17:14:20.914Z","caller":"raft/raft.go:700","msg":"129e664cf22c2231 became follower at term 81"}
Mar 11 12:14:20 localhost.localdomain microshift[26768]: {"level":"info","ts":"2022-03-11T17:14:20.914Z","caller":"raft/raft.go:383","msg":"newRaft 129e664cf22c2231 [peers: [], term: 81, commit: 162, applied: 0, lastindex: 162, lastterm: 81]"}
Mar 11 12:14:20 localhost.localdomain microshift[26768]: {"level":"warn","ts":"2022-03-11T17:14:20.916Z","caller":"auth/store.go:1366","msg":"simple token is not cryptographically signed"}
Mar 11 12:14:20 localhost.localdomain microshift[26768]: {"level":"info","ts":"2022-03-11T17:14:20.918Z","caller":"etcdserver/quota.go:98","msg":"enabled backend quota with default value","quota-name":"v3-applier","quota-size-bytes":2147483648,"quota-size":"2.1 GB"}
Mar 11 12:14:20 localhost.localdomain microshift[26768]: {"level":"info","ts":"2022-03-11T17:14:20.919Z","caller":"etcdserver/server.go:803","msg":"starting etcd server","local-member-id":"129e664cf22c2231","local-server-version":"3.4.14","cluster-version":"to_be_decided"}
Mar 11 12:14:20 localhost.localdomain microshift[26768]: {"level":"info","ts":"2022-03-11T17:14:20.921Z","caller":"etcdserver/server.go:691","msg":"starting initial election tick advance","election-ticks":10}
Mar 11 12:14:20 localhost.localdomain microshift[26768]: {"level":"info","ts":"2022-03-11T17:14:20.921Z","caller":"raft/raft.go:1530","msg":"129e664cf22c2231 switched to configuration voters=(1341622219669512753)"}
Mar 11 12:14:20 localhost.localdomain microshift[26768]: {"level":"info","ts":"2022-03-11T17:14:20.921Z","caller":"membership/cluster.go:392","msg":"added member","cluster-id":"1fbf54d7fd7fe23a","local-member-id":"129e664cf22c2231","added-peer-id":"129e664cf22c2231","added-peer-peer-urls":["https://192.168.86.38:2380"]}
Mar 11 12:14:20 localhost.localdomain microshift[26768]: {"level":"info","ts":"2022-03-11T17:14:20.922Z","caller":"membership/cluster.go:558","msg":"set initial cluster version","cluster-id":"1fbf54d7fd7fe23a","local-member-id":"129e664cf22c2231","cluster-version":"3.4"}
Mar 11 12:14:20 localhost.localdomain microshift[26768]: {"level":"info","ts":"2022-03-11T17:14:20.922Z","caller":"api/capability.go:76","msg":"enabled capabilities for version","cluster-version":"3.4"}
Mar 11 12:14:20 localhost.localdomain microshift[26768]: {"level":"info","ts":"2022-03-11T17:14:20.922Z","caller":"embed/etcd.go:726","msg":"starting with client TLS","tls-info":"cert = /var/lib/microshift/certs/etcd/etcd-serving.crt, key = /var/lib/microshift/certs/etcd/etcd-serving.key, trusted-ca = /var/lib/microshift/certs/ca-bundle/ca-bundle.crt, client-cert-auth = false, crl-file = ","cipher-suites":["TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384","TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384","TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256","TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256","TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305","TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305"]}
Mar 11 12:14:20 localhost.localdomain microshift[26768]: {"level":"info","ts":"2022-03-11T17:14:20.923Z","caller":"embed/etcd.go:592","msg":"serving peer traffic","address":"192.168.86.38:2380"}
Mar 11 12:14:20 localhost.localdomain microshift[26768]: {"level":"info","ts":"2022-03-11T17:14:20.923Z","caller":"embed/etcd.go:253","msg":"now serving peer/client/metrics","local-member-id":"129e664cf22c2231","initial-advertise-peer-urls":["https://192.168.86.38:2380"],"listen-peer-urls":["https://192.168.86.38:2380"],"advertise-client-urls":["https://192.168.86.38:2379"],"listen-client-urls":["https://127.0.0.1:2379","https://192.168.86.38:2379"],"listen-metrics-urls":["https://127.0.0.1:2381"]}
Mar 11 12:14:20 localhost.localdomain microshift[26768]: {"level":"info","ts":"2022-03-11T17:14:20.923Z","caller":"embed/etcd.go:799","msg":"serving metrics","address":"https://127.0.0.1:2381"}
Mar 11 12:14:22 localhost.localdomain microshift[26768]: {"level":"info","ts":"2022-03-11T17:14:22.715Z","caller":"raft/raft.go:923","msg":"129e664cf22c2231 is starting a new election at term 81"}
Mar 11 12:14:22 localhost.localdomain microshift[26768]: {"level":"info","ts":"2022-03-11T17:14:22.715Z","caller":"raft/raft.go:713","msg":"129e664cf22c2231 became candidate at term 82"}
Mar 11 12:14:22 localhost.localdomain microshift[26768]: {"level":"info","ts":"2022-03-11T17:14:22.715Z","caller":"raft/raft.go:824","msg":"129e664cf22c2231 received MsgVoteResp from 129e664cf22c2231 at term 82"}
Mar 11 12:14:22 localhost.localdomain microshift[26768]: {"level":"info","ts":"2022-03-11T17:14:22.715Z","caller":"raft/raft.go:765","msg":"129e664cf22c2231 became leader at term 82"}
Mar 11 12:14:22 localhost.localdomain microshift[26768]: {"level":"info","ts":"2022-03-11T17:14:22.715Z","caller":"raft/node.go:325","msg":"raft.node: 129e664cf22c2231 elected leader 129e664cf22c2231 at term 82"}
Mar 11 12:14:22 localhost.localdomain microshift[26768]: {"level":"info","ts":"2022-03-11T17:14:22.716Z","caller":"etcdserver/server.go:2037","msg":"published local member to cluster through raft","local-member-id":"129e664cf22c2231","local-member-attributes":"{Name:localhost.localdomain ClientURLs:[https://192.168.86.38:2379]}","request-path":"/0/members/129e664cf22c2231/attributes","cluster-id":"1fbf54d7fd7fe23a","publish-timeout":"7s"}
Mar 11 12:14:22 localhost.localdomain microshift[26768]: I0311 17:14:22.716380 1 etcd.go:102] etcd is ready
Mar 11 12:14:22 localhost.localdomain microshift[26768]: I0311 17:14:22.716602 1 manager.go:112] Starting kube-apiserver
Mar 11 12:14:22 localhost.localdomain microshift[26768]: E0311 17:14:22.716772 1 kube-apiserver.go:225] kube-apiserver readiness check: stat /var/lib/microshift/resources/kubeadmin/kubeconfig: no such file or directory
Mar 11 12:14:22 localhost.localdomain microshift[26768]: panic: runtime error: invalid memory address or nil pointer dereference
Mar 11 12:14:22 localhost.localdomain microshift[26768]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x30c257a]
Mar 11 12:14:22 localhost.localdomain microshift[26768]:
Mar 11 12:14:22 localhost.localdomain microshift[26768]: goroutine 306 [running]:
Mar 11 12:14:22 localhost.localdomain microshift[26768]: k8s.io/client-go/kubernetes.NewForConfig(0x0, 0xc00100efa8, 0x0, 0x0)
Mar 11 12:14:22 localhost.localdomain microshift[26768]: /opt/app-root/src/github.com/redhat-et/microshift/vendor/k8s.io/client-go/kubernetes/clientset.go:395 +0x3a
Mar 11 12:14:22 localhost.localdomain microshift[26768]: github.com/openshift/microshift/pkg/controllers.(*KubeAPIServer).Run.func1(0xc000728c60, 0xc0017bc180, 0xc001614000)
Mar 11 12:14:22 localhost.localdomain microshift[26768]: /opt/app-root/src/github.com/redhat-et/microshift/pkg/controllers/kube-apiserver.go:229 +0xa5
Mar 11 12:14:22 localhost.localdomain microshift[26768]: created by github.com/openshift/microshift/pkg/controllers.(*KubeAPIServer).Run
Mar 11 12:14:22 localhost.localdomain microshift[26768]: /opt/app-root/src/github.com/redhat-et/microshift/pkg/controllers/kube-apiserver.go:222 +0xc8
Mar 11 12:14:22 localhost.localdomain podman[26789]: 2022-03-11 12:14:22.838498262 -0500 EST m=+0.069568919 container died ea62f7801a7cc9044e17d0e2a5f86365119b15710c95f2020f2abfb7aa159f4d (image=quay.io/microshift/microshift:latest, name=microshift)
Mar 11 12:14:22 localhost.localdomain podman[26789]: 2022-03-11 12:14:22.868233541 -0500 EST m=+0.099304178 container remove ea62f7801a7cc9044e17d0e2a5f86365119b15710c95f2020f2abfb7aa159f4d (image=quay.io/microshift/microshift:latest, name=microshift, architecture=x86_64, io.openshift.expose-services=, version=8.4, io.k8s.description=The Universal Base Image Minimal is a stripped down image that uses microdnf as a package manager. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1006.osbs.prod.upshift.rdu2.redhat.com, maintainer=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8-minimal/images/8.4-212, distribution-scope=public, name=ubi8-minimal, vendor=Red Hat, Inc., com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vcs-ref=7256039d3c371a38cf13906dcf5688c19700c73b, vcs-type=git, io.containers.autoupdate=registry, com.redhat.component=ubi8-minimal-container, build-date=2021-10-26T13:03:06.167238, PODMAN_SYSTEMD_UNIT=microshift.service, io.openshift.tags=minimal rhel8, io.buildah.version=1.23.1, description=The Universal Base Image Minimal is a stripped down image that uses microdnf as a package manager. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=Red Hat Universal Base Image 8 Minimal, release=212, summary=Provides the latest release of the minimal Red Hat Universal Base Image 8.)
Mar 11 12:14:22 localhost.localdomain systemd[1]: microshift.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Mar 11 12:14:23 localhost.localdomain systemd[1]: microshift.service: Failed with result 'exit-code'.
Mar 11 12:14:23 localhost.localdomain systemd[1]: Failed to start MicroShift Containerized.
Hello @lcarva
What method are you using to install MicroShift? Is it RPM or containerized version?
I'd like to replicate your environment locally.
Thanks
Issues go stale after 90d of inactivity.
Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.
If this issue is safe to close now please do so with /close.
/lifecycle stale
@ggiguash: Closing this issue.
In response to this:
/close
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.
![openshift-ci[bot] avatar](https://avatars.githubusercontent.com/in/91280?v=4 "Published by a pub.dev verified publisher"){kind=small}
Hi Team,
I am trying to install microshift on RHEL-8.7 and I am following Red Hat documentation
https://access.redhat.com/documentation/en-us/red_hat_build_of_microshift/4.12/html/installing/microshift-install-rpm#microshift-install-system-requirements_microshift-install-rpm
Everything went well, but while I am starting the microshift service not starting. I see below issue.
[root@microshift ~]# systemctl start microshift Job for microshift.service failed because the service did not take the steps required by its unit configuration. See "systemctl status microshift.service" and "journalctl -xe" for details. [root@microshift ~]#
Is any one can help on this issue? thanks in advance.
@skylabreddy can you share the output of journalctl -u microshift? It might be large, so you may want to use a separate gist or pastebin.
It would also be useful to know more about the environment where you are running. What sort of host (laptop, VM, etc.)? What networking do you have enabled?
@dhellmann Here is the inputs
- Microshift running on RHEL-8.7 VM
- If come to network, i did not enabled any decated network for microshift,
[root@microshift ~]# journalctl -u microshift -- Logs begin at Fri 2023-03-17 20:54:09 IST, end at Fri 2023-03-17 21:09:48 IST. -- Mar 17 20:54:09 microshift systemd[1]: Starting MicroShift... Mar 17 20:54:09 microshift microshift[649415]: ??? I0317 20:54:09.958519 649415 run.go:116] Starting MicroShift Mar 17 20:54:09 microshift microshift[649415]: ??? E0317 20:54:09.958634 649415 certchains.go:122] [admin-kubeconfig-signer] rotate at: 2032-03-19 10:01:> Mar 17 20:54:09 microshift microshift[649415]: ??? E0317 20:54:09.958659 649415 certchains.go:122] [admin-kubeconfig-signer admin-kubeconfig-client] rota> Mar 17 20:54:09 microshift microshift[649415]: ??? E0317 20:54:09.958677 649415 certchains.go:122] [aggregator-signer] rotate at: 2023-11-17 10:01:49 +00> Mar 17 20:54:09 microshift microshift[649415]: ??? E0317 20:54:09.958693 649415 certchains.go:122] [aggregator-signer aggregator-client] rotate at: 2023-> Mar 17 20:54:09 microshift microshift[649415]: ??? E0317 20:54:09.958707 649415 certchains.go:122] [etcd-signer] rotate at: 2032-03-19 10:01:52 +0000 UTC Mar 17 20:54:09 microshift microshift[649415]: ??? E0317 20:54:09.958721 649415 certchains.go:122] [etcd-signer apiserver-etcd-client] rotate at: 2032-03> Mar 17 20:54:09 microshift microshift[649415]: ??? E0317 20:54:09.958734 649415 certchains.go:122] [ingress-ca] rotate at: 2032-03-19 10:01:50 +0000 UTC Mar 17 20:54:09 microshift microshift[649415]: ??? E0317 20:54:09.958748 649415 certchains.go:122] [ingress-ca router-default-serving] rotate at: 2023-11> Mar 17 20:54:09 microshift microshift[649415]: ??? E0317 20:54:09.958760 649415 certchains.go:122] [kube-apiserver-external-signer] rotate at: 2032-03-19> Mar 17 20:54:09 microshift microshift[649415]: ??? E0317 20:54:09.958775 649415 certchains.go:122] [kube-apiserver-external-signer kube-external-serving]> Mar 17 20:54:09 microshift microshift[649415]: ??? E0317 20:54:09.958788 649415 certchains.go:122] [kube-apiserver-localhost-signer] rotate at: 2032-03-1> Mar 17 20:54:09 microshift microshift[649415]: ??? E0317 20:54:09.958800 649415 certchains.go:122] [kube-apiserver-localhost-signer kube-apiserver-localh> Mar 17 20:54:09 microshift microshift[649415]: ??? E0317 20:54:09.958813 649415 certchains.go:122] [kube-apiserver-service-network-signer] rotate at: 203> Mar 17 20:54:09 microshift microshift[649415]: ??? E0317 20:54:09.958827 649415 certchains.go:122] [kube-apiserver-service-network-signer kube-apiserver-> Mar 17 20:54:09 microshift microshift[649415]: ??? E0317 20:54:09.958840 649415 certchains.go:122] [kube-apiserver-to-kubelet-signer] rotate at: 2023-11-> Mar 17 20:54:09 microshift microshift[649415]: ??? E0317 20:54:09.958853 649415 certchains.go:122] [kube-apiserver-to-kubelet-signer kube-apiserver-to-ku> Mar 17 20:54:09 microshift microshift[649415]: ??? E0317 20:54:09.958867 649415 certchains.go:122] [kube-control-plane-signer] rotate at: 2023-11-17 10:0> Mar 17 20:54:09 microshift microshift[649415]: ??? E0317 20:54:09.958883 649415 certchains.go:122] [kube-control-plane-signer cluster-policy-controller] > Mar 17 20:54:09 microshift microshift[649415]: ??? E0317 20:54:09.958896 649415 certchains.go:122] [kube-control-plane-signer kube-controller-manager] ro> Mar 17 20:54:09 microshift microshift[649415]: ??? E0317 20:54:09.958909 649415 certchains.go:122] [kube-control-plane-signer kube-scheduler] rotate at: > Mar 17 20:54:09 microshift microshift[649415]: ??? E0317 20:54:09.958921 649415 certchains.go:122] [kube-control-plane-signer route-controller-manager] r> Mar 17 20:54:09 microshift microshift[649415]: ??? E0317 20:54:09.958934 649415 certchains.go:122] [kubelet-signer] rotate at: 2023-11-17 10:01:48 +0000 > Mar 17 20:54:09 microshift microshift[649415]: ??? E0317 20:54:09.958948 649415 certchains.go:122] [kubelet-signer kube-csr-signer] rotate at: 2023-11-17> Mar 17 20:54:09 microshift microshift[649415]: ??? E0317 20:54:09.958963 649415 certchains.go:122] [kubelet-signer kube-csr-signer kubelet-client] rotate> Mar 17 20:54:09 microshift microshift[649415]: ??? E0317 20:54:09.958978 649415 certchains.go:122] [kubelet-signer kube-csr-signer kubelet-server] rotate> Mar 17 20:54:09 microshift microshift[649415]: ??? E0317 20:54:09.958990 649415 certchains.go:122] [service-ca] rotate at: 2032-03-19 10:01:49 +0000 UTC Mar 17 20:54:09 microshift microshift[649415]: ??? E0317 20:54:09.959003 649415 certchains.go:122] [service-ca route-controller-manager-serving] rotate a> Mar 17 20:54:09 microshift microshift[649415]: ??? I0317 20:54:09.959205 649415 run.go:127] Started service-manager Mar 17 20:54:09 microshift microshift[649415]: network-configuration I0317 20:54:09.959290 649415 manager.go:114] Starting network-configuration Mar 17 20:54:09 microshift microshift[649415]: etcd I0317 20:54:09.959326 649415 manager.go:114] Starting etcd Mar 17 20:54:09 microshift microshift[649415]: {"level":"info","ts":"2023-03-17T20:54:09.959+0530","caller":"embed/etcd.go:131","msg":"configuring peer li> Mar 17 20:54:09 microshift microshift[649415]: {"level":"info","ts":"2023-03-17T20:54:09.959+0530","caller":"embed/etcd.go:481","msg":"starting with peer > Mar 17 20:54:09 microshift microshift[649415]: network-configuration I0317 20:54:09.959751 649415 netconfig.go:74] "network-configuration" is ready Mar 17 20:54:09 microshift microshift[649415]: sysconfwatch-controller I0317 20:54:09.959787 649415 manager.go:114] Starting sysconfwatch-controller Mar 17 20:54:09 microshift microshift[649415]: sysconfwatch-controller I0317 20:54:09.959811 649415 sysconfwatch_linux.go:89] starting sysconfwatch-contr> Mar 17 20:54:09 microshift microshift[649415]: sysconfwatch-controller I0317 20:54:09.959827 649415 sysconfwatch_linux.go:95] sysconfwatch-controller is > Mar 17 20:54:09 microshift microshift[649415]: {"level":"info","ts":"2023-03-17T20:54:09.960+0530","caller":"embed/etcd.go:139","msg":"configuring client > Mar 17 20:54:09 microshift microshift[649415]: {"level":"info","ts":"2023-03-17T20:54:09.960+0530","caller":"embed/etcd.go:309","msg":"starting an etcd se> Mar 17 20:54:09 microshift microshift[649415]: {"level":"info","ts":"2023-03-17T20:54:09.961+0530","caller":"etcdserver/backend.go:81","msg":"opened backe> Mar 17 20:54:09 microshift microshift[649415]: {"level":"info","ts":"2023-03-17T20:54:09.982+0530","caller":"etcdserver/server.go:532","msg":"No snapshot > Mar 17 20:54:09 microshift microshift[649415]: {"level":"info","ts":"2023-03-17T20:54:09.998+0530","caller":"etcdserver/raft.go:483","msg":"restarting loc> Mar 17 20:54:10 microshift microshift[649415]: {"level":"info","ts":"2023-03-17T20:54:10.000+0530","logger":"raft","caller":"etcdserver/zap_raft.go:77","m> Mar 17 20:54:10 microshift microshift[649415]: {"level":"info","ts":"2023-03-17T20:54:10.000+0530","logger":"raft","caller":"etcdserver/zap_raft.go:77","m> Mar 17 20:54:10 microshift microshift[649415]: {"level":"info","ts":"2023-03-17T20:54:10.000+0530","logger":"raft","caller":"etcdserver/zap_raft.go:77","m> lines 1-47
You should not need a dedicated network, but we know the CNI plugin does not work with wifi interfaces, yet. As long as your VM has a default route, you should be fine.
I do not see any obvious errors in the log, but it does not look complete, either.
How is the VM configured? How much RAM and CPU, for example?
That should be plenty. It would be useful to capture more log output. Could you try running microshift outside of systemd to see if we get more information that way?
sudo /usr/bin/microshift run 2>&1 | tee /tmp/microshift.log
@dhellmann Here is the log output,
[root@microshift ~]# sudo /usr/bin/microshift run 2>&1 | tee /tmp/microshift.log
??? I0318 01:18:58.624860 1154438 run.go:116] Starting MicroShift
??? E0318 01:18:58.625050 1154438 certchains.go:122] [admin-kubeconfig-signer] rotate at: 2032-03-19 10:01:47 +0000 UTC
??? E0318 01:18:58.625062 1154438 certchains.go:122] [admin-kubeconfig-signer admin-kubeconfig-client] rotate at: 2032-03-19 10:01:48 +0000 UTC
??? E0318 01:18:58.625070 1154438 certchains.go:122] [aggregator-signer] rotate at: 2023-11-17 10:01:49 +0000 UTC
??? E0318 01:18:58.625075 1154438 certchains.go:122] [aggregator-signer aggregator-client] rotate at: 2023-11-17 10:01:49 +0000 UTC
??? E0318 01:18:58.625080 1154438 certchains.go:122] [etcd-signer] rotate at: 2032-03-19 10:01:52 +0000 UTC
??? E0318 01:18:58.625084 1154438 certchains.go:122] [etcd-signer apiserver-etcd-client] rotate at: 2032-03-19 10:01:52 +0000 UTC
??? E0318 01:18:58.625089 1154438 certchains.go:122] [ingress-ca] rotate at: 2032-03-19 10:01:50 +0000 UTC
??? E0318 01:18:58.625094 1154438 certchains.go:122] [ingress-ca router-default-serving] rotate at: 2023-11-17 10:01:50 +0000 UTC
??? E0318 01:18:58.625098 1154438 certchains.go:122] [kube-apiserver-external-signer] rotate at: 2032-03-19 10:01:51 +0000 UTC
??? E0318 01:18:58.625103 1154438 certchains.go:122] [kube-apiserver-external-signer kube-external-serving] rotate at: 2023-11-17 10:01:51 +0000 UTC
??? E0318 01:18:58.625107 1154438 certchains.go:122] [kube-apiserver-localhost-signer] rotate at: 2032-03-19 10:01:51 +0000 UTC
??? E0318 01:18:58.625112 1154438 certchains.go:122] [kube-apiserver-localhost-signer kube-apiserver-localhost-serving] rotate at: 2023-11-17 10:01:51 +0000 UTC
??? E0318 01:18:58.625116 1154438 certchains.go:122] [kube-apiserver-service-network-signer] rotate at: 2032-03-19 10:01:52 +0000 UTC
??? E0318 01:18:58.625121 1154438 certchains.go:122] [kube-apiserver-service-network-signer kube-apiserver-service-network-serving] rotate at: 2023-11-17 10:01:52 +0000 UTC
??? E0318 01:18:58.625125 1154438 certchains.go:122] [kube-apiserver-to-kubelet-signer] rotate at: 2023-11-17 10:01:47 +0000 UTC
??? E0318 01:18:58.625131 1154438 certchains.go:122] [kube-apiserver-to-kubelet-signer kube-apiserver-to-kubelet-client] rotate at: 2023-11-17 10:01:47 +0000 UTC
??? E0318 01:18:58.625135 1154438 certchains.go:122] [kube-control-plane-signer] rotate at: 2023-11-17 10:01:46 +0000 UTC
??? E0318 01:18:58.625140 1154438 certchains.go:122] [kube-control-plane-signer cluster-policy-controller] rotate at: 2023-11-17 10:01:47 +0000 UTC
??? E0318 01:18:58.625145 1154438 certchains.go:122] [kube-control-plane-signer kube-controller-manager] rotate at: 2023-11-17 10:01:46 +0000 UTC
??? E0318 01:18:58.625149 1154438 certchains.go:122] [kube-control-plane-signer kube-scheduler] rotate at: 2023-11-17 10:01:46 +0000 UTC
??? E0318 01:18:58.625154 1154438 certchains.go:122] [kube-control-plane-signer route-controller-manager] rotate at: 2023-11-17 10:01:47 +0000 UTC
??? E0318 01:18:58.625158 1154438 certchains.go:122] [kubelet-signer] rotate at: 2023-11-17 10:01:48 +0000 UTC
??? E0318 01:18:58.625162 1154438 certchains.go:122] [kubelet-signer kube-csr-signer] rotate at: 2023-11-17 10:01:48 +0000 UTC
??? E0318 01:18:58.625168 1154438 certchains.go:122] [kubelet-signer kube-csr-signer kubelet-client] rotate at: 2023-11-17 10:01:48 +0000 UTC
??? E0318 01:18:58.625173 1154438 certchains.go:122] [kubelet-signer kube-csr-signer kubelet-server] rotate at: 2023-11-17 10:01:49 +0000 UTC
??? E0318 01:18:58.625178 1154438 certchains.go:122] [service-ca] rotate at: 2032-03-19 10:01:49 +0000 UTC
??? E0318 01:18:58.625183 1154438 certchains.go:122] [service-ca route-controller-manager-serving] rotate at: 2023-11-17 10:01:50 +0000 UTC
??? I0318 01:18:58.625352 1154438 run.go:127] Started service-manager
network-configuration I0318 01:18:58.625490 1154438 manager.go:114] Starting network-configuration
network-configuration I0318 01:18:58.625828 1154438 netconfig.go:74] "network-configuration" is ready
etcd I0318 01:18:58.625866 1154438 manager.go:114] Starting etcd
sysconfwatch-controller I0318 01:18:58.625886 1154438 manager.go:114] Starting sysconfwatch-controller
sysconfwatch-controller I0318 01:18:58.625906 1154438 sysconfwatch_linux.go:89] starting sysconfwatch-controller with IP address "173.16.11.20"
sysconfwatch-controller I0318 01:18:58.625916 1154438 sysconfwatch_linux.go:95] sysconfwatch-controller is ready
{"level":"info","ts":"2023-03-18T01:18:58.626+0530","caller":"embed/etcd.go:131","msg":"configuring peer listeners","listen-peer-urls":["https://173.16.11.20:2380"]}
{"level":"info","ts":"2023-03-18T01:18:58.626+0530","caller":"embed/etcd.go:481","msg":"starting with peer TLS","tls-info":"cert = /var/lib/microshift/certs/etcd-signer/etcd-peer/peer.crt, key = /var/lib/microshift/certs/etcd-signer/etcd-peer/peer.key, client-cert=, client-key=, trusted-ca = /var/lib/microshift/certs/etcd-signer/ca.crt, client-cert-auth = false, crl-file = ","cipher-suites":["TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384","TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384","TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256","TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256","TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305","TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305"]}
{"level":"info","ts":"2023-03-18T01:18:58.627+0530","caller":"embed/etcd.go:139","msg":"configuring client listeners","listen-client-urls":["https://127.0.0.1:2379","https://173.16.11.20:2379"]}
{"level":"info","ts":"2023-03-18T01:18:58.627+0530","caller":"embed/etcd.go:309","msg":"starting an etcd server","etcd-version":"3.5.3","git-sha":"Not provided (use ./build instead of go build)","go-version":"go1.19.4","go-os":"linux","go-arch":"amd64","max-cpu-set":4,"max-cpu-available":4,"member-initialized":true,"name":"microshift","data-dir":"/var/lib/microshift/etcd","wal-dir":"","wal-dir-dedicated":"","member-dir":"/var/lib/microshift/etcd/member","force-new-cluster":false,"heartbeat-interval":"100ms","election-timeout":"1s","initial-election-tick-advance":true,"snapshot-count":100000,"snapshot-catchup-entries":5000,"initial-advertise-peer-urls":["https://173.16.11.20:2380"],"listen-peer-urls":["https://173.16.11.20:2380"],"advertise-client-urls":["https://173.16.11.20:2379"],"listen-client-urls":["https://127.0.0.1:2379","https://173.16.11.20:2379"],"listen-metrics-urls":["https://127.0.0.1:2381"],"cors":[""],"host-whitelist":[""],"initial-cluster":"","initial-cluster-state":"new","initial-cluster-token":"","quota-size-bytes":2147483648,"pre-vote":true,"initial-corrupt-check":false,"corrupt-check-time-interval":"0s","auto-compaction-mode":"","auto-compaction-retention":"0s","auto-compaction-interval":"0s","discovery-url":"","discovery-proxy":"","downgrade-check-interval":"5s","max-learners":1}
{"level":"info","ts":"2023-03-18T01:18:58.628+0530","caller":"etcdserver/backend.go:81","msg":"opened backend db","path":"/var/lib/microshift/etcd/member/snap/db","took":"257.01µs"}
{"level":"info","ts":"2023-03-18T01:18:58.656+0530","caller":"etcdserver/server.go:532","msg":"No snapshot found. Recovering WAL from scratch!"}
{"level":"info","ts":"2023-03-18T01:18:58.688+0530","caller":"etcdserver/raft.go:483","msg":"restarting local member","cluster-id":"69a5be14589482e1","local-member-id":"8c1bff2098e6483a","commit-index":21898}
{"level":"info","ts":"2023-03-18T01:18:58.689+0530","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8c1bff2098e6483a switched to configuration voters=()"}
{"level":"info","ts":"2023-03-18T01:18:58.689+0530","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8c1bff2098e6483a became follower at term 10949"}
{"level":"info","ts":"2023-03-18T01:18:58.689+0530","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"newRaft 8c1bff2098e6483a [peers: [], term: 10949, commit: 21898, applied: 0, lastindex: 21898, lastterm: 10949]"}
{"level":"warn","ts":"2023-03-18T01:18:58.690+0530","caller":"auth/store.go:1220","msg":"simple token is not cryptographically signed"}
{"level":"info","ts":"2023-03-18T01:18:58.690+0530","caller":"mvcc/kvstore.go:415","msg":"kvstore restored","current-rev":1}
{"level":"info","ts":"2023-03-18T01:18:58.691+0530","caller":"etcdserver/quota.go:94","msg":"enabled backend quota with default value","quota-name":"v3-applier","quota-size-bytes":2147483648,"quota-size":"2.1 GB"}
{"level":"info","ts":"2023-03-18T01:18:58.691+0530","caller":"etcdserver/server.go:860","msg":"starting etcd server","local-member-id":"8c1bff2098e6483a","local-server-version":"3.5.3","cluster-version":"to_be_decided"}
{"level":"info","ts":"2023-03-18T01:18:58.693+0530","caller":"etcdserver/server.go:761","msg":"starting initial election tick advance","election-ticks":10}
{"level":"info","ts":"2023-03-18T01:18:58.693+0530","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8c1bff2098e6483a switched to configuration voters=(10095943505150363706)"}
{"level":"info","ts":"2023-03-18T01:18:58.693+0530","caller":"membership/cluster.go:421","msg":"added member","cluster-id":"69a5be14589482e1","local-member-id":"8c1bff2098e6483a","added-peer-id":"8c1bff2098e6483a","added-peer-peer-urls":["https://173.16.11.20:2380"],"added-peer-is-learner":false}
{"level":"info","ts":"2023-03-18T01:18:58.693+0530","caller":"membership/cluster.go:587","msg":"set initial cluster version","cluster-id":"69a5be14589482e1","local-member-id":"8c1bff2098e6483a","cluster-version":"3.5"}
{"level":"info","ts":"2023-03-18T01:18:58.693+0530","caller":"api/capability.go:75","msg":"enabled capabilities for version","cluster-version":"3.5"}
{"level":"info","ts":"2023-03-18T01:18:58.695+0530","caller":"embed/etcd.go:690","msg":"starting with client TLS","tls-info":"cert = /var/lib/microshift/certs/etcd-signer/etcd-serving/peer.crt, key = /var/lib/microshift/certs/etcd-signer/etcd-serving/peer.key, client-cert=, client-key=, trusted-ca = /var/lib/microshift/certs/etcd-signer/ca.crt, client-cert-auth = false, crl-file = ","cipher-suites":["TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384","TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384","TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256","TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256","TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305","TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305"]}
{"level":"info","ts":"2023-03-18T01:18:58.695+0530","caller":"embed/etcd.go:278","msg":"now serving peer/client/metrics","local-member-id":"8c1bff2098e6483a","initial-advertise-peer-urls":["https://173.16.11.20:2380"],"listen-peer-urls":["https://173.16.11.20:2380"],"advertise-client-urls":["https://173.16.11.20:2379"],"listen-client-urls":["https://127.0.0.1:2379","https://173.16.11.20:2379"],"listen-metrics-urls":["https://127.0.0.1:2381"]}
{"level":"info","ts":"2023-03-18T01:18:58.695+0530","caller":"embed/etcd.go:583","msg":"serving peer traffic","address":"173.16.11.20:2380"}
{"level":"info","ts":"2023-03-18T01:18:58.695+0530","caller":"embed/etcd.go:555","msg":"cmux::serve","address":"173.16.11.20:2380"}
{"level":"info","ts":"2023-03-18T01:18:58.695+0530","caller":"embed/etcd.go:765","msg":"serving metrics","address":"https://127.0.0.1:2381"}
{"level":"info","ts":"2023-03-18T01:19:00.290+0530","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8c1bff2098e6483a is starting a new election at term 10949"}
{"level":"info","ts":"2023-03-18T01:19:00.291+0530","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8c1bff2098e6483a became pre-candidate at term 10949"}
{"level":"info","ts":"2023-03-18T01:19:00.291+0530","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8c1bff2098e6483a received MsgPreVoteResp from 8c1bff2098e6483a at term 10949"}
{"level":"info","ts":"2023-03-18T01:19:00.291+0530","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8c1bff2098e6483a became candidate at term 10950"}
{"level":"info","ts":"2023-03-18T01:19:00.291+0530","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8c1bff2098e6483a received MsgVoteResp from 8c1bff2098e6483a at term 10950"}
{"level":"info","ts":"2023-03-18T01:19:00.291+0530","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8c1bff2098e6483a became leader at term 10950"}
{"level":"info","ts":"2023-03-18T01:19:00.291+0530","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"raft.node: 8c1bff2098e6483a elected leader 8c1bff2098e6483a at term 10950"}
{"level":"info","ts":"2023-03-18T01:19:00.293+0530","caller":"embed/serve.go:98","msg":"ready to serve client requests"}
{"level":"info","ts":"2023-03-18T01:19:00.293+0530","caller":"etcdserver/server.go:2051","msg":"published local member to cluster through raft","local-member-id":"8c1bff2098e6483a","local-member-attributes":"{Name:microshift ClientURLs:[https://173.16.11.20:2379]}","request-path":"/0/members/8c1bff2098e6483a/attributes","cluster-id":"69a5be14589482e1","publish-timeout":"7s"}
etcd I0318 01:19:00.293736 1154438 etcd.go:103] etcd is ready
{"level":"info","ts":"2023-03-18T01:19:00.293+0530","caller":"embed/serve.go:98","msg":"ready to serve client requests"}
{"level":"info","ts":"2023-03-18T01:19:00.296+0530","caller":"embed/serve.go:188","msg":"serving client traffic securely","address":"127.0.0.1:2379"}
{"level":"info","ts":"2023-03-18T01:19:00.296+0530","caller":"embed/serve.go:188","msg":"serving client traffic securely","address":"173.16.11.20:2379"}
kube-apiserver I0318 01:19:00.296302 1154438 manager.go:114] Starting kube-apiserver
Flag --openshift-config has been deprecated, to be removed
kube-apiserver I0318 01:19:00.298286 1154438 flags.go:64] FLAG: --add_dir_header="false"
kube-apiserver I0318 01:19:00.298303 1154438 flags.go:64] FLAG: --admission-control="[]"
kube-apiserver I0318 01:19:00.298317 1154438 flags.go:64] FLAG: --admission-control-config-file=""
kube-apiserver I0318 01:19:00.298326 1154438 flags.go:64] FLAG: --advertise-address="
@skylabreddy , your issue should be fixed by https://github.com/openshift/microshift/pull/1505 which will be in a z-stream release soon (I don't know if it made the deadline for this week, so it might be in next week's build).
@dhellmann Is there any Tentative dates to fix the issue? is there any option to install old version of MicroShift?
@dhellmann I see still same issue with current release also, microshift-4.1.2.8.
Service start issue.
Thanks Skylab
Mar 21 19:41:20 microshift microshift[3888540]: kube-apiserver E0321 19:41:20.616357 3888540 manager.go:116] service kube-apiserver exited with error: failed to initialize admission: failed to initialize admission plugin "PodSecurity": PodSecurity configuration error: [enforce: must be one of privileged, baseline, restricted, enforce-version: must be "latest" or "v1.x", audit: must be one of privileged, baseline, restricted, audit-version: must be "latest" or "v1.x", warn: must be one of privileged, baseline, restricted, warn-version: must be "latest" or "v1.x"], stopping MicroShift Mar 21 19:41:23 microshift microshift[3888643]: Error: failed to initialize admission: failed to initialize admission plugin "PodSecurity": PodSecurity configuration error: [enforce: must be one of privileged, baseline, restricted, enforce-version: must be "latest" or "v1.x", audit: must be one of privileged, baseline, restricted, audit-version: must be "latest" or "v1.x", warn: must be one of privileged, baseline, restricted, warn-version: must be "latest" or "v1.x"]
I think we missed the build pipeline this week by a couple of hours, so it should be in the next build.