actions-runner-controller icon indicating copy to clipboard operation
actions-runner-controller copied to clipboard
verified publisher icon actions

RUNNER_TOKEN is not set for RunnerSet on 0.25.2

Open erypolovina opened this issue 3 years ago • 4 comments

Controller Version

0.25.2

Helm Chart Version

0.20.2

CertManager Version

N/A

Deployment Method

Helm

cert-manager installation

I am not using cert-manager.

Checks

  • [X] This isn't a question or user support case (For Q&A and community support, go to Discussions. It might also be a good idea to contract with any of contributors and maintainers if your business is so critical and therefore you need priority support
  • [X] I've read releasenotes before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes
  • [X] My actions-runner-controller version (v0.x.y) does support the feature
  • [X] I've already upgraded ARC (including the CRDs, see charts/actions-runner-controller/docs/UPGRADING.md for details) to the latest and it didn't fix the issue

Resource Definitions

Not working

apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerSet
metadata:
  name: github-actions-runner-set
spec:
  replicas: 1
  enterprise: <redacted>
  serviceName: github-actions-runner-set
  ephemeral: false
  selector:
    matchLabels:
      app: github-actions-runner-set
  template:
    metadata:
      labels:
        app: github-actions-runner-set
    spec:
      containers:
        - name: runner
          volumeMounts:
          - name: artifactory-creds
            mountPath: /home/runner/.docker/
          env:
            - name: RUNNER_FEATURE_FLAG_EPHEMERAL
              value: "true"
            - name: RUNNER_EPHEMERAL
              value: "true"
        - name: docker
      volumes:
        - name: artifactory-creds
          secret:
            secretName: artifactory-credentials
            items:
              - key: .dockerconfigjson
                path: config.json

working

apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
  name: github-actions-runner-deployment
spec:
  replicas: 1
  template:
    spec:
      enterprise: <redacted>
      ephemeral: true
      containers:
        - name: runner
          volumeMounts:
            - name: artifactory-creds
              mountPath: /home/runner/.docker/
          env:
          - name: RUNNER_FEATURE_FLAG_EPHEMERAL
            value: "true"
          - name: RUNNER_EPHEMERAL
            value: "true"
        - name: docker
      volumes:
        - name: artifactory-creds
          secret:
            secretName: artifactory-credentials
            items:
              - key: .dockerconfigjson
                path: config.json

To Reproduce

Deploy both RunnerSet and RunnerDeployment and observe that runnerset fails to start the runner as the RUNNER_NAME and RUNNER_TOKEN environment variables are not injected by ARC properly.

Describe the bug

We recently upgraded our ARC to helm chart from 0.15.1->0.20.2. When on 0.15.1, we were deploying the runner pods with RunnerSets and HorizontalRunnerAutoscaler.

After the upgrade (and fixing any new spec changes), we noticed we could not create pods using RunnerSet as RUNNER_NAME and RUNNER_TOKEN were not being injected into the running pod causing an ERROR or CrashLoopBackOff. This feature was working for us in 0.15.1.

When testing the same spec but for RunnerDeployment, it was working and creating the Runner pods correctly.

Describe the expected behavior

RUNNER_TOKEN and RUNNER_NAME should be injected into the Runner pods created by RunnerSet

Controller Logs

Controller logs:


I0829 17:57:58.328242       1 request.go:601] Waited for 1.047060206s due to client-side throttling, not priority and fairness, request: GET:https://10.8.1.1:443/apis/coordination.k8s.io/v1?timeout=32s
2022-08-29T17:57:58Z    INFO    controller-runtime.metrics      Metrics server is starting to listen    {"addr": "0.0.0.0:8080"}
2022-08-29T17:57:58Z    INFO    actions-runner-controller       Initializing actions-runner-controller  {"github-api-cache-duration": "9m50s", "default-scale-down-delay": "10m0s", "sync-period": "10m0s", "default-runner-image": "docker.io/summerwind/actions-runner:latest", "default-docker-image": "docker.io/docker:dind", "common-runnner-labels": null, "leader-election-enabled": true, "leader-election-id": "actions-runner-controller", "watch-namespace": "github-actions-runner"}
2022-08-29T17:57:58Z    INFO    controller-runtime.builder      Registering a mutating webhook  {"GVK": "actions.summerwind.dev/v1alpha1, Kind=Runner", "path": "/mutate-actions-summerwind-dev-v1alpha1-runner"}
2022-08-29T17:57:58Z    INFO    controller-runtime.webhook      Registering webhook     {"path": "/mutate-actions-summerwind-dev-v1alpha1-runner"}
2022-08-29T17:57:58Z    INFO    controller-runtime.builder      Registering a validating webhook        {"GVK": "actions.summerwind.dev/v1alpha1, Kind=Runner", "path": "/validate-actions-summerwind-dev-v1alpha1-runner"}
2022-08-29T17:57:58Z    INFO    controller-runtime.webhook      Registering webhook     {"path": "/validate-actions-summerwind-dev-v1alpha1-runner"}
2022-08-29T17:57:58Z    INFO    controller-runtime.builder      Registering a mutating webhook  {"GVK": "actions.summerwind.dev/v1alpha1, Kind=RunnerDeployment", "path": "/mutate-actions-summerwind-dev-v1alpha1-runnerdeployment"}
2022-08-29T17:57:58Z    INFO    controller-runtime.webhook      Registering webhook     {"path": "/mutate-actions-summerwind-dev-v1alpha1-runnerdeployment"}
2022-08-29T17:57:58Z    INFO    controller-runtime.builder      Registering a validating webhook        {"GVK": "actions.summerwind.dev/v1alpha1, Kind=RunnerDeployment", "path": "/validate-actions-summerwind-dev-v1alpha1-runnerdeployment"}
2022-08-29T17:57:58Z    INFO    controller-runtime.webhook      Registering webhook     {"path": "/validate-actions-summerwind-dev-v1alpha1-runnerdeployment"}
2022-08-29T17:57:58Z    INFO    controller-runtime.builder      Registering a mutating webhook  {"GVK": "actions.summerwind.dev/v1alpha1, Kind=RunnerReplicaSet", "path": "/mutate-actions-summerwind-dev-v1alpha1-runnerreplicaset"}
2022-08-29T17:57:58Z    INFO    controller-runtime.webhook      Registering webhook     {"path": "/mutate-actions-summerwind-dev-v1alpha1-runnerreplicaset"}
2022-08-29T17:57:58Z    INFO    controller-runtime.builder      Registering a validating webhook        {"GVK": "actions.summerwind.dev/v1alpha1, Kind=RunnerReplicaSet", "path": "/validate-actions-summerwind-dev-v1alpha1-runnerreplicaset"}
2022-08-29T17:57:58Z    INFO    controller-runtime.webhook      Registering webhook     {"path": "/validate-actions-summerwind-dev-v1alpha1-runnerreplicaset"}
2022-08-29T17:57:58Z    INFO    controller-runtime.webhook      Registering webhook     {"path": "/mutate-runner-set-pod"}
2022-08-29T17:57:58Z    INFO    actions-runner-controller       starting manager
2022-08-29T17:57:58Z    INFO    controller-runtime.webhook.webhooks     Starting webhook server
2022-08-29T17:57:58Z    INFO    Starting server {"path": "/metrics", "kind": "metrics", "addr": "[::]:8080"}
2022-08-29T17:57:58Z    INFO    controller-runtime.certwatcher  Updated current TLS certificate
2022-08-29T17:57:58Z    INFO    controller-runtime.webhook      Serving webhook server  {"host": "", "port": 9443}
2022-08-29T17:57:58Z    INFO    controller-runtime.certwatcher  Starting certificate watcher
I0829 17:57:58.784435       1 leaderelection.go:248] attempting to acquire leader lease github-actions-runner/actions-runner-controller...
I0829 17:58:14.035337       1 leaderelection.go:258] successfully acquired lease github-actions-runner/actions-runner-controller
2022-08-29T17:58:14Z    DEBUG   events  Normal  {"object": {"kind":"Lease","namespace":"github-actions-runner","name":"actions-runner-controller","uid":"5eb6c52c-3b0b-4c26-bc4d-c49577dad490","apiVersion":"coordination.k8s.io/v1","resourceVersion":"257205820"}, "reason": "LeaderElection", "message": "github-actions-runner-controller-95b5c4c9c-72g5x_691c0161-3fa9-47a6-998e-09791e20a099 became leader"}
2022-08-29T17:58:14Z    INFO    Starting EventSource    {"controller": "runnerreplicaset-controller", "controllerGroup": "actions.summerwind.dev", "controllerKind": "RunnerReplicaSet", "source": "kind source: *v1alpha1.RunnerReplicaSet"}
2022-08-29T17:58:14Z    INFO    Starting EventSource    {"controller": "runnerreplicaset-controller", "controllerGroup": "actions.summerwind.dev", "controllerKind": "RunnerReplicaSet", "source": "kind source: *v1alpha1.Runner"}
2022-08-29T17:58:14Z    INFO    Starting Controller     {"controller": "runnerreplicaset-controller", "controllerGroup": "actions.summerwind.dev", "controllerKind": "RunnerReplicaSet"}
2022-08-29T17:58:14Z    INFO    Starting EventSource    {"controller": "runner-controller", "controllerGroup": "actions.summerwind.dev", "controllerKind": "Runner", "source": "kind source: *v1alpha1.Runner"}
2022-08-29T17:58:14Z    INFO    Starting EventSource    {"controller": "runner-controller", "controllerGroup": "actions.summerwind.dev", "controllerKind": "Runner", "source": "kind source: *v1.Pod"}
2022-08-29T17:58:14Z    INFO    Starting Controller     {"controller": "runner-controller", "controllerGroup": "actions.summerwind.dev", "controllerKind": "Runner"}
2022-08-29T17:58:14Z    INFO    Starting EventSource    {"controller": "runnerpersistentvolumeclaim-controller", "controllerGroup": "", "controllerKind": "PersistentVolumeClaim", "source": "kind source: *v1.PersistentVolumeClaim"}
2022-08-29T17:58:14Z    INFO    Starting Controller     {"controller": "runnerpersistentvolumeclaim-controller", "controllerGroup": "", "controllerKind": "PersistentVolumeClaim"}
2022-08-29T17:58:14Z    INFO    Starting EventSource    {"controller": "runnerpersistentvolume-controller", "controllerGroup": "", "controllerKind": "PersistentVolume", "source": "kind source: *v1.PersistentVolume"}
2022-08-29T17:58:14Z    INFO    Starting EventSource    {"controller": "runnerpod-controller", "controllerGroup": "", "controllerKind": "Pod", "source": "kind source: *v1.Pod"}
2022-08-29T17:58:14Z    INFO    Starting Controller     {"controller": "runnerpersistentvolume-controller", "controllerGroup": "", "controllerKind": "PersistentVolume"}
2022-08-29T17:58:14Z    INFO    Starting Controller     {"controller": "runnerpod-controller", "controllerGroup": "", "controllerKind": "Pod"}
2022-08-29T17:58:14Z    INFO    Starting EventSource    {"controller": "horizontalrunnerautoscaler-controller", "controllerGroup": "actions.summerwind.dev", "controllerKind": "HorizontalRunnerAutoscaler", "source": "kind source: *v1alpha1.HorizontalRunnerAutoscaler"}
2022-08-29T17:58:14Z    INFO    Starting Controller     {"controller": "horizontalrunnerautoscaler-controller", "controllerGroup": "actions.summerwind.dev", "controllerKind": "HorizontalRunnerAutoscaler"}
2022-08-29T17:58:14Z    INFO    Starting EventSource    {"controller": "runnerset-controller", "controllerGroup": "actions.summerwind.dev", "controllerKind": "RunnerSet", "source": "kind source: *v1alpha1.RunnerSet"}
2022-08-29T17:58:14Z    INFO    Starting EventSource    {"controller": "runnerset-controller", "controllerGroup": "actions.summerwind.dev", "controllerKind": "RunnerSet", "source": "kind source: *v1.StatefulSet"}
2022-08-29T17:58:14Z    INFO    Starting Controller     {"controller": "runnerset-controller", "controllerGroup": "actions.summerwind.dev", "controllerKind": "RunnerSet"}
2022-08-29T17:58:14Z    INFO    Starting EventSource    {"controller": "runnerdeployment-controller", "controllerGroup": "actions.summerwind.dev", "controllerKind": "RunnerDeployment", "source": "kind source: *v1alpha1.RunnerDeployment"}
2022-08-29T17:58:14Z    INFO    Starting EventSource    {"controller": "runnerdeployment-controller", "controllerGroup": "actions.summerwind.dev", "controllerKind": "RunnerDeployment", "source": "kind source: *v1alpha1.RunnerReplicaSet"}
2022-08-29T17:58:14Z    INFO    Starting Controller     {"controller": "runnerdeployment-controller", "controllerGroup": "actions.summerwind.dev", "controllerKind": "RunnerDeployment"}
2022-08-29T17:58:14Z    INFO    Starting workers        {"controller": "runnerpersistentvolume-controller", "controllerGroup": "", "controllerKind": "PersistentVolume", "worker count": 1}
2022-08-29T17:58:14Z    INFO    Starting workers        {"controller": "runner-controller", "controllerGroup": "actions.summerwind.dev", "controllerKind": "Runner", "worker count": 1}
2022-08-29T17:58:14Z    INFO    Starting workers        {"controller": "horizontalrunnerautoscaler-controller", "controllerGroup": "actions.summerwind.dev", "controllerKind": "HorizontalRunnerAutoscaler", "worker count": 1}
2022-08-29T17:58:14Z    INFO    Starting workers        {"controller": "runnerpersistentvolumeclaim-controller", "controllerGroup": "", "controllerKind": "PersistentVolumeClaim", "worker count": 1}
2022-08-29T17:58:14Z    INFO    Starting workers        {"controller": "runnerpod-controller", "controllerGroup": "", "controllerKind": "Pod", "worker count": 1}
2022-08-29T17:58:14Z    INFO    Starting workers        {"controller": "runnerset-controller", "controllerGroup": "actions.summerwind.dev", "controllerKind": "RunnerSet", "worker count": 1}
2022-08-29T17:58:14Z    INFO    Starting workers        {"controller": "runnerreplicaset-controller", "controllerGroup": "actions.summerwind.dev", "controllerKind": "RunnerReplicaSet", "worker count": 1}
2022-08-29T17:58:14Z    INFO    Starting workers        {"controller": "runnerdeployment-controller", "controllerGroup": "actions.summerwind.dev", "controllerKind": "RunnerDeployment", "worker count": 1}
2022-08-29T18:00:14Z    INFO    actions-runner-controller.runnerdeployment      Created runnerreplicaset        {"runnerdeployment": "github-actions-runner/github-actions-runner-deployment", "runnerreplicaset": "github-actions-runner-deployment-xjm86"}
2022-08-29T18:00:14Z    DEBUG   actions-runner-controller.runnerreplicaset      Created replica(s)      {"runnerreplicaset": "github-actions-runner/github-actions-runner-deployment-xjm86", "lastSyncTime": null, "effectiveTime": "<nil>", "templateHashDesired": "66484d585", "replicasDesired": 1, "replicasPending": 0, "replicasRunning": 0, "replicasMaybeRunning": 0, "templateHashObserved": [], "created": 1}
2022-08-29T18:00:14Z    DEBUG   actions-runner-controller.runnerreplicaset      Skipped reconcilation because owner is not synced yet   {"runnerreplicaset": "github-actions-runner/github-actions-runner-deployment-xjm86", "owner": "github-actions-runner/github-actions-runner-deployment-xjm86-vpb64", "pods": null}
2022-08-29T18:00:14Z    DEBUG   actions-runner-controller.runnerreplicaset      Skipped reconcilation because owner is not synced yet   {"runnerreplicaset": "github-actions-runner/github-actions-runner-deployment-xjm86", "owner": "github-actions-runner/github-actions-runner-deployment-xjm86-vpb64", "pods": null}
2022-08-29T18:00:14Z    INFO    actions-runner-controller.runner        Updated registration token      {"runner": "github-actions-runner-deployment-xjm86-vpb64", "repository": ""}
2022-08-29T18:00:14Z    DEBUG   events  Normal  {"object": {"kind":"Runner","namespace":"github-actions-runner","name":"github-actions-runner-deployment-xjm86-vpb64","uid":"69c6dd2f-fe1a-4f20-8b56-f74ab5c0c466","apiVersion":"actions.summerwind.dev/v1alpha1","resourceVersion":"257206775"}, "reason": "RegistrationTokenUpdated", "message": "Successfully update registration token"}
2022-08-29T18:00:14Z    DEBUG   actions-runner-controller.runnerreplicaset      Skipped reconcilation because owner is not synced yet   {"runnerreplicaset": "github-actions-runner/github-actions-runner-deployment-xjm86", "owner": "github-actions-runner/github-actions-runner-deployment-xjm86-vpb64", "pods": null}
2022-08-29T18:00:14Z    INFO    actions-runner-controller.runner        Created runner pod      {"runner": "github-actions-runner/github-actions-runner-deployment-xjm86-vpb64", "repository": ""}
2022-08-29T18:00:14Z    DEBUG   events  Normal  {"object": {"kind":"Runner","namespace":"github-actions-runner","name":"github-actions-runner-deployment-xjm86-vpb64","uid":"69c6dd2f-fe1a-4f20-8b56-f74ab5c0c466","apiVersion":"actions.summerwind.dev/v1alpha1","resourceVersion":"257206782"}, "reason": "PodCreated", "message": "Created pod 'github-actions-runner-deployment-xjm86-vpb64'"}
2022-08-29T18:00:17Z    DEBUG   actions-runner-controller.runner        Runner appears to have been registered and running.     {"runner": "github-actions-runner/github-actions-runner-deployment-xjm86-vpb64", "podCreationTimestamp": "2022-08-29 18:00:14 +0000 UTC"}
2022-08-29T18:00:20Z    DEBUG   actions-runner-controller.runnerset     Created replica(s)      {"runnerset": "github-actions-runner/github-actions-runner-set", "lastSyncTime": null, "effectiveTime": "<nil>", "templateHashDesired": "7775c79f48", "replicasDesired": 1, "replicasPending": 0, "replicasRunning": 0, "replicasMaybeRunning": 0, "templateHashObserved": [], "created": 1}
2022-08-29T18:00:20Z    DEBUG   actions-runner-controller.runnerset     Skipped reconcilation because owner is not synced yet   {"runnerset": "github-actions-runner/github-actions-runner-set", "owner": "github-actions-runner/github-actions-runner-set-9p6nl", "pods": null}

Running kubectl describe on runnerset pod

Name:         github-actions-runner-set-9p6nl-0
Namespace:    github-actions-runner
Priority:     0
Node:         gke-infra-dev-infra-node-pool-eaf2647d-nm1n/10.8.13.206
Start Time:   Mon, 29 Aug 2022 11:00:20 -0700
Labels:       actions-runner=
              actions-runner-controller/inject-registration-token=true
              app=github-actions-runner-set
              controller-revision-hash=github-actions-runner-set-9p6nl-69d45777b
              runner-template-hash=7775c79f48
              runnerset-name=github-actions-runner-set
              statefulset.kubernetes.io/pod-name=github-actions-runner-set-9p6nl-0
Annotations:  cni.projectcalico.org/containerID: 933de3e24fbc4d47a637da978c1a307574d8a4cce8df21ca6f10baf23cd7f7f2
              cni.projectcalico.org/podIP: 10.8.32.165/32
              cni.projectcalico.org/podIPs: 10.8.32.165/32
Status:       Running
IP:           10.8.32.165
IPs:
  IP:           10.8.32.165
Controlled By:  StatefulSet/github-actions-runner-set-9p6nl
Containers:
  runner:
    Container ID:   containerd://040534ddd09ac34f14526958b352944c4748949e4ae9e136d452ed5a9bbb344b
    Image:          docker.io/summerwind/actions-runner:latest
    Image ID:       docker.io/summerwind/actions-runner@sha256:ea15a4ae3446465c376c63440b949bd76237592bf27d0d9bc6e270b5d76562e1
    Port:           <none>
    Host Port:      <none>
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Mon, 29 Aug 2022 11:36:51 -0700
      Finished:     Mon, 29 Aug 2022 11:36:51 -0700
    Ready:          False
    Restart Count:  12
    Environment:
      RUNNER_FEATURE_FLAG_EPHEMERAL:  true
      RUNNER_EPHEMERAL:               true
      RUNNER_ORG:
      RUNNER_REPO:
      RUNNER_ENTERPRISE:              <redacted>
      RUNNER_LABELS:
      RUNNER_GROUP:
      DOCKER_ENABLED:                 true
      DOCKERD_IN_RUNNER:              false
      GITHUB_URL:                     <redacted>
      RUNNER_WORKDIR:                 /runner/_work
      RUNNER_EPHEMERAL:               false
      DOCKER_HOST:                    tcp://localhost:2376
      DOCKER_TLS_VERIFY:              1
      DOCKER_CERT_PATH:               /certs/client
    Mounts:
      /certs/client from certs-client (ro)
      /home/runner/.docker/ from artifactory-creds (rw)
      /runner from runner (rw)
      /runner/_work from work (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-sq2v2 (ro)
  docker:
    Container ID:   containerd://ea7ab25c521ee68e375d8b8a5aeed7af4d24964565a86ced7f87ed8a1e84941a
    Image:          docker.io/docker:dind
    Image ID:       docker.io/docker@sha256:f92b5715c425e1066cab3a4d32be7b6ccee26c6a1ca3ca7544f86c6554df9424
    Port:           <none>
    Host Port:      <none>
    State:          Running
      Started:      Mon, 29 Aug 2022 11:00:20 -0700
    Ready:          True
    Restart Count:  0
    Environment:
      DOCKER_TLS_CERTDIR:  /certs
    Mounts:
      /certs/client from certs-client (rw)
      /runner from runner (rw)
      /runner/_work from work (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-sq2v2 (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  artifactory-creds:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  artifactory-credentials
    Optional:    false
  runner:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  work:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  certs-client:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  kube-api-access-sq2v2:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                    From               Message
  ----     ------     ----                   ----               -------
  Normal   Scheduled  39m                    default-scheduler  Successfully assigned github-actions-runner/github-actions-runner-set-9p6nl-0 to gke-infra-dev-infra-node-pool-eaf2647d-nm1n
  Normal   Started    39m                    kubelet            Started container docker
  Normal   Pulled     39m                    kubelet            Successfully pulled image "docker.io/summerwind/actions-runner:latest" in 65.304461ms
  Normal   Pulled     39m                    kubelet            Container image "docker.io/docker:dind" already present on machine
  Normal   Created    39m                    kubelet            Created container docker
  Normal   Pulled     39m                    kubelet            Successfully pulled image "docker.io/summerwind/actions-runner:latest" in 61.36297ms
  Normal   Pulled     39m                    kubelet            Successfully pulled image "docker.io/summerwind/actions-runner:latest" in 67.615471ms
  Normal   Created    38m (x4 over 39m)      kubelet            Created container runner
  Normal   Pulling    38m (x4 over 39m)      kubelet            Pulling image "docker.io/summerwind/actions-runner:latest"
  Normal   Started    38m (x4 over 39m)      kubelet            Started container runner
  Normal   Pulled     38m                    kubelet            Successfully pulled image "docker.io/summerwind/actions-runner:latest" in 65.969371ms
  Warning  BackOff    4m13s (x166 over 39m)  kubelet            Back-off restarting failed container

Running kubectl describe on runnerdeployment pod

Name:         github-actions-runner-deployment-xjm86-vpb64
Namespace:    github-actions-runner
Priority:     0
Node:         gke-infra-dev-infra-node-pool-56058494-mfn9/10.8.13.203
Start Time:   Mon, 29 Aug 2022 11:00:14 -0700
Labels:       actions-runner=
              actions-runner-controller/inject-registration-token=true
              pod-template-hash=8666d9584c
              runner-deployment-name=github-actions-runner-deployment
              runner-template-hash=66484d585
Annotations:  actions-runner/id: 1488
              cni.projectcalico.org/containerID: dfb78d99e82862f2eb8bac5cbace5fbb729ed036bdfa4e5be1a2b81b522f69df
              cni.projectcalico.org/podIP: 10.8.32.197/32
              cni.projectcalico.org/podIPs: 10.8.32.197/32
              sync-time: 2022-08-29T18:00:14Z
Status:       Running
IP:           10.8.32.197
IPs:
  IP:           10.8.32.197
Controlled By:  Runner/github-actions-runner-deployment-xjm86-vpb64
Containers:
  runner:
    Container ID:   containerd://cb259c883ebd5d728e04a5cc43e63f495a9d7db79f8ba2a209207479f6dd62bb
    Image:          docker.io/summerwind/actions-runner:latest
    Image ID:       docker.io/summerwind/actions-runner@sha256:ea15a4ae3446465c376c63440b949bd76237592bf27d0d9bc6e270b5d76562e1
    Port:           <none>
    Host Port:      <none>
    State:          Running
      Started:      Mon, 29 Aug 2022 11:00:16 -0700
    Ready:          True
    Restart Count:  0
    Environment:
      RUNNER_FEATURE_FLAG_EPHEMERAL:  true
      RUNNER_EPHEMERAL:               true
      RUNNER_ORG:
      RUNNER_REPO:
      RUNNER_ENTERPRISE:              <redacted>
      RUNNER_LABELS:
      RUNNER_GROUP:
      DOCKER_ENABLED:                 true
      DOCKERD_IN_RUNNER:              false
      GITHUB_URL:                     <redacted>
      RUNNER_WORKDIR:                 /runner/_work
      RUNNER_EPHEMERAL:               true
      DOCKER_HOST:                    tcp://localhost:2376
      DOCKER_TLS_VERIFY:              1
      DOCKER_CERT_PATH:               /certs/client
      RUNNER_NAME:                    github-actions-runner-deployment-xjm86-vpb64
      RUNNER_TOKEN:                   <redacted>
    Mounts:
      /certs/client from certs-client (ro)
      /home/runner/.docker/ from artifactory-creds (rw)
      /runner from runner (rw)
      /runner/_work from work (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-84plr (ro)
  docker:
    Container ID:   containerd://5ea6ce8cc60f720e31b00c684ce17742684040ccbf6c970aa234098578d3190b
    Image:          docker.io/docker:dind
    Image ID:       docker.io/docker@sha256:8d0980a0fc4ed2869ef9f8bd017a637a408fb4d038a64d490a6080c2318eacc8
    Port:           <none>
    Host Port:      <none>
    State:          Running
      Started:      Mon, 29 Aug 2022 11:00:17 -0700
    Ready:          True
    Restart Count:  0
    Environment:
      DOCKER_TLS_CERTDIR:  /certs
    Mounts:
      /certs/client from certs-client (rw)
      /runner from runner (rw)
      /runner/_work from work (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-84plr (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  runner:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  work:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  certs-client:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  artifactory-creds:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  artifactory-credentials
    Optional:    false
  kube-api-access-84plr:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  41m   default-scheduler  Successfully assigned github-actions-runner/github-actions-runner-deployment-xjm86-vpb64 to gke-infra-dev-infra-node-pool-56058494-mfn9
  Normal  Pulling    41m   kubelet            Pulling image "docker.io/summerwind/actions-runner:latest"
  Normal  Pulled     41m   kubelet            Successfully pulled image "docker.io/summerwind/actions-runner:latest" in 848.689786ms
  Normal  Created    41m   kubelet            Created container runner
  Normal  Started    41m   kubelet            Started container runner
  Normal  Pulled     41m   kubelet            Container image "docker.io/docker:dind" already present on machine
  Normal  Created    41m   kubelet            Created container docker
  Normal  Started    41m   kubelet            Started container docker

Runner Pod Logs

2022-08-29 18:43:42.606  DEBUG --- Github endpoint URL <GHES endpoint>
2022-08-29 18:43:42.607  ERROR --- RUNNER_NAME must be set

Additional Context

No response

erypolovina avatar Aug 29 '22 18:08 erypolovina

https://github.com/actions-runner-controller/actions-runner-controller/issues/1305#issuecomment-1103285294 docs need patching

toast-gear avatar Aug 30 '22 15:08 toast-gear

@toast-gear should that be a fix? because I have the same error, and NS is labeled correctly:

k describe ns github-action-runners                                                                                  
Name:         github-action-runners
Labels:       kubernetes.io/metadata.name=github-action-runners
              name=github-action-runners
...
Annotations:  name: github-action-runners
              owner: terraform/aws-applications
...
Status:       Active

dac73 avatar Aug 30 '22 15:08 dac73

interesting, what do your MutatingWebhookConfig kinds look like? Can you do a get and provide the yaml?

toast-gear avatar Aug 30 '22 15:08 toast-gear

Hah, I don't have it 🤔 Helm is deployed via terraform and not showing diff. I will get back here if redeployment fixes the issue. Oh well, at least I know in which direction to search now. Thanks.

dac73 avatar Aug 30 '22 15:08 dac73

@dac73 what did you find out?

smartinov avatar Nov 23 '22 17:11 smartinov

Nothing much. After redeployment, it worked. I had issues with MutatingWebhookConfig, but not sure what caused it. In the end, I reinstalled all EKS plugins and reinstalled ARC.

dac73 avatar Nov 23 '22 17:11 dac73

Yeah, I guess I have that problem with the mutating webhook since I had to delete it because it wasn't reachable. Thanks for the heads uP!

smartinov avatar Nov 23 '22 17:11 smartinov

fixed by running kubectl label ns github-actions-runner name=github-actions-runner

erypolovina avatar Mar 27 '23 22:03 erypolovina