actions-runner-controller
actions-runner-controller copied to clipboard
actions
ARC with AKS workload identity not working
Checks
- [X] I've already read https://github.com/actions/actions-runner-controller/blob/master/TROUBLESHOOTING.md and I'm sure my issue is not covered in the troubleshooting guide.
- [X] I'm not using a custom entrypoint in my runner image
Controller Version
0.27.0
Helm Chart Version
0.25.2
CertManager Version
No response
Deployment Method
Helm
cert-manager installation
Using AGIC + key vault cert (no issue)
Checks
- [X] This isn't a question or user support case (For Q&A and community support, go to Discussions. It might also be a good idea to contract with any of contributors and maintainers if your business is so critical and therefore you need priority support
- [X] I've read releasenotes before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes
- [X] My actions-runner-controller version (v0.x.y) does support the feature
- [X] I've already upgraded ARC (including the CRDs, see charts/actions-runner-controller/docs/UPGRADING.md for details) to the latest and it didn't fix the issue
- [X] I've migrated to the workflow job webhook event (if you using webhook driven scaling)
Resource Definitions
apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
name: datapipelines
spec:
template:
metadata:
labels:
app: datapipelines
azure.workload.identity/use: "true"
annotations:
azure.workload.identity/inject-proxy-sidecar: "true"
spec:
organization: organization
image:
imagePullPolicy: Always
serviceAccountName: datapipelines
labels:
- self-hosted
ephemeral: true
To Reproduce
1. Use an AKS cluster with workflow identity support enabled
2. Allow any job to queue and run (successful or not, makes no difference)
3. Job completes, the runner and related resources are not scaling down
Describe the bug
Pods are not scaling down with AKS workload identity
Describe the expected behavior
Pod Sclaes down after grace period
Whole Controller Logs
2023-01-23T12:07:57Z ERROR runnerreplicaset Failed to patch pod to have actions-runner/unregistration-request-timestamp annotation {"runnerreplicaset": "gitrunners/adfdatapipelines-simv2-runners-t5pb2", "lastSyncTime": "2023-01-23T11:50:56Z", "effectiveTime": "<nil>", "templateHashDesired": "6d74d7fd7b", "replicasDesired": 0, "replicasPending": 0, "replicasRunning": 0, "replicasMaybeRunning": 0, "templateHashObserved": ["6d74d7fd7b"], "owner": "gitrunners/adfdatapipelines-simv2-runners-t5pb2-zwl9b", "error": "Pod \"adfdatapipelines-simv2-runners-t5pb2-zwl9b\" is invalid: spec: Forbidden: pod updates may not change fields other than `spec.containers[*].image`, `spec.initContainers[*].image`, `spec.activeDeadlineSeconds`, `spec.tolerations` (only additions to existing tolerations) or `spec.terminationGracePeriodSeconds` (allow it to be set to 1 if it was previously negative)\n core.PodSpec{\n \tVolumes: {{Name: \"workload-socket\", VolumeSource: {EmptyDir: &{}}}, {Name: \"workload-certs\", VolumeSource: {EmptyDir: &{}}}, {Name: \"istio-envoy\", VolumeSource: {EmptyDir: &{Medium: \"Memory\"}}}, {Name: \"istio-data\", VolumeSource: {EmptyDir: &{}}}, ...},\n \tInitContainers: []core.Container{\n \t\t{Name: \"azwi-proxy-init\", Image: \"mcr.microsoft.com/oss/azure/workload-identity/proxy-init:v0.15.0\", Env: {{Name: \"PROXY_PORT\", Value: \"8000\"}, {Name: \"AZURE_CLIENT_ID\", Value: \"clientid\"}, {Name: \"AZURE_TENANT_ID\", Value: \"tenantid\"}, {Name: \"AZURE_FEDERATED_TOKEN_FILE\", Value: \"/var/run/secrets/azure/tokens/azure-identity-token\"}, ...}, VolumeMounts: {{Name: \"azure-identity-token\", ReadOnly: true, MountPath: \"/var/run/secrets/azure/tokens\"}, {Name: \"kube-api-access-fkxzq\", ReadOnly: true, MountPath: \"/var/run/secrets/kubernetes.io/serviceaccount\"}}, ...},\n \t\t{\n \t\t\t... // 5 identical fields\n \t\t\tPorts: nil,\n \t\t\tEnvFrom: nil,\n- \t\t\tEnv: []core.EnvVar{\n- \t\t\t\t{Name: \"AZURE_CLIENT_ID\", Value: \"clientid\"},\n- \t\t\t\t{Name: \"AZURE_TENANT_ID\", Value: \"tenantid\"},\n- \t\t\t\t{\n- \t\t\t\t\tName: \"AZURE_FEDERATED_TOKEN_FILE\",\n- \t\t\t\t\tValue: \"/var/run/secrets/azure/tokens/azure-identity-token\",\n- \t\t\t\t},\n- \t\t\t\t{Name: \"AZURE_AUTHORITY_HOST\", Value: \"https://login.microsoftonline.com/\"},\n- \t\t\t},\n+ \t\t\tEnv: nil,\n \t\t\tResources: {Limits: {s\"cpu\": {i: {...}, s: \"2\", Format: \"DecimalSI\"}, s\"memory\": {i: {...}, s: \"1Gi\", Format: \"BinarySI\"}}, Requests: {s\"cpu\": {i: {...}, s: \"10m\", Format: \"DecimalSI\"}, s\"memory\": {i: {...}, Format: \"BinarySI\"}}},\n \t\t\tVolumeMounts: []core.VolumeMount{\n \t\t\t\t{Name: \"kube-api-access-fkxzq\", ReadOnly: true, MountPath: \"/var/run/secrets/kubernetes.io/serviceaccount\"},\n- \t\t\t\t{\n- \t\t\t\t\tName: \"azure-identity-token\",\n- \t\t\t\t\tReadOnly: true,\n- \t\t\t\t\tMountPath: \"/var/run/secrets/azure/tokens\",\n- \t\t\t\t},\n \t\t\t},\n \t\t\tVolumeDevices: nil,\n \t\t\tLivenessProbe: nil,\n \t\t\t... // 10 identical fields\n \t\t},\n \t},\n \tContainers: []core.Container{\n \t\t{Name: \"runner\", Image: \"dbg-bigdata-docker-local.artifactory.dbgcloud.io/bdaa/aksrunner:\"..., Env: {{Name: \"http_proxy\", Value: \"proxy\"}, {Name: \"https_proxy\", Value: \"proxy\"}, {Name: \"no_proxy\", Value: \"proxy,.\"...}, {Name: \"RUNNER_ORG\", Value: \"organisation\"}, ...}, VolumeMounts: {{Name: \"runner\", MountPath: \"/runner\"}, {Name: \"work\", MountPath: \"/runner/_work\"}, {Name: \"certs-client\", ReadOnly: true, MountPath: \"/certs/client\"}, {Name: \"kube-api-access-fkxzq\", ReadOnly: true, MountPath: \"/var/run/secrets/kubernetes.io/serviceaccount\"}, ...}, ...},\n \t\t{Name: \"docker\", Image: \"remote-docker.artifactory.dbgcloud.io/docker:dind\", Env: {{Name: \"DOCKER_TLS_CERTDIR\", Value: \"/certs\"}, {Name: \"AZURE_CLIENT_ID\", Value: \"clientid\"}, {Name: \"AZURE_TENANT_ID\", Value: \"tenantid\"}, {Name: \"AZURE_FEDERATED_TOKEN_FILE\", Value: \"/var/run/secrets/azure/tokens/azure-identity-token\"}, ...}, VolumeMounts: {{Name: \"runner\", MountPath: \"/runner\"}, {Name: \"certs-client\", MountPath: \"/certs/client\"}, {Name: \"work\", MountPath: \"/runner/_work\"}, {Name: \"kube-api-access-fkxzq\", ReadOnly: true, MountPath: \"/var/run/secrets/kubernetes.io/serviceaccount\"}, ...}, ...},\n \t\t{Name: \"azwi-proxy\", Image: \"mcr.microsoft.com/oss/azure/workload-identity/proxy:v0.15.0\", Args: {\"--proxy-port=8000\"}, Ports: {{ContainerPort: 8000, Protocol: \"TCP\"}}, ...},\n \t\t{\n \t\t\t... // 5 identical fields\n \t\t\tPorts: {{Name: \"http-envoy-prom\", ContainerPort: 15090, Protocol: \"TCP\"}},\n \t\t\tEnvFrom: nil,\n \t\t\tEnv: []core.EnvVar{\n \t\t\t\t... // 15 identical elements\n \t\t\t\t{Name: \"ISTIO_META_MESH_ID\", Value: \"cluster.local\"},\n \t\t\t\t{Name: \"TRUST_DOMAIN\", Value: \"cluster.local\"},\n- \t\t\t\t{Name: \"AZURE_CLIENT_ID\", Value: \"clientid\"},\n- \t\t\t\t{Name: \"AZURE_TENANT_ID\", Value: \"tenantid\"},\n- \t\t\t\t{\n- \t\t\t\t\tName: \"AZURE_FEDERATED_TOKEN_FILE\",\n- \t\t\t\t\tValue: \"/var/run/secrets/azure/tokens/azure-identity-token\",\n- \t\t\t\t},\n- \t\t\t\t{Name: \"AZURE_AUTHORITY_HOST\", Value: \"https://login.microsoftonline.com/\"},\n \t\t\t},\n \t\t\tResources: {Limits: {s\"cpu\": {i: {...}, s: \"2\", Format: \"DecimalSI\"}, s\"memory\": {i: {...}, s: \"1Gi\", Format: \"BinarySI\"}}, Requests: {s\"cpu\": {i: {...}, s: \"10m\", Format: \"DecimalSI\"}, s\"memory\": {i: {...}, Format: \"BinarySI\"}}},\n \t\t\tVolumeMounts: []core.VolumeMount{\n \t\t\t\t... // 6 identical elements\n \t\t\t\t{Name: \"istio-podinfo\", MountPath: \"/etc/istio/pod\"},\n \t\t\t\t{Name: \"kube-api-access-fkxzq\", ReadOnly: true, MountPath: \"/var/run/secrets/kubernetes.io/serviceaccount\"},\n- \t\t\t\t{\n- \t\t\t\t\tName: \"azure-identity-token\",\n- \t\t\t\t\tReadOnly: true,\n- \t\t\t\t\tMountPath: \"/var/run/secrets/azure/tokens\",\n- \t\t\t\t},\n \t\t\t},\n \t\t\tVolumeDevices: nil,\n \t\t\tLivenessProbe: nil,\n \t\t\t... // 10 identical fields\n \t\t},\n \t},\n \tEphemeralContainers: nil,\n \tRestartPolicy: \"Never\",\n \t... // 26 identical fields\n }\n"}
github.com/actions/actions-runner-controller/controllers/actions%2esummerwind%2enet.annotatePodOnce
github.com/actions/actions-runner-controller/controllers/actions.summerwind.net/runner_graceful_stop.go:62
github.com/actions/actions-runner-controller/controllers/actions%2esummerwind%2enet.syncRunnerPodsOwners
github.com/actions/actions-runner-controller/controllers/actions.summerwind.net/runner_pod_owner.go:440
github.com/actions/actions-runner-controller/controllers/actions%2esummerwind%2enet.(*RunnerReplicaSetReconciler).Reconcile
github.com/actions/actions-runner-controller/controllers/actions.summerwind.net/runnerreplicaset_controller.go:131
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:121
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:320
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:273
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:234
2023-01-23T12:07:57Z ERROR Reconciler error {"controller": "runnerreplicaset-controller", "controllerGroup": "actions.summerwind.dev", "controllerKind": "RunnerReplicaSet", "RunnerReplicaSet": {"name":"adfdatapipelines-simv2-runners-t5pb2","namespace":"gitrunners"}, "namespace": "gitrunners", "name": "adfdatapipelines-simv2-runners-t5pb2", "reconcileID": "7963ab72-50be-4ced-b195-62136ec426ba", "error": "Pod \"adfdatapipelines-simv2-runners-t5pb2-zwl9b\" is invalid: spec: Forbidden: pod updates may not change fields other than `spec.containers[*].image`, `spec.initContainers[*].image`, `spec.activeDeadlineSeconds`, `spec.tolerations` (only additions to existing tolerations) or `spec.terminationGracePeriodSeconds` (allow it to be set to 1 if it was previously negative)\n core.PodSpec{\n \tVolumes: {{Name: \"workload-socket\", VolumeSource: {EmptyDir: &{}}}, {Name: \"workload-certs\", VolumeSource: {EmptyDir: &{}}}, {Name: \"istio-envoy\", VolumeSource: {EmptyDir: &{Medium: \"Memory\"}}}, {Name: \"istio-data\", VolumeSource: {EmptyDir: &{}}}, ...},\n \tInitContainers: []core.Container{\n \t\t{Name: \"azwi-proxy-init\", Image: \"mcr.microsoft.com/oss/azure/workload-identity/proxy-init:v0.15.0\", Env: {{Name: \"PROXY_PORT\", Value: \"8000\"}, {Name: \"AZURE_CLIENT_ID\", Value: \"clientid\"}, {Name: \"AZURE_TENANT_ID\", Value: \"tenantid\"}, {Name: \"AZURE_FEDERATED_TOKEN_FILE\", Value: \"/var/run/secrets/azure/tokens/azure-identity-token\"}, ...}, VolumeMounts: {{Name: \"azure-identity-token\", ReadOnly: true, MountPath: \"/var/run/secrets/azure/tokens\"}, {Name: \"kube-api-access-fkxzq\", ReadOnly: true, MountPath: \"/var/run/secrets/kubernetes.io/serviceaccount\"}}, ...},\n \t\t{\n \t\t\t... // 5 identical fields\n \t\t\tPorts: nil,\n \t\t\tEnvFrom: nil,\n- \t\t\tEnv: []core.EnvVar{\n- \t\t\t\t{Name: \"AZURE_CLIENT_ID\", Value: \"clientid\"},\n- \t\t\t\t{Name: \"AZURE_TENANT_ID\", Value: \"tenantid\"},\n- \t\t\t\t{\n- \t\t\t\t\tName: \"AZURE_FEDERATED_TOKEN_FILE\",\n- \t\t\t\t\tValue: \"/var/run/secrets/azure/tokens/azure-identity-token\",\n- \t\t\t\t},\n- \t\t\t\t{Name: \"AZURE_AUTHORITY_HOST\", Value: \"https://login.microsoftonline.com/\"},\n- \t\t\t},\n+ \t\t\tEnv: nil,\n \t\t\tResources: {Limits: {s\"cpu\": {i: {...}, s: \"2\", Format: \"DecimalSI\"}, s\"memory\": {i: {...}, s: \"1Gi\", Format: \"BinarySI\"}}, Requests: {s\"cpu\": {i: {...}, s: \"10m\", Format: \"DecimalSI\"}, s\"memory\": {i: {...}, Format: \"BinarySI\"}}},\n \t\t\tVolumeMounts: []core.VolumeMount{\n \t\t\t\t{Name: \"kube-api-access-fkxzq\", ReadOnly: true, MountPath: \"/var/run/secrets/kubernetes.io/serviceaccount\"},\n- \t\t\t\t{\n- \t\t\t\t\tName: \"azure-identity-token\",\n- \t\t\t\t\tReadOnly: true,\n- \t\t\t\t\tMountPath: \"/var/run/secrets/azure/tokens\",\n- \t\t\t\t},\n \t\t\t},\n \t\t\tVolumeDevices: nil,\n \t\t\tLivenessProbe: nil,\n \t\t\t... // 10 identical fields\n \t\t},\n \t},\n \tContainers: []core.Container{\n \t\t{Name: \"runner\", Image: \"dbg-bigdata-docker-local.artifactory.dbgcloud.io/bdaa/aksrunner:\"..., Env: {{Name: \"http_proxy\", Value: \"proxy\"}, {Name: \"https_proxy\", Value: \"proxy\"}, {Name: \"no_proxy\", Value: \"proxy,.\"...}, {Name: \"RUNNER_ORG\", Value: \"organisation\"}, ...}, VolumeMounts: {{Name: \"runner\", MountPath: \"/runner\"}, {Name: \"work\", MountPath: \"/runner/_work\"}, {Name: \"certs-client\", ReadOnly: true, MountPath: \"/certs/client\"}, {Name: \"kube-api-access-fkxzq\", ReadOnly: true, MountPath: \"/var/run/secrets/kubernetes.io/serviceaccount\"}, ...}, ...},\n \t\t{Name: \"docker\", Image: \"remote-docker.artifactory.dbgcloud.io/docker:dind\", Env: {{Name: \"DOCKER_TLS_CERTDIR\", Value: \"/certs\"}, {Name: \"AZURE_CLIENT_ID\", Value: \"clientid\"}, {Name: \"AZURE_TENANT_ID\", Value: \"tenantid\"}, {Name: \"AZURE_FEDERATED_TOKEN_FILE\", Value: \"/var/run/secrets/azure/tokens/azure-identity-token\"}, ...}, VolumeMounts: {{Name: \"runner\", MountPath: \"/runner\"}, {Name: \"certs-client\", MountPath: \"/certs/client\"}, {Name: \"work\", MountPath: \"/runner/_work\"}, {Name: \"kube-api-access-fkxzq\", ReadOnly: true, MountPath: \"/var/run/secrets/kubernetes.io/serviceaccount\"}, ...}, ...},\n \t\t{Name: \"azwi-proxy\", Image: \"mcr.microsoft.com/oss/azure/workload-identity/proxy:v0.15.0\", Args: {\"--proxy-port=8000\"}, Ports: {{ContainerPort: 8000, Protocol: \"TCP\"}}, ...},\n \t\t{\n \t\t\t... // 5 identical fields\n \t\t\tPorts: {{Name: \"http-envoy-prom\", ContainerPort: 15090, Protocol: \"TCP\"}},\n \t\t\tEnvFrom: nil,\n \t\t\tEnv: []core.EnvVar{\n \t\t\t\t... // 15 identical elements\n \t\t\t\t{Name: \"ISTIO_META_MESH_ID\", Value: \"cluster.local\"},\n \t\t\t\t{Name: \"TRUST_DOMAIN\", Value: \"cluster.local\"},\n- \t\t\t\t{Name: \"AZURE_CLIENT_ID\", Value: \"clientid\"},\n- \t\t\t\t{Name: \"AZURE_TENANT_ID\", Value: \"tenantid\"},\n- \t\t\t\t{\n- \t\t\t\t\tName: \"AZURE_FEDERATED_TOKEN_FILE\",\n- \t\t\t\t\tValue: \"/var/run/secrets/azure/tokens/azure-identity-token\",\n- \t\t\t\t},\n- \t\t\t\t{Name: \"AZURE_AUTHORITY_HOST\", Value: \"https://login.microsoftonline.com/\"},\n \t\t\t},\n \t\t\tResources: {Limits: {s\"cpu\": {i: {...}, s: \"2\", Format: \"DecimalSI\"}, s\"memory\": {i: {...}, s: \"1Gi\", Format: \"BinarySI\"}}, Requests: {s\"cpu\": {i: {...}, s: \"10m\", Format: \"DecimalSI\"}, s\"memory\": {i: {...}, Format: \"BinarySI\"}}},\n \t\t\tVolumeMounts: []core.VolumeMount{\n \t\t\t\t... // 6 identical elements\n \t\t\t\t{Name: \"istio-podinfo\", MountPath: \"/etc/istio/pod\"},\n \t\t\t\t{Name: \"kube-api-access-fkxzq\", ReadOnly: true, MountPath: \"/var/run/secrets/kubernetes.io/serviceaccount\"},\n- \t\t\t\t{\n- \t\t\t\t\tName: \"azure-identity-token\",\n- \t\t\t\t\tReadOnly: true,\n- \t\t\t\t\tMountPath: \"/var/run/secrets/azure/tokens\",\n- \t\t\t\t},\n \t\t\t},\n \t\t\tVolumeDevices: nil,\n \t\t\tLivenessProbe: nil,\n \t\t\t... // 10 identical fields\n \t\t},\n \t},\n \tEphemeralContainers: nil,\n \tRestartPolicy: \"Never\",\n \t... // 26 identical fields\n }\n"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:326
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:273
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:234
2023-01-23T12:07:57Z INFO runnerreplicaset Runner failed to register itself to GitHub in timely manner. Recreating the pod to see if it resolves the issue. CAUTION: If you see this a lot, you should investigate the root cause. See https://github.com/actions/actions-runner-controller/issues/288 {"runnerreplicaset": "gitrunners/infradeployment-simv2-runners-9psmm", "owner": "gitrunners/infradeployment-simv2-runners-9psmm-7m5l9", "creationTimestamp": "2023-01-23 11:50:19 +0000 UTC", "readyTransitionTime": "2023-01-23 11:50:24 +0000 UTC", "configuredRegistrationTimeout": "10m0s"}
2023-01-23T12:07:57Z INFO runnerreplicaset Runner failed to register itself to GitHub in timely manner. Recreating the pod to see if it resolves the issue. CAUTION: If you see this a lot, you should investigate the root cause. See https://github.com/actions/actions-runner-controller/issues/288 {"runnerreplicaset": "gitrunners/infradeployment-simv2-runners-9psmm", "owner": "gitrunners/infradeployment-simv2-runners-9psmm-sdbj2", "creationTimestamp": "2023-01-23 11:49:31 +0000 UTC", "readyTransitionTime": "2023-01-23 11:49:35 +0000 UTC", "configuredRegistrationTimeout": "10m0s"}
2023-01-23T12:07:58Z ERROR runnerreplicaset Failed to patch pod to have actions-runner/unregistration-request-timestamp annotation {"runnerreplicaset": "gitrunners/infradeployment-simv2-runners-9psmm", "lastSyncTime": "2023-01-23T11:50:19Z", "effectiveTime": "<nil>", "templateHashDesired": "6dcdfbfd65", "replicasDesired": 0, "replicasPending": 0, "replicasRunning": 0, "replicasMaybeRunning": 0, "templateHashObserved": ["6dcdfbfd65"], "owner": "gitrunners/infradeployment-simv2-runners-9psmm-7m5l9", "error": "Pod \"infradeployment-simv2-runners-9psmm-7m5l9\" is invalid: spec: Forbidden: pod updates may not change fields other than `spec.containers[*].image`, `spec.initContainers[*].image`, `spec.activeDeadlineSeconds`, `spec.tolerations` (only additions to existing tolerations) or `spec.terminationGracePeriodSeconds` (allow it to be set to 1 if it was previously negative)\n core.PodSpec{\n \tVolumes: {{Name: \"workload-socket\", VolumeSource: {EmptyDir: &{}}}, {Name: \"workload-certs\", VolumeSource: {EmptyDir: &{}}}, {Name: \"istio-envoy\", VolumeSource: {EmptyDir: &{Medium: \"Memory\"}}}, {Name: \"istio-data\", VolumeSource: {EmptyDir: &{}}}, ...},\n \tInitContainers: []core.Container{\n \t\t{Name: \"azwi-proxy-init\", Image: \"mcr.microsoft.com/oss/azure/workload-identity/proxy-init:v0.15.0\", Env: {{Name: \"PROXY_PORT\", Value: \"8000\"}, {Name: \"AZURE_CLIENT_ID\", Value: \"clientid\"}, {Name: \"AZURE_TENANT_ID\", Value: \"tenantid\"}, {Name: \"AZURE_FEDERATED_TOKEN_FILE\", Value: \"/var/run/secrets/azure/tokens/azure-identity-token\"}, ...}, VolumeMounts: {{Name: \"azure-identity-token\", ReadOnly: true, MountPath: \"/var/run/secrets/azure/tokens\"}, {Name: \"kube-api-access-66d6j\", ReadOnly: true, MountPath: \"/var/run/secrets/kubernetes.io/serviceaccount\"}}, ...},\n \t\t{\n \t\t\t... // 5 identical fields\n \t\t\tPorts: nil,\n \t\t\tEnvFrom: nil,\n- \t\t\tEnv: []core.EnvVar{\n- \t\t\t\t{Name: \"AZURE_CLIENT_ID\", Value: \"clientid\"},\n- \t\t\t\t{Name: \"AZURE_TENANT_ID\", Value: \"tenantid\"},\n- \t\t\t\t{\n- \t\t\t\t\tName: \"AZURE_FEDERATED_TOKEN_FILE\",\n- \t\t\t\t\tValue: \"/var/run/secrets/azure/tokens/azure-identity-token\",\n- \t\t\t\t},\n- \t\t\t\t{Name: \"AZURE_AUTHORITY_HOST\", Value: \"https://login.microsoftonline.com/\"},\n- \t\t\t},\n+ \t\t\tEnv: nil,\n \t\t\tResources: {Limits: {s\"cpu\": {i: {...}, s: \"2\", Format: \"DecimalSI\"}, s\"memory\": {i: {...}, s: \"1Gi\", Format: \"BinarySI\"}}, Requests: {s\"cpu\": {i: {...}, s: \"10m\", Format: \"DecimalSI\"}, s\"memory\": {i: {...}, Format: \"BinarySI\"}}},\n \t\t\tVolumeMounts: []core.VolumeMount{\n \t\t\t\t{Name: \"kube-api-access-66d6j\", ReadOnly: true, MountPath: \"/var/run/secrets/kubernetes.io/serviceaccount\"},\n- \t\t\t\t{\n- \t\t\t\t\tName: \"azure-identity-token\",\n- \t\t\t\t\tReadOnly: true,\n- \t\t\t\t\tMountPath: \"/var/run/secrets/azure/tokens\",\n- \t\t\t\t},\n \t\t\t},\n \t\t\tVolumeDevices: nil,\n \t\t\tLivenessProbe: nil,\n \t\t\t... // 10 identical fields\n \t\t},\n \t},\n \tContainers: []core.Container{\n \t\t{Name: \"runner\", Image: \"dbg-bigdata-docker-local.artifactory.dbgcloud.io/bdaa/aksrunner:\"..., Env: {{Name: \"http_proxy\", Value: \"proxy\"}, {Name: \"https_proxy\", Value: \"proxy\"}, {Name: \"no_proxy\", Value: \"proxy,.\"...}, {Name: \"RUNNER_ORG\", Value: \"organisation\"}, ...}, VolumeMounts: {{Name: \"runner\", MountPath: \"/runner\"}, {Name: \"work\", MountPath: \"/runner/_work\"}, {Name: \"certs-client\", ReadOnly: true, MountPath: \"/certs/client\"}, {Name: \"kube-api-access-66d6j\", ReadOnly: true, MountPath: \"/var/run/secrets/kubernetes.io/serviceaccount\"}, ...}, ...},\n \t\t{Name: \"docker\", Image: \"remote-docker.artifactory.dbgcloud.io/docker:dind\", Env: {{Name: \"DOCKER_TLS_CERTDIR\", Value: \"/certs\"}, {Name: \"AZURE_CLIENT_ID\", Value: \"clientid\"}, {Name: \"AZURE_TENANT_ID\", Value: \"tenantid\"}, {Name: \"AZURE_FEDERATED_TOKEN_FILE\", Value: \"/var/run/secrets/azure/tokens/azure-identity-token\"}, ...}, VolumeMounts: {{Name: \"runner\", MountPath: \"/runner\"}, {Name: \"certs-client\", MountPath: \"/certs/client\"}, {Name: \"work\", MountPath: \"/runner/_work\"}, {Name: \"kube-api-access-66d6j\", ReadOnly: true, MountPath: \"/var/run/secrets/kubernetes.io/serviceaccount\"}, ...}, ...},\n \t\t{Name: \"azwi-proxy\", Image: \"mcr.microsoft.com/oss/azure/workload-identity/proxy:v0.15.0\", Args: {\"--proxy-port=8000\"}, Ports: {{ContainerPort: 8000, Protocol: \"TCP\"}}, ...},\n \t\t{\n \t\t\t... // 5 identical fields\n \t\t\tPorts: {{Name: \"http-envoy-prom\", ContainerPort: 15090, Protocol: \"TCP\"}},\n \t\t\tEnvFrom: nil,\n \t\t\tEnv: []core.EnvVar{\n \t\t\t\t... // 15 identical elements\n \t\t\t\t{Name: \"ISTIO_META_MESH_ID\", Value: \"cluster.local\"},\n \t\t\t\t{Name: \"TRUST_DOMAIN\", Value: \"cluster.local\"},\n- \t\t\t\t{Name: \"AZURE_CLIENT_ID\", Value: \"clientid\"},\n- \t\t\t\t{Name: \"AZURE_TENANT_ID\", Value: \"tenantid\"},\n- \t\t\t\t{\n- \t\t\t\t\tName: \"AZURE_FEDERATED_TOKEN_FILE\",\n- \t\t\t\t\tValue: \"/var/run/secrets/azure/tokens/azure-identity-token\",\n- \t\t\t\t},\n- \t\t\t\t{Name: \"AZURE_AUTHORITY_HOST\", Value: \"https://login.microsoftonline.com/\"},\n \t\t\t},\n \t\t\tResources: {Limits: {s\"cpu\": {i: {...}, s: \"2\", Format: \"DecimalSI\"}, s\"memory\": {i: {...}, s: \"1Gi\", Format: \"BinarySI\"}}, Requests: {s\"cpu\": {i: {...}, s: \"10m\", Format: \"DecimalSI\"}, s\"memory\": {i: {...}, Format: \"BinarySI\"}}},\n \t\t\tVolumeMounts: []core.VolumeMount{\n \t\t\t\t... // 6 identical elements\n \t\t\t\t{Name: \"istio-podinfo\", MountPath: \"/etc/istio/pod\"},\n \t\t\t\t{Name: \"kube-api-access-66d6j\", ReadOnly: true, MountPath: \"/var/run/secrets/kubernetes.io/serviceaccount\"},\n- \t\t\t\t{\n- \t\t\t\t\tName: \"azure-identity-token\",\n- \t\t\t\t\tReadOnly: true,\n- \t\t\t\t\tMountPath: \"/var/run/secrets/azure/tokens\",\n- \t\t\t\t},\n \t\t\t},\n \t\t\tVolumeDevices: nil,\n \t\t\tLivenessProbe: nil,\n \t\t\t... // 10 identical fields\n \t\t},\n \t},\n \tEphemeralContainers: nil,\n \tRestartPolicy: \"Never\",\n \t... // 26 identical fields\n }\n"}
github.com/actions/actions-runner-controller/controllers/actions%2esummerwind%2enet.annotatePodOnce
github.com/actions/actions-runner-controller/controllers/actions.summerwind.net/runner_graceful_stop.go:62
github.com/actions/actions-runner-controller/controllers/actions%2esummerwind%2enet.syncRunnerPodsOwners
github.com/actions/actions-runner-controller/controllers/actions.summerwind.net/runner_pod_owner.go:440
github.com/actions/actions-runner-controller/controllers/actions%2esummerwind%2enet.(*RunnerReplicaSetReconciler).Reconcile
github.com/actions/actions-runner-controller/controllers/actions.summerwind.net/runnerreplicaset_controller.go:131
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:121
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:320
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:273
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:234
2023-01-23T12:07:58Z ERROR Reconciler error {"controller": "runnerreplicaset-controller", "controllerGroup": "actions.summerwind.dev", "controllerKind": "RunnerReplicaSet", "RunnerReplicaSet": {"name":"infradeployment-simv2-runners-9psmm","namespace":"gitrunners"}, "namespace": "gitrunners", "name": "infradeployment-simv2-runners-9psmm", "reconcileID": "377548e7-5cc3-4c10-8bc5-b02a931bd7de", "error": "Pod \"infradeployment-simv2-runners-9psmm-7m5l9\" is invalid: spec: Forbidden: pod updates may not change fields other than `spec.containers[*].image`, `spec.initContainers[*].image`, `spec.activeDeadlineSeconds`, `spec.tolerations` (only additions to existing tolerations) or `spec.terminationGracePeriodSeconds` (allow it to be set to 1 if it was previously negative)\n core.PodSpec{\n \tVolumes: {{Name: \"workload-socket\", VolumeSource: {EmptyDir: &{}}}, {Name: \"workload-certs\", VolumeSource: {EmptyDir: &{}}}, {Name: \"istio-envoy\", VolumeSource: {EmptyDir: &{Medium: \"Memory\"}}}, {Name: \"istio-data\", VolumeSource: {EmptyDir: &{}}}, ...},\n \tInitContainers: []core.Container{\n \t\t{Name: \"azwi-proxy-init\", Image: \"mcr.microsoft.com/oss/azure/workload-identity/proxy-init:v0.15.0\", Env: {{Name: \"PROXY_PORT\", Value: \"8000\"}, {Name: \"AZURE_CLIENT_ID\", Value: \"clientid\"}, {Name: \"AZURE_TENANT_ID\", Value: \"tenantid\"}, {Name: \"AZURE_FEDERATED_TOKEN_FILE\", Value: \"/var/run/secrets/azure/tokens/azure-identity-token\"}, ...}, VolumeMounts: {{Name: \"azure-identity-token\", ReadOnly: true, MountPath: \"/var/run/secrets/azure/tokens\"}, {Name: \"kube-api-access-66d6j\", ReadOnly: true, MountPath: \"/var/run/secrets/kubernetes.io/serviceaccount\"}}, ...},\n \t\t{\n \t\t\t... // 5 identical fields\n \t\t\tPorts: nil,\n \t\t\tEnvFrom: nil,\n- \t\t\tEnv: []core.EnvVar{\n- \t\t\t\t{Name: \"AZURE_CLIENT_ID\", Value: \"clientid\"},\n- \t\t\t\t{Name: \"AZURE_TENANT_ID\", Value: \"tenantid\"},\n- \t\t\t\t{\n- \t\t\t\t\tName: \"AZURE_FEDERATED_TOKEN_FILE\",\n- \t\t\t\t\tValue: \"/var/run/secrets/azure/tokens/azure-identity-token\",\n- \t\t\t\t},\n- \t\t\t\t{Name: \"AZURE_AUTHORITY_HOST\", Value: \"https://login.microsoftonline.com/\"},\n- \t\t\t},\n+ \t\t\tEnv: nil,\n \t\t\tResources: {Limits: {s\"cpu\": {i: {...}, s: \"2\", Format: \"DecimalSI\"}, s\"memory\": {i: {...}, s: \"1Gi\", Format: \"BinarySI\"}}, Requests: {s\"cpu\": {i: {...}, s: \"10m\", Format: \"DecimalSI\"}, s\"memory\": {i: {...}, Format: \"BinarySI\"}}},\n \t\t\tVolumeMounts: []core.VolumeMount{\n \t\t\t\t{Name: \"kube-api-access-66d6j\", ReadOnly: true, MountPath: \"/var/run/secrets/kubernetes.io/serviceaccount\"},\n- \t\t\t\t{\n- \t\t\t\t\tName: \"azure-identity-token\",\n- \t\t\t\t\tReadOnly: true,\n- \t\t\t\t\tMountPath: \"/var/run/secrets/azure/tokens\",\n- \t\t\t\t},\n \t\t\t},\n \t\t\tVolumeDevices: nil,\n \t\t\tLivenessProbe: nil,\n \t\t\t... // 10 identical fields\n \t\t},\n \t},\n \tContainers: []core.Container{\n \t\t{Name: \"runner\", Image: \"dbg-bigdata-docker-local.artifactory.dbgcloud.io/bdaa/aksrunner:\"..., Env: {{Name: \"http_proxy\", Value: \"proxy\"}, {Name: \"https_proxy\", Value: \"proxy\"}, {Name: \"no_proxy\", Value: \"proxy,.\"...}, {Name: \"RUNNER_ORG\", Value: \"organisation\"}, ...}, VolumeMounts: {{Name: \"runner\", MountPath: \"/runner\"}, {Name: \"work\", MountPath: \"/runner/_work\"}, {Name: \"certs-client\", ReadOnly: true, MountPath: \"/certs/client\"}, {Name: \"kube-api-access-66d6j\", ReadOnly: true, MountPath: \"/var/run/secrets/kubernetes.io/serviceaccount\"}, ...}, ...},\n \t\t{Name: \"docker\", Image: \"remote-docker.artifactory.dbgcloud.io/docker:dind\", Env: {{Name: \"DOCKER_TLS_CERTDIR\", Value: \"/certs\"}, {Name: \"AZURE_CLIENT_ID\", Value: \"clientid\"}, {Name: \"AZURE_TENANT_ID\", Value: \"tenantid\"}, {Name: \"AZURE_FEDERATED_TOKEN_FILE\", Value: \"/var/run/secrets/azure/tokens/azure-identity-token\"}, ...}, VolumeMounts: {{Name: \"runner\", MountPath: \"/runner\"}, {Name: \"certs-client\", MountPath: \"/certs/client\"}, {Name: \"work\", MountPath: \"/runner/_work\"}, {Name: \"kube-api-access-66d6j\", ReadOnly: true, MountPath: \"/var/run/secrets/kubernetes.io/serviceaccount\"}, ...}, ...},\n \t\t{Name: \"azwi-proxy\", Image: \"mcr.microsoft.com/oss/azure/workload-identity/proxy:v0.15.0\", Args: {\"--proxy-port=8000\"}, Ports: {{ContainerPort: 8000, Protocol: \"TCP\"}}, ...},\n \t\t{\n \t\t\t... // 5 identical fields\n \t\t\tPorts: {{Name: \"http-envoy-prom\", ContainerPort: 15090, Protocol: \"TCP\"}},\n \t\t\tEnvFrom: nil,\n \t\t\tEnv: []core.EnvVar{\n \t\t\t\t... // 15 identical elements\n \t\t\t\t{Name: \"ISTIO_META_MESH_ID\", Value: \"cluster.local\"},\n \t\t\t\t{Name: \"TRUST_DOMAIN\", Value: \"cluster.local\"},\n- \t\t\t\t{Name: \"AZURE_CLIENT_ID\", Value: \"clientid\"},\n- \t\t\t\t{Name: \"AZURE_TENANT_ID\", Value: \"tenantid\"},\n- \t\t\t\t{\n- \t\t\t\t\tName: \"AZURE_FEDERATED_TOKEN_FILE\",\n- \t\t\t\t\tValue: \"/var/run/secrets/azure/tokens/azure-identity-token\",\n- \t\t\t\t},\n- \t\t\t\t{Name: \"AZURE_AUTHORITY_HOST\", Value: \"https://login.microsoftonline.com/\"},\n \t\t\t},\n \t\t\tResources: {Limits: {s\"cpu\": {i: {...}, s: \"2\", Format: \"DecimalSI\"}, s\"memory\": {i: {...}, s: \"1Gi\", Format: \"BinarySI\"}}, Requests: {s\"cpu\": {i: {...}, s: \"10m\", Format: \"DecimalSI\"}, s\"memory\": {i: {...}, Format: \"BinarySI\"}}},\n \t\t\tVolumeMounts: []core.VolumeMount{\n \t\t\t\t... // 6 identical elements\n \t\t\t\t{Name: \"istio-podinfo\", MountPath: \"/etc/istio/pod\"},\n \t\t\t\t{Name: \"kube-api-access-66d6j\", ReadOnly: true, MountPath: \"/var/run/secrets/kubernetes.io/serviceaccount\"},\n- \t\t\t\t{\n- \t\t\t\t\tName: \"azure-identity-token\",\n- \t\t\t\t\tReadOnly: true,\n- \t\t\t\t\tMountPath: \"/var/run/secrets/azure/tokens\",\n- \t\t\t\t},\n \t\t\t},\n \t\t\tVolumeDevices: nil,\n \t\t\tLivenessProbe: nil,\n \t\t\t... // 10 identical fields\n \t\t},\n \t},\n \tEphemeralContainers: nil,\n \tRestartPolicy: \"Never\",\n \t... // 26 identical fields\n }\n"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:326
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:273
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:234
2023-01-23T12:07:58Z ERROR runnerpod Failed to update runner {"runnerpod": "gitrunners/adfdatapipelines-simv2-runners-rwtph-zmv7g", "error": "Pod \"adfdatapipelines-simv2-runners-rwtph-zmv7g\" is invalid: spec: Forbidden: pod updates may not change fields other than `spec.containers[*].image`, `spec.initContainers[*].image`, `spec.activeDeadlineSeconds`, `spec.tolerations` (only additions to existing tolerations) or `spec.terminationGracePeriodSeconds` (allow it to be set to 1 if it was previously negative)\n core.PodSpec{\n \tVolumes: {{Name: \"workload-socket\", VolumeSource: {EmptyDir: &{}}}, {Name: \"workload-certs\", VolumeSource: {EmptyDir: &{}}}, {Name: \"istio-envoy\", VolumeSource: {EmptyDir: &{Medium: \"Memory\"}}}, {Name: \"istio-data\", VolumeSource: {EmptyDir: &{}}}, ...},\n \tInitContainers: []core.Container{\n \t\t{Name: \"azwi-proxy-init\", Image: \"mcr.microsoft.com/oss/azure/workload-identity/proxy-init:v0.15.0\", Env: {{Name: \"PROXY_PORT\", Value: \"8000\"}, {Name: \"AZURE_CLIENT_ID\", Value: \"clientid\"}, {Name: \"AZURE_TENANT_ID\", Value: \"tenantid\"}, {Name: \"AZURE_FEDERATED_TOKEN_FILE\", Value: \"/var/run/secrets/azure/tokens/azure-identity-token\"}, ...}, VolumeMounts: {{Name: \"azure-identity-token\", ReadOnly: true, MountPath: \"/var/run/secrets/azure/tokens\"}, {Name: \"kube-api-access-5jcqw\", ReadOnly: true, MountPath: \"/var/run/secrets/kubernetes.io/serviceaccount\"}}, ...},\n \t\t{\n \t\t\t... // 5 identical fields\n \t\t\tPorts: nil,\n \t\t\tEnvFrom: nil,\n- \t\t\tEnv: []core.EnvVar{\n- \t\t\t\t{Name: \"AZURE_CLIENT_ID\", Value: \"clientid\"},\n- \t\t\t\t{Name: \"AZURE_TENANT_ID\", Value: \"tenantid\"},\n- \t\t\t\t{\n- \t\t\t\t\tName: \"AZURE_FEDERATED_TOKEN_FILE\",\n- \t\t\t\t\tValue: \"/var/run/secrets/azure/tokens/azure-identity-token\",\n- \t\t\t\t},\n- \t\t\t\t{Name: \"AZURE_AUTHORITY_HOST\", Value: \"https://login.microsoftonline.com/\"},\n- \t\t\t},\n+ \t\t\tEnv: nil,\n \t\t\tResources: {Limits: {s\"cpu\": {i: {...}, s: \"2\", Format: \"DecimalSI\"}, s\"memory\": {i: {...}, s: \"1Gi\", Format: \"BinarySI\"}}, Requests: {s\"cpu\": {i: {...}, s: \"10m\", Format: \"DecimalSI\"}, s\"memory\": {i: {...}, Format: \"BinarySI\"}}},\n \t\t\tVolumeMounts: []core.VolumeMount{\n \t\t\t\t{Name: \"kube-api-access-5jcqw\", ReadOnly: true, MountPath: \"/var/run/secrets/kubernetes.io/serviceaccount\"},\n- \t\t\t\t{\n- \t\t\t\t\tName: \"azure-identity-token\",\n- \t\t\t\t\tReadOnly: true,\n- \t\t\t\t\tMountPath: \"/var/run/secrets/azure/tokens\",\n- \t\t\t\t},\n \t\t\t},\n \t\t\tVolumeDevices: nil,\n \t\t\tLivenessProbe: nil,\n \t\t\t... // 10 identical fields\n \t\t},\n \t},\n \tContainers: []core.Container{\n \t\t{Name: \"runner\", Image: \"dbg-bigdata-docker-local.artifactory.dbgcloud.io/bdaa/aksrunner:\"..., Env: {{Name: \"http_proxy\", Value: \"proxy\"}, {Name: \"https_proxy\", Value: \"proxy\"}, {Name: \"no_proxy\", Value: \"proxy,.\"...}, {Name: \"RUNNER_ORG\", Value: \"organisation\"}, ...}, VolumeMounts: {{Name: \"runner\", MountPath: \"/runner\"}, {Name: \"work\", MountPath: \"/runner/_work\"}, {Name: \"certs-client\", ReadOnly: true, MountPath: \"/certs/client\"}, {Name: \"kube-api-access-5jcqw\", ReadOnly: true, MountPath: \"/var/run/secrets/kubernetes.io/serviceaccount\"}, ...}, ...},\n \t\t{Name: \"docker\", Image: \"remote-docker.artifactory.dbgcloud.io/docker:dind\", Env: {{Name: \"DOCKER_TLS_CERTDIR\", Value: \"/certs\"}, {Name: \"AZURE_CLIENT_ID\", Value: \"clientid\"}, {Name: \"AZURE_TENANT_ID\", Value: \"tenantid\"}, {Name: \"AZURE_FEDERATED_TOKEN_FILE\", Value: \"/var/run/secrets/azure/tokens/azure-identity-token\"}, ...}, VolumeMounts: {{Name: \"runner\", MountPath: \"/runner\"}, {Name: \"certs-client\", MountPath: \"/certs/client\"}, {Name: \"work\", MountPath: \"/runner/_work\"}, {Name: \"kube-api-access-5jcqw\", ReadOnly: true, MountPath: \"/var/run/secrets/kubernetes.io/serviceaccount\"}, ...}, ...},\n \t\t{Name: \"azwi-proxy\", Image: \"mcr.microsoft.com/oss/azure/workload-identity/proxy:v0.15.0\", Args: {\"--proxy-port=8000\"}, Ports: {{ContainerPort: 8000, Protocol: \"TCP\"}}, ...},\n \t\t{\n \t\t\t... // 5 identical fields\n \t\t\tPorts: {{Name: \"http-envoy-prom\", ContainerPort: 15090, Protocol: \"TCP\"}},\n \t\t\tEnvFrom: nil,\n \t\t\tEnv: []core.EnvVar{\n \t\t\t\t... // 15 identical elements\n \t\t\t\t{Name: \"ISTIO_META_MESH_ID\", Value: \"cluster.local\"},\n \t\t\t\t{Name: \"TRUST_DOMAIN\", Value: \"cluster.local\"},\n- \t\t\t\t{Name: \"AZURE_CLIENT_ID\", Value: \"clientid\"},\n- \t\t\t\t{Name: \"AZURE_TENANT_ID\", Value: \"tenantid\"},\n- \t\t\t\t{\n- \t\t\t\t\tName: \"AZURE_FEDERATED_TOKEN_FILE\",\n- \t\t\t\t\tValue: \"/var/run/secrets/azure/tokens/azure-identity-token\",\n- \t\t\t\t},\n- \t\t\t\t{Name: \"AZURE_AUTHORITY_HOST\", Value: \"https://login.microsoftonline.com/\"},\n \t\t\t},\n \t\t\tResources: {Limits: {s\"cpu\": {i: {...}, s: \"2\", Format: \"DecimalSI\"}, s\"memory\": {i: {...}, s: \"1Gi\", Format: \"BinarySI\"}}, Requests: {s\"cpu\": {i: {...}, s: \"10m\", Format: \"DecimalSI\"}, s\"memory\": {i: {...}, Format: \"BinarySI\"}}},\n \t\t\tVolumeMounts: []core.VolumeMount{\n \t\t\t\t... // 6 identical elements\n \t\t\t\t{Name: \"istio-podinfo\", MountPath: \"/etc/istio/pod\"},\n \t\t\t\t{Name: \"kube-api-access-5jcqw\", ReadOnly: true, MountPath: \"/var/run/secrets/kubernetes.io/serviceaccount\"},\n- \t\t\t\t{\n- \t\t\t\t\tName: \"azure-identity-token\",\n- \t\t\t\t\tReadOnly: true,\n- \t\t\t\t\tMountPath: \"/var/run/secrets/azure/tokens\",\n- \t\t\t\t},\n \t\t\t},\n \t\t\tVolumeDevices: nil,\n \t\t\tLivenessProbe: nil,\n \t\t\t... // 10 identical fields\n \t\t},\n \t},\n \tEphemeralContainers: nil,\n \tRestartPolicy: \"Never\",\n \t... // 26 identical fields\n }\n"}
github.com/actions/actions-runner-controller/controllers/actions%2esummerwind%2enet.(*RunnerPodReconciler).Reconcile
github.com/actions/actions-runner-controller/controllers/actions.summerwind.net/runner_pod_controller.go:119
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:121
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:320
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:273
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:234
2023-01-23T12:07:58Z ERROR Reconciler error {"controller": "runnerpod-controller", "controllerGroup": "", "controllerKind": "Pod", "Pod": {"name":"adfdatapipelines-simv2-runners-rwtph-zmv7g","namespace":"gitrunners"}, "namespace": "gitrunners", "name": "adfdatapipelines-simv2-runners-rwtph-zmv7g", "reconcileID": "6edc5ae1-dca9-4b2c-a05d-f29abaa0f79f", "error": "Pod \"adfdatapipelines-simv2-runners-rwtph-zmv7g\" is invalid: spec: Forbidden: pod updates may not change fields other than `spec.containers[*].image`, `spec.initContainers[*].image`, `spec.activeDeadlineSeconds`, `spec.tolerations` (only additions to existing tolerations) or `spec.terminationGracePeriodSeconds` (allow it to be set to 1 if it was previously negative)\n core.PodSpec{\n \tVolumes: {{Name: \"workload-socket\", VolumeSource: {EmptyDir: &{}}}, {Name: \"workload-certs\", VolumeSource: {EmptyDir: &{}}}, {Name: \"istio-envoy\", VolumeSource: {EmptyDir: &{Medium: \"Memory\"}}}, {Name: \"istio-data\", VolumeSource: {EmptyDir: &{}}}, ...},\n \tInitContainers: []core.Container{\n \t\t{Name: \"azwi-proxy-init\", Image: \"mcr.microsoft.com/oss/azure/workload-identity/proxy-init:v0.15.0\", Env: {{Name: \"PROXY_PORT\", Value: \"8000\"}, {Name: \"AZURE_CLIENT_ID\", Value: \"clientid\"}, {Name: \"AZURE_TENANT_ID\", Value: \"tenantid\"}, {Name: \"AZURE_FEDERATED_TOKEN_FILE\", Value: \"/var/run/secrets/azure/tokens/azure-identity-token\"}, ...}, VolumeMounts: {{Name: \"azure-identity-token\", ReadOnly: true, MountPath: \"/var/run/secrets/azure/tokens\"}, {Name: \"kube-api-access-5jcqw\", ReadOnly: true, MountPath: \"/var/run/secrets/kubernetes.io/serviceaccount\"}}, ...},\n \t\t{\n \t\t\t... // 5 identical fields\n \t\t\tPorts: nil,\n \t\t\tEnvFrom: nil,\n- \t\t\tEnv: []core.EnvVar{\n- \t\t\t\t{Name: \"AZURE_CLIENT_ID\", Value: \"clientid\"},\n- \t\t\t\t{Name: \"AZURE_TENANT_ID\", Value: \"tenantid\"},\n- \t\t\t\t{\n- \t\t\t\t\tName: \"AZURE_FEDERATED_TOKEN_FILE\",\n- \t\t\t\t\tValue: \"/var/run/secrets/azure/tokens/azure-identity-token\",\n- \t\t\t\t},\n- \t\t\t\t{Name: \"AZURE_AUTHORITY_HOST\", Value: \"https://login.microsoftonline.com/\"},\n- \t\t\t},\n+ \t\t\tEnv: nil,\n \t\t\tResources: {Limits: {s\"cpu\": {i: {...}, s: \"2\", Format: \"DecimalSI\"}, s\"memory\": {i: {...}, s: \"1Gi\", Format: \"BinarySI\"}}, Requests: {s\"cpu\": {i: {...}, s: \"10m\", Format: \"DecimalSI\"}, s\"memory\": {i: {...}, Format: \"BinarySI\"}}},\n \t\t\tVolumeMounts: []core.VolumeMount{\n \t\t\t\t{Name: \"kube-api-access-5jcqw\", ReadOnly: true, MountPath: \"/var/run/secrets/kubernetes.io/serviceaccount\"},\n- \t\t\t\t{\n- \t\t\t\t\tName: \"azure-identity-token\",\n- \t\t\t\t\tReadOnly: true,\n- \t\t\t\t\tMountPath: \"/var/run/secrets/azure/tokens\",\n- \t\t\t\t},\n \t\t\t},\n \t\t\tVolumeDevices: nil,\n \t\t\tLivenessProbe: nil,\n \t\t\t... // 10 identical fields\n \t\t},\n \t},\n \tContainers: []core.Container{\n \t\t{Name: \"runner\", Image: \"dbg-bigdata-docker-local.artifactory.dbgcloud.io/bdaa/aksrunner:\"..., Env: {{Name: \"http_proxy\", Value: \"proxy\"}, {Name: \"https_proxy\", Value: \"proxy\"}, {Name: \"no_proxy\", Value: \"proxy,.\"...}, {Name: \"RUNNER_ORG\", Value: \"organisation\"}, ...}, VolumeMounts: {{Name: \"runner\", MountPath: \"/runner\"}, {Name: \"work\", MountPath: \"/runner/_work\"}, {Name: \"certs-client\", ReadOnly: true, MountPath: \"/certs/client\"}, {Name: \"kube-api-access-5jcqw\", ReadOnly: true, MountPath: \"/var/run/secrets/kubernetes.io/serviceaccount\"}, ...}, ...},\n \t\t{Name: \"docker\", Image: \"remote-docker.artifactory.dbgcloud.io/docker:dind\", Env: {{Name: \"DOCKER_TLS_CERTDIR\", Value: \"/certs\"}, {Name: \"AZURE_CLIENT_ID\", Value: \"clientid\"}, {Name: \"AZURE_TENANT_ID\", Value: \"tenantid\"}, {Name: \"AZURE_FEDERATED_TOKEN_FILE\", Value: \"/var/run/secrets/azure/tokens/azure-identity-token\"}, ...}, VolumeMounts: {{Name: \"runner\", MountPath: \"/runner\"}, {Name: \"certs-client\", MountPath: \"/certs/client\"}, {Name: \"work\", MountPath: \"/runner/_work\"}, {Name: \"kube-api-access-5jcqw\", ReadOnly: true, MountPath: \"/var/run/secrets/kubernetes.io/serviceaccount\"}, ...}, ...},\n \t\t{Name: \"azwi-proxy\", Image: \"mcr.microsoft.com/oss/azure/workload-identity/proxy:v0.15.0\", Args: {\"--proxy-port=8000\"}, Ports: {{ContainerPort: 8000, Protocol: \"TCP\"}}, ...},\n \t\t{\n \t\t\t... // 5 identical fields\n \t\t\tPorts: {{Name: \"http-envoy-prom\", ContainerPort: 15090, Protocol: \"TCP\"}},\n \t\t\tEnvFrom: nil,\n \t\t\tEnv: []core.EnvVar{\n \t\t\t\t... // 15 identical elements\n \t\t\t\t{Name: \"ISTIO_META_MESH_ID\", Value: \"cluster.local\"},\n \t\t\t\t{Name: \"TRUST_DOMAIN\", Value: \"cluster.local\"},\n- \t\t\t\t{Name: \"AZURE_CLIENT_ID\", Value: \"clientid\"},\n- \t\t\t\t{Name: \"AZURE_TENANT_ID\", Value: \"tenantid\"},\n- \t\t\t\t{\n- \t\t\t\t\tName: \"AZURE_FEDERATED_TOKEN_FILE\",\n- \t\t\t\t\tValue: \"/var/run/secrets/azure/tokens/azure-identity-token\",\n- \t\t\t\t},\n- \t\t\t\t{Name: \"AZURE_AUTHORITY_HOST\", Value: \"https://login.microsoftonline.com/\"},\n \t\t\t},\n \t\t\tResources: {Limits: {s\"cpu\": {i: {...}, s: \"2\", Format: \"DecimalSI\"}, s\"memory\": {i: {...}, s: \"1Gi\", Format: \"BinarySI\"}}, Requests: {s\"cpu\": {i: {...}, s: \"10m\", Format: \"DecimalSI\"}, s\"memory\": {i: {...}, Format: \"BinarySI\"}}},\n \t\t\tVolumeMounts: []core.VolumeMount{\n \t\t\t\t... // 6 identical elements\n \t\t\t\t{Name: \"istio-podinfo\", MountPath: \"/etc/istio/pod\"},\n \t\t\t\t{Name: \"kube-api-access-5jcqw\", ReadOnly: true, MountPath: \"/var/run/secrets/kubernetes.io/serviceaccount\"},\n- \t\t\t\t{\n- \t\t\t\t\tName: \"azure-identity-token\",\n- \t\t\t\t\tReadOnly: true,\n- \t\t\t\t\tMountPath: \"/var/run/secrets/azure/tokens\",\n- \t\t\t\t},\n \t\t\t},\n \t\t\tVolumeDevices: nil,\n \t\t\tLivenessProbe: nil,\n \t\t\t... // 10 identical fields\n \t\t},\n \t},\n \tEphemeralContainers: nil,\n \tRestartPolicy: \"Never\",\n \t... // 26 identical fields\n }\n"}
Whole Runner Pod Logs
2023-01-23 12:02:28.891 NOTICE --- Runner init started with pid 10
2023-01-23 12:02:29.845 DEBUG --- Configuring the runner.
# Authentication
√ Connected to GitHub
# Runner Registration
√ Runner successfully added
√ Runner connection is good
# Runner settings
√ Settings Saved.
2023-01-23 12:02:37.820 DEBUG --- Runner successfully configured.
{
"isHostedServer": false,
"agentId": 31827,
"agentName": "datapipelines",
"poolId": 9,
"poolName": "datapipelines",
"serverUrl": "https://github.com/_services/pipelines/tg1bKSIkI103oyxiraiSvb1IhsYTwVsA6Qhr1DxAimauUGR9mk",
"gitHubUrl": "https://github.com/org",
"workFolder": "/runner/_work"
2023-01-23 12:02:37.828 DEBUG --- Docker enabled runner detected and Docker daemon wait is enabled
2023-01-23 12:02:37.830 DEBUG --- Waiting until Docker is available or the timeout of 120 seconds is reached
unable to resolve docker endpoint: open /certs/client/ca.pem: no such file or directory
unable to resolve docker endpoint: open /certs/client/ca.pem: no such file or directory
unable to resolve docker endpoint: open /certs/client/ca.pem: no such file or directory
unable to resolve docker endpoint: open /certs/client/ca.pem: no such file or directory
unable to resolve docker endpoint: open /certs/client/ca.pem: no such file or directory
unable to resolve docker endpoint: open /certs/client/ca.pem: no such file or directory
}CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
√ Connected to GitHub
Current runner version: '2.299.1'
2023-01-23 12:02:46Z: Listening for Jobs
Additional Context
No response
Hello! Thank you for filing an issue.
The maintainers will triage your issue shortly.
In the meantime, please take a look at the troubleshooting guide for bug reports.
If this is a feature request, please review our contribution guidelines.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
https://github.com/Azure/azure-workload-identity/issues/647 possibly related to this? Are you still having this issue? Surprised no one has said anything to this
