actions
Listener ServiceAccount does not allow to add Annotations
Checks
- [x] I've already read https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners-with-actions-runner-controller/troubleshooting-actions-runner-controller-errors and I'm sure my issue is not covered in the troubleshooting guide.
- [x] I am using charts that are officially provided
Controller Version
0.13.0
Deployment Method
Helm
Checks
- [x] This isn't a question or user support case (For Q&A and community support, go to Discussions).
- [x] I've read the Changelog before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes
To Reproduce
1. declare helm chart and set annotations, example is in tf
resource "helm_release" "gha_runner_scale_set" {
name = local.runner_name
namespace = local.runner_namespace
chart = "oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set"
version = var.arc_version
create_namespace = true
dependency_update = true
values = [jsonencode({
githubConfigUrl = "https://github.com/ORG_NAME"
githubConfigSecret = local.github_secret_name
controllerServiceAccount = {
namespace = local.listener_namespace
name = "arc-gha-rs-controller"
}
minRunners = 1
maxRunners = 10
# this is the annotation i want to add to the listener service account
annotations = {
"eks.amazonaws.com/role-arn" = aws_iam_role.listener_secrets_access.arn
}
listenerTemplate = {
spec = {
volumes = [{
name = "secrets-store"
csi = {
driver = "secrets-store.csi.k8s.io"
readOnly = true
volumeAttributes = {
secretProviderClass = local.github_secret_name
}
}
}]
containers = [{
name = "listener"
volumeMounts = [{
name = "secrets-store"
mountPath = "/mnt/secrets-store"
readOnly = true
}]
}]
}
}
labels = { for k, v in local.tags : k => tostring(v) }
})]
depends_on = [
kubernetes_manifest.github_app_secret_provider,
helm_release.gha_runner_scale_set_controller
]
}
2. apply and look up info of the listener ServiceAccount in the controller namespace
Name: ***-754b578d-listener
Namespace: arc-systems
Labels: actions.github.com/organization=***
actions.github.com/scale-set-name=***
actions.github.com/scale-set-namespace=arc-runners
app.kubernetes.io/component=runner-scale-set-listener
app.kubernetes.io/instance=***
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=***
app.kubernetes.io/part-of=gha-runner-scale-set
app.kubernetes.io/version=0.13.0
helm.sh/chart=gha-rs-0.13.0
Annotations: <none>
Image pull secrets: <none>
Mountable secrets: <none>
Tokens: <none>
Events: <none>
looking at the listener pod definition we see that this is the ServiceAccount that is used
Name: ***-754b578d-listener
Namespace: arc-systems
Priority: 0
Service Account: ***-754b578d-listener
[...]
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 12m default-scheduler Successfully assigned arc-systems/***-754b578d-listener to ip-10-170-16-192.eu-central-1.compute.internal
Warning FailedMount 119s (x13 over 12m) kubelet MountVolume.SetUp failed for volume "secrets-store" : rpc error: code = Unknown desc = failed to mount secrets store objects for pod arc-systems/***-754b578d-listener, err: rpc error: code = Unknown desc = eu-central-1: An IAM role must be associated with service account ***-754b578d-listener (namespace: arc-systems)
In order to avoid confusion, the ServiceAccounts in the runner namespace to get the annotations ([code](https://github.com/actions/actions-runner-controller/blob/3c1a323381a7f5ba782fb1a66c5db04767f70c7b/charts/gha-runner-scale-set/templates/kube_mode_serviceaccount.yaml#L12))
Describe the bug
It's not possible to add annotations to the listener ServiceAccount.
We are using the secret-store csi driver to sync from aws secrets manager to kubernetes secrets, we store the github app id/installation id/pem inside the aws secret and want to sync it to kubernetes.
My first problem is that there is a circular dependency issue, the the controller only creates the listener if the secret exists, but the secret is created when the volume is mounted when the listener pod is started. I worked around this by manually creating a secret, as this is a one time action and only necessary during the initial setup this is acceptable for us.
The next step is to allow the ServiceAccount to assume the iam role, so it can read the secret and sync it to the k8s secret. In order to do this you add an annotation to the ServiceAccount.
I looked up the code for the ServiceAccount for the listener, which is here https://github.com/actions/actions-runner-controller/blob/e46c9292413f2f8c7d9c341c192e76c489f9d4d9/controllers/actions.github.com/resourcebuilder.go#L433
and we see that it is not possible to pass in annotations. As far as I can tell we would only need to add the following line Annotations: autoscalingListener.Annotations,
I want to note that I had it up and running before I tried to add the csi driver secret store, using a normal kubernetes secret, so the problems are all related to that.
Describe the expected behavior
Custom annotation are correctly passed into the listener service account.
Additional Context
I added the passed in values above in tf
Controller Logs
https://gist.github.com/jonny-rimek/49eee36949fa2023f8c5b46036376c22
Runner Pod Logs
runner doesn't start, because the listener doesn't start
Here is a workaround using Terraform.
Create a kubernetes_service_account resource
resource "kubernetes_service_account" "gha_runner_scale_set" {
metadata {
name = "arc-gha-rs-controller"
namespace = local.listener_namespace
annotations = {
"eks.amazonaws.com/role-arn" = aws_iam_role.listener_secrets_access.arn
}
labels = {
"app.kubernetes.io/managed-by" = "Terraform"
}
}
}
Then in your helm values
listenerTemplate = {
spec = {
serviceAccountName = "arc-gha-rs-controller"
...
Chiming in from the product side. This is something we will consider for our next major release 0.14.0, which is expected to ship in the Jan-Apr 2026 timeframe. Our team needs to take a look at the multiple options that have been identified so far and determine the best path forward.
I have a similar problem wit annotations on "Role" and "RoleBinding" for listener resources.
I have a similar problem wit annotations on "Role" and "RoleBinding" for listener resources.
Have to fix it？
@ego93 we tried implementing the fix you mentioned, but it turns out that listenerTemplate.spec.serviceAccountName is also not passed through. Using that we still created the default ServiceAccount, that we could not add Annotations to.
In the following two links we can see that serviceAccountName does not get merged in https://github.com/actions/actions-runner-controller/blob/e46c9292413f2f8c7d9c341c192e76c489f9d4d9/controllers/actions.github.com/resourcebuilder.go#L392 https://github.com/actions/actions-runner-controller/blob/e46c9292413f2f8c7d9c341c192e76c489f9d4d9/controllers/actions.github.com/resourcebuilder.go#L314
for completeness’s sake, i'll share our current tf code again
resource "kubernetes_service_account" "gha_runner_scale_set" {
metadata {
name = "arc-gha-rs-controller"
namespace = local.listener_namespace
annotations = {
"meta.helm.sh/release-name" = "arc",
"meta.helm.sh/release-namespace" = "arc-systems"
}
labels = merge(
{ for k, v in local.tags : k => tostring(v) },
{
"app.kubernetes.io/managed-by" = "Helm",
"app.kubernetes.io/component" = "runner-scale-set-listener",
"app.kubernetes.io/part-of" = "gha-rs-controller",
}
)
}
}
resource "kubernetes_service_account" "listener_service_account" {
metadata {
name = "scacap-dev-small-listener"
namespace = local.listener_namespace
annotations = {
"eks.amazonaws.com/role-arn" = aws_iam_role.listener_secrets_access.arn
}
labels = merge(
{ for k, v in local.tags : k => tostring(v) },
{
"actions.github.com/organization" = "ScaCap"
"actions.github.com/scale-set-name" = "scacap-dev-small"
"actions.github.com/scale-set-namespace" = "arc-runners"
"app.kubernetes.io/component" = "runner-scale-set-listener"
"app.kubernetes.io/instance" = "scacap-dev-small"
"app.kubernetes.io/managed-by" = "Helm"
"app.kubernetes.io/name" = "scacap-dev-small"
"app.kubernetes.io/part-of" = "gha-runner-scale-set"
"app.kubernetes.io/version" = "0.13.0"
"helm.sh/chart" = "gha-rs-0.13.0"
"sidecar.istio.io/inject" = false
}
)
}
}
resource "helm_release" "gha_runner_scale_set" {
name = local.runner_name
namespace = local.runner_namespace
chart = "oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set"
version = var.arc_version
create_namespace = true
dependency_update = true
values = [jsonencode({
githubConfigUrl = "https://github.com/ScaCap"
githubConfigSecret = local.github_secret_name
controllerServiceAccount = {
namespace = local.listener_namespace
name = "arc-gha-rs-controller"
}
minRunners = 1
maxRunners = 10
listenerTemplate = {
spec = {
serviceAccountName = "scacap-dev-small-listener" # <- does NOT work
volumes = [{
name = "secrets-store"
csi = {
driver = "secrets-store.csi.k8s.io"
readOnly = true
volumeAttributes = {
secretProviderClass = local.github_secret_name
}
}
}]
containers = [{
name = "listener"
volumeMounts = [{
name = "secrets-store"
mountPath = "/mnt/secrets-store"
readOnly = true
}]
}]
}
}
labels = { for k, v in local.tags : k => tostring(v) }
labels = merge(
{ for k, v in local.tags : k => tostring(v) },
{
"sidecar.istio.io/inject" = false
})
})]
depends_on = [
kubernetes_manifest.github_app_secret_provider,
kubernetes_service_account.gha_runner_scale_set,
kubernetes_service_account.listener_service_account,
helm_release.gha_runner_scale_set_controller,
]
}
I tried manually adding the annotation by modifying the resource directly and everything started to work, for obvious reasons this is not a real solution for us.
Not being able to add annotations to the listener Service Account is blocking us from adopting the new version of ARC.