actions
Runner custom volumeMount is not available to mount in mode dind when using containerized actions or running any other container
Checks
- [X] I've already read https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners-with-actions-runner-controller/troubleshooting-actions-runner-controller-errors and I'm sure my issue is not covered in the troubleshooting guide.
- [X] I am using charts that are officially provided
Controller Version
0.8.1
Deployment Method
Helm
Checks
- [X] This isn't a question or user support case (For Q&A and community support, go to Discussions).
- [X] I've read the Changelog before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes
To Reproduce
In the values file, specify the `runner` container volumeMounts:
template:
spec:
containers:
- name: runner
volumeMounts:
- name: pv-gha-terraform-providers-cache
mountPath: /mnt/fsx/terraform-providers-cache
volumes:
- name: pv-gha-terraform-providers-cache
persistentVolumeClaim:
claimName: pvc-gha-terraform-providers-cache
Then, try to use a containerized action, when using one, github by default mount the `/home/runner/_work/{repo_name}` to the container path `/github/workspace`:
`"/home/runner/_work/{repo_name}":"/github/workspace"`
This is the default behavior, and the only way to expose directory to the containerized action is to expose it via `/home/runner/_work/` directory
In our workflow, after mounting the PVC to `/mnt/fsx/terraform-providers-cache` we also run a step that mount the PVC mountpoint to the GITHUB_WORKSPACE (/home/runner/_work/{repo_name} so it will be exposed to any containerized action:
`mount --bind /mnt/fsx/terraform-providers-cache/ ${GITHUB_WORKSPACE}/terraform-providers-cache`
Now, when trying to run containerized action, the actual mount is coming from the `dind` container and not from the `runner` container.
Its very easy to reproduce:
runner@arc-terraform-live-large-runners-q28b7-runner-j8dct:~$ ls -la /mnt/fsx/terraform-providers-cache/
total 65
drwxrwxrwx 3 runner runner 33280 Feb 12 13:29 .
drwxr-xr-x 3 root root 39 Feb 12 13:27 ..
drwxrwxrwx 10 root root 33280 Oct 3 13:13 registry.terraform.io
runner@arc-terraform-live-large-runners-q28b7-runner-j8dct:~$ docker run -it -v /mnt/fsx/terraform-providers-cache/:/fsx busybox
/ # ls /fsx/
/ # ls -la /fsx/
total 0
drwxr-xr-x 2 root root 6 Feb 12 13:28 .
drwxr-xr-x 1 root root 29 Feb 12 13:29 ..
After mounting the volume to the exact same location in the `dind` container it will work with no issues and the busybox container will be able to access the data in /mnt/fsx/terraform-providers-cache/ via /fsx.
Describe the bug
When using containerized actions, it's not possible to mount any external volume to the container of the containerized action, this is because its not possible to mount a custom volumes (volumeMounts) to the dind container which is the actual container that mounting his volume/local directory to the wanted container.
This is also relevant to cases where the users run any other container and trying to mount external volume (that is mounted to the runner container) to the new container, but the dind container is missing this mount, so it will mount an empty directory without the actual content.
Describe the expected behavior
When mounting a volume to the runner container, it should be also possible to mount the same volume in the dind container.
This is something that used to work when the docker daemon was inside the runner container (using the summerwinds image for example), however, since moving to runner-scale-set, this is not possible and the current helm chart does not allow any modifications to the dind container volumes.
Without allowing volume mounts to the dind container, we won't be able to mount any external volume to the containers we are running inside the runner (dind).
We need to be able to customize the dind container here: https://github.com/actions/actions-runner-controller/blob/a68aa00bd8d3c30b2a8636e2eddaccb9aec07766/charts/gha-runner-scale-set/templates/_helpers.tpl#L98C1-L116C11
Additional Context
None
Controller Logs
None
Runner Pod Logs
None
Hey @AlonAvrahami,
In case you want to customize the dind configuration, you should leave the container mode empty, and uncomment the expanded dind spec documented in values.yaml file. We documented this in Customizing container modes section of our docs.
If you follow this approach, do you still see an issue adding a custom volume mount?
Closing this one. Feel free to comment on it if you find issues with the documented approach :relaxed:
Hi @nikola-jokic
So i want to update about this :)
First, the documentation are not very clear, and i had to dig in and understand what is the default configuration of the template and copy it entirely (including the init-containers, containers, volumes and all of the configurations), so ideally, i do think it should support custom volumeMount for the dind container. Also, i think its better to provide additional option as described below.
After a long investigation, i want to update that i didn't found a solution to my problem, where in the old ARC that had runners with the docker daemon inside of it - it was working fine.
The main issue is that the runner cannot control any of the dind container mounts, and cannot send commands to it either, resulting in having to mount the external mount via the POD specs, but what about the mountpoint itself? in order to have it available for the containerized actions, it must be inside the GITHUB_WORKSPACE directory or the GITHUB_HOME that are mounted to containerized actions, which are created only after the workflow begins..
I tried to mount it manually to multiple directories (ie /home/runner/_work/_temp/_github_home/MOUNT_NAME) but it results in a readonly issue in the runner container (also tried privileged mode and fsGroup security contexts).
I think it should be considered to release a new container with the docker daemon as part of the image, so it would be able to control external mount points, and also allow mounts as part of the workflow (nfs mount command)
This is the approach we probably will go with, but for the rest of the community, i think it will be better to give the options and choose whatever fits per use-case. Summerwinds images had this options and it worked great.
When working with containerized actions, these are the available mountpoints:
"/home/runner/_work/_temp/_github_home":"/github/home"
"/home/runner/_work/_temp/_github_workflow":"/github/workflow"
"/home/runner/_work/_temp/_runner_file_commands":"/github/file_commands"
"/home/runner/_work/REPO/REPO":"/github/workspace"
so the actual mountpoint must be in one of the folders mounted to the action container.
What we did till now? we mounted the FSX/NFS in the pod, and included a step to create a bind mount between the mountpoint and the wanted directory before calling the action:
sudo mkdir -p ${GITHUB_WORKSPACE}/terraform-providers-cache
sudo mount --bind /mnt/fsx/terraform-providers-cache/ ${GITHUB_WORKSPACE}/terraform-providers-cache
once the action is done, we also had an umount step in the end:
sudo umount ${GITHUB_WORKSPACE}/terraform-providers-cache
Why we did this? because after each workflow, the GITHUB_WORKSPACE is cleaned up, so we had to have another step to unmount the external storage to avoid deleting the data in it.
Let me know if i can assist or if you have any other question, would be happy to contribute to this effort :)