Index was out of range error when using Kubernetes mode with a sidecar

[lacarvalho91]

When using k8s container mode and adding a sidecar to the hook template we consistently get the following workflow error:

Run '/home/runner/k8s/index.js'
Warning: Skipping image override: image can't be overwritten
sh: [ $(cat /etc/*release* | grep -i -e "^ID=*alpine*" -c) != 0 ] || exit 1: not found
Error: Index was out of range. Must be non-negative and less than the size of the collection. (Parameter 'index')

The logs point to the runner:

[2024-03-14 00:21:49Z ERR  StepsRunner] Caught exception from step: System.ArgumentOutOfRangeException: Index was out of range. Must be non-negative and less than the size of the collection. (Parameter 'index')
   at GitHub.Runner.Worker.Container.ContainerHooks.ContainerHookManager.UpdateJobContext(IExecutionContext context, ContainerInfo jobContainer, List`1 serviceContainers, PrepareJobResponse response)
   at GitHub.Runner.Worker.Container.ContainerHooks.ContainerHookManager.PrepareJobAsync(IExecutionContext context, List`1 containers)
   at GitHub.Runner.Worker.ContainerOperationProvider.StartContainersAsync(IExecutionContext executionContext, Object data)
   at GitHub.Runner.Worker.JobExtensionRunner.RunAsync()
   at GitHub.Runner.Worker.StepsRunner.RunStepAsync(IStep step, CancellationToken jobCancellationToken)
[2024-03-14 00:21:49Z INFO StepsRunner] Step result: Failed

This issue in the runner repo seems to be reporting the same thing. But I thought I should raise it here as it seems that sidecars just don't work with kubernetes mode, even though the examples have one defined and the ADR seems to suggest it is intended to work.

If we remove the sidecar then its fine.

We're using a very similar template to what is in the examples, we also tried one without the service container:

    metadata:
      annotations:
        annotated-by: "extension"
      labels:
        labeled-by: "extension"
    spec:
      securityContext:
        runAsUser: 1000
        runAsGroup: 3000
      restartPolicy: Never
      containers:
        - name: $job # overwrites job container
          env:
            - name: ENV1
              value: "value1"
          imagePullPolicy: Always
          image: "busybox:1.28" # Ignored
          command:
            - sh
          args:
            - -c
            - sleep 50
        - name: $redis # overwrites redis service
          env:
            - name: ENV2
              value: "value2"
          image: "busybox:1.28" # Ignored
          resources:
            requests:
              memory: "1Mi"
              cpu: "1"
            limits:
              memory: "1Gi"
              cpu: "2"
        - name: side-car
          image: "ubuntu:latest" # required
          command:
            - sh
          args:
            - -c
            - sleep 60

[lacarvalho91]

Hoping theres a workaround 🙏

[nikola-jokic]

Hey @lacarvalho91,

Thank you for reporting this! The problem is on the runner side, but we will fix it on the hook side. Essentially, we put side-car into services context, which causes the out of bound exception... Sorry this happened, I'll introduce a fix soon :relaxed:

Also, sorry for the delay!

[lacarvalho91]

@nikola-jokic awesome that sounds promising. I was very much hoping it could be fixed on this side. Thank you 😊

[lacarvalho91]

@nikola-jokic when that PR is merged what needs to happen for it to be included in the runners (images) themselves?

[israel-morales]

We are experiencing the same behavior when attempting to enable istio at the namespace level:

kubectl label namespace $NAMESPACE istio-injection=enabled

We are looking to gather http-level observability metrics and are blocked at the moment.