Create Kubernetes specific install docs

[loshz]

I recently started migrating from a competitor CI to Buildkite and so far I am loving the service! I am running my own agents in a self-hosted Kubernetes cluster with no problems so far.

Even though I am running on AWS, I loosely followed the GCloud docs and got things set up fairly quickly. However, I have a couple of "concerns" about using the approach the docs recommend.

Firstly, from my experience, it is not recommended to mount the host Docker socket (Docker-on-Docker) into a pod as this could potentially expose the following security issues:

• Malicious code on the agent could have full access to the Docker daemon on the host machine.
• Any Docker resources created via an agent build would be unknown to the kubelet and could cause scheduling/resource issues.
• Running the buildkite-agent with a privileged security context could again mean code ran via a build could potentially have root access on the host machine.

I am by no means a security expert, but after using Kubernetes in production for several years I have come to learn that some of the above practices could be dangerous.

That being said, I would like to propose a new, Kubernetes specific, set of installation docs- and I would be more than happy to hep write them alongside anyone else!

I have the following Docker-in-Docker (dind) setup running in one of my clusters:

spec:
  containers:
    - name: dind
      image: 'docker:18.06-dind' # must be the same version of Docker as the host machine
      securityContext:
        privileged: true
      volumeMounts:
        - name: dind-storage
          mountPath: /var/lib/docker
    - name: buildkite-agent
      image: 'buildkite/agent:3.22.0'
      livenessProbe:
        httpGet:
          path: /
          port: 6000
        initialDelaySeconds: 30
        periodSeconds: 3
      env:
        - name: DOCKER_HOST
          value: 'tcp://localhost:2375'
        - name: BUILDKITE_AGENT_HEALTH_CHECK_ADDR
          value: ':6000'
        - name: BUILDKITE_AGENT_TOKEN
          valueFrom:
            secretKeyRef:
              name: buildkite-agent
              key: token
  volumes:
    - name: dind-storage
      emptyDir: {}

Happy to provide any further information!

[yob]

We'd love to improve our documentation around running the agent on Kubernetes, thanks for offering to help out.

As you point out, the security issues are a hurdle. I suspect one of the reasons we haven't managed to publish more official documentation yet is the best approach isn't necessarily obvious.

You're right that mounting the host docker socket has security implications (and it may not always be an option, as some kube hosts experiment with containerd instead of docker). However, you're also correct that the docker-in-docker approach isn't particularly secure either - privileged: true means any workload run by the agent is able to escape the pod and access the host as root.

One option is we document a way to run the agent on kubernetes without docker. That's a secure configuration, however it requires builds of a very different shape (using something like kaniko to build images, docker-compose can't be used to start a database, etc).

Is docker-in-docker the approach that's working bets for you?

[loshz]

I absolutely agree- running this sort of setup in k8s can be a bit of a minefield and there's definitely not a one solution to fit all!

I would suggest that the new docs start with potential warnings and informs the user of any assumptions made, that way we can keep it as generic as possible.

DinD has been working perfectly for me. No issues so far.