[EKS] [request]: Allow running all managed add ons as non-root where possible

[iandrewt]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Tell us about your request We are being instructed by our security team that all pods in our EKS clusters need to run as non-root or go through a lengthy exception process for each container that cannot fit this requirement. We would like to be able to run the EBS and EFS driver pods as non-root, or at least a subset of the containers in the pod if there are containers that fundamentally cannot run as non-root.

Which service(s) is this request for? EKS

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard? All of the containers in the EBS and EFS driver pods run as root, and while the helm chart does offer the ability to set a security context on each container, this is not exposed through EKS Add-ons, and even if it was, there is no documentation for which container are safe to set to a non-root user and what UID/GID should be used. It would also be good to be able to drop all Linux capabilities and only grant capabilities relevant to the task associated with each container, along with any other securityContext options to lock down the pods.

Are you currently working around this issue? Currently just gathering information to prepare for a security exception, assuming this isn't something that can be resolved easily.

Additional context This would extend to beyond just EBS and EFS pods. I haven't done a full check of all EKS Add-ons to see where they're running as non-root where they might not need to be

Attachments

[ConnorJC3]

The ebs-csi-node pods require root access for several reasons:

• The /var/lib/kubelet/ subdirectories that it mounts to are typically owned by root
• Mounting a volume on Linux requires CAP_SYS_ADMIN (effectively equivalent to root): https://www.man7.org/linux/man-pages/man7/capabilities.7.html#:~:text=mount%282%29%2C%20umount%282%29%2C
• In order for the mounts to propagate to the host, the pod requires a Bidirectional (Kubernetes name for rshared) mount of /var/lib/kubelet/, which requires a privileged pod on Kubernetes: https://kubernetes.io/docs/concepts/storage/volumes/#mount-propagation:~:text=therefore%20it%20is%20allowed%20only%20in%20privileged%20containers

Thus, the ebs-plugin container (which performs the mounts) in such pods already runs with what is likely the minimal set of permissions possible. I'll add an item to my team's backlog to investigate the other two containers in that pod, but those may not be possible (in particular, they need to access Unix sockets likely to be owned by root).

The ebs-csi-controller pods do not need root and already run with as a non-root user.

[iandrewt]

@ConnorJC3 thanks for that info, very helpful!

I've also noticed that the scanning tools our security team are using are checking the pod specs, not the status, where it is sometimes reported what UID/GID a container is actually using despite not mentioning it in the security context. This does make sense to scan this way, since admission controllers can only check the spec before scheduling.

Because of this, they're flagging a false-positive of CoreDNS running as root. I can see in the pod status that it runs as UID 65532, which would have been defined in a USER statement in the Dockerfile, but the lack of a runAsNonRoot: true statement in the pod spec means there's nothing to stop the pod from starting if the image is replaced with one that does run as root.

I know most admission controllers allow excluding namespaces, but our security team wants to manage exceptions on an application by application basis, regardless of namespace. It would be weird to need an exception for CoreDNS when it is in fact running as a non-root user.

We are exploring using https://docs.aws.amazon.com/eks/latest/userguide/kubernetes-field-management.html to set these options ourselves, but it would be great if the defaults were already the full set of best practices. I did try this for the ebs-csi-node pod - I wasn't able to start the liveness-probe and node-driver-registrar containers as non-root due to the permissions needed for the Unix sockets, but I was able to set privileged: false (edit: nope. the scanner complained that within the pod, a shared volume was used between privileged and unprivileged containers. I really can't win here...) and capabilities: {drop: ["ALL"]} without issues. The ebs-plugin container worked with capabilities: {drop: ["ALL"], add: ["CAP_SYS_ADMIN"]}, but that probably doesn't provide any value if the container is privileged: true anyway right?