Install helm chart with non-root user

[dnskr]

Problem

The helm chart cannot be successfully installed if non-root user is used to run the opencost-ui container.

The issue exists at least on OpenShift clusters, because OpenShift runs containers using users with random large number id (it adds runAsNonRoot: true and runAsUser: 123456789 to securityContext). The issue can be reproduced in other environments with the configs provided below.

Prerequisites - deploy Prometheus as mentioned in the documentation:

helm install prometheus --repo https://prometheus-community.github.io/helm-charts prometheus --namespace prometheus-system --create-namespace --set prometheus-pushgateway.enabled=false --set alertmanager.enabled=false -f https://raw.githubusercontent.com/opencost/opencost/develop/kubernetes/prometheus/extraScrapeConfigs.yaml

How to reproduce

Create config file values-test.yaml:

opencost:
  exporter:
    securityContext:
      runAsNonRoot: true
      runAsUser: 123456789
  ui:
    securityContext:
      runAsNonRoot: true
      runAsUser: 123456789

Install opencost with values-test.yaml config file:

helm install opencost opencost/opencost -f values-test.yaml

Issue 1 The pod cannot start because of Permission denied in /var/www/ directory, see opencost-ui container logs:

cp: can't create '/var/www/MaterialIcons-Regular.12b3b105.woff': Permission denied
cp: can't create '/var/www/MaterialIcons-Regular.333251c4.ttf': Permission denied
cp: can't create '/var/www/MaterialIcons-Regular.b99eb5ce.woff2': Permission denied
cp: can't create '/var/www/MaterialIcons-Regular.e9e55c63.eot': Permission denied
cp: can't create '/var/www/favicon.7eff484d.ico': Permission denied
cp: can't create '/var/www/index.3406705b.css': Permission denied
cp: can't create '/var/www/index.3406705b.css.map': Permission denied
cp: can't create '/var/www/index.4170b679.js': Permission denied
cp: can't create '/var/www/index.4170b679.js.map': Permission denied
cp: can't create '/var/www/index.6d70e179.js': Permission denied
cp: can't create '/var/www/index.6d70e179.js.map': Permission denied
cp: can't create '/var/www/index.html': Permission denied
cp: can't create '/var/www/index.runtime.256cff63.js': Permission denied
cp: can't create '/var/www/index.runtime.256cff63.js.map': Permission denied
cp: can't create '/var/www/index.runtime.997a1df7.js': Permission denied
cp: can't create '/var/www/index.runtime.997a1df7.js.map': Permission denied
cp: can't create '/var/www/logo.b9464e00.png': Permission denied

Possible fix (workaround) for issue 1 Add emptyDir volume to opencost-ui container:

opencost:
  exporter:
    securityContext:
      runAsNonRoot: true
      runAsUser: 123456789
  ui:
    securityContext:
      runAsNonRoot: true
      runAsUser: 123456789
    extraVolumeMounts:
      - name: empty-var-www
        mountPath: /var/www

extraVolumes:
  - name: empty-var-www
    emptyDir: {}

Issue 2 The pod still cannot start because of Permission denied for /etc/nginx/conf.d/default.nginx.conf, see opencost-ui container logs:

'/opt/ui/dist/MaterialIcons-Regular.12b3b105.woff' -> '/var/www/MaterialIcons-Regular.12b3b105.woff'
'/opt/ui/dist/MaterialIcons-Regular.333251c4.ttf' -> '/var/www/MaterialIcons-Regular.333251c4.ttf'
'/opt/ui/dist/MaterialIcons-Regular.b99eb5ce.woff2' -> '/var/www/MaterialIcons-Regular.b99eb5ce.woff2'
'/opt/ui/dist/MaterialIcons-Regular.e9e55c63.eot' -> '/var/www/MaterialIcons-Regular.e9e55c63.eot'
'/opt/ui/dist/favicon.7eff484d.ico' -> '/var/www/favicon.7eff484d.ico'
'/opt/ui/dist/index.3406705b.css' -> '/var/www/index.3406705b.css'
'/opt/ui/dist/index.3406705b.css.map' -> '/var/www/index.3406705b.css.map'
'/opt/ui/dist/index.4170b679.js' -> '/var/www/index.4170b679.js'
'/opt/ui/dist/index.4170b679.js.map' -> '/var/www/index.4170b679.js.map'
'/opt/ui/dist/index.6d70e179.js' -> '/var/www/index.6d70e179.js'
'/opt/ui/dist/index.6d70e179.js.map' -> '/var/www/index.6d70e179.js.map'
'/opt/ui/dist/index.html' -> '/var/www/index.html'
'/opt/ui/dist/index.runtime.256cff63.js' -> '/var/www/index.runtime.256cff63.js'
'/opt/ui/dist/index.runtime.256cff63.js.map' -> '/var/www/index.runtime.256cff63.js.map'
'/opt/ui/dist/index.runtime.997a1df7.js' -> '/var/www/index.runtime.997a1df7.js'
'/opt/ui/dist/index.runtime.997a1df7.js.map' -> '/var/www/index.runtime.997a1df7.js.map'
'/opt/ui/dist/logo.b9464e00.png' -> '/var/www/logo.b9464e00.png'
running with BASE_URL=/model
/usr/local/bin/docker-entrypoint.sh: line 21: can't create /etc/nginx/conf.d/default.nginx.conf: Permission denied

[mattray]

I'll try to investigate, I suspect the fix will be in opencost/opencost rather than here.

[asdfgugus]

Looks like the same issue as here: https://github.com/opencost/opencost-ui/issues/4

[dnskr]

Looks like the same issue as here: opencost/opencost-ui#4

Right, it is very similar issue.

[brito-rafa]

FYI, from OpenShift official documentation to let containers to run as arbitrarily container, they say "directories and files that are written to by processes in the image must be owned by the root group and be read/writable by that group": https://docs.openshift.com/container-platform/4.15/openshift_images/create-images.html#use-uid_create-images

Perhaps we would need to tweak the opencost-ui dockerfile and test on OpenShift/non-OpenShift setup

[asdfgugus]

Well the issue is in the docker-entrypoint.sh script:

• cp -rv /opt/ui/dist/* /var/www
• envsubst '$API_PORT $API_SERVER $UI_PORT' < /etc/nginx/conf.d/default.nginx.conf.template > /etc/nginx/conf.d/default.nginx.conf

[asdfgugus]

Perhaps we would need to tweak the opencost-ui dockerfile and test on OpenShift/non-OpenShift setup

I totally agree on that. We should make it read-only again.

[hobti01]

The dockerfile is creating non-root user 1001 and /var/www is owned by this user and group 1000 (https://github.com/opencost/opencost-ui/blob/main/Dockerfile#L38-L39).

For OpenShift with random UID and root group I understand that you should use RunAsGroup if you are mounting an external volume for /var/www

opencost:
  ui:
    securityContext:
      runAsGroup: 0
      runAsNonRoot: true
      runAsUser: 123456789

Running in Kubernetes without an additional volume, one must disable readOnlyRootFilesystem and use the correct user (doc example uses 1000 which is incorrect):

opencost:
  ui:
    securityContext:
      allowPrivilegeEscalation: false
      capabilities:
        drop:
          - ALL
      # Cannot copy webUI files with read-only root filesystem unless mounting a volume to /var/www
      readOnlyRootFilesystem: false
      runAsNonRoot: true
      runAsUser: 1001
      seccompProfile:
        type: RuntimeDefault

[DriesS]

Has anyone found a solution or workaround for this issue?

We're running OpenShift, and user 1001 is not permitted. The pod fails to start with any other user because the permissions for /etc/nginx/conf.d/default.nginx.conf are set to 1001:1001.

[mattray]

I tried to test this but was running into other issues related to 1.110.0. Everything in 1.111.0 seems to be working now, I'll try to circle back but I don't have a lot of cycles to test fixes myself.

[jamalsms]

In version 1.111.0 I got the problem witch UI:

/usr/local/bin/docker-entrypoint.sh: line 21: can't open /etc/nginx/conf.d/default.nginx.conf.template: no such file

My security-context:

          securityContext:
            runAsUser: 1001
            runAsGroup: 1001
            readOnlyRootFilesystem: false
            allowPrivilegeEscalation: false
            capabilities:
              drop:
                - ALL
            runAsNonRoot: true
            seccompProfile:
              type: RuntimeDefault

[toscott]

Closing in favor of https://github.com/opencost/opencost-ui/issues/4 where a change will need to be made.