RFE: steampipe can run in a container as anonymous user

[erikerlandson]

Is your feature request related to a problem? Please describe. In OpenShift, containers always run as an anonymous, random, uid, and group 0): for example 13223455:0 to run a containerized steampipe server in OpenShift, it is desirable to install steampipe as some user (USER steampipe:0), but then configure the install so that some other user (e.g. 13223455:0) can run that install.

Typically this is accomplished by giving the install directory tree something like: chmod -R g+rwX /opt/steampipe - that is, give group-0 rwX, so that an anonyumous user can read and write into the application tree.

However: doing this actually seems to break steampipe. I will provide examples of my experiments below.

Describe the solution you'd like Have the examples below run correctly. In other words be able to run a steampipe install as another user, different than the user used to install.

Describe alternatives you've considered One alternative, that does work, is to set anyuid privileges for the steampipe container - this tells OpenShift that it can run this container as steampipe:0, and since that is the user it was installed as, it all works. This is generally considered bad practice, and so the more general solution above is desirable.

Here I will describe experiments I've done to better explain how it is currently failing.

Consider this Containerfile (aka Dockerfile):

FROM registry.access.redhat.com/ubi8/ubi-minimal:8.4

USER root:0
RUN microdnf install wget curl tar gzip shadow-utils \
 && useradd --uid 9998 --gid 0 steampipe --base-dir /opt

RUN chown -R steampipe:0 /opt/steampipe

# install steampipe from their install.sh
ENV STEAMPIPE_VERSION=0.5.1
RUN echo \\
 && cd /tmp \
 && wget -nv https://raw.githubusercontent.com/turbot/steampipe/${STEAMPIPE_VERSION}/install.sh \
 && chmod a+rx ./install.sh \
 && ./install.sh ${STEAMPIPE_VERSION} \
 && rm -rf /tmp/* \
 && echo

# plugin installs and initialization need to be as non-root
USER steampipe:0
WORKDIR /opt/steampipe
# note there seems to be an initial error message but it is working
RUN echo \
 # the usual techniques of setting group-0 wrX access to run anonymously break steampipe
 #&& umask 002 \
 #&& chmod -R g+rwX /opt/steampipe \
 && steampipe plugin install steampipe kubernetes \
 && echo

# This query runs, because we are still the original install user
RUN steampipe query "select name from steampipe_registry_plugin"

If I attempted to add the following to the build, it will fail, because user 9999:0 does not have write access to .steampipe:

# the following will fail because steampipe wants you to have write permissions
# steampipe will not work when run as different user than the one that installed it
USER 9999:0

ENV STEAMPIPE_INSTALL_DIR=/opt/steampipe/.steampipe \
    STEAMPIPE_UPDATE_CHECK=false \
    HOME=/opt/steampipe \
    USER=steampipe

# this query will fail
RUN steampipe query "select name from steampipe_registry_plugin"

The normal fix for this is to do a chmod -R g+rwX /opt/steampipe:

RUN echo \
 && steampipe plugin install steampipe kubernetes \
 # give other users with gid 0 rwX access:
 && chmod -R g+rwX /opt/steampipe \
 && echo

# This will now fail!  Even though we are still 'steampipe:0'
RUN steampipe query "select name from steampipe_registry_plugin"

However, if I do that, steampipe will fail even if the user is still steampipe:0:

STEP 13: RUN steampipe query "select name from steampipe_registry_plugin"
Error: Could not start steampipe service: Error: could not establish connection with database

Note, the only thing that changed was adding "group-0" rwX - that breaks the install for some reason.

I did a second experiment where I set umask 002 prior to install:

RUN echo \
 && chmod -R g+rwX /opt/steampipe \
 && umask 002 \
 && steampipe plugin install steampipe kubernetes \
 && echo

In this case, the query does work for steampipe:0, but if I try to run as 9999:0 it crashes:

# set up a non-install user
USER 9999:0

ENV STEAMPIPE_INSTALL_DIR=/opt/steampipe/.steampipe \
    STEAMPIPE_UPDATE_CHECK=false \
    HOME=/opt/steampipe \
    USER=steampipe

# this query will crash
RUN steampipe query "select name from steampipe_registry_plugin"

The crash error is this:

STEP 16: RUN steampipe query "select name from steampipe_registry_plugin"
Error: Could not start steampipe service: Error: runtime error: invalid memory address or nil pointer dereference

[e-gineer]

Thanks @erikerlandson ... this is very comprehensive and helpful research! Improving "service management" in Steampipe and running in containers in general is something we are keen to do, but to set expectations for this issue, I'm not sure exactly when we'll get into it in detail.

[erikerlandson]

@e-gineer, no problem - I wanted to get a tracking issue on the books, as something to organize future investigation around.

[johnsmyth]

@erikerlandson I looked at this a bit today.

When starting steampipe with chmod -R g+rwX /opt/steampipe and running as the 9999:0 user, the postgres error is:

2021-06-24 22:27:39.578 UTC [25] FATAL:  data directory "/opt/steampipe/.steampipe/db/12.1.0/data" has wrong ownership
2021-06-24 22:27:39.578 UTC [25] HINT:  The server must be started by the user that owns the data directory.

I found a stackoverflow article that suggests that there may be a solution using nss_wrapper, but I have not yet investigated it further.

FYI, when starting steampipe with chmod -R g+rwX /opt/steampipe and running as the steampipe user, the postgres error is:

2021-06-24 22:20:14.155 UTC [57] FATAL:  data directory "/opt/steampipe/.steampipe/db/12.1.0/data" has invalid permissions
2021-06-24 22:20:14.155 UTC [57] DETAIL:  Permissions should be u=rwx (0700) or u=rwx,g=rx (0750).

Doing chmod 750 /opt/steampipe/.steampipe/db/12.1.0/data fixes the issue when running as the steampipe:0 user, but does not work for the 9999 user so doesn't ultimately fix your issue, but explains why steampipe failed even if the user is still steampipe:0 after you did chmod -R g+rwX /opt/steampipe

In case its helpful in troubleshooting, I found these errors by:

• running a shell in the container: docker run --rm -it --name steampipe steampipe /bin/sh
• setting the steampipe log level to trace: export STEAMPIPE_LOG=trace
• runnning steampipe query:

$ steampipe query
2021-06-24T23:03:34.105Z [TRACE] steampipe: db.EnsureDbAndStartService start
2021-06-24T23:03:34.105Z [TRACE] steampipe: ensure installed
2021-06-24T23:03:34.106Z [TRACE] steampipe: ensured installed
2021-06-24T23:03:34.106Z [TRACE] steampipe: GetRunStatus - pid 22 does not exist
2021-06-24T23:03:34.106Z [TRACE] steampipe: start service
Error: Could not start steampipe service: 2021-06-24T23:03:34.121Z [TRACE] steampipe: ensure installed
2021-06-24T23:03:34.122Z [TRACE] steampipe: ensured installed
2021-06-24T23:03:34.122Z [TRACE] steampipe: GetRunStatus - loadRunningInstanceInfo returned nil
2021-06-24T23:03:34.122Z [TRACE] steampipe: postgres start command:  /opt/steampipe/.steampipe/db/12.1.0/postgres/bin/postgres -p 9193 -c listen_addresses="localhost" -c application_name=steampipe -c cluster_name=steampipe -c autovacuum=off -c bgwriter_lru_maxpages=0 -c effective-cache-size=64kB -c fsync=off -c full_page_writes=off -c maintenance-work-mem=1024kB -c password_encryption=scram-sha-256 -c random-page-cost=0.01 -c seq-page-cost=0.01 -c temp-buffers=800kB -c timezone=UTC -c track_activities=off -c track_counts=off -c wal-buffers=32kB -c work-mem=64kB -c jit=off -c log_statement=all -c log_min_duration_statement=2000 -c logging_collector=on -c log_min_error_statement=error -c log_directory=/opt/steampipe/.steampipe/logs -c log_filename=database-%Y-%m-%d.log -D /opt/steampipe/.steampipe/db/12.1.0/data
2021-06-24T23:03:34.123Z [TRACE] steampipe: createDbClient
2021-06-24T23:03:34.123Z [TRACE] steampipe: status:  &{40 9193 [localhost 127.0.0.1] local query 5580-44fa-8d20 steampipe steampipe}
2021-06-24T23:03:34.123Z [TRACE] steampipe: Connection string:  host=localhost port=9193 user=steampipe dbname=steampipe sslmode=disable
Error: could not establish connection with database

• Running the start command from the trace log output:

sh-4.4$ /opt/steampipe/.steampipe/db/12.1.0/postgres/bin/postgres -p 9193 -c listen_addresses="localhost" -c application_name=steampipe -c cluster_name=steampipe -c autovacuum=off -c bgwriter_lru_maxpages=0 -c effective-cache-size=64kB -c fsync=off -c full_page_writes=off -c maintenance-work-mem=1024kB -c password_encryption=scram-sha-256 -c random-page-cost=0.01 -c seq-page-cost=0.01 -c temp-buffers=800kB -c timezone=UTC -c track_activities=off -c track_counts=off -c wal-buffers=32kB -c work-mem=64kB -c jit=off -c log_statement=all -c log_min_duration_statement=2000 -c logging_collector=on -c log_min_error_statement=error -c log_directory=/opt/steampipe/.steampipe/logs -c log_filename=database-%Y-%m-%d.log -D /opt/steampipe/.steampipe/db/12.1.0/data
2021-06-24 23:05:53.619 UTC [43] FATAL:  data directory "/opt/steampipe/.steampipe/db/12.1.0/data" has wrong ownership
2021-06-24 23:05:53.619 UTC [43] HINT:  The server must be started by the user that owns the data directory.

[harlequin]

Pleased reopen, because Clean solution seems to be missing

[e-gineer]

@harlequin Could you please share more about your use case for this please? Understand the problem and why this would be nice to have, but wondering why specifically our current docker container is insufficient (need to have). Thanks!

[harlequin]

@e-gineer We are operating an Openshift Cluster, where every namespace have a random uid and gid when starting a container.

For example uid will be 1000870000

When the container start the error message

2022-04-04 06:06:21.777 UTC [TRACE] steampipe: initdb start: /data/db/12.1.0/postgres/bin/initdb --auth=trust --username=root --pgdata=/data/db/12.1.0/data --encoding=UTF-8 --wal-segsize=1 --debug
2022-04-04 06:06:21.837 UTC [TRACE] steampipe: initdb failed:
 Running in debug mode.
initdb: could not look up effective user ID 1000870000: user does not exist
2022-04-04 06:06:21.837 UTC [TRACE] steampipe: initDatabase failed: exit status 1
Error: Initializing database... FAILED!

will pop up.

[erikerlandson]

Would still be nice if we could run steampipe in OpenShift without setting anyuid, which is not a secure policy

[tbugfinder]

This also applies to our environment. Basically anyuid is declined by the team operating the platform.

[johnsmyth]

I have no experience with OpenShift, but it appears to be kubernetes based. I believe it supports init containers

1. Can you run an initContainer as root or any uid?
2. Do you know the UID that you will be using in advance (can you code it in the pod spec?) If yes to both, you could chmod the files in the init container. For example, create a data dir volume (can even be emptyDir):

      volumes:
      - emptyDir: {}
        name: sp-datadir

Then add an initcontainer, something like this:

      initContainers:
      - args:
        - -c
        - chown -R 9193:0 /home/steampipe/.steampipe/db/14.2.0/data 
        command:
        - /bin/sh
        image: busybox:1.33.1
        imagePullPolicy: IfNotPresent
        name: workspace-init
        securityContext:
          readOnlyRootFilesystem: true
          runAsGroup: 0
          runAsNonRoot: false
          runAsUser: 0
        volumeMounts:
        - mountPath: /home/steampipe/.steampipe/db/14.2.0/data
          name: sp-datadir

Then mount that same data dir to the same path in your steampipe container:

        volumeMounts:
        - mountPath: /home/steampipe/.steampipe/db/14.2.0/data
          name: sp-datadir

[luisffc]

I was thinking on simply deleting the data base at build time, so Steampipe will recreate it at run time, hence, random user will own them. It can work if Openshift creates the random user on the container

Changed @erikerlandson 's Dockerfile to the following:

FROM registry.access.redhat.com/ubi8/ubi-minimal:8.4

USER root:0
RUN microdnf install wget curl tar gzip shadow-utils \
 && useradd --uid 9998 --gid 0 steampipe --base-dir /opt \
 # Emulating OpenShift creating the random user. So, remove the line bellow when running on OpenShift
 && useradd --uid 9999 --gid 0 randomuser --base-dir /opt

RUN chown -R steampipe:0 /opt/steampipe

# Install steampipe
RUN echo \
 && cd /tmp \
 && wget -nv https://raw.githubusercontent.com/turbot/steampipe/main/install.sh \
 && chmod a+rx ./install.sh \
 && ./install.sh \
 && rm -rf /tmp/* \
 && echo

# plugin installs and initialization need to be as non-root
USER steampipe:0
WORKDIR /opt/steampipe

RUN steampipe plugin install steampipe kubernetes \
 && chmod -R g+rwX /opt/steampipe \
 && rm -rf /opt/steampipe/.steampipe/db

# Emulates a ramdon user
USER 9999:0

ENV STEAMPIPE_INSTALL_DIR=/opt/steampipe/.steampipe \
    STEAMPIPE_UPDATE_CHECK=false \
    HOME=/opt/steampipe \
    STEAMPIPE_LOG=trace

# Running steampipe as a random user
# At this time, Steampipe will notice the db doesn't exist and will recreate it, so the db files will belong to running user
RUN steampipe query "select name from steampipe_registry_plugin"

The above works on Docker, but not sure it will work on OpenShift

To see the output of docker build I've added --progress=plain --no-cache parameters to the docker build command

[erikerlandson]

@luisffc The random uid is created by openshift when it creates the pod, so it is at container run time. If the steampipe query could be run by the container cmd or entrypoint perhaps, and created then?

[luisffc]

@erikerlandson yep, by the time Steampipe runs, it will then recreate the database

[erikerlandson]

@luisffc If I'm understanding your idea, then this is where you remove the db: rm -rf /opt/steampipe/.steampipe/db

But this part also happens at podman build time, and would re-create it: RUN steampipe query "select name from steampipe_registry_plugin"

So I'd guess that it might work to do that rm part, but not the second steampipe query part.

Maybe steampipe will just re-create its db the first time someone hits the running container with a query?

[luisffc]

Correct @erikerlandson, in this example the RUN steampipe query line would recreate the database while still at build time. Please consider the following:

FROM registry.access.redhat.com/ubi8/ubi-minimal:8.4

USER root:0
RUN microdnf install wget curl tar gzip shadow-utils \
 && useradd --uid 9998 --gid 0 steampipe --base-dir /opt \
 # Emulates OpenShift creating the random user. So, remove the line bellow when running on OpenShift
 && useradd --uid 9999 --gid 0 randomuser --base-dir /opt

RUN chown -R steampipe:0 /opt/steampipe \
 && chmod -R g+rwX /opt/steampipe \
 && rm -rf /opt/steampipe/.steampipe/db/14.2.0/data/

# Install steampipe
RUN echo \
 && cd /tmp \
 && wget -nv https://raw.githubusercontent.com/turbot/steampipe/main/install.sh \
 && chmod a+rx ./install.sh \
 && ./install.sh \
 && rm -rf /tmp/* \
 && echo

# plugin installs and initialization need to be as non-root
USER steampipe:0
WORKDIR /opt/steampipe

RUN steampipe plugin install steampipe kubernetes

ENV STEAMPIPE_INSTALL_DIR=/opt/steampipe/.steampipe \
    STEAMPIPE_UPDATE_CHECK=false \
    HOME=/opt/steampipe

Running with a emulated random user 9999: docker run --user 9999:0 --rm -it steampipe-new steampipe query "select name from steampipe_registry_plugin"

You will noticed a message saying: "Initializing db"

The trade off here is that the db initialization adds 4 seconds to the running time of the container. Based on my local tests, running that docker run statement above went from 6 sec to 10 sec

[erikerlandson]

@luisffc thanks, this is very interesting, and might work! Adding a few extra seconds to pod startup time would not really be a problem, so if it works this would be a very reasonable solution. I will give it a try when I have some time.