docker icon indicating copy to clipboard operation
docker copied to clipboard
verified publisher icon librenms

Seemingly incorrect ownership on /opt/librenms/cache directory

Open jcostom opened this issue 3 years ago • 1 comments

Behaviour

This is present in at least version 22.4.1 of the librenms/librenms container. I would guess it's likely in most versions as well. Honestly, I'm not sure if it's actually impairing function, or more of a nuisance/log-spam type issue.

The default ownership of the /opt/librenms/cache directory is nobody:nogroup. This impacts the device discovery process, resulting in errors I've most often observed with the os_defs.cache update performed during the discovery process.

Steps to reproduce this issue

  1. Deploy the container in the typical way.
  2. Discover a device. (I forced it via Capture -> Discover)

Expected behaviour

I'd expect (naturally) to have the permissions correct and not get the error. I know, captain obvious here. And of course, I can correct it easily enough, at least until there's a new release, and I update, at which point I get to update the permissions once more.

Changing ownership:

$ docker exec librenms chown librenms:librenms /opt/librenms/cache

$ docker exec librenms ls -ld /opt/librenms/cache
drwxr-xr-x 1 librenms librenms 4096 Apr 24 10:25 /opt/librenms/cache

After changing permissions, this is what that same discovery op looks like:

===================================
Version info:
Commit SHA: 
Commit Date: 
DB Schema: 2022_04_08_085504_isis_adjacencies_table_add_index (236)
PHP: 7.4.29
MySQL: 10.7.3-MariaDB-1:10.7.3+maria~focal
RRDTool: 1.7.2
SNMP: 5.9.1
==================================DEBUG!

Updating os_def.cache
[.......]

Actual behaviour

Demonstrate the issue:

$ docker exec librenms ls -ld /opt/librenms/cache
drwxr-xr-x 2 nobody nogroup 4096 Apr 24 10:25 /opt/librenms/cache

With the default permissions in-place, here's the error that is seen in all discovery ops, check near the top of the log.

===================================
Version info:
Commit SHA: 
Commit Date: 
DB Schema: 2022_04_08_085504_isis_adjacencies_table_add_index (236)
PHP: 7.4.29
MySQL: 10.7.3-MariaDB-1:10.7.3+maria~focal
RRDTool: 1.7.2
SNMP: 5.9.1
==================================DEBUG!

Updating os_def.cache  


Warning: file_put_contents(/opt/librenms/cache/os_defs.cache): failed to open stream: Permission denied in /opt/librenms/LibreNMS/Util/OS.php on line 111
[.......]

Configuration

This really doesn't appear to be system or Docker version dependent, but I'll fill it all in anyhow.. I've also tested on both Ubuntu 20 and 22.

  • Docker version: Docker version 20.10.12, build 20.10.12-0ubuntu4
  • Docker compose version: Whatever's embedded in Portainer-CE 2.11.1, it's what I'm deploying with.
  • Platform: Ubuntu 20.04 & 22.04
  • System info: 5.15.0-27-generic #28-Ubuntu SMP Thu Apr 14 04:55:28 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
  • Include all necessary configuration files : Here's the compose file I'm using for the Portainer stack...
---
version: '3'

services:
  mariadb:
    image: mariadb:10.7
    container_name: mariadb
    command:
      - "mysqld"
      - "--innodb-file-per-table=1"
      - "--lower-case-table-names=0"
      - "--character-set-server=utf8mb4"
      - "--collation-server=utf8mb4_unicode_ci"
    volumes:
      - /var/docks/mariadb:/var/lib/mysql
    environment:
      - TZ=America/New_York
      - MYSQL_ALLOW_EMPTY_PASSWORD=yes
      - MYSQL_DATABASE=librenms
      - MYSQL_USER=librenms
      - MYSQL_PASSWORD=xxREDACTEDxx
    restart: unless-stopped
    networks:
      - librenms

  oxidized:
    image: oxidized/oxidized:latest
    container_name: oxidized
    tty: true
    volumes:
      - /var/docks/oxidized:/root/.config/oxidized
    environment:
      - CONFIG_RELOAD_INTERVAL=3600
    restart: unless-stopped
    networks:
      - librenms
  
  redis:
    image: redis:6
    container_name: redis
    user: 1000:1000
    restart: unless-stopped
    environment:
      - TZ=America/New_York
    volumes:
      - /var/docks/redis:/data
    networks:
      - librenms

  librenms:
    image: librenms/librenms:latest
    container_name: librenms
    hostname: librenms
    depends_on:
      - mariadb
      - redis
    volumes:
      - /var/docks/librenms:/data
    environment:
      - TZ=America/New_York
      - PUID=1000
      - PGID=1000
      - DB_HOST=mariadb
      - DB_NAME=librenms
      - DB_USER=librenms
      - DB_PASSWORD=xxREDACTEDxx
      - DB_TIMEOUT=60
      - REDIS_HOST=redis
      - REDIS_PORT=6379
      - REDIS_DB=0
      - MEMORY_LIMIT=256M
      - UPLOAD_MAX_SIZE=16M
      - OPCACHE_MEM_SIZE=128
      - REAL_IP_FROM=0.0.0.0/32
      - REAL_IP_HEADER=X-Forwarded-For
      - LOG_IP_VAR=http_x_forwarded_for
      - LIBRENMS_SNMP_COMMUNITY=xxREDACTEDxx
      - LIBRENMS_WEATHERMAP=false
      - LIBRENMS_WEATHERMAP_SCHEDULE=*/5 * * * *
    restart: unless-stopped
    networks:
      - librenms
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.librenms.entrypoints=http,https"
      - "traefik.http.routers.librenms.rule=Host(`manager.xxREDACTEDxx.org`)"
      - "traefik.http.routers.librenms.tls=true"
      - "traefik.http.services.librenms.loadbalancer.server.port=8000"
      - "traefik.http.routers.librenms.middlewares=default-headers"
  
  dispatcher:
    image: librenms/librenms:latest
    container_name: dispatcher
    hostname: dispatcher
    depends_on:
      - mariadb
      - redis
      - librenms
    volumes:
      - /var/docks/librenms:/data
    environment:
      - TZ=America/New_York
      - PUID=1000
      - PGID=1000
      - DB_HOST=mariadb
      - DB_NAME=librenms
      - DB_USER=librenms
      - DB_PASSWORD=xxREDACTEDxx
      - DB_TIMEOUT=60
      - DISPATCHER_NODE_ID=dispatcher1
      - REDIS_HOST=redis
      - REDIS_PORT=6379
      - REDIS_DB=0
      - MEMORY_LIMIT=256M
      - UPLOAD_MAX_SIZE=16M
      - OPCACHE_MEM_SIZE=128
      - REAL_IP_FROM=0.0.0.0/32
      - REAL_IP_HEADER=X-Forwarded-For
      - LOG_IP_VAR=http_x_forwarded_for
      - LIBRENMS_SNMP_COMMUNITY=xxREDACTEDxx
      - LIBRENMS_WEATHERMAP=false
      - LIBRENMS_WEATHERMAP_SCHEDULE=*/5 * * * *
      - SIDECAR_DISPATCHER=1
    restart: unless-stopped
    networks:
      - librenms

  syslogng:
    image: librenms/librenms:latest
    container_name: syslogng
    hostname: syslogng
    depends_on:
      - redis
      - librenms
    ports:
      - 514:514/tcp
      - 514:514/udp
    volumes:
      - /var/docks/librenms:/data
    environment:
      - TZ=America/New_York
      - PUID=1000
      - PGID=1000
      - DB_HOST=mariadb
      - DB_NAME=librenms
      - DB_USER=librenms
      - DB_PASSWORD=xxREDACTEDxx
      - DB_TIMEOUT=60
      - REDIS_HOST=redis
      - REDIS_PORT=6379
      - REDIS_DB=0
      - MEMORY_LIMIT=256M
      - UPLOAD_MAX_SIZE=16M
      - OPCACHE_MEM_SIZE=128
      - REAL_IP_FROM=0.0.0.0/32
      - REAL_IP_HEADER=X-Forwarded-For
      - LOG_IP_VAR=http_x_forwarded_for
      - LIBRENMS_SNMP_COMMUNITY=xxREDACTEDxx
      - LIBRENMS_WEATHERMAP=false
      - LIBRENMS_WEATHERMAP_SCHEDULE=*/5 * * * *
      - SIDECAR_SYSLOGNG=1
    restart: unless-stopped
    networks:
      - librenms

  snmptrapd:
    image: librenms/librenms:latest
    container_name: snmptrapd
    hostname: snmptrapd
    depends_on:
      - librenms
    ports:
      - 162:162/tcp
      - 162:162/udp
    volumes:
      - /var/docks/librenms:/data
    environment:
      - TZ=America/New_York
      - PUID=1000
      - PGID=1000
      - DB_HOST=mariadb
      - DB_NAME=librenms
      - DB_USER=librenms
      - DB_PASSWORD=xxREDACTEDxx
      - DB_TIMEOUT=60
      - MEMORY_LIMIT=256M
      - UPLOAD_MAX_SIZE=16M
      - OPCACHE_MEM_SIZE=128
      - REAL_IP_FROM=0.0.0.0/32
      - REAL_IP_HEADER=X-Forwarded-For
      - LOG_IP_VAR=http_x_forwarded_for
      - LIBRENMS_SNMP_COMMUNITY=xxREDACTEDxx
      - LIBRENMS_WEATHERMAP=false
      - LIBRENMS_WEATHERMAP_SCHEDULE=*/5 * * * *
      - SIDECAR_SNMPTRAPD=1
    restart: unless-stopped
    networks:
      - librenms

networks:
  librenms:
    external: true

Docker info

$ docker info
Client:
 Context:    default
 Debug Mode: false

Server:
 Containers: 18
  Running: 17
  Paused: 0
  Stopped: 1
 Images: 15
 Server Version: 20.10.12
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version:
 runc version:
 init version:
 Security Options:
  apparmor
  seccomp
   Profile: default
  cgroupns
 Kernel Version: 5.15.0-27-generic
 Operating System: Ubuntu 22.04 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 6
 Total Memory: 31.28GiB
 Name: lorien
 ID: GSOQ:QGUJ:BOZD:6F47:ARZ6:4SK4:7SCQ:SB7Z:G7MF:T3GH:7H7Z:5T4C
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Logs

I looked high and low through the librenms and dispatcher container logs looking for any errors at all. Not a single error in sight, sadly. It sort of leads me to wonder - is this really mostly a cosmetic issue?

Discovery_0-4(INFO):Discovering device 3
Alerting(INFO):Checking alerts
Alerting(INFO):Completed alerting run for alerts in 0.29s
Poller_0-15(INFO):Completed poller run for 3 in 23.34s
Discovery_0-4(INFO):Completed discovery run for 3 in 23.53s```

jcostom avatar Apr 30 '22 17:04 jcostom

Luckily, the cache only needs to be updated when the definitions are updated, which only happens when the container is updated. So yes, it is a bug, but it has no ill effects in this case (other than log spam).

murrant avatar May 27 '22 15:05 murrant

When I try running Capture > Poller > Run I'm getting this error, which causes the whole thing to fail:

Updating os_def.cache  

file_put_contents(/opt/librenms/cache/os_defs.cache): Failed to open stream: Permission denied {"exception":"[object] (ErrorException(code: 0): file_put_contents(/opt/librenms/cache/os_defs.cache): Failed to open stream: Permission denied at /opt/librenms/LibreNMS/Util/OS.php:113)"} 



In OS.php line 113:
                                                                               
  [ErrorException]                                                             
  file_put_contents(/opt/librenms/cache/os_defs.cache): Failed to open stream  
  : Permission denied

karlshea avatar Dec 10 '23 20:12 karlshea

I can confirm, effects are clearly visible when a polling of a single host is launched via the WebUI. The polling fails.

To fix the problem, you need to run a chown inside the container: docker exec -it librenms chown -R 1000:1000 /opt/librenms

or just: docker exec -it librenms chown -R 1000:1000 /opt/librenms/cache

This is very restrictive since you have to run this command each time the container image is updated on the host.

What I do not understand is why Dockerfile initializes the working dir of LibreNMS with nobody:nogroup: Dockerfile#L134

Knowing that LibreNMS has its own user and group, wouldn't it be simpler to initialize the directory with librenms:librenms aka 1000:1000 ?

Thank you for any clarifications you can give us.

undefinedid avatar Dec 21 '23 18:12 undefinedid

Ok, I think I have an idea of what is happening. Polling is done in the dispatcher sidecar, which has this cache file correctly created. When you run the capture from the web page it is run in the web server container, which does not have the cache generated. It tries to create the file from the web server user.

murrant avatar Dec 21 '23 22:12 murrant