workspaces-issues [Bug] - Containers not starting after system update

Describe the bug After running apt upgrade and rebooted the systems, containers won't start with error:

An Unexpected Error occurred creating the Kasm. Please contact an Administrator

My understanding is that for some reason, kasm_guac is not starting. I can see the following logs:

kasm_guac     |
kasm_guac     | > [email protected] start
kasm_guac     | > NODE_ENV=production node app.js
kasm_guac     |
kasm_guac     | /gclient/logger.js:61
kasm_guac     |   errorEventName: config.kasmguac.logging.errorEventName,
kasm_guac     |                          ^
kasm_guac     |
kasm_guac     | TypeError: Cannot read properties of undefined (reading 'kasmguac')
kasm_guac     |     at Object.<anonymous> (/gclient/logger.js:61:26)
kasm_guac     |     at Module._compile (node:internal/modules/cjs/loader:1191:14)
kasm_guac     |     at Object.Module._extensions..js (node:internal/modules/cjs/loader:1245:10)
kasm_guac     |     at Module.load (node:internal/modules/cjs/loader:1069:32)
kasm_guac     |     at Function.Module._load (node:internal/modules/cjs/loader:904:12)
kasm_guac     |     at Module.require (node:internal/modules/cjs/loader:1093:19)
kasm_guac     |     at require (node:internal/modules/cjs/helpers:108:18)
kasm_guac     |     at Object.<anonymous> (/gclient/app.js:12:16)
kasm_guac     |     at Module._compile (node:internal/modules/cjs/loader:1191:14)
kasm_guac     |     at Object.Module._extensions..js (node:internal/modules/cjs/loader:1245:10)
kasm_guac exited with code 1

2023-05-14 22:20:23,978 [DEBUG] client_api_server: Requesting Hello for Server(fd5432bd-c5b0-4815-aea2-95168cce25cc) via URL: (https://proxy:443/agent/api/v1/create_container/)
2023-05-14 22:20:36,253 [ERROR] client_api_server: Error during Create request for Server(fd5432bd-c5b0-4815-aea2-95168cce25cc) : (Exception creating Kasm: Traceback (most recent call last):
  File "__init__.py", line 538, in post
  File "provision.py", line 1199, in provision
  File "provision.py", line 1351, in generate_nginx_config
Exception: Nginx failed to reload after generating config for container (67a668aa3636f5e07ad473624f5f644f64f2e1f5dea070ce264ee370b35639bd)
)
2023-05-14 22:20:36,254 [DEBUG] client_api_server: Function (provider_manager.create_kasm_from_slot) executed in (12.282997131347656) seconds
2023-05-14 22:20:36,254 [DEBUG] client_api_server: Function (provider_manager.get_container) executed in (12.308872938156128) seconds
2023-05-14 22:20:36,255 [ERROR] client_api_server: An Unexpected Error occurred creating the Kasm. Please contact an Administrator : Error during Create request for Server(fd5432bd-c5b0-4815-aea2-95168cce25cc) : (Exception creating Kasm: Traceback (most recent call last):
  File "__init__.py", line 538, in post
  File "provision.py", line 1199, in provision
  File "provision.py", line 1351, in generate_nginx_config
Exception: Nginx failed to reload after generating config for container (67a668aa3636f5e07ad473624f5f644f64f2e1f5dea070ce264ee370b35639bd)
)
2023-05-14 22:20:36,256 [DEBUG] client_api_server: Function (client_api.request_kasm) executed in (12.330260515213013) seconds

To Reproduce In my case, all I changed was to create a new workspace for vs-code, and ran apt update/upgrade and rebooted.

Expected behavior Workspaces should start

Workspaces Version Version 1.13

Workspaces Installation Method Single Server

Client Browser (please complete the following information):

OS: Windows 11
Browser: chrome
Version: 113.0.5672.93

Workspace Server Information (please provide the output of the following commands):

uname -a : Linux kasm 5.15.0-71-generic #78-Ubuntu SMP Tue Apr 18 09:00:29 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
cat /etc/os-release: PRETTY_NAME="Ubuntu 22.04.2 LTS" NAME="Ubuntu" VERSION_ID="22.04" VERSION="22.04.2 LTS (Jammy Jellyfish)" VERSION_CODENAME=jammy ID=ubuntu ID_LIKE=debian HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" UBUNTU_CODENAME=jammy
sudo docker info Client: Context: default Debug Mode: false Plugins: buildx: Docker Buildx (Docker Inc.) Version: v0.10.4 Path: /usr/libexec/docker/cli-plugins/docker-buildx compose: Docker Compose (Docker Inc.) Version: v2.5.0 Path: /usr/local/lib/docker/cli-plugins/docker-compose Server: Containers: 8 Running: 7 Paused: 0 Stopped: 1 Images: 15 Server Version: 23.0.6 Storage Driver: overlay2 Backing Filesystem: extfs Supports d_type: true Using metacopy: false Native Overlay Diff: true userxattr: false Logging Driver: json-file Cgroup Driver: systemd Cgroup Version: 2 Plugins: Volume: local Network: bridge host ipvlan macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog Swarm: inactive Runtimes: io.containerd.runc.v2 runc Default Runtime: runc Init Binary: docker-init containerd version: 3dce8eb055cbb6872793272b4f20ed16117344f8 runc version: v1.1.7-0-g860f061 init version: de40ad0 Security Options: apparmor seccomp Profile: builtin cgroupns Kernel Version: 5.15.0-71-generic Operating System: Ubuntu 22.04.2 LTS OSType: linux Architecture: x86_64 CPUs: 4 Total Memory: 23.47GiB Name: kasm ID: 7f57ad97-db5b-439b-bd5e-6472934d8b6b Docker Root Dir: /var/lib/docker Debug Mode: false Registry: https://index.docker.io/v1/ Experimental: false Insecure Registries: 127.0.0.0/8 Live Restore Enabled: false
sudo docker ps | grep kasm c5aa49242e05 kasmweb/nginx:latest "/docker-entrypoint.…" 4 weeks ago Up 24 minutes 80/tcp, 0.0.0.0:443->443/tcp, :::443->443/tcp kasm_proxy 940d3dee4c62 kasmweb/share:1.13.0 "/bin/sh -c '/usr/bi…" 4 weeks ago Up 24 minutes (healthy) 8182/tcp kasm_share 6d2146481f80 kasmweb/agent:1.13.0 "/bin/sh -c '/usr/bi…" 4 weeks ago Up 24 minutes (healthy) 4444/tcp kasm_agent 3867ff1c0bd1 redis:5-alpine "docker-entrypoint.s…" 4 weeks ago Up 24 minutes 6379/tcp kasm_redis b16aa80ebd96 kasmweb/manager:1.13.0 "/bin/sh -c '/usr/bi…" 4 weeks ago Up 24 minutes (healthy) 8181/tcp kasm_manager b96d497e27d6 kasmweb/kasm-guac:1.13.0 "/dockerentrypoint.sh" 4 weeks ago Restarting (1) 57 seconds ago kasm_guac 00216db40e0a postgres:12-alpine "docker-entrypoint.s…" 4 weeks ago Up 24 minutes (healthy) 5432/tcp kasm_db d2cc8b15c307 kasmweb/api:1.13.0 "/bin/sh -c '/usr/bi…" 4 weeks ago Up 24 minutes (healthy) 8080/tcp kasm_api

May 14 '23 22:05 mciantar

same problem here

Aug 29 '23 04:08 JackyF737

I am also having this issue on Kasm 1.15.0 installed on debian 12. Started right after I upgraded the host packages but restoring to a time before the upgrade doesn't fix it. Kasm is current unusable for me.

Aug 19 '24 08:08 silmarine

Based on @mciantar's logs, the kasm_guac container is not starting, and since it does not start, nginx in kasm_proxy can't start because it can't resolve the container's hostname. Based on the logs, it seems that the kasm_guac config file may be corrupt or missing. Can you please provide the contents of this file?

/opt/kasm/current/conf/app/kasmguac.app.config.yaml

Aug 19 '24 09:08 mmcclaskey

For me it's empty. I used the cat command and it had no output at all.

Aug 19 '24 09:08 silmarine

We have a KB article that covers getting things restored if this happens to you. The update and reboot was likely not the cause. The most likely cause, but not only, is that the system disk was full some time while Kasm was running. Kasm continually writes to this file and if the disk is full the write fails and you end up with a blank file. Kasm continues to work normally because everything in that file is in memory, but on a reboot it attempts to read the values from that file and thats when it fails. So the reboot did not cause the issue, it merely made you aware of the issue that was there waiting.

https://kasmweb.atlassian.net/servicedesk/customer/portal/3/article/8126468

Aug 19 '24 13:08 mmcclaskey