Web interface access failing after running for a few days

[MisterAnderson91]

Hey, reimaged last week around Thursday, the web interface was accessible.

It started coming up with 504 Gateway Time-out after any changes were made in the system tab and reboot was needed.

It now won't allow any login at all, reboot or not, it is stuck on 504 Gateway Time-out. SSH access is still working and it's playing the last schedule of assets set before lockout. I can grab log files if you can tell me which ones you need.

This is a raspberry pi OS lite install from script.

[nicomiguelino]

@MisterAnderson91, can you provide me the logs for anthias-server and anthias-viewer?

[nicomiguelino]

@MisterAnderson91 are you still experiencing the same issue? I did a fresh install on Pi 4 recently, and it seems that there's no issue so far.

Have you tried running at least ./bin/upgrade_containers.sh inside the ~/screenly directory? (Let's see if running the command above will update the containers and resolve the issue. It might be caused by migrating the codebase from Python 2 to Python 3).

[MisterAnderson91]

@nicomiguelino Hi Nico, can you tell me where the log files are stored? I don't see them within the home folder or ~/screenly folder.

The web interface is still dead but I want to grab the log files before upgrading containers.

[nicomiguelino]

Hi @MisterAnderson91, thanks for getting back. I'm not sure yet on where to retrieve the log as files, but you can do docker compose logs -f anthias-viewer anthias-server to get the server and viewer logs at the same time.

[nicomiguelino]

If you'd like to view logs of just one service (at a time), you can try the following inside the ~/screenly directory for instance:

docker compose logs -f anthias-viewer

Alternatively, you can try the following:

docker logs -f screenly-anthias-server-1

and

docker logs -f screenly-anthias-viewer-1

[MisterAnderson91]

My apologies for the delay, I haven't been able to run upgrade_containers.sh yet because it's a massive hassle to provide it with direct internet access once installed.

Here are the log files for the moment. anthais-viewer.log anthais-server.log

[MisterAnderson91]

Running upgrade_containers once broke Anthias completely. After running a couple more times, I am back to the "Screenly" black screen with no web interface. The web interface no longer gives a 504 error, it just doesn't exist when connecting to the IP address and I get a standard browser "Unable to connect" error.

How do I see the log files for the component that runs the web configuration interface?

[nicomiguelino]

You can check out the anthias-server logs by running the following...

docker-compose logs -f anthias-server

It's one of the components where configs are loaded. Viewer's using some configs too.

Let me know if you have more questions.

[MisterAnderson91]

EDIT: please ignore the previous comment, I forgot you have to run those commands inside the screenly folder.

The log has the upper section repeated a bunch, then it just keeps trying to connect to github but can't. Is that an issue? Why does it need to connect to github after the containers upgrade anyway?

screenly-anthias-server-1  | [2023-03-14 09:04:40 +1000] [19] [INFO] Starting gunicorn 19.10.0
screenly-anthias-server-1  | [2023-03-14 09:04:40 +1000] [19] [INFO] Listening at: http://0.0.0.0:8080 (19)
screenly-anthias-server-1  | [2023-03-14 09:04:40 +1000] [19] [INFO] Using worker: threads
screenly-anthias-server-1  | [2023-03-14 09:04:40 +1000] [27] [INFO] Booting worker with pid: 27
screenly-anthias-server-1  | Running migration...
screenly-anthias-server-1  | Obsolete column (filename) is not present
screenly-anthias-server-1  | Column (is_enabled,nocache) already present
screenly-anthias-server-1  | already has primary key
screenly-anthias-server-1  | Column (play_order) already present
screenly-anthias-server-1  | Column (is_processing) already present
screenly-anthias-server-1  | Column (skip_asset_check) already present
screenly-anthias-server-1  | Migration done.
screenly-anthias-server-1  | [2023-03-14 09:11:00 +1000] [19] [INFO] Starting gunicorn 19.10.0
screenly-anthias-server-1  | [2023-03-14 09:11:00 +1000] [19] [INFO] Listening at: http://0.0.0.0:8080 (19)
screenly-anthias-server-1  | [2023-03-14 09:11:00 +1000] [19] [INFO] Using worker: threads
screenly-anthias-server-1  | [2023-03-14 09:11:00 +1000] [27] [INFO] Booting worker with pid: 27
screenly-anthias-server-1  | ConnectionError fetching remote branch availability from GitHub: no data
screenly-anthias-server-1  | Remote Git branch not available
screenly-anthias-server-1  | Unable to get latest version from GitHub
screenly-anthias-server-1  | ConnectionError fetching remote branch availability from GitHub: no data
screenly-anthias-server-1  | Remote Git branch not available
screenly-anthias-server-1  | Unable to get latest version from GitHub
screenly-anthias-server-1  | ConnectionError fetching remote branch availability from GitHub: no data
screenly-anthias-server-1  | Remote Git branch not available
screenly-anthias-server-1  | Unable to get latest version from GitHub
screenly-anthias-server-1  | ConnectionError fetching remote branch availability from GitHub: no data
screenly-anthias-server-1  | Remote Git branch not available
screenly-anthias-server-1  | Unable to get latest version from GitHub

[MisterAnderson91]

I have given up on the previous install and wiped it and done a fresh install. I will run it for a week or so for testing and see.

I will note that the web interface still takes forever to become accessible after reboot, but at least the assets begin playing quickly.

[MisterAnderson91]

After leaving it for about a week, I'm am randomly getting either Error 504 gateway timeout, or the web interface loads. Doesn't seem to be a pattern and the web interface seems to stay accessible for a while once it starts working again.

[traeu]

It looks like I have a similar problem here. I have 6 Pis with freshly installed Screenly Anthias (installed on fresh Bullseye Raspi-OS this month using install-anthias.srly.io) I enabled daily reboots early in the morning to exclude any issues that appear only in long term use. But every day, when I open all web-GUIs from all 6 Pis in new tabs, I always get the same picture: Some web-GUIs load normally, some others load a minute and then show "504 Gateway Time-out" from NGINX. But its not always the same Pis that do this, the 504 Error happens on all Pis randomly. After waiting some time and reloading all tabs, the previous 504 showing Web-GUIs are loading normally, but some others show 504. Something must be unstable with the Web-GUI, but it only happens temporarily and after waiting some minutes, they start to respond normally again.

[vpetersson]

@traeu Please provide some logs from Docker.

[traeu]

@vpetersson I am happy to help with that. Can you please give me some advice how to get logs? is "docker compose logs -f anthias-viewer anthias-server" enough? edit: this results in "no configuration file provided: not found". "docker-compose logs -f anthias-server" as advised by @nicomiguelino also does not work, docker-compose is not a valid command.

[vpetersson]

@traeu Yeah docker-compose has been sunsetted in favor of docker compose. In any case, I'd probably do docker compose logs instead to make sure you get everything, not just the server (even though that is most likely where the issue is).

[traeu]

@vpetersson I guess I'm doing something wrong here. "docker compose logs" gives me "no configuration file provided: not found".

[vpetersson]

You need to be in the ~/anthias or ~/screenly folder (depending on the age of your installation), since it is parsing the docker-compose.yml file.

[nicomiguelino]

Thanks, @vpetersson. My bad @traeu. Alternatively you can view the logs by just running a docker logs command. It looks something like this:

docker logs -f $CONTAINER_NAME

First, you have to get the list of all the running containers. You can run the command anywhere.

docker ps

The console output will look like the following:

CONTAINER ID   IMAGE                                   COMMAND                  CREATED       STATUS          PORTS                               NAMES
ce4f0bd2d161   screenly/anthias-nginx:latest-pi4       "/usr/bin/entry.sh n…"   10 days ago   Up 14 minutes   0.0.0.0:80->80/tcp, :::80->80/tcp   screenly-anthias-nginx-1
0de22ce16f17   screenly/anthias-viewer:latest-pi4      "/usr/bin/entry.sh b…"   10 days ago   Up 14 minutes                                       screenly-anthias-viewer-1
9957b7014da9   screenly/anthias-celery:latest-pi4      "/usr/bin/entry.sh /…"   10 days ago   Up 14 minutes                                       screenly-anthias-celery-1
21f7d84ca262   screenly/anthias-websocket:latest-pi4   "/usr/bin/entry.sh p…"   10 days ago   Up 14 minutes                                       screenly-anthias-websocket-1
1b1f80368b32   screenly/anthias-server:latest-pi4      "/usr/bin/entry.sh b…"   10 days ago   Up 14 minutes                                       screenly-anthias-server-1
f7c4b28fc489   screenly/anthias-redis:latest-pi4       "/usr/bin/entry.sh r…"   10 days ago   Up 14 minutes   127.0.0.1:6379->6379/tcp            screenly-redis-1

If you only want to display just a few columns (say the CONTAINER ID and the NAMES), you can run:

docker ps --format "table {{.ID}}\t{{.Names}}"

Here's the corresponding console output:

CONTAINER ID   NAMES
ce4f0bd2d161   screenly-anthias-nginx-1
0de22ce16f17   screenly-anthias-viewer-1
9957b7014da9   screenly-anthias-celery-1
21f7d84ca262   screenly-anthias-websocket-1
1b1f80368b32   screenly-anthias-server-1
f7c4b28fc489   screenly-redis-1

So if we want to view the viewer logs, run:

docker logs -f screenly-anthias-viewer-1

Using the -f option tells Docker to continuously display the output in real-time. This is useful if you want to monitor the logs without the need to re-run the command.

[traeu]

I have collected some log files from all of my Pis. Just before I created all log files (via "docker compose logs > /home/pi/anthias_10X.log" executed inside /home/pi/screenly directory), I checked the web-guis: Pi 100, Pi 101, Pi 102 and Pi 105 did not load the web gui and showed nginx error 504. Pi 103 and Pi 104 had a normal working gui. Collecting log files did work, but Pi 102 and Pi 104 gave me this error: pi@XXXXXXX:~/screenly $ docker compose logs > /home/pi/anthias.log error from daemon in stream: Error grabbing logs: invalid character '\x00' looking for beginning of value

Also, the log files I got are really big (some are 10 MB or more). Thats why I attached them as ZIP. screenly logs.zip

By the way: my screenly/anthias installation is brand new. But I don't have a folder called "anthias" inside /home/pi, it is still named "screenly" even if my installation was just made some days ago. @vpetersson says the name of this folder depends on the age of the installtion.

[traeu]

I think I found something @vpetersson @nicomiguelino : Usually, my Pis are in a separate network without internet access. I enable internet access only for updates. now I noticed that the web-guis are stable and always available as soon as I switch on internet access. Then I remembered an old issue here on github: https://github.com/Screenly/Anthias/issues/1494 I then only enabled access to api.github.com and it looks like the web guis are still stable. In this old bug report 1494 the problem was that screenly did not show anything without access to api.github.com Now it looks like it is kind of working even without internet access (even the "update available" notification disappears when there is no internet) but it seems that it is not running stable without access to api.github.com

[vpetersson]

Running Anthias air-gapped is a bit of a soar point. Yes, it can work in theory (if you use an NTP server), but it's not very well tested.

[traeu]

yes, I use an internal NTP server distributed over DHCP. I also noticed that if I choose Y(es) to let anthias manage network connectivity (second question at installer), distributing NTP servers over DHCP doesn't work anymore.

But now, with disabled network management, local NTP server distributed over DHCP and a firewall exception for tcp 443 to api.github.com it looks like its running smoothly. I think it would be an improvement if the stability of anthias would not depend on the availability of api.github.com

[vpetersson]

I also noticed that if I choose Y(es) to let anthias manage network connectivity (second question at installer), distributing NTP servers over DHCP doesn't work anymore.

Interesting. That switch toggles to using Network Manager (NM). It might not support NTP provisioning from DHCP.

But now, with disabled network management, local NTP server distributed over DHCP and a firewall exception for tcp 443 to api.github.com it looks like its running smoothly. I think it would be an improvement if the stability of anthias would not depend on the availability of api.github.com

Agreed. There should be a graceful degradation happening here.

[traeu]

Interesting. That switch toggles to using Network Manager (NM). It might not support NTP provisioning from DHCP.

I enable "option ntp_servers" in /etc/dhcpcd.conf. Without network manager, after rebooting I see the correct IP of my local NTP server when checking timedatectl timesync-status. With enabled network manager, this change in dhcpcd.conf has no effect. I guess, with network manager enabled this config file is not used, but I don't know for sure.

[MisterAnderson91]

I enable "option ntp_servers" in /etc/dhcpcd.conf.

I believe I have been using systemd-timesyncd.service in screenly/anthias and set the servers in I think/etc/systemd/timesyncd.conf and that has worked fine. I haven't tried since March due to this issue remaining open but that should at least give you working NTP while NetworkManager is active.