MichaIng
DietPi-Update | SSH connection fails during APT update
Creating a bug report/issue
- [x ] I have searched the existing open and closed issues
Required Information
-
DietPi version G_DIETPI_VERSION_CORE=8 G_DIETPI_VERSION_SUB=23 G_DIETPI_VERSION_RC=3 G_GITBRANCH='master' G_GITOWNER='MichaIng' G_LIVE_PATCH_STATUS[0]='applied' G_LIVE_PATCH_STATUS[1]='not applied' G_LIVE_PATCH_STATUS[2]='not applicable'
-
Distro version: bookworm
-
Kernel version:
Linux NanoPi Neo Air 6.1.53-current-sunxi #3 SMP Wed Sep 13 07:43:05 UTC 2023 armv7l GNU/Linux -
SBC model: NanoPi NEO Air (armv7l)
-
Power supply used: 5V 1.5A
-
SD card used: EMMC
Additional Information (if applicable)
macbookpro:~ macuser$ ssh 192.168.0.28 -l username
[email protected]'s password:
The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.
Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
─────────────────────────────────────────────────────
DietPi v8.23.3 : Update available
─────────────────────────────────────────────────────
- Device model : NanoPi NEO Air (armv7l)
- CPU temp : 29 °C / 84 °F : Who put me in the freezer!
- LAN IP : 192.168.0.28 (wlan0)
- MOTD : Stay safe and have a good start into 2024
─────────────────────────────────────────────────────
DietPi Team : https://github.com/MichaIng/DietPi#the-dietpi-project-team
Patreon Legends : Chris Gelatt, ADSB.im
Website : https://dietpi.com/ | https://twitter.com/DietPi_
Contribute : https://dietpi.com/contribute.html
Web Hosting by : https://myvirtualserver.com
dietpi-update : Run now to update DietPi from v8.23.3 to v8.25.1
dietpi-launcher : All the DietPi programs in one place
dietpi-config : Feature rich configuration tool for your device
dietpi-software : Select optimised software for installation
htop : Resource monitor
cpu : Shows CPU information and stats
username@NanopiNeoAir:~$ su -
Password:
─────────────────────────────────────────────────────
DietPi v8.23.3 : Update available
─────────────────────────────────────────────────────
- Device model : NanoPi NEO Air (armv7l)
- CPU temp : 29 °C / 84 °F : Who put me in the freezer!
- LAN IP : 192.168.0.28 (wlan0)
- MOTD : Stay safe and have a good start into 2024
─────────────────────────────────────────────────────
DietPi Team : https://github.com/MichaIng/DietPi#the-dietpi-project-team
Patreon Legends : Chris Gelatt, ADSB.im
Website : https://dietpi.com/ | https://twitter.com/DietPi_
Contribute : https://dietpi.com/contribute.html
Web Hosting by : https://myvirtualserver.com
dietpi-update : Run now to update DietPi from v8.23.3 to v8.25.1
dietpi-launcher : All the DietPi programs in one place
NanoPi NEO Air (armv7l) | IP: 192.168.0.28
dietpi-config : Feature rich configuration tool for your device
dietpi-software : Select optimised software for installation
htop : Resource monitor
cpu : Shows CPU information and stats
root@NanopiNeoAir:~# dietpi-update
DietPi-Update
─────────────────────────────────────────────────────
Phase: Checking for available DietPi update
[ OK ] DietPi-Update | Checking IPv4 network connectivity
[ OK ] DietPi-Update | Checking DNS resolver
[ INFO ] DietPi-Update | Getting latest version from: https://raw.githubusercontent.com/MichaIng/DietPi/master/.update/version
[ OK ] DietPi-Update | Got valid latest version: 8.25.1
[ OK ] DietPi-Update | Update available:
[ INFO ] DietPi-Update | Current version : v8.23.3
[ INFO ] DietPi-Update | Latest version : v8.25.1
DietPi-Update
─────────────────────────────────────────────────────
Phase: Checking for update pre-requirements
[ OK ] DietPi-Update | DietPi-Userdata validation: /mnt/dietpi_userdata
[ OK ] DietPi-Update | Free space check: path=/ | available=24471 MiB | required=100 MiB
[ OK ] DietPi-Update | curl -sSfLO https://raw.githubusercontent.com/MichaIng/DietPi/master/CHANGELOG.txt
[ SUB1 ] DietPi-Services > stop
[ OK ] DietPi-Services | stop : cron
[ OK ] DietPi-Services | stop : docker
[ OK ] DietPi-Services | stop : influxdb
[ OK ] DietPi-Services | stop : mariadb
DietPi-Update
─────────────────────────────────────────────────────
Phase: Applying pre-patches
[ OK ] DietPi-Update | Downloading pre-patches
[ OK ] DietPi-Update | Applying execute permission
[ OK ] DietPi-Update | Successfully applied pre-patches
DietPi-Update
─────────────────────────────────────────────────────
Phase: Upgrading APT packages
[ INFO ] DietPi-Update | APT update, please wait...
Get:1 https://deb.debian.org/debian bookworm InRelease [151 kB]
Get:2 https://download.docker.com/linux/debian bookworm InRelease [43.3 kB]
Get:3 https://repos.influxdata.com/debian stable InRelease [6901 B]
Get:5 https://repos.influxdata.com/debian stable/main armhf Packages [4201 B]
Get:6 https://deb.debian.org/debian bookworm-updates InRelease [52.1 kB]
Get:7 https://deb.debian.org/debian-security bookworm-security InRelease [48.0 kB]
Get:8 https://download.docker.com/linux/debian bookworm/stable armhf Packages [13.4 kB]
Get:4 https://stpete-mirror.armbian.com/apt bookworm InRelease [53.3 kB]
Get:9 https://deb.debian.org/debian bookworm-backports InRelease [56.5 kB]
Get:10 https://deb.debian.org/debian bookworm/non-free armhf Packages [55.9 kB]
Get:11 https://deb.debian.org/debian bookworm/main armhf Packages [8497 kB]
Get:12 https://stpete-mirror.armbian.com/apt bookworm/main armhf Packages [132 kB]
Get:13 https://stpete-mirror.armbian.com/apt bookworm/main all Packages [19.8 kB]
client_loop: send disconnect: Broken pipe
macbookpro:~ macuser$ ssh 192.168.0.28 -l username
ssh: connect to host 192.168.0.28 port 22: Operation timed out
macbookpro:~ macuser$
Expected behaviour
OS upgrade to v8.25.1
Actual behaviour
Connection gets disconnected -> upgrade fails
Extra details
Aftermatch: Started manually stopped services. Would it be good idea to run upgrade on screen? Please note that I DON'T have console access.
Can you reconnect to SSH after the connection was lost? If so, lets check SSH server logs:
journalctl -u dropbear
The update did not even reach a stage where any packages are actually upgraded. Also, even an upgraded and restarted SSH server won't terminate any SSH connection. So as long as the network connection between client and server is not somehow known to be flaky, this looks more like the system itself crashed, power/voltage issue being the reason in most such cases.
Would it be good idea to run upgrade on screen?
If the network connection is known to be flaky, yes. Another nice alternative is GNU Screen: https://www.gnu.org/software/screen/
apt install screen
screen
So you can re-attach to the shell session of the aborted SSH connection, which just continues to run in background.
If the system itself crashed, then of course this needs to be investigated first.
-- No entries -- when executed journalctl -u dropbear
Uptime is up 1 day, 1:20 (NanoPi boots via the cron frequently) so the upgrade did not make OS to crash. However, watchdog wrote that "No response from the GW" and tried to boot ( but it failed due the bug..)
So...should I apply renice-command for the upgrade, while it is running on screen? :)
Not sure if I understand correctly.
However, watchdog wrote that "No response from the GW" and tried to boot ( but it failed due the bug..)
You see this as kernel error on the NanoPi? And what do you mean with "tried to boot", and what bug?
-- No entries -- when executed journalctl -u dropbear
Did you switch to OpenSSH server? In case:
journalctl -u ssh
Do you see any other kernel errors?
dmesg -l 0,1,2,3
So...should I apply renice-command for the upgrade, while it is running on screen? :)
No, renice is never needed for anything if the system runs stable, and it does not help but can make things worse when the system runs unstable. We need to find the cause for system and/or network to be unstable.
About the watchdog, systemctl status watchdog showed the mentioned "No response from the GW". From your reply I got an idea: The network IS slow; 0.5Mb DL and 5MB UL. This is "IOT setup" where the needed bandwidth is minimal. Output of "dmesg -l 0,1,2,3" is empty. As we know, the HW-resources with NanoPi Neo Air are limited and from that I got an idea to use renice. Output of "journalctl -u ssh" show only my typos when typing password :D No OOM visible with dmesg -T, with that command only notable output is: [Sun Jan 14 17:03:08 2024] brcmfmac mmc1:0001:1: Direct firmware load for brcm/brcmfmac43430-sdio.friendlyarm,nanopi-neo-air.bin failed with error -2 [Sun Jan 14 17:03:08 2024] brcmfmac mmc1:0001:1: Falling back to sysfs fallback for: brcm/brcmfmac43430-sdio.friendlyarm,nanopi-neo-air.bin
About the watchdog, systemctl status watchdog showed the mentioned "No response from the GW"
Ah okay. Then I am actually quite sure that it is a network issue, not so much the WAN bandwidth, but the connection between SSH client and NanoPi as well NanoPi and gateway (the watchdog error), hence LAN-internal. Not sure about the quality/range of the onboard WiFi adapter of the NanoPi NEO Air. Probably you can put it to a better position or attach some antenna for a better/stable signal.
As we know, the HW-resources with NanoPi Neo Air are limited and from that I got an idea to use renice.
There are however much more important processes than a foreground system setup, like init system, system logging, udev and stuff like that, and of course the SSH server you are connecting through. Shifting resources via nice/priority to the foreground shell/script could cause more issues. In this case, as can be seen from missing kernel and SSH server errors, the issue is most likely the WiFi connection, hence the nice/priority of the foreground process has no effect on this. When system resources are exhausted, such foreground setup/install processes can run slower, but they should never become unstable, unless there are other/indirect issues like voltage or temperature. Renice can be reasonable when you have time-critical/RT processes or high quality audio processing. Other than that, it can make sense to lower the priority background/cron jobs if you feel that they disturb other processes.
The upgrade completed. The trick was to utilise screen and running dietpi-upgrade there. I also disabled and stopped watchdog. The upgrade took almost 2h to complete (due the 0.5MB downlink). NanoPi NEO Air wifi signal strength is excellent, the device is located about 50cm from the 4G-router and there is nothing between the router and NanoPi Neo Air to block signal. Thank you :)
Do you connect remotely (via Internet) with your SSH client? Probably this "No response from the GW" error can also show up when the router/gateway does take too long get a request. SSH is pretty cheap on bandwidth, but probably when doing APT updates/upgrades, it is exhausted, breaking SSH as well.
I was looking for a way to prioritise SSH via something like QoS, and indeed there is a way: https://debian-handbook.info/browse/stable/sect.quality-of-service.html
Would be interesting whether this helps:
apt install wondershaper iptables
wondershaper wlan0 4000 40000
iptables -t mangle -A PREROUTING -p tcp --sport 22 -j DSCP --set-dscp 4
iptables -t mangle -A PREROUTING -p tcp --dport 22 -j DSCP --set-dscp 4
- With
wondershaper, this limits traffic to 4000 kbps download and 40,000 kbps upload, a little below your theoretical bandwidth? In case adjust those values to be a little below the available bandwidth. And I took DL indeed as "download to the server" and UL as "upload from the server", while usually the download bandwidth is larger than the upload bandwidth and you mean this from client side? In case swap the two values: https://manpages.debian.org/wondershaper - With those
iptablesrules (I hope I translated them correctly from thenftablesrules of the above guide), we set the DSCP bit for SSH packets for low latency/high reliability/real-time interactive applications: https://en.wikipedia.org/wiki/Differentiated_services#Class_Selector, https://linuxreviews.org/Type_of_Service_(ToS)and_DSCP_Values#Iptables.26_ToS_.26_DSCP_Values
If this indeed helps to keep up the SSH connection during APT upgrades/downloads etc, these can be added to network configs to be applied automatically at boot/when the network is brought up.
FYI: ───────────────────────────────────────────────────── DietPi v9.0.2 : 14:53 - Wed 01/24/24 ─────────────────────────────────────────────────────
- Device model : NanoPi NEO Air (armv7l)
- CPU temp : 29 °C / 84 °F : Who put me in the freezer!
Successfully upgraded to v9.0.2 :)
Did you apply the suggested bandwidth sharpening and/or DSCP bits, or did it finally work without those?
Hi. I logged in via reverse-ssh and the dietpi-update was executed on screen.
EDIT: Ah whoops, I mixed up the issues. Would be still good to know whether the above steps help to keep a non-reverse SSH session active.