MichaIng
Tailscale installed via dietpi-software does not update cleanly
Creating a bug report/issue
Required Information
- DietPi version | 8.20.1
- Distro version | bullseye
- Kernel version | Linux fountain 6.1.21-v8+
- SBC model | RPi 4 Model B (aarch64)
- Power supply used | RPi
- SD card used | Samsung
Additional Information (if applicable)
As of v1.36 and above, Tailscale now includes a functionality for self-updating (sudo tailscale update). I've confirmed this works fine on another machine running Raspbian, but on Dietpi 8.20.1 it fails with the following message:
open /etc/apt/sources.list.d/tailscale.list: no such file or directory
Looking at /etc/apt/sources.list.d on my dietpi machine, there is a file called "dietpi-tailscale.list" which I assume is named that way to denote that it was installed via dietpi-software, but it seems to be causing an issue with Tailscale's self-update functionality that there's no tailscale.list file present in /etc/apt
Expected behaviour
Tailscale should self-update cleanly
Actual behaviour
The command fails with the above error message
Extra details
- ...
Strange, the self-updater checks for a hardcoded APT list name? What exactly does it actually do? Since it is an APT package, updates are easily done via apt upgrade. What does tailscale update different than that?
I guess I'll have to wait until the next Tailscale update to investigate further, will post back with more info after the next update drops.
Probably you can downgrade it. Check for available versions via:
apt show -a tailscale
Then downgrade to a specific version via:
apt install tailscale=a.b.c
Replace "a.b.c" with the version string.
-
So after installing Tailscale from dietpi-software, it seems that only dietpi-tailscale.list is created in /etc/apt/sources.list.d/ - there is no tailscale.list file created when installing Tailscale via dietpi-software
-
I just ran 'apt install tailscale=1.46.1' to downgrade to that version, and it gives me the following error message:
W: Target Packages (main/binary-amd64/Packages) is configured multiple times in /etc/apt/sources.list.d/dietpi-tailscale.list:1 and /etc/apt/sources.list.d/tailscale.list:2
W: Target Packages (main/binary-all/Packages) is configured multiple times in /etc/apt/sources.list.d/dietpi-tailscale.list:1 and /etc/apt/sources.list.d/tailscale.list:2
Do you want to continue? [Y/n]
So installing / downgrading via apt (rather than dietpi-software) now appears to create tailscale.list, but there's a clash with the dietpi-tailscale.list file that was already there.
-
I entered Y and the downgrade proceeded. There are now 2 tailscale files in /etc/apt/sources.list.d - dietpi-tailscale.list, and tailscale.list
-
I ran 'tailscale update' again to retry the self-update functionality, and it worked without issue, because there is now a tailscale.list file in /etc/apt/sources.list.d.
So it's fixable when using apt directly, but the root issue appears to be that installing Tailscale via dietpi-software in the first place does not create a tailscale.list file, but installing it via apt does create it.
So after installing Tailscale from dietpi-software, it seems that only dietpi-tailscale.list is created in /etc/apt/sources.list.d/ - there is no tailscale.list file created when installing Tailscale via dietpi-software
Yes this is intentional for all APT lists we installed in dietpi-software, to have them unique and identifiable so that we can e.g. safely apply patches to them without risking to overwrite custom admin files, or generally such we cannot be certain about what the content/intention is.
So installing / downgrading via apt (rather than dietpi-software) now appears to create tailscale.list
No, the APT package does not install a list file. That would be a chicken and egg issue 😉. I could imagine that tailscale update created it 🤔.
I just tried it myself and neither installing nor downgrading the package creates this list file. Are you sure that you did not manually create it in attempt to solve the issue?
I also renamed the file, and tailscale update does nothing else than apt update && apt install tailscale, quite unnecessary IMO. It has a --track option to switch to the stable/unstable suite, in which case it edits the tailscale.list. But the implementation is broken and it additionally passes unstable as version string, which of course does not exist:
root@VM-Bookworm:/etc/apt/sources.list.d# tailscale update --track unstable
This will update Tailscale from 1.48.1 to unstable. Continue? [y/n] y
Updated /etc/apt/sources.list.d/tailscale.list to use the unstable track
Get:1 https://pkgs.tailscale.com/unstable/debian bookworm InRelease
Get:2 https://pkgs.tailscale.com/unstable/debian bookworm/main amd64 Packages [34.7 kB]
Get:3 https://pkgs.tailscale.com/unstable/debian bookworm/main all Packages [354 B]
Fetched 41.7 kB in 2s (19.2 kB/s)
Reading package lists... Done
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Package tailscale is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source
E: Version 'unstable' for 'tailscale' was not found
exit status 100
I have no idea why someone would create and maintain an APT CLI wrapper for the own software, just with limited options and enough complexity that above bugs can happen 😄.
Forget about this, just use apt update (or wait for the DietPi banner to show/offer you upgrades) and apt upgrade like for every other APT package.
OK thanks for that, I've tried it another way but I'm still getting an issue that doesn't exist on other machines running Raspbian:
- Install Tailscale via dietpi-software - this installs the latest version (currently 1.48.1)
- Run 'tailscale up' to authenticate the machine and add it to my Tailnet
- SSH into the machine over the Tailscale connection
- Run 'apt install tailscale=1.46.1' to downgrade to the previous version (so I can test updating)
- Run 'dietpi-update' - this runs 'apt update' and correctly identifies that a new version of Tailscale is available.
- Run 'apt upgrade' - this fails with the following message:
E: dpkg was interrupted, you must manually run 'dpkg --configure -a' to correct the problem.
- Run 'dpkg --configure -a' (per the error message) and the following happens:
Setting up tailscale (1.46.1) ... Connection to dietpi closed by remote host. Connection to dietpi closed.
- SSHing back in over Tailscale, I still get a notification that there's an update to install. Running 'apt upgrade' just produces the same error from step 6 again.
- On the machine directly (not via SSH over Tailscale), run 'apt upgrade' - I receive the same error from step 6.
- Again on the machine directly, run 'dpkg --configure -a' and it appears to work correctly, with the below output:
Setting up tailscale (1.46.1) ...
- Run 'tailscale version' and it's confirm it's only 1.46.1 - it hasn't been upgraded to 1.48.1 despite having run 'apt upgrade'
- Run 'apt upgrade' again (on the machine directly), and it works OK
So trying the 'apt update && apt upgrade' method fails when you do it while SSH'd into the machine over the Tailscale connection itself. On other Raspbian machines, you can update Tailscale this way without issues - you do lose the SSH connection at the "Setting up tailscale (version)..." step, but when you SSH back in again over the Tailscale connection, the setup has completed.
Are you trying to upgrade Tailscale while you are connected through Tailscale? It seems like the package upgrade aborts the connection which aborts the upgrade, causing this error.
It is normal that aborted SSH connections abort any running processes from this session. This is done via SIGTERM, AFAIK, so whether this aborts an APT package install in a bad state, or whether it even finishes is just a question of timing, and e.g. how far the package install/configuration went with previous attempts. So likely it finally worked because previous dpkg --configure -a run did already configure some stuff so that the last attempt just finished quickly enough, before the process is killed by the system.
Raspbian uses OpenSSH by default instead of Dropbear. It might give processes a little more grace time before killing them, but generally behaves the same. So if/as a Tailscale package upgrade does abort all open connections, you simply should not upgrade it while being connected through Tailscale.
If there is really no other way, better use something like GNU/screen (apt install screen) to have the shell session within a dedicated process that is detached from the SSH session. This way, when network connection or SSH session is aborted for any reason, the shell with all its processes just continues detached and when you connect back you can re-attach to this shell.
Yes I was trying to upgrade Tailscale while connected through Tailscale. On Raspbian this works fine - the SSH connection drops during the process, but once you log back in everything has completed successfully and there's none of the issues that I'm running into with Dietpi, regardless of whether you use 'tailscale update' or 'apt update && apt upgrade'.
Not sure how else to dig into finding the issue at this stage.
Thanks for the tip about screen, I might try that next time there's a Tailscale update to install.
as stated by @MichaIng , on DietPi we use a different SSH on a default configuration. Simply switch to OpenSSH and check again. However way better to use GNU/screen (apt install screen) to have the shell session within a dedicated process that is detached from the SSH session.
And again: This is not an issue but perfectly expected behaviour with all SSH servers. It is pure luck and timing whether a task is able to finish before being killed or not, and it is pure luck whether a killed task leaves its work in a graceful state or broken. Especially with APT upgrades this is quite risky: Think of a kernel upgrade where the large kernel image is not fully written, or essential modules missing, or the initramfs not fully created, and then the system coincidentally powers off for some other reason and won't be able to boot anymore. Do not rely on this working, even if it did work well a few times on RPi OS.
It would be actually a nice feature request, if possible, to allow Tailscale package upgrades without having connections aborted. E.g. this is how SSH servers are upgraded: The master service is restarted, but the connection workers remain intact, so it is no problem to upgrade an SSH server while being connection through SSH. There may be reasons why this is impossible or unwanted with Tailscale though.
as stated by @MichaIng , on DietPi we use a different SSH on a default configuration. Simply switch to OpenSSH and check again. However way better to use GNU/screen (apt install screen) to have the shell session within a dedicated process that is detached from the SSH session.
Having the same issues with OpenSSH selected as the SSH server, but if screen is gonna be the better way to do it I'll try that on the next update for sure.
A new Tailscale version (1.48.2) just dropped so I tried updating via screen and there are still issues:
- SSH into my dietpi machine via Tailscale (the SSH server running on this machine is OpenSSH)
- run 'screen'
- run 'sudo apt update && sudo apt upgrade' - it eventually asks me if I want to apply the tailscale upgrade and I select Y.
- The SSH session eventually drops, as expected
- SSH back into the machine over Tailscale - it works correctly.
- run 'screen -rd' to reconnect to the screen session from earlier - it says "There is no screen to be detached.", which is unexpected
- Check the Tailscale update applied correctly ('tailscale version') - it did (it reports version 1.48.2)
- Run 'sudo apt update && sudo apt upgrade' again - it gives the following error:
E: dpkg was interrupted, you must manually run 'sudo dpkg --configure -a' to correct the problem.
- run 'sudo dpkg --configure -a' - it says "Setting up tailscale (1.48.2) ..." but then the SSH connection drops
- re-connect to the machine via Tailscale, run 'screen', and then re-run 'sudo dpkg --configure -a' - it says "Setting up tailscale (1.48.2) ..." but then the SSH connection drops
Even with 'screen', Tailscale cannot be updated cleanly. The only clean way to complete the upgrade was:
- SSH into the machine directly (not over Tailscale)
- run 'sudo dpkg --configure -a' - it gave the same "Setting up tailscale (1.48.2) ..." output, but then completed correctly and returned me to the command line
- I ran 'sudo apt update && sudo apt upgrade" once more just to double check there were no continuing issues, and there weren't any.
When I upgraded another machine running Raspbian, here's what happened (it was successful):
- SSH in over Tailscale
- run 'screen'
- run 'sudo apt update && sudo apt upgrade' - the upgrade proceeds, I hit Y to confirm the upgrade, and then the SSH session drops as expected
- I SSH back into the machine over Tailscale, and run 'screen -rd' - unlike on the dietpi machine, this time the screen session is resumed correctly, and I can see the output of the resumed session shows that the installation completed correctly.
- I run 'exit' to get back to a plain shell
- I run 'tailscale version' to check the updated version (1.48.2) is running, and it is
- I run 'sudo apt update && sudo apt upgrade' to check it runs cleanly (without the dpkg error from above) and it does.
Hmm, were you able to replicate this? "There is no screen to be detached." sounds like the screen was not detached but closed, like CTRL+D instead of CTRL+A+D would do it. The related packages on Raspbian and DietPi are the same, especially when you switched to OpenSSH on DietPi, and generally screen behaves the same, neither being affected by crashing internal processes, nor by crashing/terminated external connections.
Doesn't seem like I'll be able to try again until another Tailscale update drops.
To my mind, it seems to point back to something in my original post - I installed Tailscale via dietpi-software and there is no tailscale.list file in /etc/apt/sources.list.d, and this is what's causing all the issues with all the various methods of updating that I've tried, even when using screen.
Upgrading to Tailscale 1.50 and 1.50.1 have produced the same issue. I've also tried uninstalling Tailscale with dietpi-software and installing it using their own install script (https://tailscale.com/download/linux) but the same issue persists. You cannot cleanly update Tailscale over a Tailscale connection, whether running the update directly or via screen, but another machine running Raspian can do it without issues (via either method).
Let me know if there's anything else to try but I'm running out of ideas myself.
I also have this Tailscale 1.54 update issue on two different Raspberry PIs. While trying to update DietPi from v8.23.3 to v8.25.1
Board 1 96201b58-f225-40a6-a098-ec06d47d84fd Board 2 8d180c74-096d-4998-b0e3-5398bed99249
Details:
- Date | Thu Jan 4 21:54:21 GMT 2024
- Bug report | 96201b58-f225-40a6-a098-ec06d47d84fd
- DietPi version | v8.23.3 (MichaIng/master)
- Image creator |
- Pre-image |
- Hardware | RPi 3 Model B (aarch64) (ID=3)
- Kernel version |
Linux ts-de 6.1.21-v8+ #1642 SMP PREEMPT Mon Apr 3 17:24:16 BST 2023 aarch64 GNU/Linux - Distro | bullseye (ID=6,RASPBIAN=0)
- Command |
apt-get -y --with-new-pkgs upgrade - Exit code | 100
- Software title | DietPi-Update
Steps to reproduce:
- run dietpi-update
- or run an apt update
- or run 'dpkg --configure -a'
Expected behaviour:
dietpi-update to finish without errors
- ...
Actual behaviour:
dietpi-update errors with E: dpkg was interrupted, you must manually run 'dpkg --configure -a' to correct the problem. Running dpkg --configure -a starts to install Tailscale then kills the remote connection.
- ...
Extra details:
The update of the tailscale package seems to cause the issue
- ...
Additional logs:
E: dpkg was interrupted, you must manually run 'dpkg --configure -a' to correct the problem.
@jacko0 during the update you are connected to the systems via Tailscale ??
Please see the issue at the Tailscale repo: https://github.com/tailscale/tailscale/issues/9780 When I tested it, the active Tailscale connection remained active during the package upgrade, so I am unable to replicate it. I have no idea why a service restart kills the connection in your both cases, and more strangely even the screen session in OP case, while it all remains up here. We really only install the default APT package from their repository, matching 100% what you get when installing with their own scripts/docs. But maybe Tailscale devs have an idea in which circumstances and why a service restart kills the connection.
Thanks for the info. My workaround was to get physical access to the PI and run 'dpkg --configure -a' it finished with no problem. Then I ran dietpi-update and it's all working again.
@jacko0
Did you run tailscale up or tailscale up --ssh when establishing the Tailscale connection?
This flag lets Tailscale use an internal SSH server. A theory was that, if this is used and you restart the Tailscale service, that internal SSH server is terminated and terminates also active sessions (unlike Dropbear and OpenSSH). Otherwise it is strange that the SSH sessions are terminated in your both cases, while it remains active on my tests.
As a workaround I was able to update the device remotely by first installing the dietpi dashboard and updating packages by using the terminal shell via the dashboard.
So you are facing the same issue, that the SSH connection is aborted? Very strange that I was never able to replicate it. Does a screen session survive in your case?
I am wondering whether OOM kills might be the reason. Quite some coincidence and unlikely if all system memory is consumed right at the time when the Tailscale service is restarted, but just to rule it out: There is more than enough free system memory?
So you are facing the same issue, that the SSH connection is aborted?
yes
Does a screen session survive in your case?
No
IMO I don't believe it could be because of system memory because the installation went alright through dietpi dashboard.
As a side note, tailscale now lets you enable auto updates on the clients. But this won't work on dietpi installation because of the dietpi- suffix in dietpi-tailscale.list sources list.
Yes, that feature IMO does not make any sense, especially not the way it was implemented. Just enable APT auto-upgrades, if you want such. I mean imagine Debian or any distro wants to add Tailscale to their APT repositories, which would be awesome. Now this builtin feature throws this error, hence Debian cannot implement Tailscale without patching the source code to remove the auto-update feature first.
Can one of you guys attack a screen+keyboard, then replicate the issue? I guess even a reinstall would work
apt install --reinstall tailscale
Then check SSH server and Tailscale daemon logs, in the hope to see something which killed the sessions:
journalctl -u dropbear -u ssh -u tailscaled
And kernel logs:
dmesg -l 0,1,2,3