access-manager icon indicating copy to clipboard operation
access-manager copied to clipboard

Your packaging technique breaks `unattended-upgrades` on Ubuntu, please consider using `debhelper`.

Open threepistons opened this issue 4 months ago • 9 comments

Is your feature request related to a problem?

When I try to use unattended-upgrades to upgrade LithnetAccessManagerAgent3 on Ubuntu automatically, I get

Package LithnetAccessManagerAgent3 has conffile prompt and needs to be upgraded manually

and the package is not upgraded. When I use apt upgrade to manually upgrade the Lithnet Access Manager, I don't get prompted to deal with any config files, because I didn't change anything relating to Lithnet on the filesystem.

In the deb file, conffiles inside control.tar.gz lists /etc/systemd/system/LithnetAccessManagerAgent.service as a config file. When the package is upgraded, it deletes the unit file from /etc/systemd/system and replaces it, even when it is unchanged between versions of the package.

I've diffed the unit files from four successive versions of the package:

# threepistons @ phylab16-014 in ~/Ephemera/lithnet [13:19:31] 
$ diff 3.0.1309.0-LithnetAccessManagerAgent.service 3.0.1312.0-LithnetAccessManagerAgent.service

# threepistons @ phylab16-014 in ~/Ephemera/lithnet [13:19:57] 
$ diff 3.0.1312.0-LithnetAccessManagerAgent.service 3.0.1325.0-LithnetAccessManagerAgent.service

# threepistons @ phylab16-014 in ~/Ephemera/lithnet [13:20:16] 
$ diff 3.0.1325.0-LithnetAccessManagerAgent.service 3.0.1519.0-LithnetAccessManagerAgent.service
7c7
< ExecStart=/opt/LithnetAccessManagerAgent/Lithnet.AccessManager.Agent
---
> ExecStart=/opt/LithnetAccessManagerAgent/Lithnet.AccessManager.Agent run

For two upgrades, the package didn't change the contents of its unit file, yet it:

  • deleted the unit file from the disk and recreated it
  • listed it in conffiles in the deb package, preventing unattended-upgrades from upgrading it.

This is frustrating because we have opted to use Lithnet to rotate the root password to improve security, we want to use unattended-upgrades to automate security patching to also improve security, but unattended-upgrades cannot patch the Lithnet agent.

Describe the solution you'd like

I suspect, looking at the package's contents, that you are deleting and recreating the unit file as part of forcibly restarting the agent during the upgrade. There are better ways to do this.

Debian have made tools for dealing with installing, enabling, and restarting systemd units in packages. What some of the output of that can look like is found in the control files of XRDP, with the creation based on the rules file of the XRDP source package. You access these tools by building your package with debhelper. If you cannot use debhelper, please consider mimicking its output in your control files.

Related to this: the usual way to handle systemd units on the filesystem, as adopted by packages like XRDP and GDM3, is to add the unit to /usr/lib/systemd/system and symlink it to /etc/systemd/system/. Then, the user can replace the unit file in /etc/systemd/system/ if they want without breaking unattended-upgrades for the rest of us, although they would be wiser to use drop-in files in /etc/systemd/system/foo.service.d/local.conf (ibid.).

Describe alternatives you've considered

I cannot set

Dpkg::Options {
   "--force-confnew";
};

to overcome this because that will make Apt overwrite every config file with the maintainer's version for every package we install, and we need to maintain persistent site-specific configuration for some packages. If I set --force-confold, it won't update the service file when you do change it. It's not possible to configure only unattended-upgrades to obey that setting.

Additional context

To Reproduce

Steps to reproduce the behaviour:

  1. Add ". ${distro_codename}:${distro_codename}"; to /etc/apt/apt.conf.d/50unattended-upgrades.
  2. Run unattended-upgrades as root to manually invoke the automated upgrade task.

Expected behaviour

  • Lithnet is upgraded without any error message.

Access Manager installation

  • OS: Ubuntu 24.04
  • Access Manager Version: all versions of the agent.

Additional context

In my use case, using unattended-upgrades to manage security patching in production is a must. We have desktops and laptops used for academic research, so the laptops are unreachable over the network a lot of the time and I cannot just SSH in or use a fleet management tool like Puppet to patch these systems manually.

Console output

On a system that I've downgraded to 3.0.1312.0:

root@testbox:~# apt install lithnetaccessmanageragent3=3.0.1325.0
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following packages will be upgraded:
  lithnetaccessmanageragent3
1 to upgrade, 0 to newly install, 0 to remove and 0 not to upgrade.
Need to get 34.1 MB of archives.
After this operation, 0 B of additional disk space will be used.
Get:1 https://packages.lithnet.io/linux/deb/prod/repos/ubuntu noble/main amd64 lithnetaccessmanageragent3 amd64 3.0.1325.0 [34.1 MB]
Fetched 34.1 MB in 1s (24.3 MB/s)                     
(Reading database ... 179000 files and directories currently installed.)
Preparing to unpack .../lithnetaccessmanageragent3_3.0.1325.0_amd64.deb ...
Removed "/etc/systemd/system/multi-user.target.wants/LithnetAccessManagerAgent.service".
Warning: The unit file, source configuration file or drop-ins of LithnetAccessManagerAgent.service changed on disk. Run 'systemctl daemon-reload' to reload units.
Unpacking lithnetaccessmanageragent3 (3.0.1325.0) over (3.0.1312.0) ...
Setting up lithnetaccessmanageragent3 (3.0.1325.0) ...
Created symlink /etc/systemd/system/multi-user.target.wants/LithnetAccessManagerAgent.service → /etc/systemd/system/LithnetAccessManagerAgent.service.
View logs with 'sudo journalctl -u LithnetAccessManagerAgent -p 7 -a -f'

threepistons avatar Oct 16 '25 16:10 threepistons

I now have a workaround, a cron.weekly job that runs

apt install -y -o Dpkg::Options::="--force-confnew" lithnetaccessmanageragent3

So this is no longer urgent, but it would be nice not to have to do this.

threepistons avatar Oct 17 '25 09:10 threepistons

@threepistons thank you for this and the other report, and for taking the time to provide an extensive level of information for us. We've acknowledged these and are internally reviewing them.

I suspect your org might have a support contract with us, if so, do you want to reach out to us via the support system or enterprise support email? That way we can get this tracked and assigned to your account and associated SLAs.

ryannewington avatar Oct 18 '25 00:10 ryannewington

@threepistons these changes are now live in our preview repo. You can update your sources.list.d file from prod to https://packages.lithnet.io/linux/deb/preview/repos/debian/...

or use the commands below on a fresh machine.

Please let us know if this resolves this issue and #263 for you

#!/bin/bash
​
# Install prerequisites
sudo apt install \
    apt-transport-https \
    ca-certificates \
    curl \
    gnupg \
    lsb-release
​
# Import the Lithnet GPG signing keys
sudo mkdir -p /etc/apt/keyrings
curl -fsSL https://packages.lithnet.io/keys/lithnet.asc | sudo gpg --dearmor -o /etc/apt/keyrings/lithnet.gpg
sudo chmod a+r /etc/apt/keyrings/lithnet.gpg
​
# Add the Lithnet repository, specific to your build and architecture
echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/lithnet.gpg] \
  https://packages.lithnet.io/linux/deb/preview/repos/debian/ \
  $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/lithnet.list > /dev/null
​
# Fetch the new repo
sudo apt update

ryannewington avatar Nov 08 '25 05:11 ryannewington

@threepistons Just checking in to see if these changes allowed you to set up the automatic updates correctly?

ryannewington avatar Nov 24 '25 21:11 ryannewington

This testing is queued for next week. I'm sorry for not getting to it sooner. I've had a lot dropped on me recently.

threepistons avatar Nov 28 '25 16:11 threepistons

No problems at all @threepistons

The build has moved off preview and into prod, so you dont need to do anything special to the repo file.

ryannewington avatar Nov 29 '25 01:11 ryannewington

I got this interesting output on my Ubuntu 22.04 daily driver:

root@mybox:/# apt update
Hit:1 http://archive.canonical.com/ubuntu jammy InRelease
[lines omitted]
Get:10 https://packages.lithnet.io/linux/deb/prod/repos/ubuntu jammy InRelease [10.8 kB]                            
[more lines omitted]
E: Repository 'https://packages.lithnet.io/linux/deb/prod/repos/ubuntu jammy InRelease' changed its 'Origin' value from '. jammy' to 'Lithnet Ubuntu prod'
E: Repository 'https://packages.lithnet.io/linux/deb/prod/repos/ubuntu jammy InRelease' changed its 'Label' value from '. jammy' to 'Lithnet Ubuntu prod-jammy'
N: This must be accepted explicitly before updates for this repository can be applied. See apt-secure(8) manpage for details.
Do you want to accept these changes and continue updating from this repository? [y/N] y

I'm guessing that's because you've addressed #263.

On freshly-built 24.04 with /etc/apt/apt.conf.d/52unattended-upgrades-local as follows:

Unattended-Upgrade::Allowed-Origins {
	"${distro_id}:${distro_codename}";
	"${distro_id}:${distro_codename}-security";
	// Extended Security Maintenance; doesn't necessarily exist for
	// every release and this system may not have it installed, but if
	// available, the policy for updates is such that unattended-upgrades
	// should also install from here by default.
	"${distro_id}ESMApps:${distro_codename}-apps-security";
	"${distro_id}ESM:${distro_codename}-infra-security";
	"${distro_id}:${distro_codename}-updates";
//	"${distro_id}:${distro_codename}-proposed";
//	"${distro_id}:${distro_codename}-backports";
// Lithnet root password rotation agent
  "Lithnet Ubuntu prod:${distro_codename}";
// Microsoft Defender is "microsoft-ubuntu-noble-prod noble".
  "microsoft-ubuntu-${distro_codename}-prod ${distro_codename}:${distro_codename}";
};

and our custom sudo rule and matching set of Polkit policies that prompt for root's password, not the user's own password, which is why Lithnet is desirable in the first place, I get:

testuser@testbox:~$ sudo -i
[sudo] password for root:

root@testbox:~# apt list lithnetaccessmanageragent3 -a
Listing... Done
lithnetaccessmanageragent3/noble,now 3.0.1548.0 amd64 [installed]
lithnetaccessmanageragent3/noble 3.0.1519.0 amd64
lithnetaccessmanageragent3/noble 3.0.1325.0 amd64
lithnetaccessmanageragent3/noble 3.0.1312.0 amd64
lithnetaccessmanageragent3/noble 3.0.1309.0 amd64
lithnetaccessmanageragent3/noble 3.0.1270.0 amd64
lithnetaccessmanageragent3/noble 3.0.1257.0 amd64
lithnetaccessmanageragent3/noble 3.0.1229.0 amd64
lithnetaccessmanageragent3/noble 3.0.1210.0 amd64
lithnetaccessmanageragent3/noble 3.0.1206.0 amd64

root@testbox:~# apt install lithnetaccessmanageragent3=3.0.1519.0
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following packages will be DOWNGRADED:
  lithnetaccessmanageragent3
0 to upgrade, 0 to newly install, 1 to downgrade, 0 to remove and 0 not to upgrade.
Need to get 34.2 MB of archives.
After this operation, 0 B of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 https://packages.lithnet.io/linux/deb/prod/repos/ubuntu noble/main amd64 lithnetaccessmanageragent3 amd64 3.0.1519.0 [34.2 MB]
Fetched 34.2 MB in 23s (1,509 kB/s)                                                                                                                                                    
dpkg: warning: downgrading lithnetaccessmanageragent3 from 3.0.1548.0 to 3.0.1519.0
(Reading database ... 178883 files and directories currently installed.)
Preparing to unpack .../lithnetaccessmanageragent3_3.0.1519.0_amd64.deb ...
Removed "/etc/systemd/system/multi-user.target.wants/LithnetAccessManagerAgent.service".
Warning: The unit file, source configuration file or drop-ins of LithnetAccessManagerAgent.service changed on disk. Run 'systemctl daemon-reload' to reload units.
Unpacking lithnetaccessmanageragent3 (3.0.1519.0) over (3.0.1548.0) ...
Setting up lithnetaccessmanageragent3 (3.0.1519.0) ...
Created symlink /etc/systemd/system/multi-user.target.wants/LithnetAccessManagerAgent.service → /etc/systemd/system/LithnetAccessManagerAgent.service.
View logs with 'sudo journalctl -u LithnetAccessManagerAgent -p 7 -a -f'

root@testbox:~# unattended-upgrades
/usr/bin/unattended-upgrades:567: DeprecationWarning: This process (pid=6251) is multi-threaded, use of fork() may lead to deadlocks in the child.
  pid = os.fork()

root@testbox:~# cat /var/log/apt/history.log

# [[many Apt operations omitted]]
# [[the following "Requested-By" reflects that this is in testuser's sudo session]]

Start-Date: 2025-12-02  16:09:56
Commandline: apt install lithnetaccessmanageragent3=3.0.1519.0
Requested-By: testuser (1001)
Downgrade: lithnetaccessmanageragent3:amd64 (3.0.1548.0, 3.0.1519.0)
End-Date: 2025-12-02  16:10:01

Start-Date: 2025-12-02  16:10:16
Commandline: /usr/bin/unattended-upgrades
Requested-By: testuser (1001)
Upgrade: lithnetaccessmanageragent3:amd64 (3.0.1519.0, 3.0.1548.0)
End-Date: 2025-12-02  16:10:20

This looks like expected behaviour to me.

threepistons avatar Dec 02 '25 16:12 threepistons

Thanks for confirming @threepistons - Yes that message is expected due to the label change. Doesn't seem to be a way we can bypass it just because we didnt set it correctly when we first published.

Thanks for making us aware and hopefully this allows you to configure the agent as you need to.

ryannewington avatar Dec 03 '25 20:12 ryannewington

Yes, I'm able to remove the workaround cronstub now.

threepistons avatar Dec 04 '25 13:12 threepistons

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs.

stale[bot] avatar Dec 11 '25 13:12 stale[bot]