gitbls
Possible apt update blockage from use of aptcache?
I'm having a very annoying time trying to update the set of Pi that I configured with sdm and the aptcache plugin. Initially it seemed perfectly fine, then a few months ago(?) there was a short period where any attempt at updating would fail with complaints about the repository/password/sometihng. It went away on its own shortly after, so I forgot about it.
Now the problem is back and being very annoying. I've whacked around with google to try to find suggestions but none of them make any difference :-( It's clearly sometihng that occurs for a lot of people running a lot of variants of linux, so not a Pi-specific thing.
However, my latest test was to use the same base .img that I had done my sdm work with, write that to an SD card, stick it in a Pi 5 and (after basic manual setup) do an update; no problem. The obvious difference relating to updates is that this version has no connection to the aptcache thingy.
So my first question is how to turn off the aptcache client? If I can do that (preferably non-destructively) on one of the affected Pi and it solves my problem then we have learnt something potentially useful. I have the same issue on the Pi running the aptcache server too, which may indicate that I'm chewing the wrong bubblegum.
There is a short thread about this on https://forums.raspberrypi.com/viewtopic.php?t=377277 which may be useful. Help me Obi-Wan, you're my only hope!
My observation on apt-cacher-ng: Love it when it works, and annoyed when it screws up and needs a good swift kick.
You can disable it on an sdm-configured system by renaming /etc/apt/apt.conf.d/02proxy to .02proxy (or whatever, of course). It should then go to the internet.
Every 3-4 months apt-cacher-ng has a mental breakdown. When that happens, I run this script:
#!/bin/bash
echo "Stop apt-cacher-ng"
systemctl stop apt-cacher-ng
echo "Clear apt-cacher-ng cache"
rm -rf /var/cache/apt-cacher-ng
echo "Create apt-cacher-ng directory and set ownership"
mkdir -p /var/cache/apt-cacher-ng
chown -R apt-cacher-ng:apt-cacher-ng /var/cache/apt-cacher-ng
echo "Start apt-cacher-ng"
systemctl start apt-cacher-ng
echo "Done"
Yea, it's a pain. But, even so, downloading these packages every so often is pretty painless in the grand scheme of things.
Thank you!
On 2024-10-01, at 6:24 PM, Benn @.***> wrote:
My observation on apt-cacher-ng: Love it when it works, and annoyed when it screws up and needs a good swift kick.
You can disable it on an sdm-configured system by renaming /etc/apt/apt.conf.d/02proxy to .02proxy (or whatever, of course). It should then go to the internet.
That made no difference at all.
Every 3-4 months apt-cacher-ng has a mental breakdown. When that happens, I run this script:
That solved it immediately both on the 'server' and a client. Excellent.
IIRC you have at least one cron job related to this installed by default(?) so perhaps adding that script would be worth it? Magic that detects the problem and auto-does the cleanup would be nice, while we're wishing for pink unicorns and chocolate teapots. Pink chocolate unicorns? Hmm.
Also, I really should move the aptcache server stuff to a Pi 5 with SSD for improved performance...
tim
tim Rowledge; @.***; http://www.rowledge.org/tim Fractured Idiom:- PORTE-KOCHERE - Sacramental wine
Great that this fixed it.
sdm currently doesn't install any sdm-related cron jobs at all. I'm trying to keep sdm focused on configuration creation and not move sdm itself into "configuration management and monitoring", although sometimes it seems inevitable😵
Installing this reset script when apt-cacher-ng (acng) server is installed is a good idea.
Since it randomly happens to me, I've been aware of acng issues, but nobody else seemed to be having problems. I attributed it, along with every other randomly occurring problem on my network, to sunspots or Martians.😝
But now that there are at least two of us with issues, I'll have another look. I've found several threads discussing acng issues, but no resolution. Not sure that it's at all obvious how to detect that acng has entered a problem state (from the server side) , and even less sure that I want the acng server to randomly try to fix itself on its own but will see what I can find.
Aside: I moved acng to a Pi5. It definitely runs faster than it did when it was on a Pi4. In fact, rebuilding my "standard img" improved at least 20% elapsed time.
PS did you mention you were using acng in the forum post? If so, apologies, I missed it. I would have popped the fixit script there if I had seen it.
On 2024-10-02, at 6:27 AM, Benn @.***> wrote:
Great that this fixed it.
sdm currently doesn't install any sdm-related cron jobs at all. I'm trying to keep sdm focused on configuration creation and not move sdm itself into "configuration management and monitoring", although sometimes it seems inevitable😵
Hmm. I could have sworn I spotted some cron stuff in one of the plugins. One of the joys of senility I guess...
Installing this reset script when apt-cacher-ng (acng) server is installed is a good idea.
Since it randomly happens to me, I've been aware of acng issues, but nobody else seemed to be having problems. I attributed it, along with every other randomly occurring problem on my network, to sunspots or Martians.😝
My suggestion for problem detection would be parsing the output of the apt update stage. Certainly too simplistic for a true solution but it would catch the problems I've seen mentioned online.
But now that there are at least two of us with issues, I'll have another look. I've found several threads discussing acng issues, but no resolution. Not sure that it's at all obvious how to detect that acng has entered a problem state (from the server side) , and even less sure that I want the acng server to randomly try to fix itself on its own but will see what I can find.
Aside: I moved acng to a Pi5. It definitely runs faster than it did when it was on a Pi4. In fact, rebuilding my "standard img" improved at least 20% elapsed time.
PS did you mention you were using acng in the forum post? If so, apologies, I missed it. I would have popped the fixit script there if I had seen it.
I hadn't, mostly because it hadn't yet occurred to me. It took making the vanilla install to realise it might be something installed, if you see what I mean.
tim
tim Rowledge; @.***; http://www.rowledge.org/tim Strange OpCodes: RPSW: Randomize Program Status Word
Nah, you're not completely senile. The system plugin has an argument to switch from using the chatty cron daemon to using systemd timers. It's amazing how much less crap there is in the system log when switching.
I'll take a look at whether there's a meaningful set of errors that sdm can search for in /etc/sdm/apt.log. But this won't do anything for deployed systems, which is where this seems to inevitably show up.
Understand how not suspecting apt-cacher-ng can happen. I only asked about the forum post to make sure that I didn't miss it, which would have been personally annoying.
That's weird; I had to redo the aptcache cleaning this morning. After that I could update the machine I had temporarily put a new SD card into and returned to its prior state.
That's weird; I had to _re_do the aptcache cleaning this morning. After that I could update the machine I had temporarily put a new SD card into and returned to its prior state.
Bummer. I've never had that happen.
Curious little note on https://wiki.debian.org/AptCacherNg that may or may not be relevant. I'm going to experiment with it (use http://ftp.us.debian.org instead of http://deb.debian.org) but the referenced bug report (skimmed briefly) seemed to indicate that this was solved.
I'll also have sdm alert if there's a 'signature verification' or 'signatures were invalid' message in apt.log, similar to what it does if the disk or IMG fills up. Unfortunately this doesn't help when your client system barfs, but at least it's a well-understood and intensely annoying issue.
Small update - several days later updating is still ok. Sofa, so good, as JDVance says.
I get hit every month or two. The errors I see are different than yours, which is why I didn't comment on them in the forum.
Basically, any time I have an apt problem, apt-cacher-ng is guilty until proven innocent. But in spite of being a repeat offender, it's still better than always downloading from the internet.
Since it randomly happens to me, I've been aware of acng issues, but nobody else seemed to be having problems. I attributed it, along with every other randomly occurring problem on my network, to sunspots or Martians.😝 But now that there are at least two of us with issues, I'll have another look.
Make that three of us with the same problem. When it happens, I run a similar script, the cut-down one found in the comments at the bottom of sdm/sdm-apt-cacher.
I just put it up earlier this afternoon on a system booting off of eMMC and writing its data on an external SATA SSD. I've been pulling updates across a bunch of machines and it blew up on the last set of two or three done in a semi-parallel mode. That suggests where the bug is and I can go find & fix it. Or, I throw up my hands and configure my Sonatype Nexus repository for apt caching as well as docker caching.
And yes, clearing the contents of the cache from the cache server fixed everything - a bit drastic, but it worked. Thanks, @gitbls !
Sweet baby Jesus they need to fix this. I had this issue months ago and after a few days I was able to fix it. Fast forward 8 months and my updates are having issues again. I remember I've fixed this before but didn't write down what I did. Took me 4 days and all the hours to find this issue. @gitbls You Sir are my savior. I have now added the script into cron to run monthly.
Sweet baby Jesus they need to fix this. I had this issue months ago and after a few days I was able to fix it. Fast forward 8 months and my updates are having issues again. I remember I've fixed this before but didn't write down what I did. Took me 4 days and all the hours to find this issue. @gitbls You Sir are my savior. I have now added the script into cron to run monthly.
You're welcome. A monthly cron refresh is one way to solve it.
I use the "Are you feeling lucky, punk?" method. I wait until it fails and reset the cacher then. I've had it fail in a couple of weeks, and other times...literally months.
Well not only do I have it in cron but the script lives in the home directory ready to fire anytime I sense danger
This has been on my list of annoying things for a while. I asked copilot about it the other day, and it suggested:
Fix: Disable Range Operations Add the following to your Apt-Cacher NG config (typically /etc/apt-cacher-ng/acng.conf):
VfileUseRangeOps:0
This forces Apt-Cacher NG to download full files instead of relying on partial updates, avoiding the corruption issue.
I edited that file, found
# VfileUseRangeOps: 1
Added
VfileUseRangeOps: 0
immediately after that and restarted apt-cacher-ng. I've been running with that for a few days with no ill effects, and no more acng crapouts.
If you are still using acng could you give this a try and let me know the result after you've run with it for a while?