nextcloud
Use a background job (instead of cron)
Well not until we have a better cron system in place. Else it can happen that somebody uploads all holliday pictures and all of a sudden their system is unsuable because we try to generate 300 previews. Which might be fine at 01:00 but maybe not at 15:00.
Well just make it a hourly running job and in the run method check the current server time?
well the use case is not that simple. Because if you have a raspberry pi you might want to run it at night. But if you have move power you might not care and want to do it every hour or so. And I don't want to basically implement cron with a bunch of settings for this app ;)
@rullzer your background job can influence/control what is running. With perhaps some sane defaults settings, one could say "genereate just 10 previews this time" on weak hardware, for instance. Also mind, on some hosted system the Nextcloud admin perhaps does not have control over cron jobs.
I really don't want to add to much dark magic checks into the job. If you can't control cron then this app just isn't for you right now.
Also I have no idea how to determine if something is weak hardware or not
Q, Can this be setup to run on webcron?
Ie https://www.easycron.com if so what would the path / settings be?
No it can't. The issue is that preview generation can take a long time (espeically on slow hardware). Anything other than the system cron has the risk to time out.
I just uploaded the whole picture collection of my wife to our nextcloud instance, including documentation photos for her work, all together >100 GB. The uploading itself via desktop client took several days of partly active laptop and one night at last to fully finish. The preview generator was set to pre-generate every 30 minutes and I guess was running though out the whole upload time until now, ~30 hours after the upload was finished.
This is on a Raspberry Pi 2 btw with data folder on USB hdd dock with 3 TB WD Red inside, so weak hardware overall ;).
Of course this is an exceptional situation. On important production systems one should stop the pre-generate cronjob during upload and then choose some calm hours to do generate-all, breaking it during active times. But of course this strongly depends on hardware, active hours and on how (which apps in what way) the nextcloud instance and other software on the server is used. In the end sometimes you give the previews simply high priority because you want to share them soon with others, being able to smoothly watch them with gallery app. Sometimes it doesn't matter too much. I would also not recommend to implement too much checks and therefore run the cron job the one or the other way. I think there are just too much parameters and cases that count. Of course some sort of web ui or occ command options for [max.] process time, files/time etc. would be nice to start the pre-generation/generate-all command with fitting settings for such individual cases. But in the end e.g. I had no idea how long the pre-generation of the whole collection would take and still will and how much the usage of the server/nextcloud instance could be effected. So even some nice options could have been chosen wrong.
Another way to solve this would be to some how make the process pause/stop by e.g. web ui access, the standard nextcloud cron.php, file accesses etc., go on again ~10 minutes after the last access or just start again with the next cron execution. Maybe this can be achieved moreless by giving the process a very low priority on hardware usage?
Also I asked myself if/how it is checked, if preview generation is already running. Found this in execution command:
if (($this->time->getTime() - $lastActivity) < 30 * 60) {
$output->writeln('Command is already running.');
return 2;
}
Does this mean, that the generation just starts, if it's last activity is 30 minutes ago? Therefore the readme suggestion "I run it every 10 minutes" would not make too much sense, or did I get something wrong?
No that is just if a run fails (with a php fatal error) (which we don't catch). Then we don't unlock. So if there is 30 minutes of no activity we assume that the previously running job died. And therefore we continue. Else you could have multiple generators running at the same time.
I like the idea to make the usage of this app a little easier by only having to activating the app. There may be the occ-commands, if you must intervene manually. Otherwise the cronjob stuff should be done via the main cronjob of Nextcloud. However, I must admit that I don't have any knowledge about Nextcloud's internals to judge this. I also like rullzer's idea of keeping everything simple.
So... There are at least 2 commands to avoid overlapping cronjobs: withlock & run-one (search Google or your repo for them). I use them together with "nice" at rullzer's suggested interval of 10 minutes. No need to hassle with lockings, deadlocks, strong/weak hardware any longer... Maybe this helps you, too!
*/30 0-6 * * * nice nice run-one php /srv/www/vhosts/nextcloud.xxx.xxx/htdocs/occ preview:pre-generate &>/dev/null
*/10 7-23 * * * nice nice run-one php /srv/www/vhosts/nextcloud.xxx.xxx/htdocs/occ preview:pre-generate &>/dev/null
@MichaIng you have 100GB of pictures you care about, and you can't even allocate to at least mirror your storage? You have a world of pain in your future, friend.
Sorry to be off-topic, but that is ludicrously incredible.
@BloodyIron I am not sure if I understood right, sorry in case, but I have a backup drive of course, being rsynced every night, all together four instances of the picture collection, main Nextcloud data drive, backup drive and two clients (desktop and laptop). I think this is quite enough to be save against data loss.
Yeah better hardware, a raid drive or even SSD data drives would be nice to handle stuff faster, but for my personal use, as I don't care too much about speed, I prefer to invest my money into other things 🙂.
The enormous preview generation and uploading time was a oneshot as said, due to uploading a whole new life time picture gallery into Nextcloud. I let it run also to test and review how things are handled, how much time it really takes. Would I have taken care about speed, I would have copied the gallery directly into the drive instead of uploading via Nextcloud client at least, then did a separate files:scan and preview generation afterwards. Would I need to upload large amounts of pictures regularly and make them available on Nextcloud fast, including previews, I would definitely invest in proper hardware 😉.
@MichaIng "but I have a backup drive of course"... PHEW! That's good to hear ;) Never mind then! ;D
As long as you have backups, I can holster myself. If you are interested in a tool recommendation, perhaps consider FreeNAS.
@BloodyIron Thanks for the recommendation. Indeed storing pictures is more a small part of my Nextcloud use, most important and actively used is calendar, tasks and contacts for self-organization, by times to share with others, friends and family. To have everything in one platform, Nextcloud does it's job very well on my RPi 🙂.
But back to topic, the gallery upload scenario would be still the same, thus preview pre-generation is not yet ready as by default implemented background job. It would need somehow a low priority to never slow down real time usage.
@MichaIng This is why I use "nice" in my crontab when launching the generator to lower the process priority and works perfectly.
@marvicz Of course, simple solution 👍.
I am not sure how exactly cron.php internal background jobs are executed, if it is possible to start a job from within cron.php with nice, because actually it is just as single php process, executing php commands, not shell commands. Maybe there is some workaround to archive this?
But for the separate cron job this is nice, I implemented it with mine and will compare preview generation times and parallel server usage.
€: Hard so analyse on 4 cores, as the process max. takes 1 core for 100%, though I guess there are tools or method to do better/deeper analyse 😉. But navigating Nextcloud, doing some search, copy and other tasks on terminal does absolutely not feel slowed down. I used maximum nice -19 and thanks to 4 cores and obviously fast CPU time/usage allocation, the preview generation did not take noticeably longer, ~1:45 min per 4k image with and without nice.
@MichaIng sudo -u www-data crontab -e 0 */1 * * * /usr/bin/nice /usr/bin/php -f /var/www/html/occ preview:pre-generate
I'm running one of my clouds on an old Acer one netbook (its motherboard better say) with CPU - Intel(R) Atom(TM) CPU N270 @ 1.60GHz, 2 core and 1GB RAM- and I can say I can recognize that NICE does its job. When applied like this, the work is remarkably faster than before. But haven't done any deep analyses, just enough to know, it works well.
A lot of us are running on shared hosting, unable to use system cron. So some solution to use this very useful preview plugin is much wanted :)
It would be great if it could be run by web cron, maybe by using a settable time limit in xxxx seconds per call during nights, and yyyy seconds during day. Another idea: can the cron just run something that set a flag of some kind that trigs the conversion, and the cron job itself exit quickly? And to run kind-of-nicely a setting to pause xxx milliseconds between each picture?
(Ideas from am mechatronics designer who have only little idea how webserver and NC works...)
Adding my 2 cents here by talking about the recognize app:
A settings page, displaying what will happen every 5 minutes (recognize is executed every 5 minutes, could be something else):
- How many pictures are handled on each job
- How many thread are used for a job
I think it could be way enough for most instances =)
