firmware [Kernel5.4] Lowering arm_freq_min leads to system hang/crash

Describe the bug I was upgrading to the newest firmware + kernel packages, which resulted in system hangs and/or crashes. I narrowed down the issue to arm_freq_min which I lowered to 150 or 300 (tested both) to allow the system clocking below 600 Mhz. Commenting the setting leads to a stable system, setting/reducing it leads to a quickly hanging or crashing system.

To reproduce

Upgrade the kernel on Raspberry Pi 2 Model B Rev 1.1 to current package release 5.4.51-v7+.
Set arm_freq_min to 300 (gpu_mem=16, if relevant)
reboot
play around in the file system, which some executables (like htop) until it either hangs or crashes. The last time I triggered it with vcgencmd measure_clock gpu.

Expected behaviour Add a clear and concise description of what you expected to happen.

Actual behaviour Setting arm_freq_min to 300 should not lead to system crashes.

System Copy and paste the results of the raspinfo command in to this section. Alternatively, copy and paste a pastebin link, or add answers to the following questions:

Which model of Raspberry Pi? Raspberry Pi 2 Model B Rev 1.1
Which OS and version (cat /etc/rpi-issue)? Raspbian GNU/Linux bullseye/sid
Which firmware version (vcgencmd version)? version 21a15cb094f41c7506ad65d2cb9b29c550693057 (clean) (release) (start_cd)
Which kernel version (uname -a)?

Linux micha.gnoedi.org 5.4.51-v7+ #1326 SMP Fri Jul 17 10:45:25 BST 2020 armv7l GNU/Linux

Logs

[  189.433811] 8<--- cut here ---
[  189.433874] Unable to handle kernel NULL pointer dereference at virtual address 0000000c
[  189.433930] pgd = 89d1d828
[  189.433974] [0000000c] *pgd=36ce9835, *pte=00000000, *ppte=00000000
[  189.434029] Internal error: Oops: 17 [#1] SMP ARM
[  189.434055] Modules linked in:
[  189.434089] CPU: 3 PID: 487 Comm: bash Not tainted 5.4.51-v7+ #1326
[  189.434116] Hardware name: BCM2835
[  189.434151] PC is at filemap_map_pages+0x118/0x448
[  189.434181] LR is at filemap_map_pages+0x43c/0x448
[  189.434207] pc : [<80272f9c>]    lr : [<802732c0>]    psr: 80000113
[  189.434235] sp : b82afe40  ip : b82afe40  fp : b82afe9c
[  189.434261] r10: b8e9d868  r9 : b82afeb4  r8 : 000000c3
[  189.434289] r7 : 80d04f48  r6 : 00000406  r5 : 000000cf  r4 : ba319480
[  189.434318] r3 : 00000004  r2 : 000000c3  r1 : b9406130  r0 : 00000008
[  189.434350] Flags: Nzcv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment user
[  189.434382] Control: 10c5387d  Table: 3832006a  DAC: 00000055
[  189.434412] Process bash (pid: 487, stack limit = 0x41237b4b)
[  189.434441] Stack: (0xb82afe40 to 0xb82b0000)
[  189.434475] fe40: 000000c3 00000000 b82afeb4 b832ba80 80d04f48 b8e9d86c 000000c3 00030000
[  189.434518] fe60: b9406130 00000000 00000000 b19558f1 00000001 000000cf 80d04f48 000000c0
[  189.434561] fe80: b82afeb4 000d7000 00000007 00000000 b82aff1c b82afea0 802ab58c 80272e90
[  189.434604] fea0: 8010bbbc 80d04f48 00000000 000d7cfc b82afef4 b6d06360 00000054 00100cca
[  189.434646] fec0: 000000c7 000d3000 b8320000 b8320000 00000000 00000000 00000000 00000000
[  189.434690] fee0: b6ce934c ba309d34 00000000 b19558f1 b82aff1c b82affb0 b6c88000 000d7cfc
[  189.434733] ff00: b8833400 00000017 b8833440 00000000 b82aff74 b82aff20 808d37cc 802aadd4
[  189.434776] ff20: 8012fcb0 8020b210 b82aff54 b82aff38 8010cfdc 8010cf4c 8010bbbc 80d04f48
[  189.434819] ff40: 00000000 00000054 b82aff8c 80d0a7d4 00000017 808d34ac 000d7cfc b82affb0
[  189.434862] ff60: 000d7110 00000000 b82affac b82aff78 80115854 808d34b8 80101d1c 00000001
[  189.434905] ff80: b82affac b82aff90 8020b3fc 00071ae0 20000010 ffffffff 10c5387d 10c5387d
[  189.434948] ffa0: 00000000 b82affb0 80101d24 80115818 00000014 00000000 000d7cfc 00000000
[  189.434991] ffc0: 00000014 000fe744 00103f60 000001e7 00d9b0e8 000d7110 00000000 001052ac
[  189.435035] ffe0: 000fe078 7eb3b408 00054684 00071ae0 20000010 ffffffff 00000000 00000000
[  189.435067] Backtrace:
[  189.435104] [<80272e84>] (filemap_map_pages) from [<802ab58c>] (handle_mm_fault+0x7c4/0xa4c)
[  189.435151]  r10:00000000 r9:00000007 r8:000d7000 r7:b82afeb4 r6:000000c0 r5:80d04f48
[  189.435186]  r4:000000cf
[  189.435220] [<802aadc8>] (handle_mm_fault) from [<808d37cc>] (do_page_fault+0x320/0x3a8)
[  189.435266]  r10:00000000 r9:b8833440 r8:00000017 r7:b8833400 r6:000d7cfc r5:b6c88000
[  189.435301]  r4:b82affb0
[  189.435333] [<808d34ac>] (do_page_fault) from [<80115854>] (do_DataAbort+0x48/0xc4)
[  189.435377]  r10:00000000 r9:000d7110 r8:b82affb0 r7:000d7cfc r6:808d34ac r5:00000017
[  189.435412]  r4:80d0a7d4
[  189.435443] [<8011580c>] (do_DataAbort) from [<80101d24>] (__dabt_usr+0x44/0x60)
[  189.435478] Exception stack(0xb82affb0 to 0xb82afff8)
[  189.435508] ffa0:                                     00000014 00000000 000d7cfc 00000000
[  189.435551] ffc0: 00000014 000fe744 00103f60 000001e7 00d9b0e8 000d7110 00000000 001052ac
[  189.435593] ffe0: 000fe078 7eb3b408 00054684 00071ae0 20000010 ffffffff
[  189.435629]  r8:10c5387d r7:10c5387d r6:ffffffff r5:20000010 r4:00071ae0
[  189.435666] Code: 0a00000d e2830005 e2833001 e0810100 (e5904004)
[  189.435719] ---[ end trace a011ff3c127a31f8 ]---
[  210.449124] rcu: INFO: rcu_sched self-detected stall on CPU
[  210.449193] rcu:     3-....: (2099 ticks this GP) idle=e0a/1/0x40000002 softirq=1262/1262 fqs=1049
[  210.449234]  (t=2100 jiffies g=1173 q=148)
[  210.449261] NMI backtrace for cpu 3
[  210.449293] CPU: 3 PID: 487 Comm: bash Tainted: G      D           5.4.51-v7+ #1326
[  210.449326] Hardware name: BCM2835
[  210.449347] Backtrace:
[  210.449391] [<8010d458>] (dump_backtrace) from [<8010d750>] (show_stack+0x20/0x24)
[  210.449434]  r6:b82ae000 r5:00000000 r4:80d93ff4 r3:b19558f1
[  210.449473] [<8010d730>] (show_stack) from [<808b22a4>] (dump_stack+0xe0/0x124)
[  210.449518] [<808b21c4>] (dump_stack) from [<808b9b58>] (nmi_cpu_backtrace+0xc8/0xcc)
[  210.449562]  r8:00000140 r7:8090202c r6:00000003 r5:00000000 r4:00000003 r3:b19558f1
[  210.449606] [<808b9a90>] (nmi_cpu_backtrace) from [<808b9c58>] (nmi_trigger_cpumask_backtrace+0xfc/0x138)
[  210.449647]  r5:80d07c8c r4:8010f340
[  210.449682] [<808b9b5c>] (nmi_trigger_cpumask_backtrace) from [<80110560>] (arch_trigger_cpumask_backtrace+0x20/0x24)
[  210.449728]  r7:80d05004 r6:80000193 r5:8090201c r4:00000003
[  210.449769] [<80110540>] (arch_trigger_cpumask_backtrace) from [<80195084>] (rcu_dump_cpu_stacks+0xb4/0xe4)
[  210.449821] [<80194fd0>] (rcu_dump_cpu_stacks) from [<80194758>] (rcu_sched_clock_irq+0x868/0xa80)
[  210.449868]  r10:3d91c000 r9:80dd1018 r8:80d04ff4 r7:80ca2ec0 r6:be5beec0 r5:80da4b64
[  210.449905]  r4:80d109c0 r3:ffffdd04
[  210.449942] [<80193ef0>] (rcu_sched_clock_irq) from [<8019dbb4>] (update_process_times+0x3c/0x64)
[  210.449989]  r10:801b11f8 r9:be5b85f0 r8:be5b8540 r7:00000030 r6:ff33a1c1 r5:b82afae0
[  210.450023]  r4:00000000
[  210.450057] [<8019db78>] (update_process_times) from [<801b0950>] (tick_sched_handle+0x64/0x70)
[  210.450096]  r4:be5b8870 r3:20000113
[  210.450129] [<801b08ec>] (tick_sched_handle) from [<801b1254>] (tick_sched_timer+0x5c/0xb8)
[  210.450176] [<801b11f8>] (tick_sched_timer) from [<8019eb34>] (__hrtimer_run_queues+0x164/0x324)
[  210.450219]  r7:b82ae000 r6:be5b8540 r5:be5b8580 r4:be5b8870
[  210.450259] [<8019e9d0>] (__hrtimer_run_queues) from [<8019f548>] (hrtimer_interrupt+0x130/0x2a4)
[  210.450306]  r10:be5b85c8 r9:be5b85f0 r8:be5b8540 r7:ffffffff r6:7fffffff r5:00000003
[  210.450340]  r4:20000193
[  210.450377] [<8019f418>] (hrtimer_interrupt) from [<8071dee8>] (arch_timer_handler_phys+0x40/0x48)
[  210.450424]  r10:b82afc68 r9:b82ae000 r8:80d63338 r7:b98aad00 r6:000000a2 r5:b9802fc0
[  210.450459]  r4:80ca22a4
[  210.450495] [<8071dea8>] (arch_timer_handler_phys) from [<80186e40>] (handle_percpu_devid_irq+0x88/0x23c)
[  210.450546] [<80186db8>] (handle_percpu_devid_irq) from [<80180384>] (generic_handle_irq+0x34/0x44)
[  210.450594]  r9:b82ae000 r8:b989d000 r7:00000001 r6:00000000 r5:00000000 r4:80ca22a4
[  210.450638] [<80180350>] (generic_handle_irq) from [<80180ad0>] (__handle_domain_irq+0x6c/0xc4)
[  210.450687] [<80180a64>] (__handle_domain_irq) from [<80102228>] (bcm2836_arm_irqchip_handle_irq+0x60/0xa4)
[  210.450735]  r8:b82afc68 r7:b82afb14 r6:ffffffff r5:20000113 r4:00000003 r3:b82afae0
[  210.450779] [<801021c8>] (bcm2836_arm_irqchip_handle_irq) from [<80101a3c>] (__irq_svc+0x5c/0x7c)
[  210.450818] Exception stack(0xb82afae0 to 0xb82afb28)
[  210.450853] fae0: ba309d34 00000000 0000000e 0000000d 00011000 b6ce9040 000ee000 80b15000
[  210.450897] fb00: b82afc68 00010000 b82afc68 b82afb3c b82afb40 b82afb30 802a8a70 808d326c
[  210.450933] fb20: 20000113 ffffffff
[  210.450958]  r4:808d326c r3:b19558f1
[  210.450993] [<808d322c>] (_raw_spin_lock) from [<802a8a70>] (unmap_page_range+0x190/0x734)
[  210.451039] [<802a88e0>] (unmap_page_range) from [<802a9060>] (unmap_single_vma+0x4c/0x54)
[  210.451085]  r10:b6c88000 r9:b8833440 r8:00000000 r7:00000000 r6:b82afc68 r5:ffffffff
[  210.451119]  r4:b6d06360
[  210.451149] [<802a9014>] (unmap_single_vma) from [<802a91e0>] (unmap_vmas+0x64/0x78)
[  210.451195] [<802a917c>] (unmap_vmas) from [<802af958>] (exit_mmap+0xdc/0x178)
[  210.451237]  r8:0000000b r7:00000001 r6:00000000 r5:80d04f48 r4:b6d06360
[  210.451276] [<802af87c>] (exit_mmap) from [<8011c734>] (mmput+0x58/0x108)
[  210.451309]  r6:00000000 r5:00000000 r4:b8833400
[  210.451345] [<8011c6dc>] (mmput) from [<80123170>] (do_exit+0x364/0xb20)
[  210.451376]  r5:b8833400 r4:b6c88000
[  210.451409] [<80122e0c>] (do_exit) from [<8010d9a8>] (die+0x254/0x358)
[  210.451437]  r7:7f000000
[  210.451468] [<8010d754>] (die) from [<801159ec>] (__do_kernel_fault.part.0+0x88/0x98)
[  210.451513]  r10:00000000 r9:b8833440 r8:00000017 r7:b8833400 r6:b82afdf0 r5:00000017
[  210.451547]  r4:0000000c
[  210.451579] [<80115964>] (__do_kernel_fault.part.0) from [<808d3848>] (do_page_fault+0x39c/0x3a8)
[  210.451618]  r7:b8833400 r3:b82afdf0
[  210.451650] [<808d34ac>] (do_page_fault) from [<80115854>] (do_DataAbort+0x48/0xc4)
[  210.451694]  r10:b8e9d868 r9:b82ae000 r8:b82afdf0 r7:0000000c r6:808d34ac r5:00000017
[  210.451729]  r4:80d0a7d4
[  210.451758] [<8011580c>] (do_DataAbort) from [<801019b4>] (__dabt_svc+0x54/0x80)
[  210.451793] Exception stack(0xb82afdf0 to 0xb82afe38)
[  210.451823] fde0:                                     00000008 b9406130 000000c3 00000004
[  210.451867] fe00: ba319480 000000cf 00000406 80d04f48 000000c3 b82afeb4 b8e9d868 b82afe9c
[  210.451908] fe20: b82afe40 b82afe40 802732c0 80272f9c 80000113 ffffffff
[  210.451944]  r8:000000c3 r7:b82afe24 r6:ffffffff r5:80000113 r4:80272f9c
[  210.451984] [<80272e84>] (filemap_map_pages) from [<802ab58c>] (handle_mm_fault+0x7c4/0xa4c)
[  210.452030]  r10:00000000 r9:00000007 r8:000d7000 r7:b82afeb4 r6:000000c0 r5:80d04f48
[  210.452064]  r4:000000cf
[  210.452095] [<802aadc8>] (handle_mm_fault) from [<808d37cc>] (do_page_fault+0x320/0x3a8)
[  210.452140]  r10:00000000 r9:b8833440 r8:00000017 r7:b8833400 r6:000d7cfc r5:b6c88000
[  210.452174]  r4:b82affb0
[  210.452205] [<808d34ac>] (do_page_fault) from [<80115854>] (do_DataAbort+0x48/0xc4)
[  210.452249]  r10:00000000 r9:000d7110 r8:b82affb0 r7:000d7cfc r6:808d34ac r5:00000017
[  210.452283]  r4:80d0a7d4
[  210.452313] [<8011580c>] (do_DataAbort) from [<80101d24>] (__dabt_usr+0x44/0x60)
[  210.452348] Exception stack(0xb82affb0 to 0xb82afff8)
[  210.452378] ffa0:                                     00000014 00000000 000d7cfc 00000000
[  210.452421] ffc0: 00000014 000fe744 00103f60 000001e7 00d9b0e8 000d7110 00000000 001052ac
[  210.452462] ffe0: 000fe078 7eb3b408 00054684 00071ae0 20000010 ffffffff
[  210.452498]  r8:10c5387d r7:10c5387d r6:ffffffff r5:20000010 r4:00071ae0

Additional context

2020-07-21 19:43:53 root@micha:~# cat /sys/devices/system/cpu/cpufreq/policy0/scaling_available_frequencies
300000 360000 450000 600000 900000

This is new and probably the reason for the crashes when lowering minimum frequency. When leaving at 600, there are only two pstates 600 and 900 and with kernel 4.19 there are always only two. I was actually hoping for that feature, so great work, however sadly at least my RPi model does not work fine with it.

Jul 21 '20 17:07 MichaIng

I can reproduce this. The backtraces in kernel seemed pretty random to me, so probably a clock/voltage issue, rather than a kernel bug.

Jul 21 '20 18:07 popcornmix

I can reproduce this.

You mean you "can" or you "can't" reproduce it? The issue is present with default clocks+voltage in my case, with only the minimum arm clock reduced and never ever any voltage warnings even when overclocked. We probably just found a second case with RPi Zero.

Probably related as well: https://www.raspberrypi.org/forums/viewtopic.php?p=1685668#p1685668

Jul 22 '20 15:07 MichaIng

I can see the crash on a Pi3+. I couldn't provoke it on a Pi4. I added logging and last thing that occurred was a switch from ~300MHz to 1400MHz. Everything looked as expected (e.g. the core voltage was raised before the frequency). Possibly there is an issue with clocks/PLLs switching by large amounts (and perhaps overshooting), but that is just guessing currently.

Workaround for now is to disable the arm_freq_min. I'll let you know when it's safe to add back in.

Jul 22 '20 15:07 popcornmix

Strange only that pre-5.4 the same large clocks jump was never an issue and with only lowest and highest clocks as only two pstates the jump was always the largest possible.

One could make a test with adjusting arm_freq and arm_freq_min so that only three as close as possible pstages are available. There seem to be fixed closks, 900, 600, 450, 360, 300, ?, 200, 150, so this should be easy. With this the intermediate pstages themselves could be ruled out as issue, which at least this was my very guess.

Jul 22 '20 15:07 MichaIng

Had the same problem on a model 1b from 2013 that runs off a battery bank and solar. Don't remember why I had arm_freq_min set so low since it did not make a huge difference in power draw. When I updated yesturday to 5.4 and then rebooted everything seemed to be fine. Ran top and within about 30 seconds the screen froze. I then used the default config.txt and after a reboot there was no more freezing when cpu load increased. Was going to continue trouble shooting today but checked the github issues and bingo Michalng saved me some time, thankyou.

Jul 23 '20 21:07 jeffmd

May I express the urgency I see in resolving or working around this bug? This has the potential to destroy systems by causing file corruption in unconditionally crashed services, e.g. databases and similar.

E.g. postinst could remove/comment arm_freq_min at least on affected systems (all but RPi4 it seems). There was a package upgrade yesterday but I don't see a hint that it has been fixed or worked around.

Let me know if there is anything I can test to help getting this resolved quickly.

Jul 25 '20 13:07 MichaIng

We have a workaround in latest rpi-update firmware that will disallow arm_freq_min below 600. A fuller fix is being worked on but will need more work/testing so will wait until after the stable release is settled.

Aug 04 '20 18:08 popcornmix

Kind of a noob here: is this fix available now (using apt-update/upgrade)?

Aug 05 '20 04:08 vmachiel

No, you will need to use rpi-update. No schedule on apt.

Aug 05 '20 08:08 JamesH65

Ok thanks

Aug 05 '20 11:08 vmachiel

And remember that this is only a workaround for user which are not yet aware of the issue. In your case you simply arm_freq_min to value of 600 or higher to have the exact same result (regarding this specific issue) 😉.

Aug 05 '20 18:08 MichaIng

And remember that this is only a workaround for user which are not yet aware of the issue. In your case you simply arm_freq_min to value of 600 or higher to have the exact same result (regarding this specific issue) 😉.

So If i just comment out my arm_freq_min=300, it will go to the default 600 right? And I can change it back once a full update is out that fixes this? (I'm not using rpi-update, I'm waiting for apt, the warning scared me 🙂)

Aug 06 '20 13:08 vmachiel

So If i just comment out my arm_freq_min=300, it will go to the default 600 right?

Yes exactly. On RPi1+Zero it's 700 Mhz but all defaults work fine.

Aug 06 '20 14:08 MichaIng

Maybe this is the wrong place to ask, but how do I know when a version is out that fixes this?

Aug 20 '20 14:08 vmachiel

Subscribe to this issue, I'm sure we'll get a dev notice once a real fix is merged and I'll anyway keep an eye on it as well and search through release commits when I recognise them and will post here in case.

Aug 20 '20 14:08 MichaIng

There is a proper fix for this as in internal PR. I'll let you know when it reached rpi-update.

Aug 20 '20 15:08 popcornmix

Latest rpi-update firmware contains a fix for this issue that doesn't involve limiting arm_freq_min. Please update and test.

Aug 25 '20 13:08 popcornmix

Jep seems to work fine. Just tested on RPi2 with arm_freq_min=150 which enables a lowest pstate of 200 MHz and all states are used, no hang or crash until now:

2020-08-25 23:05:25 root@micha:~# cat /sys/devices/system/cpu/cpufreq/policy0/stats/time_in_state
200000 36921
225000 152
257142 195
300000 225
360000 282
450000 307
600000 1073
900000 8511

~~200 MHz seems to be the lowest supported frequency, right?~~ EDIT: 100 MHz works as well, whether reasonable or not:

2020-09-16 13:27:28 root@micha:~# cat /sys/devices/system/cpu/cpufreq/policy0/stats/time_in_state
100000 200567
200000 2108
300000 1295
400000 854
500000 721
600000 1684
700000 799
800000 738
900000 5749

The steps are beautiful here, each 100 MHz, compared to varying ranges above.

Aug 25 '20 21:08 MichaIng

That's great! But as I understand, rpi-update is for pre-release stuff right? It will come to apt eventually?

Aug 26 '20 07:08 vmachiel

Yes, rpi-update by default loads the current master branch, compared to the stable branch that matches the apt packages. It would be great if you could test it as well, but only if you have a full SD card backup that you can recover, just in case any issues appear.

Aug 26 '20 08:08 MichaIng

It will come to apt but will take a while as we've just finalised a stable apt version and a commit like this (which affects clocks/plls) could do with some time on the testing branch.

Aug 26 '20 09:08 popcornmix

It will come to apt but will take a while as we've just finalised a stable apt version and a commit like this (which affects clocks/plls) could do with some time on the testing branch.

Ok cool. Totally get that, I’m just happy it’s getting a fix! I’ll keep my eye on this thread for info on the apt release.

Aug 28 '20 14:08 vmachiel

So I’m seeing another update round of updates of the firmware in apt. Does this contain the fix? Thanks

Sep 20 '20 13:09 vmachiel

No, only a single commit has been merged that has nothing to do with this issue: https://github.com/raspberrypi/firmware/commit/2b41f509710d99758a5b8efa88d95dd0e9169c0a Must have been an urgent one as well to justify a full firmware + kernel + bootloader upgrade for this single change 😉.

Sep 20 '20 14:09 MichaIng

Okay not sure if it is related, but with newest kernel I get another hang/crash:

dmesg

[  407.680929] rcu: INFO: rcu_sched self-detected stall on CPU
[  407.680961] rcu:     3-....: (2099 ticks this GP) idle=9a2/1/0x40000002 softirq=9183/9183 fqs=1049
[  407.680969]  (t=2100 jiffies g=35521 q=628)
[  407.680977] NMI backtrace for cpu 3
[  407.680989] CPU: 3 PID: 633 Comm: mysqld Not tainted 5.4.68-v7+ #1343
[  407.680993] Hardware name: BCM2835
[  407.680998] Backtrace:
[  407.681022] [<8010d4f4>] (dump_backtrace) from [<8010d7ec>] (show_stack+0x20/0x24)
[  407.681032]  r6:b5f48000 r5:00000000 r4:80e97abc r3:02ff4cf1
[  407.681045] [<8010d7cc>] (show_stack) from [<80908944>] (dump_stack+0xe0/0x124)
[  407.681059] [<80908864>] (dump_stack) from [<80910398>] (nmi_cpu_backtrace+0xc8/0xcc)
[  407.681069]  r8:00000140 r7:80a0202c r6:00000003 r5:00000000 r4:00000003 r3:02ff4cf1
[  407.681083] [<809102d0>] (nmi_cpu_backtrace) from [<80910498>] (nmi_trigger_cpumask_backtrace+0xfc/0x138)
[  407.681090]  r5:80e07d3c r4:8010f3dc
[  407.681104] [<8091039c>] (nmi_trigger_cpumask_backtrace) from [<801105fc>] (arch_trigger_cpumask_backtrace+0x20/0x24)
[  407.681112]  r7:80e05044 r6:80000193 r5:80a0201c r4:00000003
[  407.681125] [<801105dc>] (arch_trigger_cpumask_backtrace) from [<80196744>] (rcu_dump_cpu_stacks+0xb4/0xe4)
[  407.681135] [<80196690>] (rcu_dump_cpu_stacks) from [<80195e18>] (rcu_sched_clock_irq+0x868/0xa80)
[  407.681147]  r10:3d818000 r9:80ed5b18 r8:80e05034 r7:80da6f40 r6:be5bef40 r5:80ea8904
[  407.681153]  r4:80e10d80 r3:00002a0f
[  407.681166] [<801955b0>] (rcu_sched_clock_irq) from [<8019f674>] (update_process_times+0x3c/0xa8)
[  407.681177]  r10:801b2e50 r9:be5b85f0 r8:be5b8540 r7:0000005e r6:eb9f3387 r5:b5f49dd0
[  407.681181]  r4:00000000
[  407.681196] [<8019f638>] (update_process_times) from [<801b25a8>] (tick_sched_handle+0x64/0x70)
[  407.681202]  r4:be5b8870 r3:80000013
[  407.681213] [<801b2544>] (tick_sched_handle) from [<801b2eac>] (tick_sched_timer+0x5c/0xb8)
[  407.681224] [<801b2e50>] (tick_sched_timer) from [<801a0638>] (__hrtimer_run_queues+0x164/0x324)
[  407.681233]  r7:b5f48000 r6:be5b8540 r5:be5b8580 r4:be5b8870
[  407.681243] [<801a04d4>] (__hrtimer_run_queues) from [<801a104c>] (hrtimer_interrupt+0x130/0x2a4)
[  407.681254]  r10:be5b85c8 r9:be5b85f0 r8:be5b8540 r7:ffffffff r6:7fffffff r5:00000003
[  407.681259]  r4:20000193
[  407.681273] [<801a0f1c>] (hrtimer_interrupt) from [<80771744>] (arch_timer_handler_phys+0x40/0x48)
[  407.681284]  r10:80e04f88 r9:b5f48000 r8:80e66b54 r7:b946dd00 r6:000000a2 r5:b9402f40
[  407.681288]  r4:80da62e4
[  407.681301] [<80771704>] (arch_timer_handler_phys) from [<80188500>] (handle_percpu_devid_irq+0x88/0x23c)
[  407.681314] [<80188478>] (handle_percpu_devid_irq) from [<80181a00>] (generic_handle_irq+0x34/0x44)
[  407.681324]  r9:b5f48000 r8:b9460000 r7:00000001 r6:00000000 r5:00000000 r4:80da62e4
[  407.681336] [<801819cc>] (generic_handle_irq) from [<8018214c>] (__handle_domain_irq+0x6c/0xc4)
[  407.681348] [<801820e0>] (__handle_domain_irq) from [<80102228>] (bcm2836_arm_irqchip_handle_irq+0x60/0xa4)
[  407.681359]  r8:b5f49e40 r7:b5f49e04 r6:ffffffff r5:80000013 r4:00000003 r3:b5f49dd0
[  407.681370] [<801021c8>] (bcm2836_arm_irqchip_handle_irq) from [<80101a3c>] (__irq_svc+0x5c/0x7c)
[  407.681376] Exception stack(0xb5f49dd0 to 0xb5f49e18)
[  407.681384] 9dc0:                                     b9651e88 00000000 0000b5f4 0000de24
[  407.681395] 9de0: 00000001 80e04f88 b9651e84 b9651e84 b5f49e40 ffffffff 80e04f88 b5f49e2c
[  407.681403] 9e00: b5f49e30 b5f49e20 801b4e58 8092a02c 80000013 ffffffff
[  407.681410]  r4:8092a02c r3:02ff4cf1
[  407.681424] [<80929fec>] (_raw_spin_lock) from [<801b4e58>] (futex_wake+0xc4/0x1c0)
[  407.681436] [<801b4d94>] (futex_wake) from [<801b79dc>] (do_futex+0x160/0xcfc)
[  407.681447]  r10:80e04f88 r9:02648194 r8:00000081 r7:000000f0 r6:7fffffff r5:80e04f88
[  407.681451]  r4:00000001
[  407.681464] [<801b787c>] (do_futex) from [<801b8780>] (sys_futex_time32+0x9c/0x19c)
[  407.681475]  r10:7fffffff r9:02648194 r8:00000081 r7:000000f0 r6:00000000 r5:80e04f88
[  407.681479]  r4:00000001
[  407.681491] [<801b86e4>] (sys_futex_time32) from [<801011a4>] (__sys_trace_return+0x0/0x1c)
[  407.681497] Exception stack(0xb5f49fa8 to 0xb5f49ff0)
[  407.681506] 9fa0:                   00000001 00000000 02648194 00000081 7fffffff 00000000
[  407.681515] 9fc0: 00000001 00000000 00000001 000000f0 000008f0 00000001 00000000 0264816c
[  407.681523] 9fe0: 02648188 58ff9c38 02648168 76d00a14
[  407.681534]  r10:000000f0 r9:b5f48000 r8:801011c4 r7:000000f0 r6:00000001 r5:00000000
[  407.681538]  r4:00000001
[  470.710923] rcu: INFO: rcu_sched self-detected stall on CPU
[  470.710951] rcu:     3-....: (8402 ticks this GP) idle=9a2/1/0x40000002 softirq=9183/9183 fqs=4200
[  470.710959]  (t=8403 jiffies g=35521 q=2757)
[  470.710966] NMI backtrace for cpu 3
[  470.710977] CPU: 3 PID: 633 Comm: mysqld Not tainted 5.4.68-v7+ #1343
[  470.710981] Hardware name: BCM2835
[  470.710986] Backtrace:
[  470.711009] [<8010d4f4>] (dump_backtrace) from [<8010d7ec>] (show_stack+0x20/0x24)
[  470.711019]  r6:b5f48000 r5:00000000 r4:80e97abc r3:02ff4cf1
[  470.711032] [<8010d7cc>] (show_stack) from [<80908944>] (dump_stack+0xe0/0x124)
[  470.711045] [<80908864>] (dump_stack) from [<80910398>] (nmi_cpu_backtrace+0xc8/0xcc)
[  470.711056]  r8:00000140 r7:80a0202c r6:00000003 r5:00000000 r4:00000003 r3:02ff4cf1
[  470.711070] [<809102d0>] (nmi_cpu_backtrace) from [<80910498>] (nmi_trigger_cpumask_backtrace+0xfc/0x138)
[  470.711076]  r5:80e07d3c r4:8010f3dc
[  470.711090] [<8091039c>] (nmi_trigger_cpumask_backtrace) from [<801105fc>] (arch_trigger_cpumask_backtrace+0x20/0x24)
[  470.711099]  r7:80e05044 r6:80000193 r5:80a0201c r4:00000003
[  470.711111] [<801105dc>] (arch_trigger_cpumask_backtrace) from [<80196744>] (rcu_dump_cpu_stacks+0xb4/0xe4)
[  470.711121] [<80196690>] (rcu_dump_cpu_stacks) from [<80195e18>] (rcu_sched_clock_irq+0x868/0xa80)
[  470.711133]  r10:3d818000 r9:80ed5b18 r8:80e05034 r7:80da6f40 r6:be5bef40 r5:80ea8904
[  470.711138]  r4:80e10d80 r3:000042ae
[  470.711152] [<801955b0>] (rcu_sched_clock_irq) from [<8019f674>] (update_process_times+0x3c/0xa8)
[  470.711164]  r10:801b2e50 r9:be5b85f0 r8:be5b8540 r7:0000006d r6:9880a7d9 r5:b5f49dd0
[  470.711168]  r4:00000000
[  470.711183] [<8019f638>] (update_process_times) from [<801b25a8>] (tick_sched_handle+0x64/0x70)
[  470.711189]  r4:be5b8870 r3:80000013
[  470.711201] [<801b2544>] (tick_sched_handle) from [<801b2eac>] (tick_sched_timer+0x5c/0xb8)
[  470.711212] [<801b2e50>] (tick_sched_timer) from [<801a0638>] (__hrtimer_run_queues+0x164/0x324)
[  470.711221]  r7:b5f48000 r6:be5b8540 r5:be5b8580 r4:be5b8870
[  470.711231] [<801a04d4>] (__hrtimer_run_queues) from [<801a104c>] (hrtimer_interrupt+0x130/0x2a4)
[  470.711242]  r10:be5b85c8 r9:be5b85f0 r8:be5b8540 r7:ffffffff r6:7fffffff r5:00000003
[  470.711247]  r4:20000193
[  470.711261] [<801a0f1c>] (hrtimer_interrupt) from [<80771744>] (arch_timer_handler_phys+0x40/0x48)
[  470.711272]  r10:80e04f88 r9:b5f48000 r8:80e66b54 r7:b946dd00 r6:000000a2 r5:b9402f40
[  470.711277]  r4:80da62e4
[  470.711289] [<80771704>] (arch_timer_handler_phys) from [<80188500>] (handle_percpu_devid_irq+0x88/0x23c)
[  470.711301] [<80188478>] (handle_percpu_devid_irq) from [<80181a00>] (generic_handle_irq+0x34/0x44)
[  470.711312]  r9:b5f48000 r8:b9460000 r7:00000001 r6:00000000 r5:00000000 r4:80da62e4
[  470.711324] [<801819cc>] (generic_handle_irq) from [<8018214c>] (__handle_domain_irq+0x6c/0xc4)
[  470.711336] [<801820e0>] (__handle_domain_irq) from [<80102228>] (bcm2836_arm_irqchip_handle_irq+0x60/0xa4)
[  470.711347]  r8:b5f49e40 r7:b5f49e04 r6:ffffffff r5:80000013 r4:00000003 r3:b5f49dd0
[  470.711357] [<801021c8>] (bcm2836_arm_irqchip_handle_irq) from [<80101a3c>] (__irq_svc+0x5c/0x7c)
[  470.711363] Exception stack(0xb5f49dd0 to 0xb5f49e18)
[  470.711372] 9dc0:                                     b9651e88 00000000 0000b5f4 0000de24
[  470.711383] 9de0: 00000001 80e04f88 b9651e84 b9651e84 b5f49e40 ffffffff 80e04f88 b5f49e2c
[  470.711392] 9e00: b5f49e30 b5f49e20 801b4e58 8092a02c 80000013 ffffffff
[  470.711398]  r4:8092a02c r3:02ff4cf1
[  470.711412] [<80929fec>] (_raw_spin_lock) from [<801b4e58>] (futex_wake+0xc4/0x1c0)
[  470.711423] [<801b4d94>] (futex_wake) from [<801b79dc>] (do_futex+0x160/0xcfc)
[  470.711435]  r10:80e04f88 r9:02648194 r8:00000081 r7:000000f0 r6:7fffffff r5:80e04f88
[  470.711439]  r4:00000001
[  470.711451] [<801b787c>] (do_futex) from [<801b8780>] (sys_futex_time32+0x9c/0x19c)
[  470.711462]  r10:7fffffff r9:02648194 r8:00000081 r7:000000f0 r6:00000000 r5:80e04f88
[  470.711467]  r4:00000001
[  470.711478] [<801b86e4>] (sys_futex_time32) from [<801011a4>] (__sys_trace_return+0x0/0x1c)
[  470.711484] Exception stack(0xb5f49fa8 to 0xb5f49ff0)
[  470.711493] 9fa0:                   00000001 00000000 02648194 00000081 7fffffff 00000000
[  470.711503] 9fc0: 00000001 00000000 00000001 000000f0 000008f0 00000001 00000000 0264816c
[  470.711510] 9fe0: 02648188 58ff9c38 02648168 76d00a14
[  470.711521]  r10:000000f0 r9:b5f48000 r8:801011c4 r7:000000f0 r6:00000001 r5:00000000
[  470.711525]  r4:00000001
[  533.740915] rcu: INFO: rcu_sched self-detected stall on CPU
[  533.740937] rcu:     3-....: (14705 ticks this GP) idle=9a2/1/0x40000002 softirq=9183/9183 fqs=7352
[  533.740944]  (t=14706 jiffies g=35521 q=5868)
[  533.740951] NMI backtrace for cpu 3
[  533.740962] CPU: 3 PID: 633 Comm: mysqld Not tainted 5.4.68-v7+ #1343
[  533.740966] Hardware name: BCM2835
[  533.740971] Backtrace:
[  533.740992] [<8010d4f4>] (dump_backtrace) from [<8010d7ec>] (show_stack+0x20/0x24)
[  533.741002]  r6:b5f48000 r5:00000000 r4:80e97abc r3:02ff4cf1
[  533.741014] [<8010d7cc>] (show_stack) from [<80908944>] (dump_stack+0xe0/0x124)
[  533.741028] [<80908864>] (dump_stack) from [<80910398>] (nmi_cpu_backtrace+0xc8/0xcc)
[  533.741039]  r8:00000140 r7:80a0202c r6:00000003 r5:00000000 r4:00000003 r3:02ff4cf1
[  533.741053] [<809102d0>] (nmi_cpu_backtrace) from [<80910498>] (nmi_trigger_cpumask_backtrace+0xfc/0x138)
[  533.741059]  r5:80e07d3c r4:8010f3dc
[  533.741073] [<8091039c>] (nmi_trigger_cpumask_backtrace) from [<801105fc>] (arch_trigger_cpumask_backtrace+0x20/0x24)
[  533.741082]  r7:80e05044 r6:80000193 r5:80a0201c r4:00000003
[  533.741094] [<801105dc>] (arch_trigger_cpumask_backtrace) from [<80196744>] (rcu_dump_cpu_stacks+0xb4/0xe4)
[  533.741104] [<80196690>] (rcu_dump_cpu_stacks) from [<80195e18>] (rcu_sched_clock_irq+0x868/0xa80)
[  533.741115]  r10:3d818000 r9:80ed5b18 r8:80e05034 r7:80da6f40 r6:be5bef40 r5:80ea8904
[  533.741121]  r4:80e10d80 r3:00005b4e
[  533.741136] [<801955b0>] (rcu_sched_clock_irq) from [<8019f674>] (update_process_times+0x3c/0xa8)
[  533.741147]  r10:801b2e50 r9:be5b85f0 r8:be5b8540 r7:0000007c r6:45621d63 r5:b5f49dd0
[  533.741152]  r4:00000000
[  533.741166] [<8019f638>] (update_process_times) from [<801b25a8>] (tick_sched_handle+0x64/0x70)
[  533.741172]  r4:be5b8870 r3:80000013
[  533.741183] [<801b2544>] (tick_sched_handle) from [<801b2eac>] (tick_sched_timer+0x5c/0xb8)
[  533.741195] [<801b2e50>] (tick_sched_timer) from [<801a0638>] (__hrtimer_run_queues+0x164/0x324)
[  533.741203]  r7:b5f48000 r6:be5b8540 r5:be5b8580 r4:be5b8870
[  533.741214] [<801a04d4>] (__hrtimer_run_queues) from [<801a104c>] (hrtimer_interrupt+0x130/0x2a4)
[  533.741225]  r10:be5b85c8 r9:be5b85f0 r8:be5b8540 r7:ffffffff r6:7fffffff r5:00000003
[  533.741229]  r4:20000193
[  533.741244] [<801a0f1c>] (hrtimer_interrupt) from [<80771744>] (arch_timer_handler_phys+0x40/0x48)
[  533.741255]  r10:80e04f88 r9:b5f48000 r8:80e66b54 r7:b946dd00 r6:000000a2 r5:b9402f40
[  533.741260]  r4:80da62e4
[  533.741272] [<80771704>] (arch_timer_handler_phys) from [<80188500>] (handle_percpu_devid_irq+0x88/0x23c)
[  533.741284] [<80188478>] (handle_percpu_devid_irq) from [<80181a00>] (generic_handle_irq+0x34/0x44)
[  533.741295]  r9:b5f48000 r8:b9460000 r7:00000001 r6:00000000 r5:00000000 r4:80da62e4
[  533.741307] [<801819cc>] (generic_handle_irq) from [<8018214c>] (__handle_domain_irq+0x6c/0xc4)
[  533.741319] [<801820e0>] (__handle_domain_irq) from [<80102228>] (bcm2836_arm_irqchip_handle_irq+0x60/0xa4)
[  533.741330]  r8:b5f49e40 r7:b5f49e04 r6:ffffffff r5:80000013 r4:00000003 r3:b5f49dd0
[  533.741340] [<801021c8>] (bcm2836_arm_irqchip_handle_irq) from [<80101a3c>] (__irq_svc+0x5c/0x7c)
[  533.741346] Exception stack(0xb5f49dd0 to 0xb5f49e18)
[  533.741355] 9dc0:                                     b9651e88 00000000 0000b5f4 0000de24
[  533.741365] 9de0: 00000001 80e04f88 b9651e84 b9651e84 b5f49e40 ffffffff 80e04f88 b5f49e2c
[  533.741374] 9e00: b5f49e30 b5f49e20 801b4e58 8092a02c 80000013 ffffffff
[  533.741380]  r4:8092a02c r3:02ff4cf1
[  533.741394] [<80929fec>] (_raw_spin_lock) from [<801b4e58>] (futex_wake+0xc4/0x1c0)
[  533.741405] [<801b4d94>] (futex_wake) from [<801b79dc>] (do_futex+0x160/0xcfc)
[  533.741416]  r10:80e04f88 r9:02648194 r8:00000081 r7:000000f0 r6:7fffffff r5:80e04f88
[  533.741421]  r4:00000001
[  533.741433] [<801b787c>] (do_futex) from [<801b8780>] (sys_futex_time32+0x9c/0x19c)
[  533.741444]  r10:7fffffff r9:02648194 r8:00000081 r7:000000f0 r6:00000000 r5:80e04f88
[  533.741449]  r4:00000001
[  533.741460] [<801b86e4>] (sys_futex_time32) from [<801011a4>] (__sys_trace_return+0x0/0x1c)
[  533.741466] Exception stack(0xb5f49fa8 to 0xb5f49ff0)
[  533.741475] 9fa0:                   00000001 00000000 02648194 00000081 7fffffff 00000000
[  533.741485] 9fc0: 00000001 00000000 00000001 000000f0 000008f0 00000001 00000000 0264816c
[  533.741492] 9fe0: 02648188 58ff9c38 02648168 76d00a14
[  533.741503]  r10:000000f0 r9:b5f48000 r8:801011c4 r7:000000f0 r6:00000001 r5:00000000
[  533.741508]  r4:00000001
[  596.770914] rcu: INFO: rcu_sched self-detected stall on CPU
[  596.770933] rcu:     3-....: (21008 ticks this GP) idle=9a2/1/0x40000002 softirq=9183/9183 fqs=10503
[  596.770940]  (t=21009 jiffies g=35521 q=7101)
[  596.770946] NMI backtrace for cpu 3
[  596.770956] CPU: 3 PID: 633 Comm: mysqld Not tainted 5.4.68-v7+ #1343
[  596.770960] Hardware name: BCM2835
[  596.770964] Backtrace:
[  596.770985] [<8010d4f4>] (dump_backtrace) from [<8010d7ec>] (show_stack+0x20/0x24)
[  596.770994]  r6:b5f48000 r5:00000000 r4:80e97abc r3:02ff4cf1
[  596.771006] [<8010d7cc>] (show_stack) from [<80908944>] (dump_stack+0xe0/0x124)
[  596.771020] [<80908864>] (dump_stack) from [<80910398>] (nmi_cpu_backtrace+0xc8/0xcc)
[  596.771031]  r8:00000140 r7:80a0202c r6:00000003 r5:00000000 r4:00000003 r3:02ff4cf1
[  596.771045] [<809102d0>] (nmi_cpu_backtrace) from [<80910498>] (nmi_trigger_cpumask_backtrace+0xfc/0x138)
[  596.771051]  r5:80e07d3c r4:8010f3dc
[  596.771065] [<8091039c>] (nmi_trigger_cpumask_backtrace) from [<801105fc>] (arch_trigger_cpumask_backtrace+0x20/0x24)
[  596.771073]  r7:80e05044 r6:80000193 r5:80a0201c r4:00000003
[  596.771086] [<801105dc>] (arch_trigger_cpumask_backtrace) from [<80196744>] (rcu_dump_cpu_stacks+0xb4/0xe4)
[  596.771096] [<80196690>] (rcu_dump_cpu_stacks) from [<80195e18>] (rcu_sched_clock_irq+0x868/0xa80)
[  596.771107]  r10:3d818000 r9:80ed5b18 r8:80e05034 r7:80da6f40 r6:be5bef40 r5:80ea8904
[  596.771113]  r4:80e10d80 r3:000073ed
[  596.771127] [<801955b0>] (rcu_sched_clock_irq) from [<8019f674>] (update_process_times+0x3c/0xa8)
[  596.771139]  r10:801b2e50 r9:be5b85f0 r8:be5b8540 r7:0000008a r6:f2439697 r5:b5f49dd0
[  596.771143]  r4:00000000
[  596.771157] [<8019f638>] (update_process_times) from [<801b25a8>] (tick_sched_handle+0x64/0x70)
[  596.771163]  r4:be5b8870 r3:80000013
[  596.771174] [<801b2544>] (tick_sched_handle) from [<801b2eac>] (tick_sched_timer+0x5c/0xb8)
[  596.771185] [<801b2e50>] (tick_sched_timer) from [<801a0638>] (__hrtimer_run_queues+0x164/0x324)
[  596.771194]  r7:b5f48000 r6:be5b8540 r5:be5b8580 r4:be5b8870
[  596.771204] [<801a04d4>] (__hrtimer_run_queues) from [<801a104c>] (hrtimer_interrupt+0x130/0x2a4)
[  596.771215]  r10:be5b85c8 r9:be5b85f0 r8:be5b8540 r7:ffffffff r6:7fffffff r5:00000003
[  596.771220]  r4:20000193
[  596.771234] [<801a0f1c>] (hrtimer_interrupt) from [<80771744>] (arch_timer_handler_phys+0x40/0x48)
[  596.771245]  r10:80e04f88 r9:b5f48000 r8:80e66b54 r7:b946dd00 r6:000000a2 r5:b9402f40
[  596.771250]  r4:80da62e4
[  596.771262] [<80771704>] (arch_timer_handler_phys) from [<80188500>] (handle_percpu_devid_irq+0x88/0x23c)
[  596.771275] [<80188478>] (handle_percpu_devid_irq) from [<80181a00>] (generic_handle_irq+0x34/0x44)
[  596.771286]  r9:b5f48000 r8:b9460000 r7:00000001 r6:00000000 r5:00000000 r4:80da62e4
[  596.771297] [<801819cc>] (generic_handle_irq) from [<8018214c>] (__handle_domain_irq+0x6c/0xc4)
[  596.771309] [<801820e0>] (__handle_domain_irq) from [<80102228>] (bcm2836_arm_irqchip_handle_irq+0x60/0xa4)
[  596.771320]  r8:b5f49e40 r7:b5f49e04 r6:ffffffff r5:80000013 r4:00000003 r3:b5f49dd0
[  596.771331] [<801021c8>] (bcm2836_arm_irqchip_handle_irq) from [<80101a3c>] (__irq_svc+0x5c/0x7c)
[  596.771336] Exception stack(0xb5f49dd0 to 0xb5f49e18)
[  596.771345] 9dc0:                                     b9651e88 00000000 0000b5f4 0000de24
[  596.771355] 9de0: 00000001 80e04f88 b9651e84 b9651e84 b5f49e40 ffffffff 80e04f88 b5f49e2c
[  596.771364] 9e00: b5f49e30 b5f49e20 801b4e58 8092a02c 80000013 ffffffff
[  596.771370]  r4:8092a02c r3:02ff4cf1
[  596.771383] [<80929fec>] (_raw_spin_lock) from [<801b4e58>] (futex_wake+0xc4/0x1c0)
[  596.771395] [<801b4d94>] (futex_wake) from [<801b79dc>] (do_futex+0x160/0xcfc)
[  596.771406]  r10:80e04f88 r9:02648194 r8:00000081 r7:000000f0 r6:7fffffff r5:80e04f88
[  596.771411]  r4:00000001
[  596.771423] [<801b787c>] (do_futex) from [<801b8780>] (sys_futex_time32+0x9c/0x19c)
[  596.771434]  r10:7fffffff r9:02648194 r8:00000081 r7:000000f0 r6:00000000 r5:80e04f88
[  596.771438]  r4:00000001
[  596.771449] [<801b86e4>] (sys_futex_time32) from [<801011a4>] (__sys_trace_return+0x0/0x1c)
[  596.771455] Exception stack(0xb5f49fa8 to 0xb5f49ff0)
[  596.771464] 9fa0:                   00000001 00000000 02648194 00000081 7fffffff 00000000
[  596.771474] 9fc0: 00000001 00000000 00000001 000000f0 000008f0 00000001 00000000 0264816c
[  596.771481] 9fe0: 02648188 58ff9c38 02648168 76d00a14
[  596.771492]  r10:000000f0 r9:b5f48000 r8:801011c4 r7:000000f0 r6:00000001 r5:00000000
[  596.771497]  r4:00000001
[  612.311034] INFO: task dietpi-set_cpu:1592 blocked for more than 122 seconds.
[  612.311049]       Not tainted 5.4.68-v7+ #1343
[  612.311057] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  612.311067] dietpi-set_cpu  D    0  1592   1538 0x00000001
[  612.311081] Backtrace:
[  612.311110] [<80923ad8>] (__schedule) from [<80924334>] (schedule+0x5c/0xcc)
[  612.311126]  r10:a5743d4c r9:a5742000 r8:7fffffff r7:00000002 r6:80e04f88 r5:a5742000
[  612.311134]  r4:a2f10000
[  612.311151] [<809242d8>] (schedule) from [<80928778>] (schedule_timeout+0x168/0x39c)
[  612.311161]  r5:7fffffff r4:a5743d54
[  612.311177] [<80928610>] (schedule_timeout) from [<80925a28>] (wait_for_completion+0xd0/0x15c)
[  612.311192]  r10:a5743d4c r9:a5742000 r8:80e04f88 r7:00000002 r6:a5743d58 r5:7fffffff
[  612.311200]  r4:a5743d54
[  612.311217] [<80925958>] (wait_for_completion) from [<8018e040>] (__wait_rcu_gp+0x150/0x17c)
[  612.311232]  r9:a5743d64 r8:80191e34 r7:00000001 r6:00000000 r5:a5743d48 r4:00000000
[  612.311249] [<8018def0>] (__wait_rcu_gp) from [<80194fa8>] (synchronize_rcu+0x68/0x8c)
[  612.311264]  r10:b8ddf780 r9:80f570bc r8:80e106f0 r7:b79fa6c0 r6:80e05088 r5:b8e31800
[  612.311272]  r4:80e04f88
[  612.311289] [<80194f40>] (synchronize_rcu) from [<801765d0>] (sugov_stop+0x50/0x70)
[  612.311298]  r4:00000004
[  612.311315] [<80176580>] (sugov_stop) from [<80743230>] (cpufreq_stop_governor.part.1+0x30/0x34)
[  612.311327]  r7:80e88458 r6:b8e31868 r5:80e04f88 r4:b8e31800
[  612.311344] [<80743200>] (cpufreq_stop_governor.part.1) from [<8074555c>] (cpufreq_set_policy+0x1d8/0x27c)
[  612.311361] [<80745384>] (cpufreq_set_policy) from [<80745844>] (store_scaling_governor+0x100/0x114)
[  612.311375]  r10:b5ee5490 r9:b5ee2b00 r8:b8e31924 r7:0000000c r6:80e88458 r5:b8e31800
[  612.311383]  r4:80e04f88
[  612.311398] [<80745744>] (store_scaling_governor) from [<807425e4>] (store+0x88/0xa4)
[  612.311410]  r7:b8e31800 r6:b5ee2b00 r5:0000000c r4:80e8833c
[  612.311427] [<8074255c>] (store) from [<803a2a94>] (sysfs_kf_write+0x54/0x58)
[  612.311442]  r8:a5743f60 r7:00000000 r6:00000000 r5:b5ee5480 r4:b5ee2b00 r3:0000000c
[  612.311457] [<803a2a40>] (sysfs_kf_write) from [<803a1d44>] (kernfs_fop_write+0xec/0x20c)
[  612.311466]  r4:0000000c r3:803a2a40
[  612.311482] [<803a1c58>] (kernfs_fop_write) from [<80301f10>] (__vfs_write+0x48/0x1e0)
[  612.311497]  r10:00000004 r9:0000000c r8:a5743f60 r7:a5743f60 r6:02ff4cf1 r5:a2f60300
[  612.311505]  r4:80e04f88
[  612.311519] [<80301ec8>] (__vfs_write) from [<80304e9c>] (vfs_write+0xb4/0x1c8)
[  612.311534]  r9:a5742000 r8:019a2a38 r7:a5743f60 r6:019a2a38 r5:a2f60300 r4:0000000c
[  612.311548] [<80304de8>] (vfs_write) from [<8030517c>] (ksys_write+0x70/0xec)
[  612.311562]  r8:019a2a38 r7:0000000c r6:a2f60300 r5:80e04f88 r4:a2f60300
[  612.311576] [<8030510c>] (ksys_write) from [<80305210>] (sys_write+0x18/0x1c)
[  612.311589]  r8:801011c4 r7:00000004 r6:76f2cd08 r5:76fb34d0 r4:00000001
[  612.311603] [<803051f8>] (sys_write) from [<80101000>] (ret_fast_syscall+0x0/0x28)
[  612.311613] Exception stack(0xa5743fa8 to 0xa5743ff0)
[  612.311626] 3fa0:                   00000001 76fb34d0 00000001 019a2a38 0000000c 00000000
[  612.311640] 3fc0: 00000001 76fb34d0 76f2cd08 00000004 76f2d2c4 76f2ce2c 00000000 00000000
[  612.311651] 3fe0: 0000000a 7e9af2f8 76ea9bb0 76ea9bcc
[  659.800916] rcu: INFO: rcu_sched self-detected stall on CPU
[  659.800937] rcu:     3-....: (27311 ticks this GP) idle=9a2/1/0x40000002 softirq=9183/9183 fqs=13654
[  659.800944]  (t=27312 jiffies g=35521 q=8549)
[  659.800951] NMI backtrace for cpu 3
[  659.800962] CPU: 3 PID: 633 Comm: mysqld Not tainted 5.4.68-v7+ #1343
[  659.800966] Hardware name: BCM2835
[  659.800971] Backtrace:
[  659.800992] [<8010d4f4>] (dump_backtrace) from [<8010d7ec>] (show_stack+0x20/0x24)
[  659.801001]  r6:b5f48000 r5:00000000 r4:80e97abc r3:02ff4cf1
[  659.801014] [<8010d7cc>] (show_stack) from [<80908944>] (dump_stack+0xe0/0x124)
[  659.801028] [<80908864>] (dump_stack) from [<80910398>] (nmi_cpu_backtrace+0xc8/0xcc)
[  659.801038]  r8:00000140 r7:80a0202c r6:00000003 r5:00000000 r4:00000003 r3:02ff4cf1
[  659.801053] [<809102d0>] (nmi_cpu_backtrace) from [<80910498>] (nmi_trigger_cpumask_backtrace+0xfc/0x138)
[  659.801059]  r5:80e07d3c r4:8010f3dc
[  659.801073] [<8091039c>] (nmi_trigger_cpumask_backtrace) from [<801105fc>] (arch_trigger_cpumask_backtrace+0x20/0x24)
[  659.801081]  r7:80e05044 r6:80000193 r5:80a0201c r4:00000003
[  659.801093] [<801105dc>] (arch_trigger_cpumask_backtrace) from [<80196744>] (rcu_dump_cpu_stacks+0xb4/0xe4)
[  659.801103] [<80196690>] (rcu_dump_cpu_stacks) from [<80195e18>] (rcu_sched_clock_irq+0x868/0xa80)
[  659.801114]  r10:3d818000 r9:80ed5b18 r8:80e05034 r7:80da6f40 r6:be5bef40 r5:80ea8904
[  659.801120]  r4:80e10d80 r3:00008c8b
[  659.801134] [<801955b0>] (rcu_sched_clock_irq) from [<8019f674>] (update_process_times+0x3c/0xa8)
[  659.801146]  r10:801b2e50 r9:be5b85f0 r8:be5b8540 r7:00000099 r6:9f251033 r5:b5f49dd0
[  659.801150]  r4:00000000
[  659.801164] [<8019f638>] (update_process_times) from [<801b25a8>] (tick_sched_handle+0x64/0x70)
[  659.801171]  r4:be5b8870 r3:80000013
[  659.801182] [<801b2544>] (tick_sched_handle) from [<801b2eac>] (tick_sched_timer+0x5c/0xb8)
[  659.801193] [<801b2e50>] (tick_sched_timer) from [<801a0638>] (__hrtimer_run_queues+0x164/0x324)
[  659.801202]  r7:b5f48000 r6:be5b8540 r5:be5b8580 r4:be5b8870
[  659.801212] [<801a04d4>] (__hrtimer_run_queues) from [<801a104c>] (hrtimer_interrupt+0x130/0x2a4)
[  659.801223]  r10:be5b85c8 r9:be5b85f0 r8:be5b8540 r7:ffffffff r6:7fffffff r5:00000003
[  659.801228]  r4:20000193
[  659.801241] [<801a0f1c>] (hrtimer_interrupt) from [<80771744>] (arch_timer_handler_phys+0x40/0x48)
[  659.801252]  r10:80e04f88 r9:b5f48000 r8:80e66b54 r7:b946dd00 r6:000000a2 r5:b9402f40
[  659.801257]  r4:80da62e4
[  659.801269] [<80771704>] (arch_timer_handler_phys) from [<80188500>] (handle_percpu_devid_irq+0x88/0x23c)
[  659.801281] [<80188478>] (handle_percpu_devid_irq) from [<80181a00>] (generic_handle_irq+0x34/0x44)
[  659.801292]  r9:b5f48000 r8:b9460000 r7:00000001 r6:00000000 r5:00000000 r4:80da62e4
[  659.801304] [<801819cc>] (generic_handle_irq) from [<8018214c>] (__handle_domain_irq+0x6c/0xc4)
[  659.801316] [<801820e0>] (__handle_domain_irq) from [<80102228>] (bcm2836_arm_irqchip_handle_irq+0x60/0xa4)
[  659.801327]  r8:b5f49e40 r7:b5f49e04 r6:ffffffff r5:80000013 r4:00000003 r3:b5f49dd0
[  659.801337] [<801021c8>] (bcm2836_arm_irqchip_handle_irq) from [<80101a3c>] (__irq_svc+0x5c/0x7c)
[  659.801343] Exception stack(0xb5f49dd0 to 0xb5f49e18)
[  659.801352] 9dc0:                                     b9651e88 00000000 0000b5f4 0000de24
[  659.801362] 9de0: 00000001 80e04f88 b9651e84 b9651e84 b5f49e40 ffffffff 80e04f88 b5f49e2c
[  659.801371] 9e00: b5f49e30 b5f49e20 801b4e58 8092a02c 80000013 ffffffff
[  659.801378]  r4:8092a02c r3:02ff4cf1
[  659.801391] [<80929fec>] (_raw_spin_lock) from [<801b4e58>] (futex_wake+0xc4/0x1c0)
[  659.801403] [<801b4d94>] (futex_wake) from [<801b79dc>] (do_futex+0x160/0xcfc)
[  659.801414]  r10:80e04f88 r9:02648194 r8:00000081 r7:000000f0 r6:7fffffff r5:80e04f88
[  659.801418]  r4:00000001
[  659.801431] [<801b787c>] (do_futex) from [<801b8780>] (sys_futex_time32+0x9c/0x19c)
[  659.801442]  r10:7fffffff r9:02648194 r8:00000081 r7:000000f0 r6:00000000 r5:80e04f88
[  659.801447]  r4:00000001
[  659.801458] [<801b86e4>] (sys_futex_time32) from [<801011a4>] (__sys_trace_return+0x0/0x1c)
[  659.801463] Exception stack(0xb5f49fa8 to 0xb5f49ff0)
[  659.801473] 9fa0:                   00000001 00000000 02648194 00000081 7fffffff 00000000
[  659.801482] 9fc0: 00000001 00000000 00000001 000000f0 000008f0 00000001 00000000 0264816c
[  659.801490] 9fe0: 02648188 58ff9c38 02648168 76d00a14
[  659.801501]  r10:000000f0 r9:b5f48000 r8:801011c4 r7:000000f0 r6:00000001 r5:00000000
[  659.801505]  r4:00000001
[  722.830917] rcu: INFO: rcu_sched self-detected stall on CPU
[  722.830940] rcu:     3-....: (33614 ticks this GP) idle=9a2/1/0x40000002 softirq=9183/9183 fqs=16806
[  722.830947]  (t=33615 jiffies g=35521 q=20364)
[  722.830954] NMI backtrace for cpu 3
[  722.830964] CPU: 3 PID: 633 Comm: mysqld Not tainted 5.4.68-v7+ #1343
[  722.830968] Hardware name: BCM2835
[  722.830973] Backtrace:
[  722.830994] [<8010d4f4>] (dump_backtrace) from [<8010d7ec>] (show_stack+0x20/0x24)
[  722.831004]  r6:b5f48000 r5:00000000 r4:80e97abc r3:02ff4cf1
[  722.831016] [<8010d7cc>] (show_stack) from [<80908944>] (dump_stack+0xe0/0x124)
[  722.831030] [<80908864>] (dump_stack) from [<80910398>] (nmi_cpu_backtrace+0xc8/0xcc)
[  722.831041]  r8:00000140 r7:80a0202c r6:00000003 r5:00000000 r4:00000003 r3:02ff4cf1
[  722.831055] [<809102d0>] (nmi_cpu_backtrace) from [<80910498>] (nmi_trigger_cpumask_backtrace+0xfc/0x138)
[  722.831061]  r5:80e07d3c r4:8010f3dc
[  722.831074] [<8091039c>] (nmi_trigger_cpumask_backtrace) from [<801105fc>] (arch_trigger_cpumask_backtrace+0x20/0x24)
[  722.831083]  r7:80e05044 r6:80000193 r5:80a0201c r4:00000003
[  722.831095] [<801105dc>] (arch_trigger_cpumask_backtrace) from [<80196744>] (rcu_dump_cpu_stacks+0xb4/0xe4)
[  722.831105] [<80196690>] (rcu_dump_cpu_stacks) from [<80195e18>] (rcu_sched_clock_irq+0x868/0xa80)
[  722.831116]  r10:3d818000 r9:80ed5b18 r8:80e05034 r7:80da6f40 r6:be5bef40 r5:80ea8904
[  722.831122]  r4:80e10d80 r3:0000a52b
[  722.831136] [<801955b0>] (rcu_sched_clock_irq) from [<8019f674>] (update_process_times+0x3c/0xa8)
[  722.831147]  r10:801b2e50 r9:be5b85f0 r8:be5b8540 r7:000000a8 r6:4c0689cf r5:b5f49dd0
[  722.831152]  r4:00000000
[  722.831165] [<8019f638>] (update_process_times) from [<801b25a8>] (tick_sched_handle+0x64/0x70)
[  722.831171]  r4:be5b8870 r3:80000013
[  722.831183] [<801b2544>] (tick_sched_handle) from [<801b2eac>] (tick_sched_timer+0x5c/0xb8)
[  722.831194] [<801b2e50>] (tick_sched_timer) from [<801a0638>] (__hrtimer_run_queues+0x164/0x324)
[  722.831203]  r7:b5f48000 r6:be5b8540 r5:be5b8580 r4:be5b8870
[  722.831213] [<801a04d4>] (__hrtimer_run_queues) from [<801a104c>] (hrtimer_interrupt+0x130/0x2a4)
[  722.831224]  r10:be5b85c8 r9:be5b85f0 r8:be5b8540 r7:ffffffff r6:7fffffff r5:00000003
[  722.831229]  r4:20000193
[  722.831243] [<801a0f1c>] (hrtimer_interrupt) from [<80771744>] (arch_timer_handler_phys+0x40/0x48)
[  722.831255]  r10:80e04f88 r9:b5f48000 r8:80e66b54 r7:b946dd00 r6:000000a2 r5:b9402f40
[  722.831259]  r4:80da62e4
[  722.831271] [<80771704>] (arch_timer_handler_phys) from [<80188500>] (handle_percpu_devid_irq+0x88/0x23c)
[  722.831283] [<80188478>] (handle_percpu_devid_irq) from [<80181a00>] (generic_handle_irq+0x34/0x44)
[  722.831294]  r9:b5f48000 r8:b9460000 r7:00000001 r6:00000000 r5:00000000 r4:80da62e4
[  722.831306] [<801819cc>] (generic_handle_irq) from [<8018214c>] (__handle_domain_irq+0x6c/0xc4)
[  722.831318] [<801820e0>] (__handle_domain_irq) from [<80102228>] (bcm2836_arm_irqchip_handle_irq+0x60/0xa4)
[  722.831329]  r8:b5f49e40 r7:b5f49e04 r6:ffffffff r5:80000013 r4:00000003 r3:b5f49dd0
[  722.831340] [<801021c8>] (bcm2836_arm_irqchip_handle_irq) from [<80101a3c>] (__irq_svc+0x5c/0x7c)
[  722.831345] Exception stack(0xb5f49dd0 to 0xb5f49e18)
[  722.831354] 9dc0:                                     b9651e88 00000000 0000b5f4 0000de24
[  722.831364] 9de0: 00000001 80e04f88 b9651e84 b9651e84 b5f49e40 ffffffff 80e04f88 b5f49e2c
[  722.831373] 9e00: b5f49e30 b5f49e20 801b4e58 8092a02c 80000013 ffffffff
[  722.831379]  r4:8092a02c r3:02ff4cf1
[  722.831392] [<80929fec>] (_raw_spin_lock) from [<801b4e58>] (futex_wake+0xc4/0x1c0)
[  722.831404] [<801b4d94>] (futex_wake) from [<801b79dc>] (do_futex+0x160/0xcfc)
[  722.831415]  r10:80e04f88 r9:02648194 r8:00000081 r7:000000f0 r6:7fffffff r5:80e04f88
[  722.831419]  r4:00000001
[  722.831432] [<801b787c>] (do_futex) from [<801b8780>] (sys_futex_time32+0x9c/0x19c)
[  722.831443]  r10:7fffffff r9:02648194 r8:00000081 r7:000000f0 r6:00000000 r5:80e04f88
[  722.831447]  r4:00000001
[  722.831459] [<801b86e4>] (sys_futex_time32) from [<801011a4>] (__sys_trace_return+0x0/0x1c)
[  722.831464] Exception stack(0xb5f49fa8 to 0xb5f49ff0)
[  722.831473] 9fa0:                   00000001 00000000 02648194 00000081 7fffffff 00000000
[  722.831483] 9fc0: 00000001 00000000 00000001 000000f0 000008f0 00000001 00000000 0264816c
[  722.831490] 9fe0: 02648188 58ff9c38 02648168 76d00a14
[  722.831501]  r10:000000f0 r9:b5f48000 r8:801011c4 r7:000000f0 r6:00000001 r5:00000000
[  722.831506]  r4:00000001
[  723.030970] rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 3-... } 2123 jiffies s: 165 root: 0x8/.
[  723.030999] rcu: blocking rcu_node structures:
[  723.031011] Task dump for CPU 3:
[  723.031019] mysqld          R  running task        0   633      1 0x00000082
[  723.031035] Backtrace:
[  723.031057] Backtrace aborted due to bad frame pointer <9e54fcbd>
[  735.191035] INFO: task dietpi-set_cpu:1592 blocked for more than 245 seconds.
[  735.191317]       Not tainted 5.4.68-v7+ #1343
[  735.191472] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  735.191737] dietpi-set_cpu  D    0  1592   1538 0x00000001
[  735.191928] Backtrace:
[  735.192037] [<80923ad8>] (__schedule) from [<80924334>] (schedule+0x5c/0xcc)
[  735.192285]  r10:a5743d4c r9:a5742000 r8:7fffffff r7:00000002 r6:80e04f88 r5:a5742000
[  735.192542]  r4:a2f10000
[  735.192645] [<809242d8>] (schedule) from [<80928778>] (schedule_timeout+0x168/0x39c)
[  735.192910]  r5:7fffffff r4:a5743d54
[  735.193047] [<80928610>] (schedule_timeout) from [<80925a28>] (wait_for_completion+0xd0/0x15c)
[  735.193345]  r10:a5743d4c r9:a5742000 r8:80e04f88 r7:00000002 r6:a5743d58 r5:7fffffff
[  735.193602]  r4:a5743d54
[  735.193707] [<80925958>] (wait_for_completion) from [<8018e040>] (__wait_rcu_gp+0x150/0x17c)
[  735.193998]  r9:a5743d64 r8:80191e34 r7:00000001 r6:00000000 r5:a5743d48 r4:00000000
[  735.194262] [<8018def0>] (__wait_rcu_gp) from [<80194fa8>] (synchronize_rcu+0x68/0x8c)
[  735.194536]  r10:b8ddf780 r9:80f570bc r8:80e106f0 r7:b79fa6c0 r6:80e05088 r5:b8e31800
[  735.194798]  r4:80e04f88
[  735.194902] [<80194f40>] (synchronize_rcu) from [<801765d0>] (sugov_stop+0x50/0x70)
[  735.195160]  r4:00000004
[  735.195261] [<80176580>] (sugov_stop) from [<80743230>] (cpufreq_stop_governor.part.1+0x30/0x34)
[  735.195561]  r7:80e88458 r6:b8e31868 r5:80e04f88 r4:b8e31800
[  735.195765] [<80743200>] (cpufreq_stop_governor.part.1) from [<8074555c>] (cpufreq_set_policy+0x1d8/0x27c)
[  735.196098] [<80745384>] (cpufreq_set_policy) from [<80745844>] (store_scaling_governor+0x100/0x114)
[  735.196411]  r10:b5ee5490 r9:b5ee2b00 r8:b8e31924 r7:0000000c r6:80e88458 r5:b8e31800
[  735.196671]  r4:80e04f88
[  735.196774] [<80745744>] (store_scaling_governor) from [<807425e4>] (store+0x88/0xa4)
[  735.197041]  r7:b8e31800 r6:b5ee2b00 r5:0000000c r4:80e8833c
[  735.197242] [<8074255c>] (store) from [<803a2a94>] (sysfs_kf_write+0x54/0x58)
[  735.197488]  r8:a5743f60 r7:00000000 r6:00000000 r5:b5ee5480 r4:b5ee2b00 r3:0000000c
[  735.197750] [<803a2a40>] (sysfs_kf_write) from [<803a1d44>] (kernfs_fop_write+0xec/0x20c)
[  735.198025]  r4:0000000c r3:803a2a40
[  735.198159] [<803a1c58>] (kernfs_fop_write) from [<80301f10>] (__vfs_write+0x48/0x1e0)
[  735.198435]  r10:00000004 r9:0000000c r8:a5743f60 r7:a5743f60 r6:02ff4cf1 r5:a2f60300
[  735.198692]  r4:80e04f88
[  735.198794] [<80301ec8>] (__vfs_write) from [<80304e9c>] (vfs_write+0xb4/0x1c8)
[  735.199046]  r9:a5742000 r8:019a2a38 r7:a5743f60 r6:019a2a38 r5:a2f60300 r4:0000000c
[  735.199313] [<80304de8>] (vfs_write) from [<8030517c>] (ksys_write+0x70/0xec)
[  735.199559]  r8:019a2a38 r7:0000000c r6:a2f60300 r5:80e04f88 r4:a2f60300
[  735.199792] [<8030510c>] (ksys_write) from [<80305210>] (sys_write+0x18/0x1c)
[  735.200038]  r8:801011c4 r7:00000004 r6:76f2cd08 r5:76fb34d0 r4:00000001
[  735.200266] [<803051f8>] (sys_write) from [<80101000>] (ret_fast_syscall+0x0/0x28)
[  735.200523] Exception stack(0xa5743fa8 to 0xa5743ff0)
[  735.200701] 3fa0:                   00000001 76fb34d0 00000001 019a2a38 0000000c 00000000
[  735.201007] 3fc0: 00000001 76fb34d0 76f2cd08 00000004 76f2d2c4 76f2ce2c 00000000 00000000
[  735.201275] 3fe0: 0000000a 7e9af2f8 76ea9bb0 76ea9bcc
[  785.860917] rcu: INFO: rcu_sched self-detected stall on CPU
[  785.861151] rcu:     3-....: (39917 ticks this GP) idle=9a2/1/0x40000002 softirq=9183/9183 fqs=19958
[  785.861451]  (t=39918 jiffies g=35521 q=25461)
[  785.861605] NMI backtrace for cpu 3
[  785.861729] CPU: 3 PID: 633 Comm: mysqld Not tainted 5.4.68-v7+ #1343
[  785.861942] Hardware name: BCM2835
[  785.862058] Backtrace:
[  785.862160] [<8010d4f4>] (dump_backtrace) from [<8010d7ec>] (show_stack+0x20/0x24)
[  785.862418]  r6:b5f48000 r5:00000000 r4:80e97abc r3:02ff4cf1
[  785.862613] [<8010d7cc>] (show_stack) from [<80908944>] (dump_stack+0xe0/0x124)
[  785.862864] [<80908864>] (dump_stack) from [<80910398>] (nmi_cpu_backtrace+0xc8/0xcc)
[  785.863128]  r8:00000140 r7:80a0202c r6:00000003 r5:00000000 r4:00000003 r3:02ff4cf1
[  785.863382] [<809102d0>] (nmi_cpu_backtrace) from [<80910498>] (nmi_trigger_cpumask_backtrace+0xfc/0x138)
[  785.863699]  r5:80e07d3c r4:8010f3dc
[  785.863831] [<8091039c>] (nmi_trigger_cpumask_backtrace) from [<801105fc>] (arch_trigger_cpumask_backtrace+0x20/0x24)
[  785.864183]  r7:80e05044 r6:80000193 r5:80a0201c r4:00000003
[  785.864376] [<801105dc>] (arch_trigger_cpumask_backtrace) from [<80196744>] (rcu_dump_cpu_stacks+0xb4/0xe4)
[  785.864701] [<80196690>] (rcu_dump_cpu_stacks) from [<80195e18>] (rcu_sched_clock_irq+0x868/0xa80)
[  785.865004]  r10:3d818000 r9:80ed5b18 r8:80e05034 r7:80da6f40 r6:be5bef40 r5:80ea8904
[  785.865263]  r4:80e10d80 r3:0000bdcb
[  785.865395] [<801955b0>] (rcu_sched_clock_irq) from [<8019f674>] (update_process_times+0x3c/0xa8)
[  785.865694]  r10:801b2e50 r9:be5b85f0 r8:be5b8540 r7:000000b6 r6:f8e802cf r5:b5f49dd0
[  785.865947]  r4:00000000
[  785.866043] [<8019f638>] (update_process_times) from [<801b25a8>] (tick_sched_handle+0x64/0x70)
[  785.866331]  r4:be5b8870 r3:80000013
[  785.866460] [<801b2544>] (tick_sched_handle) from [<801b2eac>] (tick_sched_timer+0x5c/0xb8)
[  785.866742] [<801b2e50>] (tick_sched_timer) from [<801a0638>] (__hrtimer_run_queues+0x164/0x324)
[  785.867038]  r7:b5f48000 r6:be5b8540 r5:be5b8580 r4:be5b8870
[  785.867234] [<801a04d4>] (__hrtimer_run_queues) from [<801a104c>] (hrtimer_interrupt+0x130/0x2a4)
[  785.867532]  r10:be5b85c8 r9:be5b85f0 r8:be5b8540 r7:ffffffff r6:7fffffff r5:00000003
[  785.867778]  r4:20000193
[  785.867877] [<801a0f1c>] (hrtimer_interrupt) from [<80771744>] (arch_timer_handler_phys+0x40/0x48)
[  785.868186]  r10:80e04f88 r9:b5f48000 r8:80e66b54 r7:b946dd00 r6:000000a2 r5:b9402f40
[  785.868442]  r4:80da62e4
[  785.868539] [<80771704>] (arch_timer_handler_phys) from [<80188500>] (handle_percpu_devid_irq+0x88/0x23c)
[  785.868861] [<80188478>] (handle_percpu_devid_irq) from [<80181a00>] (generic_handle_irq+0x34/0x44)
[  785.869165]  r9:b5f48000 r8:b9460000 r7:00000001 r6:00000000 r5:00000000 r4:80da62e4
[  785.869420] [<801819cc>] (generic_handle_irq) from [<8018214c>] (__handle_domain_irq+0x6c/0xc4)
[  785.869713] [<801820e0>] (__handle_domain_irq) from [<80102228>] (bcm2836_arm_irqchip_handle_irq+0x60/0xa4)
[  785.870039]  r8:b5f49e40 r7:b5f49e04 r6:ffffffff r5:80000013 r4:00000003 r3:b5f49dd0
[  785.870295] [<801021c8>] (bcm2836_arm_irqchip_handle_irq) from [<80101a3c>] (__irq_svc+0x5c/0x7c)
[  785.870587] Exception stack(0xb5f49dd0 to 0xb5f49e18)
[  785.870759] 9dc0:                                     b9651e88 00000000 0000b5f4 0000de24
[  785.871000] 9de0: 00000001 80e04f88 b9651e84 b9651e84 b5f49e40 ffffffff 80e04f88 b5f49e2c
[  785.871266] 9e00: b5f49e30 b5f49e20 801b4e58 8092a02c 80000013 ffffffff
[  785.871480]  r4:8092a02c r3:02ff4cf1
[  785.871612] [<80929fec>] (_raw_spin_lock) from [<801b4e58>] (futex_wake+0xc4/0x1c0)
[  785.871872] [<801b4d94>] (futex_wake) from [<801b79dc>] (do_futex+0x160/0xcfc)
[  785.872121]  r10:80e04f88 r9:02648194 r8:00000081 r7:000000f0 r6:7fffffff r5:80e04f88
[  785.872368]  r4:00000001
[  785.872462] [<801b787c>] (do_futex) from [<801b8780>] (sys_futex_time32+0x9c/0x19c)
[  785.872721]  r10:7fffffff r9:02648194 r8:00000081 r7:000000f0 r6:00000000 r5:80e04f88
[  785.872962]  r4:00000001
[  785.873055] [<801b86e4>] (sys_futex_time32) from [<801011a4>] (__sys_trace_return+0x0/0x1c)
[  785.873330] Exception stack(0xb5f49fa8 to 0xb5f49ff0)
[  785.873503] 9fa0:                   00000001 00000000 02648194 00000081 7fffffff 00000000
[  785.873745] 9fc0: 00000001 00000000 00000001 000000f0 000008f0 00000001 00000000 0264816c
[  785.873994] 9fe0: 02648188 58ff9c38 02648168 76d00a14
[  785.874169]  r10:000000f0 r9:b5f48000 r8:801011c4 r7:000000f0 r6:00000001 r5:00000000
[  785.874411]  r4:00000001
[  786.390965] rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 3-... } 8459 jiffies s: 165 root: 0x8/.
[  786.391359] rcu: blocking rcu_node structures:
[  786.391537] Task dump for CPU 3:
[  786.391669] mysqld          R  running task        0   633      1 0x00000082
[  786.391906] Backtrace:
[  786.392011] Backtrace aborted due to bad frame pointer <9e54fcbd>
[  848.890917] rcu: INFO: rcu_sched self-detected stall on CPU
[  848.891151] rcu:     3-....: (46220 ticks this GP) idle=9a2/1/0x40000002 softirq=9183/9183 fqs=23109
[  848.891481]  (t=46221 jiffies g=35521 q=37096)
[  848.891655] NMI backtrace for cpu 3
[  848.891784] CPU: 3 PID: 633 Comm: mysqld Not tainted 5.4.68-v7+ #1343
[  848.892001] Hardware name: BCM2835
[  848.892126] Backtrace:
[  848.892236] [<8010d4f4>] (dump_backtrace) from [<8010d7ec>] (show_stack+0x20/0x24)
[  848.892507]  r6:b5f48000 r5:00000000 r4:80e97abc r3:02ff4cf1
[  848.892711] [<8010d7cc>] (show_stack) from [<80908944>] (dump_stack+0xe0/0x124)
[  848.892974] [<80908864>] (dump_stack) from [<80910398>] (nmi_cpu_backtrace+0xc8/0xcc)
[  848.893253]  r8:00000140 r7:80a0202c r6:00000003 r5:00000000 r4:00000003 r3:02ff4cf1
[  848.893523] [<809102d0>] (nmi_cpu_backtrace) from [<80910498>] (nmi_trigger_cpumask_backtrace+0xfc/0x138)
[  848.893867]  r5:80e07d3c r4:8010f3dc
[  848.894010] [<8091039c>] (nmi_trigger_cpumask_backtrace) from [<801105fc>] (arch_trigger_cpumask_backtrace+0x20/0x24)
[  848.894436]  r7:80e05044 r6:80000193 r5:80a0201c r4:00000003
[  848.894815] [<801105dc>] (arch_trigger_cpumask_backtrace) from [<80196744>] (rcu_dump_cpu_stacks+0xb4/0xe4)
[  848.895324] [<80196690>] (rcu_dump_cpu_stacks) from [<80195e18>] (rcu_sched_clock_irq+0x868/0xa80)
[  848.895646]  r10:3d818000 r9:80ed5b18 r8:80e05034 r7:80da6f40 r6:be5bef40 r5:80ea8904
[  848.895911]  r4:80e10d80 r3:0000d66a
[  848.896047] [<801955b0>] (rcu_sched_clock_irq) from [<8019f674>] (update_process_times+0x3c/0xa8)
[  848.896358]  r10:801b2e50 r9:be5b85f0 r8:be5b8540 r7:000000c5 r6:a5c97c9f r5:b5f49dd0
[  848.896667]  r4:00000000
[  848.896797] [<8019f638>] (update_process_times) from [<801b25a8>] (tick_sched_handle+0x64/0x70)
[  848.897110]  r4:be5b8870 r3:80000013
[  848.897259] [<801b2544>] (tick_sched_handle) from [<801b2eac>] (tick_sched_timer+0x5c/0xb8)
[  848.897591] [<801b2e50>] (tick_sched_timer) from [<801a0638>] (__hrtimer_run_queues+0x164/0x324)
[  848.897942]  r7:b5f48000 r6:be5b8540 r5:be5b8580 r4:be5b8870
[  848.898221] [<801a04d4>] (__hrtimer_run_queues) from [<801a104c>] (hrtimer_interrupt+0x130/0x2a4)
[  848.898615]  r10:be5b85c8 r9:be5b85f0 r8:be5b8540 r7:ffffffff r6:7fffffff r5:00000003
[  848.898890]  r4:20000193
[  848.899022] [<801a0f1c>] (hrtimer_interrupt) from [<80771744>] (arch_timer_handler_phys+0x40/0x48)
[  848.899330]  r10:80e04f88 r9:b5f48000 r8:80e66b54 r7:b946dd00 r6:000000a2 r5:b9402f40
[  848.899585]  r4:80da62e4
[  848.899684] [<80771704>] (arch_timer_handler_phys) from [<80188500>] (handle_percpu_devid_irq+0x88/0x23c)
[  848.900007] [<80188478>] (handle_percpu_devid_irq) from [<80181a00>] (generic_handle_irq+0x34/0x44)
[  848.900310]  r9:b5f48000 r8:b9460000 r7:00000001 r6:00000000 r5:00000000 r4:80da62e4
[  848.900566] [<801819cc>] (generic_handle_irq) from [<8018214c>] (__handle_domain_irq+0x6c/0xc4)
[  848.900860] [<801820e0>] (__handle_domain_irq) from [<80102228>] (bcm2836_arm_irqchip_handle_irq+0x60/0xa4)
[  848.901190]  r8:b5f49e40 r7:b5f49e04 r6:ffffffff r5:80000013 r4:00000003 r3:b5f49dd0
[  848.901446] [<801021c8>] (bcm2836_arm_irqchip_handle_irq) from [<80101a3c>] (__irq_svc+0x5c/0x7c)
[  848.901739] Exception stack(0xb5f49dd0 to 0xb5f49e18)
[  848.901911] 9dc0:                                     b9651e88 00000000 0000b5f4 0000de24
[  848.902151] 9de0: 00000001 80e04f88 b9651e84 b9651e84 b5f49e40 ffffffff 80e04f88 b5f49e2c
[  848.902417] 9e00: b5f49e30 b5f49e20 801b4e58 8092a02c 80000013 ffffffff
[  848.902631]  r4:8092a02c r3:02ff4cf1
[  848.902763] [<80929fec>] (_raw_spin_lock) from [<801b4e58>] (futex_wake+0xc4/0x1c0)
[  848.903025] [<801b4d94>] (futex_wake) from [<801b79dc>] (do_futex+0x160/0xcfc)
[  848.903272]  r10:80e04f88 r9:02648194 r8:00000081 r7:000000f0 r6:7fffffff r5:80e04f88
[  848.903521]  r4:00000001
[  848.903617] [<801b787c>] (do_futex) from [<801b8780>] (sys_futex_time32+0x9c/0x19c)
[  848.903876]  r10:7fffffff r9:02648194 r8:00000081 r7:000000f0 r6:00000000 r5:80e04f88
[  848.904119]  r4:00000001
[  848.904213] [<801b86e4>] (sys_futex_time32) from [<801011a4>] (__sys_trace_return+0x0/0x1c)
[  848.904488] Exception stack(0xb5f49fa8 to 0xb5f49ff0)
[  848.904661] 9fa0:                   00000001 00000000 02648194 00000081 7fffffff 00000000
[  848.904903] 9fc0: 00000001 00000000 00000001 000000f0 000008f0 00000001 00000000 0264816c
[  848.905152] 9fe0: 02648188 58ff9c38 02648168 76d00a14
[  848.905327]  r10:000000f0 r9:b5f48000 r8:801011c4 r7:000000f0 r6:00000001 r5:00000000
[  848.905569]  r4:00000001
[  852.950976] rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 3-... } 15115 jiffies s: 165 root: 0x8/.
[  852.951381] rcu: blocking rcu_node structures:
[  852.951578] Task dump for CPU 3:
[  852.951696] mysqld          R  running task        0   633      1 0x00000082
[  852.951933] Backtrace:
[  852.952038] Backtrace aborted due to bad frame pointer <9e54fcbd>
[  858.071050] INFO: task dietpi-set_cpu:1592 blocked for more than 368 seconds.
[  858.071336]       Not tainted 5.4.68-v7+ #1343
[  858.071493] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  858.071758] dietpi-set_cpu  D    0  1592   1538 0x00000001
[  858.071949] Backtrace:
[  858.072061] [<80923ad8>] (__schedule) from [<80924334>] (schedule+0x5c/0xcc)
[  858.072308]  r10:a5743d4c r9:a5742000 r8:7fffffff r7:00000002 r6:80e04f88 r5:a5742000
[  858.072565]  r4:a2f10000
[  858.072668] [<809242d8>] (schedule) from [<80928778>] (schedule_timeout+0x168/0x39c)
[  858.072932]  r5:7fffffff r4:a5743d54
[  858.073066] [<80928610>] (schedule_timeout) from [<80925a28>] (wait_for_completion+0xd0/0x15c)
[  858.073363]  r10:a5743d4c r9:a5742000 r8:80e04f88 r7:00000002 r6:a5743d58 r5:7fffffff
[  858.073619]  r4:a5743d54
[  858.073723] [<80925958>] (wait_for_completion) from [<8018e040>] (__wait_rcu_gp+0x150/0x17c)
[  858.074013]  r9:a5743d64 r8:80191e34 r7:00000001 r6:00000000 r5:a5743d48 r4:00000000
[  858.074276] [<8018def0>] (__wait_rcu_gp) from [<80194fa8>] (synchronize_rcu+0x68/0x8c)
[  858.074549]  r10:b8ddf780 r9:80f570bc r8:80e106f0 r7:b79fa6c0 r6:80e05088 r5:b8e31800
[  858.074811]  r4:80e04f88
[  858.074915] [<80194f40>] (synchronize_rcu) from [<801765d0>] (sugov_stop+0x50/0x70)
[  858.075171]  r4:00000004
[  858.075272] [<80176580>] (sugov_stop) from [<80743230>] (cpufreq_stop_governor.part.1+0x30/0x34)
[  858.075572]  r7:80e88458 r6:b8e31868 r5:80e04f88 r4:b8e31800
[  858.075776] [<80743200>] (cpufreq_stop_governor.part.1) from [<8074555c>] (cpufreq_set_policy+0x1d8/0x27c)
[  858.076109] [<80745384>] (cpufreq_set_policy) from [<80745844>] (store_scaling_governor+0x100/0x114)
[  858.076423]  r10:b5ee5490 r9:b5ee2b00 r8:b8e31924 r7:0000000c r6:80e88458 r5:b8e31800
[  858.076682]  r4:80e04f88
[  858.076785] [<80745744>] (store_scaling_governor) from [<807425e4>] (store+0x88/0xa4)
[  858.077051]  r7:b8e31800 r6:b5ee2b00 r5:0000000c r4:80e8833c
[  858.077251] [<8074255c>] (store) from [<803a2a94>] (sysfs_kf_write+0x54/0x58)
[  858.077497]  r8:a5743f60 r7:00000000 r6:00000000 r5:b5ee5480 r4:b5ee2b00 r3:0000000c
[  858.077758] [<803a2a40>] (sysfs_kf_write) from [<803a1d44>] (kernfs_fop_write+0xec/0x20c)
[  858.078033]  r4:0000000c r3:803a2a40
[  858.078166] [<803a1c58>] (kernfs_fop_write) from [<80301f10>] (__vfs_write+0x48/0x1e0)
[  858.078440]  r10:00000004 r9:0000000c r8:a5743f60 r7:a5743f60 r6:02ff4cf1 r5:a2f60300
[  858.078698]  r4:80e04f88
[  858.078800] [<80301ec8>] (__vfs_write) from [<80304e9c>] (vfs_write+0xb4/0x1c8)
[  858.079052]  r9:a5742000 r8:019a2a38 r7:a5743f60 r6:019a2a38 r5:a2f60300 r4:0000000c
[  858.079318] [<80304de8>] (vfs_write) from [<8030517c>] (ksys_write+0x70/0xec)
[  858.079563]  r8:019a2a38 r7:0000000c r6:a2f60300 r5:80e04f88 r4:a2f60300
[  858.079798] [<8030510c>] (ksys_write) from [<80305210>] (sys_write+0x18/0x1c)
[  858.080055]  r8:801011c4 r7:00000004 r6:76f2cd08 r5:76fb34d0 r4:00000001
[  858.080291] [<803051f8>] (sys_write) from [<80101000>] (ret_fast_syscall+0x0/0x28)
[  858.080550] Exception stack(0xa5743fa8 to 0xa5743ff0)
[  858.080729] 3fa0:                   00000001 76fb34d0 00000001 019a2a38 0000000c 00000000
[  858.081033] 3fc0: 00000001 76fb34d0 76f2cd08 00000004 76f2d2c4 76f2ce2c 00000000 00000000
[  858.081312] 3fe0: 0000000a 7e9af2f8 76ea9bb0 76ea9bcc
[  858.081498] INFO: task sessionclean:1776 blocked for more than 122 seconds.
[  858.081740]       Not tainted 5.4.68-v7+ #1343
[  858.081895] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  858.082162] sessionclean    D    0  1776      1 0x00000004
[  858.082352] Backtrace:
[  858.082462] [<80923ad8>] (__schedule) from [<80924334>] (schedule+0x5c/0xcc)
[  858.082711]  r10:000000f8 r9:b7841440 r8:89e8fe50 r7:80e10d80 r6:80e04f88 r5:89e8e000
[  858.082973]  r4:a2f83d00
[  858.083080] [<809242d8>] (schedule) from [<80194f34>] (synchronize_rcu_expedited+0x358/0x364)
[  858.083367]  r5:000000a8 r4:80e10e60
[  858.083505] [<80194bdc>] (synchronize_rcu_expedited) from [<8032b7b8>] (namespace_unlock+0xc8/0x13c)
[  858.083820]  r8:00000000 r7:00000001 r6:00000000 r5:80e04f88 r4:89e8feac
[  858.084048] [<8032b6f0>] (namespace_unlock) from [<8032d1c8>] (drop_collected_mounts+0x78/0x80)
[  858.084345]  r7:00000001 r6:00000000 r5:a562d790 r4:80e04c40
[  858.084551] [<8032d150>] (drop_collected_mounts) from [<8032f980>] (put_mnt_ns+0x4c/0x58)
[  858.084831]  r5:00000558 r4:a83b3680
[  858.084968] [<8032f934>] (put_mnt_ns) from [<801448e4>] (free_nsproxy+0x28/0xcc)
[  858.085223]  r4:b966f940 r3:000000c8
[  858.085358] [<801448bc>] (free_nsproxy) from [<80144abc>] (switch_task_namespaces+0x84/0x88)
[  858.085646]  r5:00000558 r4:a2f83d00
[  858.085783] [<80144a38>] (switch_task_namespaces) from [<80144adc>] (exit_task_namespaces+0x1c/0x20)
[  858.086097]  r6:00000000 r5:b7841400 r4:a2f83d00 r3:00000284
[  858.086299] [<80144ac0>] (exit_task_namespaces) from [<80124840>] (do_exit+0x3d8/0xb18)
[  858.086576] [<80124468>] (do_exit) from [<80125018>] (do_group_exit+0x4c/0xc4)
[  858.086822]  r7:000000f8
[  858.086924] [<80124fcc>] (do_group_exit) from [<801250b0>] (__wake_up_parent+0x0/0x30)
[  858.087195]  r6:000117b4 r5:00000000 r4:00000004 r3:00036010
[  858.087392] [<80125090>] (sys_exit_group) from [<80101000>] (ret_fast_syscall+0x0/0x28)
[  858.087664] Exception stack(0x89e8ffa8 to 0x89e8fff0)
[  858.087843] ffa0:                   00000004 00000000 00000000 00000000 00000000 00036010
[  858.088092] ffc0: 00000004 00000000 000117b4 000000f8 00000000 00000000 76fa6000 00000000
[  858.088345] ffe0: 00035ee0 7ed0f998 0001fb10 76eb15d4
[  911.920917] rcu: INFO: rcu_sched self-detected stall on CPU
[  911.921152] rcu:     3-....: (52523 ticks this GP) idle=9a2/1/0x40000002 softirq=9183/9183 fqs=26260
[  911.921450]  (t=52524 jiffies g=35521 q=45249)
[  911.921604] NMI backtrace for cpu 3
[  911.921728] CPU: 3 PID: 633 Comm: mysqld Not tainted 5.4.68-v7+ #1343
[  911.921940] Hardware name: BCM2835
[  911.922056] Backtrace:
[  911.922159] [<8010d4f4>] (dump_backtrace) from [<8010d7ec>] (show_stack+0x20/0x24)
[  911.922415]  r6:b5f48000 r5:00000000 r4:80e97abc r3:02ff4cf1
[  911.922612] [<8010d7cc>] (show_stack) from [<80908944>] (dump_stack+0xe0/0x124)
[  911.922872] [<80908864>] (dump_stack) from [<80910398>] (nmi_cpu_backtrace+0xc8/0xcc)
[  911.923140]  r8:00000140 r7:80a0202c r6:00000003 r5:00000000 r4:00000003 r3:02ff4cf1
[  911.923395] [<809102d0>] (nmi_cpu_backtrace) from [<80910498>] (nmi_trigger_cpumask_backtrace+0xfc/0x138)
[  911.923711]  r5:80e07d3c r4:8010f3dc
[  911.923845] [<8091039c>] (nmi_trigger_cpumask_backtrace) from [<801105fc>] (arch_trigger_cpumask_backtrace+0x20/0x24)
[  911.924197]  r7:80e05044 r6:80000193 r5:80a0201c r4:00000003
[  911.924389] [<801105dc>] (arch_trigger_cpumask_backtrace) from [<80196744>] (rcu_dump_cpu_stacks+0xb4/0xe4)
[  911.924714] [<80196690>] (rcu_dump_cpu_stacks) from [<80195e18>] (rcu_sched_clock_irq+0x868/0xa80)
[  911.925018]  r10:3d818000 r9:80ed5b18 r8:80e05034 r7:80da6f40 r6:be5bef40 r5:80ea8904
[  911.925275]  r4:80e10d80 r3:0000ef09
[  911.925409] [<801955b0>] (rcu_sched_clock_irq) from [<8019f674>] (update_process_times+0x3c/0xa8)
[  911.925707]  r10:801b2e50 r9:be5b85f0 r8:be5b8540 r7:000000d4 r6:52aaf5d3 r5:b5f49dd0
[  911.925961]  r4:00000000
[  911.926057] [<8019f638>] (update_process_times) from [<801b25a8>] (tick_sched_handle+0x64/0x70)
[  911.926344]  r4:be5b8870 r3:80000013
[  911.926473] [<801b2544>] (tick_sched_handle) from [<801b2eac>] (tick_sched_timer+0x5c/0xb8)
[  911.926755] [<801b2e50>] (tick_sched_timer) from [<801a0638>] (__hrtimer_run_queues+0x164/0x324)
[  911.927050]  r7:b5f48000 r6:be5b8540 r5:be5b8580 r4:be5b8870
[  911.927246] [<801a04d4>] (__hrtimer_run_queues) from [<801a104c>] (hrtimer_interrupt+0x130/0x2a4)
[  911.927544]  r10:be5b85c8 r9:be5b85f0 r8:be5b8540 r7:ffffffff r6:7fffffff r5:00000003
[  911.927792]  r4:20000193
[  911.927889] [<801a0f1c>] (hrtimer_interrupt) from [<80771744>] (arch_timer_handler_phys+0x40/0x48)
[  911.928194]  r10:80e04f88 r9:b5f48000 r8:80e66b54 r7:b946dd00 r6:000000a2 r5:b9402f40
[  911.928447]  r4:80da62e4
[  911.928545] [<80771704>] (arch_timer_handler_phys) from [<80188500>] (handle_percpu_devid_irq+0x88/0x23c)
[  911.928867] [<80188478>] (handle_percpu_devid_irq) from [<80181a00>] (generic_handle_irq+0x34/0x44)
[  911.929171]  r9:b5f48000 r8:b9460000 r7:00000001 r6:00000000 r5:00000000 r4:80da62e4
[  911.929426] [<801819cc>] (generic_handle_irq) from [<8018214c>] (__handle_domain_irq+0x6c/0xc4)
[  911.929720] [<801820e0>] (__handle_domain_irq) from [<80102228>] (bcm2836_arm_irqchip_handle_irq+0x60/0xa4)
[  911.930045]  r8:b5f49e40 r7:b5f49e04 r6:ffffffff r5:80000013 r4:00000003 r3:b5f49dd0
[  911.930300] [<801021c8>] (bcm2836_arm_irqchip_handle_irq) from [<80101a3c>] (__irq_svc+0x5c/0x7c)
[  911.930593] Exception stack(0xb5f49dd0 to 0xb5f49e18)
[  911.930766] 9dc0:                                     b9651e88 00000000 0000b5f4 0000de24
[  911.931007] 9de0: 00000001 80e04f88 b9651e84 b9651e84 b5f49e40 ffffffff 80e04f88 b5f49e2c
[  911.931273] 9e00: b5f49e30 b5f49e20 801b4e58 8092a02c 80000013 ffffffff
[  911.931487]  r4:8092a02c r3:02ff4cf1
[  911.931619] [<80929fec>] (_raw_spin_lock) from [<801b4e58>] (futex_wake+0xc4/0x1c0)
[  911.931879] [<801b4d94>] (futex_wake) from [<801b79dc>] (do_futex+0x160/0xcfc)
[  911.932127]  r10:80e04f88 r9:02648194 r8:00000081 r7:000000f0 r6:7fffffff r5:80e04f88
[  911.932375]  r4:00000001
[  911.932469] [<801b787c>] (do_futex) from [<801b8780>] (sys_futex_time32+0x9c/0x19c)
[  911.932728]  r10:7fffffff r9:02648194 r8:00000081 r7:000000f0 r6:00000000 r5:80e04f88
[  911.932970]  r4:00000001
[  911.933063] [<801b86e4>] (sys_futex_time32) from [<801011a4>] (__sys_trace_return+0x0/0x1c)
[  911.933342] Exception stack(0xb5f49fa8 to 0xb5f49ff0)
[  911.933518] 9fa0:                   00000001 00000000 02648194 00000081 7fffffff 00000000
[  911.933759] 9fc0: 00000001 00000000 00000001 000000f0 000008f0 00000001 00000000 0264816c
[  911.934009] 9fe0: 02648188 58ff9c38 02648168 76d00a14
[  911.934184]  r10:000000f0 r9:b5f48000 r8:801011c4 r7:000000f0 r6:00000001 r5:00000000
[  911.934427]  r4:00000001
[  919.510972] rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 3-... } 21771 jiffies s: 165 root: 0x8/.
[  919.511379] rcu: blocking rcu_node structures:
[  919.511574] Task dump for CPU 3:
[  919.511693] mysqld          R  running task        0   633      1 0x00000082
[  919.511929] Backtrace:
[  919.512034] Backtrace aborted due to bad frame pointer <9e54fcbd>
[  974.950918] rcu: INFO: rcu_sched self-detected stall on CPU
[  974.951153] rcu:     3-....: (58826 ticks this GP) idle=9a2/1/0x40000002 softirq=9183/9183 fqs=29410
[  974.951454]  (t=58827 jiffies g=35521 q=51791)
[  974.951608] NMI backtrace for cpu 3
[  974.951731] CPU: 3 PID: 633 Comm: mysqld Not tainted 5.4.68-v7+ #1343
[  974.951944] Hardware name: BCM2835
[  974.952060] Backtrace:
[  974.952163] [<8010d4f4>] (dump_backtrace) from [<8010d7ec>] (show_stack+0x20/0x24)
[  974.952419]  r6:b5f48000 r5:00000000 r4:80e97abc r3:02ff4cf1
[  974.952615] [<8010d7cc>] (show_stack) from [<80908944>] (dump_stack+0xe0/0x124)
[  974.952865] [<80908864>] (dump_stack) from [<80910398>] (nmi_cpu_backtrace+0xc8/0xcc)
[  974.953130]  r8:00000140 r7:80a0202c r6:00000003 r5:00000000 r4:00000003 r3:02ff4cf1
[  974.953383] [<809102d0>] (nmi_cpu_backtrace) from [<80910498>] (nmi_trigger_cpumask_backtrace+0xfc/0x138)
[  974.953698]  r5:80e07d3c r4:8010f3dc
[  974.953831] [<8091039c>] (nmi_trigger_cpumask_backtrace) from [<801105fc>] (arch_trigger_cpumask_backtrace+0x20/0x24)
[  974.954183]  r7:80e05044 r6:80000193 r5:80a0201c r4:00000003
[  974.954375] [<801105dc>] (arch_trigger_cpumask_backtrace) from [<80196744>] (rcu_dump_cpu_stacks+0xb4/0xe4)
[  974.954701] [<80196690>] (rcu_dump_cpu_stacks) from [<80195e18>] (rcu_sched_clock_irq+0x868/0xa80)
[  974.955005]  r10:3d818000 r9:80ed5b18 r8:80e05034 r7:80da6f40 r6:be5bef40 r5:80ea8904
[  974.955263]  r4:80e10d80 r3:000107a7
[  974.955395] [<801955b0>] (rcu_sched_clock_irq) from [<8019f674>] (update_process_times+0x3c/0xa8)
[  974.955694]  r10:801b2e50 r9:be5b85f0 r8:be5b8540 r7:000000e2 r6:ff8c7073 r5:b5f49dd0
[  974.955948]  r4:00000000
[  974.956043] [<8019f638>] (update_process_times) from [<801b25a8>] (tick_sched_handle+0x64/0x70)
[  974.956331]  r4:be5b8870 r3:80000013
[  974.956460] [<801b2544>] (tick_sched_handle) from [<801b2eac>] (tick_sched_timer+0x5c/0xb8)
[  974.956742] [<801b2e50>] (tick_sched_timer) from [<801a0638>] (__hrtimer_run_queues+0x164/0x324)
[  974.957037]  r7:b5f48000 r6:be5b8540 r5:be5b8580 r4:be5b8870
[  974.957233] [<801a04d4>] (__hrtimer_run_queues) from [<801a104c>] (hrtimer_interrupt+0x130/0x2a4)
[  974.957530]  r10:be5b85c8 r9:be5b85f0 r8:be5b8540 r7:ffffffff r6:7fffffff r5:00000003
[  974.957776]  r4:20000193
[  974.957873] [<801a0f1c>] (hrtimer_interrupt) from [<80771744>] (arch_timer_handler_phys+0x40/0x48)
[  974.958176]  r10:80e04f88 r9:b5f48000 r8:80e66b54 r7:b946dd00 r6:000000a2 r5:b9402f40
[  974.958430]  r4:80da62e4
[  974.958528] [<80771704>] (arch_timer_handler_phys) from [<80188500>] (handle_percpu_devid_irq+0x88/0x23c)
[  974.958850] [<80188478>] (handle_percpu_devid_irq) from [<80181a00>] (generic_handle_irq+0x34/0x44)
[  974.959153]  r9:b5f48000 r8:b9460000 r7:00000001 r6:00000000 r5:00000000 r4:80da62e4
[  974.959407] [<801819cc>] (generic_handle_irq) from [<8018214c>] (__handle_domain_irq+0x6c/0xc4)
[  974.959700] [<801820e0>] (__handle_domain_irq) from [<80102228>] (bcm2836_arm_irqchip_handle_irq+0x60/0xa4)
[  974.960026]  r8:b5f49e40 r7:b5f49e04 r6:ffffffff r5:80000013 r4:00000003 r3:b5f49dd0
[  974.960281] [<801021c8>] (bcm2836_arm_irqchip_handle_irq) from [<80101a3c>] (__irq_svc+0x5c/0x7c)
[  974.960573] Exception stack(0xb5f49dd0 to 0xb5f49e18)
[  974.960745] 9dc0:                                     b9651e88 00000000 0000b5f4 0000de24
[  974.960986] 9de0: 00000001 80e04f88 b9651e84 b9651e84 b5f49e40 ffffffff 80e04f88 b5f49e2c
[  974.961252] 9e00: b5f49e30 b5f49e20 801b4e58 8092a02c 80000013 ffffffff
[  974.961466]  r4:8092a02c r3:02ff4cf1
[  974.961598] [<80929fec>] (_raw_spin_lock) from [<801b4e58>] (futex_wake+0xc4/0x1c0)
[  974.961859] [<801b4d94>] (futex_wake) from [<801b79dc>] (do_futex+0x160/0xcfc)
[  974.962107]  r10:80e04f88 r9:02648194 r8:00000081 r7:000000f0 r6:7fffffff r5:80e04f88
[  974.962354]  r4:00000001
[  974.962448] [<801b787c>] (do_futex) from [<801b8780>] (sys_futex_time32+0x9c/0x19c)
[  974.962707]  r10:7fffffff r9:02648194 r8:00000081 r7:000000f0 r6:00000000 r5:80e04f88
[  974.962950]  r4:00000001
[  974.963043] [<801b86e4>] (sys_futex_time32) from [<801011a4>] (__sys_trace_return+0x0/0x1c)
[  974.963318] Exception stack(0xb5f49fa8 to 0xb5f49ff0)
[  974.963491] 9fa0:                   00000001 00000000 02648194 00000081 7fffffff 00000000
[  974.963732] 9fc0: 00000001 00000000 00000001 000000f0 000008f0 00000001 00000000 0264816c
[  974.963981] 9fe0: 02648188 58ff9c38 02648168 76d00a14
[  974.964156]  r10:000000f0 r9:b5f48000 r8:801011c4 r7:000000f0 r6:00000001 r5:00000000
[  974.964399]  r4:00000001
[  980.951014] INFO: task apache2:682 blocked for more than 122 seconds.
[  980.951269]       Not tainted 5.4.68-v7+ #1343
[  980.951424] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  980.951689] apache2         D    0   682      1 0x00000004
[  980.951873] Backtrace:
[  980.951985] [<80923ad8>] (__schedule) from [<80924334>] (schedule+0x5c/0xcc)
[  980.952233]  r10:000000f8 r9:b784ce40 r8:b5f5fe50 r7:80e10d80 r6:80e04f88 r5:b5f5e000
[  980.952493]  r4:b6f3eac0
[  980.952600] [<809242d8>] (schedule) from [<80194ef8>] (synchronize_rcu_expedited+0x31c/0x364)
[  980.952887]  r5:000000ac r4:80e10e6c
[  980.953024] [<80194bdc>] (synchronize_rcu_expedited) from [<8032b7b8>] (namespace_unlock+0xc8/0x13c)
[  980.953336]  r8:00000000 r7:00000001 r6:00000000 r5:80e04f88 r4:b5f5feac
[  980.953562] [<8032b6f0>] (namespace_unlock) from [<8032d1c8>] (drop_collected_mounts+0x78/0x80)
[  980.953856]  r7:00000001 r6:00000000 r5:b5c66e50 r4:80e04c40
[  980.954053] [<8032d150>] (drop_collected_mounts) from [<8032f980>] (put_mnt_ns+0x4c/0x58)
[  980.954330]  r5:00000558 r4:b966c700
[  980.954466] [<8032f934>] (put_mnt_ns) from [<801448e4>] (free_nsproxy+0x28/0xcc)
[  980.954719]  r4:b5f39b40 r3:00000322
[  980.954854] [<801448bc>] (free_nsproxy) from [<80144abc>] (switch_task_namespaces+0x84/0x88)
[  980.955141]  r5:00000558 r4:b6f3eac0
[  980.955275] [<80144a38>] (switch_task_namespaces) from [<80144adc>] (exit_task_namespaces+0x1c/0x20)
[  980.955586]  r6:00000000 r5:b784ce00 r4:b6f3eac0 r3:00000174
[  980.955787] [<80144ac0>] (exit_task_namespaces) from [<80124840>] (do_exit+0x3d8/0xb18)
[  980.956062] [<80124468>] (do_exit) from [<80125018>] (do_group_exit+0x4c/0xc4)
[  980.956309]  r7:000000f8
[  980.956410] [<80124fcc>] (do_group_exit) from [<801250b0>] (__wake_up_parent+0x0/0x30)
[  980.956682]  r6:00000000 r5:76e67418 r4:00000001 r3:66dc5800
[  980.956878] [<80125090>] (sys_exit_group) from [<80101000>] (ret_fast_syscall+0x0/0x28)
[  980.957147] Exception stack(0xb5f5ffa8 to 0xb5f5fff0)
[  980.957322] ffa0:                   00000001 76e67418 00000000 00000000 00000000 66dc5800
[  980.957574] ffc0: 00000001 76e67418 00000000 000000f8 76e676c8 00000000 0000000c 76e66000
[  980.957832] ffe0: 00000000 7eeb1b38 76d4b384 76dbd5d4
[  980.958009] INFO: task dietpi-set_cpu:1592 blocked for more than 491 seconds.
[  980.958249]       Not tainted 5.4.68-v7+ #1343
[  980.958401] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  980.958664] dietpi-set_cpu  D    0  1592      1 0x00000001
[  980.958851] Backtrace:
[  980.958953] [<80923ad8>] (__schedule) from [<80924334>] (schedule+0x5c/0xcc)
[  980.959200]  r10:a5743d4c r9:a5742000 r8:7fffffff r7:00000002 r6:80e04f88 r5:a5742000
[  980.959458]  r4:a2f10000
[  980.959561] [<809242d8>] (schedule) from [<80928778>] (schedule_timeout+0x168/0x39c)
[  980.959825]  r5:7fffffff r4:a5743d54
[  980.959959] [<80928610>] (schedule_timeout) from [<80925a28>] (wait_for_completion+0xd0/0x15c)
[  980.960256]  r10:a5743d4c r9:a5742000 r8:80e04f88 r7:00000002 r6:a5743d58 r5:7fffffff
[  980.960513]  r4:a5743d54
[  980.960617] [<80925958>] (wait_for_completion) from [<8018e040>] (__wait_rcu_gp+0x150/0x17c)
[  980.960949]  r9:a5743d64 r8:80191e34 r7:00000001 r6:00000000 r5:a5743d48 r4:00000000
[  980.961232] [<8018def0>] (__wait_rcu_gp) from [<80194fa8>] (synchronize_rcu+0x68/0x8c)
[  980.961521]  r10:b8ddf780 r9:80f570bc r8:80e106f0 r7:b79fa6c0 r6:80e05088 r5:b8e31800
[  980.961791]  r4:80e04f88
[  980.961899] [<80194f40>] (synchronize_rcu) from [<801765d0>] (sugov_stop+0x50/0x70)
[  980.962162]  r4:00000004
[  980.962265] [<80176580>] (sugov_stop) from [<80743230>] (cpufreq_stop_governor.part.1+0x30/0x34)
[  980.962567]  r7:80e88458 r6:b8e31868 r5:80e04f88 r4:b8e31800
[  980.962773] [<80743200>] (cpufreq_stop_governor.part.1) from [<8074555c>] (cpufreq_set_policy+0x1d8/0x27c)
[  980.963113] [<80745384>] (cpufreq_set_policy) from [<80745844>] (store_scaling_governor+0x100/0x114)
[  980.963429]  r10:b5ee5490 r9:b5ee2b00 r8:b8e31924 r7:0000000c r6:80e88458 r5:b8e31800
[  980.963698]  r4:80e04f88
[  980.963802] [<80745744>] (store_scaling_governor) from [<807425e4>] (store+0x88/0xa4)
[  980.964072]  r7:b8e31800 r6:b5ee2b00 r5:0000000c r4:80e8833c
[  980.964275] [<8074255c>] (store) from [<803a2a94>] (sysfs_kf_write+0x54/0x58)
[  980.964523]  r8:a5743f60 r7:00000000 r6:00000000 r5:b5ee5480 r4:b5ee2b00 r3:0000000c
[  980.964786] [<803a2a40>] (sysfs_kf_write) from [<803a1d44>] (kernfs_fop_write+0xec/0x20c)
[  980.965063]  r4:0000000c r3:803a2a40
[  980.965196] [<803a1c58>] (kernfs_fop_write) from [<80301f10>] (__vfs_write+0x48/0x1e0)
[  980.965472]  r10:00000004 r9:0000000c r8:a5743f60 r7:a5743f60 r6:02ff4cf1 r5:a2f60300
[  980.965731]  r4:80e04f88
[  980.965833] [<80301ec8>] (__vfs_write) from [<80304e9c>] (vfs_write+0xb4/0x1c8)
[  980.966087]  r9:a5742000 r8:019a2a38 r7:a5743f60 r6:019a2a38 r5:a2f60300 r4:0000000c
[  980.966354] [<80304de8>] (vfs_write) from [<8030517c>] (ksys_write+0x70/0xec)
[  980.966600]  r8:019a2a38 r7:0000000c r6:a2f60300 r5:80e04f88 r4:a2f60300
[  980.966835] [<8030510c>] (ksys_write) from [<80305210>] (sys_write+0x18/0x1c)
[  980.967081]  r8:801011c4 r7:00000004 r6:76f2cd08 r5:76fb34d0 r4:00000001
[  980.967310] [<803051f8>] (sys_write) from [<80101000>] (ret_fast_syscall+0x0/0x28)
[  980.967570] Exception stack(0xa5743fa8 to 0xa5743ff0)
[  980.967749] 3fa0:                   00000001 76fb34d0 00000001 019a2a38 0000000c 00000000
[  980.968002] 3fc0: 00000001 76fb34d0 76f2cd08 00000004 76f2d2c4 76f2ce2c 00000000 00000000
[  980.968305] 3fe0: 0000000a 7e9af2f8 76ea9bb0 76ea9bcc
[  980.968489] INFO: task sessionclean:1776 blocked for more than 245 seconds.
[  980.968728]       Not tainted 5.4.68-v7+ #1343
[  980.968881] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  980.969146] sessionclean    D    0  1776      1 0x00000004
[  980.969335] Backtrace:
[  980.969441] [<80923ad8>] (__schedule) from [<80924334>] (schedule+0x5c/0xcc)
[  980.969689]  r10:000000f8 r9:b7841440 r8:89e8fe50 r7:80e10d80 r6:80e04f88 r5:89e8e000
[  980.969951]  r4:a2f83d00
[  980.970057] [<809242d8>] (schedule) from [<80194f34>] (synchronize_rcu_expedited+0x358/0x364)
[  980.970346]  r5:000000a8 r4:80e10e60
[  980.970484] [<80194bdc>] (synchronize_rcu_expedited) from [<8032b7b8>] (namespace_unlock+0xc8/0x13c)
[  980.970796]  r8:00000000 r7:00000001 r6:00000000 r5:80e04f88 r4:89e8feac
[  980.971061] [<8032b6f0>] (namespace_unlock) from [<8032d1c8>] (drop_collected_mounts+0x78/0x80)
[  980.971365]  r7:00000001 r6:00000000 r5:a562d790 r4:80e04c40
[  980.971565] [<8032d150>] (drop_collected_mounts) from [<8032f980>] (put_mnt_ns+0x4c/0x58)
[  980.971843]  r5:00000558 r4:a83b3680
[  980.971980] [<8032f934>] (put_mnt_ns) from [<801448e4>] (free_nsproxy+0x28/0xcc)
[  980.972234]  r4:b966f940 r3:000000c8
[  980.972393] [<801448bc>] (free_nsproxy) from [<80144abc>] (switch_task_namespaces+0x84/0x88)
[  980.972687]  r5:00000558 r4:a2f83d00
[  980.972823] [<80144a38>] (switch_task_namespaces) from [<80144adc>] (exit_task_namespaces+0x1c/0x20)
[  980.973135]  r6:00000000 r5:b7841400 r4:a2f83d00 r3:00000284
[  980.973336] [<80144ac0>] (exit_task_namespaces) from [<80124840>] (do_exit+0x3d8/0xb18)
[  980.973614] [<80124468>] (do_exit) from [<80125018>] (do_group_exit+0x4c/0xc4)
[  980.973860]  r7:000000f8
[  980.973962] [<80124fcc>] (do_group_exit) from [<801250b0>] (__wake_up_parent+0x0/0x30)
[  980.974235]  r6:000117b4 r5:00000000 r4:00000004 r3:00036010
[  980.974431] [<80125090>] (sys_exit_group) from [<80101000>] (ret_fast_syscall+0x0/0x28)
[  980.974700] Exception stack(0x89e8ffa8 to 0x89e8fff0)
[  980.974880] ffa0:                   00000004 00000000 00000000 00000000 00000000 00036010
[  980.975132] ffc0: 00000004 00000000 000117b4 000000f8 00000000 00000000 76fa6000 00000000
[  980.975386] ffe0: 00035ee0 7ed0f998 0001fb10 76eb15d4
[  980.975570] INFO: task apachectl:1871 blocked for more than 122 seconds.
[  980.975802]       Not tainted 5.4.68-v7+ #1343
[  980.975956] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  980.976220] apachectl       D    0  1871      1 0x00000004
[  980.976405] Backtrace:
[  980.976509] [<80923ad8>] (__schedule) from [<80924334>] (schedule+0x5c/0xcc)
[  980.976757]  r10:80e10f50 r9:00000000 r8:00000002 r7:89b60000 r6:00000000 r5:89b60000
[  980.977004]  r4:89b18f40
[  980.977108] [<809242d8>] (schedule) from [<809247a0>] (schedule_preempt_disabled+0x18/0x1c)
[  980.977390]  r5:89b61dcc r4:89b60000
[  980.977527] [<80924788>] (schedule_preempt_disabled) from [<80926b8c>] (__mutex_lock.constprop.9+0x2c4/0x5b0)
[  980.977865] [<809268c8>] (__mutex_lock.constprop.9) from [<80926f94>] (__mutex_lock_slowpath+0x1c/0x20)
[  980.978185]  r10:000000f8 r9:b7841a40 r8:00000002 r7:80e10d80 r6:80e04f88 r5:000000ac
[  980.978440]  r4:80e10f50
[  980.978542] [<80926f78>] (__mutex_lock_slowpath) from [<80926ff4>] (mutex_lock+0x5c/0x60)
[  980.978827] [<80926f98>] (mutex_lock) from [<80194d54>] (synchronize_rcu_expedited+0x178/0x364)
[  980.979120]  r4:00000000 r3:0000002a
[  980.979252] [<80194bdc>] (synchronize_rcu_expedited) from [<8032b7b8>] (namespace_unlock+0xc8/0x13c)
[  980.979565]  r8:00000000 r7:00000001 r6:00000000 r5:80e04f88 r4:89b61eac
[  980.979793] [<8032b6f0>] (namespace_unlock) from [<8032d1c8>] (drop_collected_mounts+0x78/0x80)
[  980.980089]  r7:00000001 r6:00000000 r5:b5630910 r4:80e04c40
[  980.980286] [<8032d150>] (drop_collected_mounts) from [<8032f980>] (put_mnt_ns+0x4c/0x58)
[  980.980563]  r5:00000558 r4:a83b3b00
[  980.980698] [<8032f934>] (put_mnt_ns) from [<801448e4>] (free_nsproxy+0x28/0xcc)
[  980.980980]  r4:b966f960 r3:0000009a
[  980.981123] [<801448bc>] (free_nsproxy) from [<80144abc>] (switch_task_namespaces+0x84/0x88)
[  980.981414]  r5:00000558 r4:89b18f40
[  980.981550] [<80144a38>] (switch_task_namespaces) from [<80144adc>] (exit_task_namespaces+0x1c/0x20)
[  980.981862]  r6:00000000 r5:b7841a00 r4:89b18f40 r3:0000009a
[  980.982064] [<80144ac0>] (exit_task_namespaces) from [<80124840>] (do_exit+0x3d8/0xb18)
[  980.982347] [<80124468>] (do_exit) from [<80125018>] (do_group_exit+0x4c/0xc4)
[  980.982595]  r7:000000f8
[  980.982696] [<80124fcc>] (do_group_exit) from [<801250b0>] (__wake_up_parent+0x0/0x30)
[  980.982972]  r6:000117b4 r5:00000000 r4:00000004 r3:00036010
[  980.983169] [<80125090>] (sys_exit_group) from [<80101000>] (ret_fast_syscall+0x0/0x28)
[  980.983439] Exception stack(0x89b61fa8 to 0x89b61ff0)
[  980.983618] 1fa0:                   00000004 00000000 00000000 00000000 00000000 00036010
[  980.983866] 1fc0: 00000004 00000000 000117b4 000000f8 00000000 00000000 76fc3000 00000000
[  980.984119] 1fe0: 00035ee0 7ea28958 0001fb10 76ece5d4
[  986.070975] rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 3-... } 28427 jiffies s: 165 root: 0x8/.
[  986.071378] rcu: blocking rcu_node structures:
[  986.071571] Task dump for CPU 3:
[  986.071689] mysqld          R  running task        0   633      1 0x00000082
[  986.071925] Backtrace:
[  986.072032] Backtrace aborted due to bad frame pointer <9e54fcbd>
[ 1037.980917] rcu: INFO: rcu_sched self-detected stall on CPU
[ 1037.981150] rcu:     3-....: (65129 ticks this GP) idle=9a2/1/0x40000002 softirq=9183/9183 fqs=32561
[ 1037.981451]  (t=65130 jiffies g=35521 q=58578)
[ 1037.981605] NMI backtrace for cpu 3
[ 1037.981729] CPU: 3 PID: 633 Comm: mysqld Not tainted 5.4.68-v7+ #1343
[ 1037.981940] Hardware name: BCM2835
[ 1037.982056] Backtrace:
[ 1037.982159] [<8010d4f4>] (dump_backtrace) from [<8010d7ec>] (show_stack+0x20/0x24)
[ 1037.982416]  r6:b5f48000 r5:00000000 r4:80e97abc r3:02ff4cf1
[ 1037.982610] [<8010d7cc>] (show_stack) from [<80908944>] (dump_stack+0xe0/0x124)
[ 1037.982861] [<80908864>] (dump_stack) from [<80910398>] (nmi_cpu_backtrace+0xc8/0xcc)
[ 1037.983125]  r8:00000140 r7:80a0202c r6:00000003 r5:00000000 r4:00000003 r3:02ff4cf1
[ 1037.983379] [<809102d0>] (nmi_cpu_backtrace) from [<80910498>] (nmi_trigger_cpumask_backtrace+0xfc/0x138)
[ 1037.983695]  r5:80e07d3c r4:8010f3dc
[ 1037.983829] [<8091039c>] (nmi_trigger_cpumask_backtrace) from [<801105fc>] (arch_trigger_cpumask_backtrace+0x20/0x24)
[ 1037.984180]  r7:80e05044 r6:80000193 r5:80a0201c r4:00000003
[ 1037.984373] [<801105dc>] (arch_trigger_cpumask_backtrace) from [<80196744>] (rcu_dump_cpu_stacks+0xb4/0xe4)
[ 1037.984697] [<80196690>] (rcu_dump_cpu_stacks) from [<80195e18>] (rcu_sched_clock_irq+0x868/0xa80)
[ 1037.985000]  r10:3d818000 r9:80ed5b18 r8:80e05034 r7:80da6f40 r6:be5bef40 r5:80ea8904
[ 1037.985257]  r4:80e10d80 r3:00012047
[ 1037.985389] [<801955b0>] (rcu_sched_clock_irq) from [<8019f674>] (update_process_times+0x3c/0xa8)
[ 1037.985687]  r10:801b2e50 r9:be5b85f0 r8:be5b8540 r7:000000f1 r6:ac6de93f r5:b5f49dd0
[ 1037.985941]  r4:00000000
[ 1037.986037] [<8019f638>] (update_process_times) from [<801b25a8>] (tick_sched_handle+0x64/0x70)
[ 1037.986324]  r4:be5b8870 r3:80000013
[ 1037.986453] [<801b2544>] (tick_sched_handle) from [<801b2eac>] (tick_sched_timer+0x5c/0xb8)
[ 1037.986734] [<801b2e50>] (tick_sched_timer) from [<801a0638>] (__hrtimer_run_queues+0x164/0x324)
[ 1037.987029]  r7:b5f48000 r6:be5b8540 r5:be5b8580 r4:be5b8870
[ 1037.987224] [<801a04d4>] (__hrtimer_run_queues) from [<801a104c>] (hrtimer_interrupt+0x130/0x2a4)
[ 1037.987521]  r10:be5b85c8 r9:be5b85f0 r8:be5b8540 r7:ffffffff r6:7fffffff r5:00000003
[ 1037.987767]  r4:20000193
[ 1037.987865] [<801a0f1c>] (hrtimer_interrupt) from [<80771744>] (arch_timer_handler_phys+0x40/0x48)
[ 1037.988168]  r10:80e04f88 r9:b5f48000 r8:80e66b54 r7:b946dd00 r6:000000a2 r5:b9402f40
[ 1037.988422]  r4:80da62e4
[ 1037.988520] [<80771704>] (arch_timer_handler_phys) from [<80188500>] (handle_percpu_devid_irq+0x88/0x23c)
[ 1037.988841] [<80188478>] (handle_percpu_devid_irq) from [<80181a00>] (generic_handle_irq+0x34/0x44)
[ 1037.989143]  r9:b5f48000 r8:b9460000 r7:00000001 r6:00000000 r5:00000000 r4:80da62e4
[ 1037.989398] [<801819cc>] (generic_handle_irq) from [<8018214c>] (__handle_domain_irq+0x6c/0xc4)
[ 1037.989691] [<801820e0>] (__handle_domain_irq) from [<80102228>] (bcm2836_arm_irqchip_handle_irq+0x60/0xa4)
[ 1037.990016]  r8:b5f49e40 r7:b5f49e04 r6:ffffffff r5:80000013 r4:00000003 r3:b5f49dd0
[ 1037.990272] [<801021c8>] (bcm2836_arm_irqchip_handle_irq) from [<80101a3c>] (__irq_svc+0x5c/0x7c)
[ 1037.990565] Exception stack(0xb5f49dd0 to 0xb5f49e18)
[ 1037.990738] 9dc0:                                     b9651e88 00000000 0000b5f4 0000de24
[ 1037.990979] 9de0: 00000001 80e04f88 b9651e84 b9651e84 b5f49e40 ffffffff 80e04f88 b5f49e2c
[ 1037.991248] 9e00: b5f49e30 b5f49e20 801b4e58 8092a02c 80000013 ffffffff
[ 1037.991463]  r4:8092a02c r3:02ff4cf1
[ 1037.991595] [<80929fec>] (_raw_spin_lock) from [<801b4e58>] (futex_wake+0xc4/0x1c0)
[ 1037.991856] [<801b4d94>] (futex_wake) from [<801b79dc>] (do_futex+0x160/0xcfc)
[ 1037.992102]  r10:80e04f88 r9:02648194 r8:00000081 r7:000000f0 r6:7fffffff r5:80e04f88
[ 1037.992350]  r4:00000001
[ 1037.992444] [<801b787c>] (do_futex) from [<801b8780>] (sys_futex_time32+0x9c/0x19c)
[ 1037.992703]  r10:7fffffff r9:02648194 r8:00000081 r7:000000f0 r6:00000000 r5:80e04f88
[ 1037.992946]  r4:00000001
[ 1037.993039] [<801b86e4>] (sys_futex_time32) from [<801011a4>] (__sys_trace_return+0x0/0x1c)
[ 1037.993314] Exception stack(0xb5f49fa8 to 0xb5f49ff0)
[ 1037.993487] 9fa0:                   00000001 00000000 02648194 00000081 7fffffff 00000000
[ 1037.993729] 9fc0: 00000001 00000000 00000001 000000f0 000008f0 00000001 00000000 0264816c
[ 1037.993979] 9fe0: 02648188 58ff9c38 02648168 76d00a14
[ 1037.994153]  r10:000000f0 r9:b5f48000 r8:801011c4 r7:000000f0 r6:00000001 r5:00000000
[ 1037.994397]  r4:00000001

Actually I just wanted to switch governor back to performance, but that tasks hangs now as well. Service stops hang, every command that hangs cannot even be killed, the whole shell hangs, so I need to spawn a new shell (new SSH session/screen) to be able to investigate. Finally rebooted, which hangs as well 😄.

Applied performance governor via reboot (power cycle), no issues since then. Will wait until tomorrow, then try to again remove arm_freq_min. I was also playing with core_freq_min and gpu_freq_min, could those cause similar issues? Although kernel errors clearly refer to CPU scheduling.

Sep 30 '20 23:09 MichaIng

I don't see any connection with that backtrace and this issue. You'll need to find a way of provoking this reliably. Then confirm if issue still occurs without arm_freq_min. Also confirm if this issue is new to a recent rpi-update kernel/firmware (e.g, report first version it occurs with and last version is didn't occur with)

Oct 01 '20 10:10 popcornmix

Very latest kernel/firmware, to test the initial_turbo+performance governor solution. Just reverted min frequencies to defaults, booted with performance governor, which gives 900 MHz (RPi2) now reliable with initial_turbo, then switched to schedutil governor, which works fine until now:

2020-10-01 12:21:26 root@micha:~# cat /sys/devices/system/cpu/cpufreq/policy0/{scaling_governor,/stats/time_in_state}
schedutil
600000 127355
700000 892
800000 831
900000 31905
2020-10-01 12:21:36 root@micha:~# uname -a
Linux micha.gnoedi.org 5.4.68-v7+ #1343 SMP Mon Sep 28 12:38:29 BST 2020 armv7l GNU/Linux

I'll leave it for a while, then re-enable arm_freq_min=300 to see if that triggers the issues.

Definitely fine was #1336, which I used with arm_freq_min=150 and schedutil governor for a long time. To run tests with the other issue, I rpi-update'd to #1341. Since it was to test performance governor and only very short sessions with over governors, I didn't recognise any issue there, but indeed I had a complete system crash a while later when bringing things back to production, leading to SD card corruptions which is why I flashed a new system from scratch with stable firmware packages. Due to much back and forth and testing I didn't think much about this, nor was I able to see any crash-related system log entries (analysing SD card on external system), but now I think it might be related to what I see just now after rpi-updating to #1343.

So if I can replicate, #1336 - #1341 are the version to go through, but since the issue often took a while to become apparent, I'll let every version run for a longer time to assure it's working, so may take a while until I can narrow it down.

Oct 01 '20 10:10 MichaIng

Okay I was able to successfully replicate a system crash multiple times by running a CPU intense task, mysqlcheck worked well in this particular case to trigger the crash in two of three times. I had dmesg -w running but sadly this time the crash broke SSH immediately, so I was not able to see any kernel errors. The symptoms are pretty much the same as before. Either tasks start to hang, producing this kind of random kernel errors, like rcu: INFO: rcu_sched self-detected stall on CPU, tick_sched_handle, tick_sched_timer which is found in first post log and new logs as well, and a few other matches around CPU scheduling, or the system crashes completely with no chance of leaving any logs left, even persistent journal is empty. This was #1343 + schedutil + arm_freq_min=300 after arm_freq_min=600 worked very well the whole night and half day and arm_freq_min=300 + performance worked well the other half of the day.

#1343 + schedutil + arm_freq_min=300 => crashes easily when inducing some load or after a while
#1343 + performance + arm_freq_min=300 => unbreakable
#1343 + schedutil + arm_freq_min=600 => unbreakable
#1337 + schedutil + arm_freq_min=300 => Wasn't able to break it first, but then it suddenly crashed on it's own when I was doing another thing.
#1336 + schedutil + arm_freq_min=300 => WIP... Stress test passed, letting it run through the night and tomorrow, followed by a few more stress and mysqlcheck tests just to be sure.
- Breaking commit would be: https://github.com/raspberrypi/firmware/commit/1a2131e68b556d781cd11f544ec0633beff82cba
- Probably https://github.com/raspberrypi/linux/pull/3815 which changed the scaling steps, but otherwise only affects RPi 1 🤔? Everything else of above commit seems completely unrelated.

Oct 01 '20 18:10 MichaIng

@popcornmix As of above testing, it is something between #1336 and #1337 that reintroduced the instability which was solved with #1336. #1336 runs rock solid through a bunch of stress tests, including and excluding RAM, SD card and USB HDD disk writes, while #1337 breaks with pretty much identical symptoms as reported originally with this thread. Probably something related to the fix has been reverted accidentally? Based on my limited insights, https://github.com/raspberrypi/linux/pull/3815 and the related commits (not directly part of the PR) around CPUfreq seem to be the only changes that could have affected it.

Oct 02 '20 12:10 MichaIng

@MichaIng is all your testing with a Pi2? https://github.com/raspberrypi/linux/pull/3815 does enable additional arm frequency points on all Pi models. I suspect prior to 3815 300MHz wasn't being used when idle (check with vcgencmd measure_clock arm)

Is the only non-default setting in config.txt arm_freq_min=300 ?

Oct 02 '20 12:10 popcornmix