earlyoom Add PSI support (threshold for swapin/swapout)

earlyoom is only aware of the amount of free swap, but I've seen many times that with large swap, system stop responding due to heavy swapin/swapout, while earlyoom did nothing since there's still plenty of free swap. Setting a low free swap minimum is no better than completely disabling swap and letting kernel oom to handle low memory situation.

Ref #34

Nov 10 '18 04:11 yangfl

while earlyoom did nothing

What was he supposed to do? Please describe in detail the algorithm you want to implement.

Nov 10 '18 15:11 hakavlad

The rate of filling of a swap partition should be a trigger?

Nov 10 '18 16:11 hakavlad

Seems like the psi (pressure stall information) was merged recently: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=eb414681d5a07d28d2ff90dc05f69ec6b232ebd2 It should bei in linux 4.20. This would probably the best source of information when to start killing things, but it will not be available in older kernels.

Nov 10 '18 16:11 rfjakob

Do you plan to support PSI in earlyoom 1.3+?

Nov 18 '18 13:11 hakavlad

Once my distribution has it, yes! (Fedora)

Nov 18 '18 20:11 rfjakob

Congratulations! Linux 4.20 released with PSI and is already in Kernel-vanilla-stable https://fedoraproject.org/wiki/Kernel_Vanilla_Repositories http://www.leemhuis.info/files/kernel-vanilla/repostatus.txt

Dec 24 '18 18:12 hakavlad

:)

Reading through https://lwn.net/Articles/759781/ , one problem with psi may be that you have to wait for the thrashing to happen before you can react. Maybe earlyoom will look at both psi and free mem (but much lower threshhold, maybe 1%)

Dec 25 '18 11:12 rfjakob

Next q.: How to find a victim when using psi?

Dec 25 '18 15:12 hakavlad

I guess same as before, highest oom_score

Dec 25 '18 15:12 rfjakob

you have to wait

10 sec maximum

Dec 26 '18 17:12 hakavlad

it works

Dec 26 '18 17:12 hakavlad

PSI support implemented in nohang. It works pretty fast. The wait is not too long. https://i.imgur.com/vEJbGMR.png

I just added 2 thresholds: sigterm_psi = 60 sigkill_psi = 80

It uses some avg10 to quickly respond.

Dec 30 '18 19:12 hakavlad

The next problem: PSI mem avg10 slowly falls after corrective action (may need more then 20 sec sleep to prevent multiple killings).

Jan 01 '19 15:01 hakavlad

Good catch. I guess this means we should use "total" instead of the averaged number.

Jan 01 '19 15:01 rfjakob

Seems like total always grows. How we can use it?

Jan 01 '19 16:01 hakavlad

https://youtu.be/YYJ9Af8Syyg demo with PSI

Jan 01 '19 16:01 hakavlad

Eureka! avg = (total2 - total1) / time Now I need test it.

Jan 01 '19 16:01 hakavlad

Yes exactly, you check if it increases

Jan 01 '19 17:01 rfjakob

avg = (total1 - total0) / (time1 - time0) / 10000 It works.

Good news: it can works very quick and reacts if swapping starts. http://okturing.com/src/5628/body (in this output avg10 is avg found from formula above)

Bad news: it may give false positives. I just ran FF, and it was killed. http://okturing.com/src/5630/body

Maybe we should use avg1 or avg2 as middle way.

Jan 01 '19 17:01 hakavlad

I will try this settings: $SIGTERM_AVG = 60 $AVG_TIME = 1 I want to give the user to choose any minimum amount of time for which AVG is calculated

Jan 01 '19 18:01 hakavlad

http://okturing.com/src/5636/body avg1 monitor (or monitor for any time, change avg_min_time)

example of output with not very quick memhogging http://okturing.com/src/5637/body

avg time: 1.0
PSI mem avg: 0.0
some avg10=2.67 avg60=5.62 avg300=9.39 total=229884775 

avg time: 1.0
PSI mem avg: 0.0
some avg10=2.67 avg60=5.62 avg300=9.39 total=229884775 

avg time: 1.0
PSI mem avg: 0.0
some avg10=2.19 avg60=5.43 avg300=9.32 total=229884775 


# START USING SWAP


avg time: 1.0
PSI mem avg: 16.92
some avg10=2.19 avg60=5.43 avg300=9.32 total=230056914 

avg time: 1.1
PSI mem avg: 84.1
some avg10=8.68 avg60=6.50 avg300=9.52 total=230972454 

avg time: 1.0
PSI mem avg: 87.9
some avg10=8.68 avg60=6.50 avg300=9.52 total=231888109 

avg time: 1.1
PSI mem avg: 99.97
some avg10=23.95 avg60=9.33 avg300=10.09 total=232967719 

avg time: 1.1
PSI mem avg: 99.99
some avg10=23.95 avg60=9.33 avg300=10.09 total=234037558 

avg time: 1.0
PSI mem avg: 94.99
some avg10=37.55 avg60=12.26 avg300=10.70 total=235007672 

avg time: 1.0
PSI mem avg: 98.21
some avg10=37.55 avg60=12.26 avg300=10.70 total=236033919 

avg time: 1.1
PSI mem avg: 99.01
some avg10=48.13 avg60=15.00 avg300=11.28 total=237084562 

avg time: 1.1
PSI mem avg: 71.53
some avg10=48.13 avg60=15.00 avg300=11.28 total=237881926 

avg time: 1.1
PSI mem avg: 74.66
some avg10=53.54 avg60=17.06 avg300=11.74 total=238669873 

avg time: 1.0
PSI mem avg: 91.74
some avg10=53.54 avg60=17.06 avg300=11.74 total=239630293 

avg time: 1.0
PSI mem avg: 95.02
some avg10=59.60 avg60=19.35 avg300=12.25 total=240594804 

avg time: 1.0
PSI mem avg: 99.95
some avg10=59.60 avg60=19.35 avg300=12.25 total=241643845 

avg time: 1.0
PSI mem avg: 98.67
some avg10=66.38 avg60=21.89 avg300=12.83 total=242666213 

avg time: 1.0
PSI mem avg: 49.99
some avg10=66.38 avg60=21.89 avg300=12.83 total=243167690 

avg time: 1.0
PSI mem avg: 84.66
some avg10=67.40 avg60=23.53 avg300=13.24 total=244038897 

avg time: 1.0
PSI mem avg: 70.76
some avg10=67.40 avg60=23.53 avg300=13.24 total=244771122 

avg time: 1.0
PSI mem avg: 92.33
some avg10=69.50 avg60=25.35 avg300=13.68 total=245703671 

avg time: 1.1
PSI mem avg: 99.31
some avg10=74.12 avg60=27.62 avg300=14.24 total=246781382 

avg time: 1.1
PSI mem avg: 99.97
some avg10=74.12 avg60=27.62 avg300=14.24 total=247864738 

avg time: 1.1
PSI mem avg: 99.97
some avg10=78.62 avg60=29.96 avg300=14.82 total=248954990 

avg time: 1.0
PSI mem avg: 99.44
some avg10=78.62 avg60=29.96 avg300=14.82 total=249980594 

avg time: 1.1
PSI mem avg: 99.06
some avg10=82.31 avg60=32.22 avg300=15.40 total=251050410 

avg time: 1.1
PSI mem avg: 95.75
some avg10=82.31 avg60=32.22 avg300=15.40 total=252087053 

avg time: 1.0
PSI mem avg: 97.02
some avg10=84.79 avg60=34.30 avg300=15.95 total=253064046 

avg time: 1.0
PSI mem avg: 99.41
some avg10=84.79 avg60=34.30 avg300=15.95 total=254100130 

avg time: 1.1
PSI mem avg: 98.17
some avg10=87.18 avg60=36.39 avg300=16.51 total=255139188 

avg time: 1.1
PSI mem avg: 84.71
some avg10=87.18 avg60=36.39 avg300=16.51 total=256069110 

avg time: 1.1
PSI mem avg: 83.95
some avg10=86.24 avg60=37.88 avg300=16.96 total=256952564 

avg time: 1.1
PSI mem avg: 98.74
some avg10=86.24 avg60=37.88 avg300=16.96 total=258030294 

avg time: 1.1
PSI mem avg: 99.98
some avg10=88.55 avg60=39.88 avg300=17.52 total=259080932 

avg time: 1.1
PSI mem avg: 80.52
some avg10=88.55 avg60=39.88 avg300=17.52 total=259943098 

avg time: 1.1
PSI mem avg: 60.02
some avg10=88.45 avg60=41.45 avg300=18.00 total=260590551 

avg time: 1.0
PSI mem avg: 84.79
some avg10=88.45 avg60=41.45 avg300=18.00 total=261455903 

avg time: 1.0
PSI mem avg: 77.87
some avg10=85.29 avg60=42.42 avg300=18.36 total=262255772 

avg time: 1.0
PSI mem avg: 86.76
some avg10=84.69 avg60=43.72 avg300=18.80 total=263132918 

avg time: 1.0
PSI mem avg: 89.88
some avg10=84.69 avg60=43.72 avg300=18.80 total=264042776 

avg time: 1.0
PSI mem avg: 69.67
some avg10=83.66 avg60=44.87 avg300=19.21 total=264758629 

avg time: 1.0
PSI mem avg: 70.96
some avg10=83.66 avg60=44.87 avg300=19.21 total=265486766 

avg time: 1.0
PSI mem avg: 88.18
some avg10=82.63 avg60=45.95 avg300=19.61 total=266382658 

avg time: 1.0
PSI mem avg: 82.89
some avg10=82.63 avg60=45.95 avg300=19.61 total=267224852 

avg time: 1.0
PSI mem avg: 83.4
some avg10=82.52 avg60=47.13 avg300=20.04 total=268071133 

avg time: 1.0
PSI mem avg: 84.65
some avg10=82.52 avg60=47.13 avg300=20.04 total=268926961 

avg time: 1.0
PSI mem avg: 86.77
some avg10=82.97 avg60=48.37 avg300=20.48 total=269808080 

avg time: 1.0
PSI mem avg: 87.66
some avg10=82.97 avg60=48.37 avg300=20.48 total=270705197 

avg time: 1.0
PSI mem avg: 83.86
some avg10=83.52 avg60=49.60 avg300=20.93 total=271558446 

avg time: 1.0
PSI mem avg: 91.06
some avg10=83.52 avg60=49.60 avg300=20.93 total=272485445 

avg time: 1.0
PSI mem avg: 89.83
some avg10=84.51 avg60=50.89 avg300=21.40 total=273399845 

avg time: 1.0
PSI mem avg: 99.46
some avg10=84.51 avg60=50.89 avg300=21.40 total=274409564 

avg time: 1.0
PSI mem avg: 95.79
some avg10=86.77 avg60=52.40 avg300=21.91 total=275390717 

avg time: 1.0
PSI mem avg: 84.82
some avg10=86.77 avg60=52.40 avg300=21.91 total=276246728 

avg time: 1.0
PSI mem avg: 99.99
some avg10=87.54 avg60=53.66 avg300=22.39 total=277262795 

avg time: 1.0
PSI mem avg: 98.98
some avg10=87.54 avg60=53.66 avg300=22.39 total=278271242 

avg time: 1.0
PSI mem avg: 92.34
some avg10=89.25 avg60=55.08 avg300=22.90 total=279209716 

avg time: 1.0
PSI mem avg: 90.39
some avg10=89.25 avg60=55.08 avg300=22.90 total=280125479 

avg time: 1.0
PSI mem avg: 96.1
some avg10=89.93 avg60=56.32 avg300=23.37 total=281097948 

avg time: 1.0
PSI mem avg: 90.81
some avg10=89.93 avg60=56.32 avg300=23.37 total=282033579 

avg time: 1.0
PSI mem avg: 94.96
some avg10=90.30 avg60=57.49 avg300=23.84 total=283005589 

avg time: 1.0
PSI mem avg: 99.84
some avg10=90.30 avg60=57.49 avg300=23.84 total=284014404 

avg time: 1.0
PSI mem avg: 95.37
some avg10=91.70 avg60=58.81 avg300=24.35 total=284980212 

avg time: 1.0
PSI mem avg: 95.8
some avg10=91.70 avg60=58.81 avg300=24.35 total=285948039 

avg time: 1.0
PSI mem avg: 99.98
some avg10=92.29 avg60=60.00 avg300=24.83 total=286968119 

avg time: 1.0
PSI mem avg: 99.94
some avg10=92.29 avg60=60.00 avg300=24.83 total=287974126 

avg time: 1.0
PSI mem avg: 97.2
some avg10=93.33 avg60=61.24 avg300=25.34 total=288957779 

avg time: 1.0
PSI mem avg: 93.15
some avg10=93.33 avg60=61.24 avg300=25.34 total=289909118 

avg time: 1.0
PSI mem avg: 85.54
some avg10=93.09 avg60=62.25 avg300=25.79 total=290779470 

avg time: 1.0
PSI mem avg: 92.57
some avg10=93.09 avg60=62.25 avg300=25.79 total=291724998 

avg time: 1.0
PSI mem avg: 99.11
some avg10=92.89 avg60=63.22 avg300=26.24 total=292728724 

avg time: 1.0
PSI mem avg: 99.97
some avg10=92.89 avg60=63.22 avg300=26.24 total=293748081 

avg time: 1.0
PSI mem avg: 99.99
some avg10=93.99 avg60=64.39 avg300=26.74 total=294776929 

avg time: 1.0
PSI mem avg: 99.98
some avg10=93.99 avg60=64.39 avg300=26.74 total=295785163 

avg time: 1.0
PSI mem avg: 99.95
some avg10=94.90 avg60=65.52 avg300=27.24 total=296787807 

avg time: 1.0
PSI mem avg: 99.98
some avg10=94.90 avg60=65.52 avg300=27.24 total=297821363 

avg time: 1.0
PSI mem avg: 98.74
some avg10=95.64 avg60=66.62 avg300=27.73 total=298820563 

# OOM!

Jan 02 '19 06:01 hakavlad

nohang output with avg3 http://okturing.com/src/5638/body

Jan 02 '19 08:01 hakavlad

sigterm_psi = 60 sigkill_psi = 95 avg_min_time = 4 psi_min_sleep_time_after_action = 16

output:

MemAvail: 1589 M, 79.8 % | SwapFree: 732 M, 100.0 %
MemAvail: 1391 M, 69.9 % | SwapFree: 732 M, 100.0 %
some avg10=0.00 avg60=0.00 avg300=0.00 total=0
PSI mem some avg: 0.0, PSI avg time: 4.1
PSI is OK  or  psi_min_sleep_time_after_action did not pass
MemAvail: 1050 M, 52.8 % | SwapFree: 732 M, 100.0 %
MemAvail:  755 M, 37.9 % | SwapFree: 732 M, 100.0 %
MemAvail:  496 M, 24.9 % | SwapFree: 732 M, 100.0 %
MemAvail:  260 M, 13.1 % | SwapFree: 732 M, 100.0 %
MemAvail:   54 M,  2.7 % | SwapFree: 727 M,  99.3 %
MemAvail:    0 M,  0.0 % | SwapFree: 718 M,  98.1 %
MemAvail:    0 M,  0.0 % | SwapFree: 718 M,  98.1 %
MemAvail:    0 M,  0.0 % | SwapFree: 715 M,  97.6 %
MemAvail:    0 M,  0.0 % | SwapFree: 704 M,  96.1 %
MemAvail:    0 M,  0.0 % | SwapFree: 701 M,  95.8 %
MemAvail:    0 M,  0.0 % | SwapFree: 694 M,  94.8 %
MemAvail:    0 M,  0.0 % | SwapFree: 688 M,  93.9 %
MemAvail:    0 M,  0.0 % | SwapFree: 685 M,  93.6 %
MemAvail:    0 M,  0.0 % | SwapFree: 680 M,  92.8 %
MemAvail:    0 M,  0.0 % | SwapFree: 676 M,  92.3 %
MemAvail:    0 M,  0.0 % | SwapFree: 667 M,  91.1 %
some avg10=9.41 avg60=1.70 avg300=0.35 total=1752059
PSI mem some avg: 41.83, PSI avg time: 4.2
PSI is OK  or  psi_min_sleep_time_after_action did not pass
MemAvail:    0 M,  0.0 % | SwapFree: 661 M,  90.2 %
MemAvail:    0 M,  0.0 % | SwapFree: 656 M,  89.6 %
MemAvail:    0 M,  0.0 % | SwapFree: 653 M,  89.2 %
MemAvail:    0 M,  0.0 % | SwapFree: 648 M,  88.5 %
MemAvail:    0 M,  0.0 % | SwapFree: 641 M,  87.6 %
MemAvail:    0 M,  0.0 % | SwapFree: 638 M,  87.1 %
MemAvail:    0 M,  0.0 % | SwapFree: 631 M,  86.1 %
MemAvail:    0 M,  0.0 % | SwapFree: 626 M,  85.5 %
MemAvail:    0 M,  0.0 % | SwapFree: 621 M,  84.8 %
MemAvail:    0 M,  0.0 % | SwapFree: 621 M,  84.8 %
MemAvail:    0 M,  0.0 % | SwapFree: 612 M,  83.6 %
MemAvail:    0 M,  0.0 % | SwapFree: 606 M,  82.7 %
MemAvail:    0 M,  0.0 % | SwapFree: 599 M,  81.8 %
MemAvail:    0 M,  0.0 % | SwapFree: 596 M,  81.4 %
MemAvail:    0 M,  0.0 % | SwapFree: 590 M,  80.5 %
MemAvail:    0 M,  0.0 % | SwapFree: 586 M,  80.1 %
MemAvail:    0 M,  0.0 % | SwapFree: 582 M,  79.5 %
MemAvail:    0 M,  0.0 % | SwapFree: 578 M,  79.0 %
MemAvail:    0 M,  0.0 % | SwapFree: 576 M,  78.7 %
MemAvail:    0 M,  0.0 % | SwapFree: 571 M,  77.9 %
some avg10=32.08 avg60=6.61 avg300=1.41 total=5262760
PSI mem some avg: 84.16, PSI avg time: 4.2
avg (84.16) > sigterm_psi (60)
Found the victim with highest badness:
  Name:    python3
  PID:     1077
  UID:     1000
  Badness: 663, oom_score: 663, oom_score_adj: 0
  VmSize:  1830 MiB
  VmRSS:   1768 MiB (Anon: 1762 MiB, File: 6 MiB, Shmem: 0 MiB)
  VmSwap:    35 MiB
  CmdLine: python3 /usr/bin/oom-trigger 
Implement corrective action:
  Sending SIGTERM to the victim; OK; response time: 16 ms

Duration of work: 45 sec; number of corrective actions:
  - Send SIGTERM to python3: 1
MemAvail: 1779 M, 89.4 % | SwapFree: 597 M,  81.5 %
MemAvail: 1791 M, 90.0 % | SwapFree: 585 M,  80.0 %
MemAvail: 1801 M, 90.5 % | SwapFree: 574 M,  78.4 %
MemAvail: 1811 M, 91.0 % | SwapFree: 564 M,  77.0 %
MemAvail: 1815 M, 91.2 % | SwapFree: 561 M,  76.6 %
MemAvail: 1814 M, 91.2 % | SwapFree: 561 M,  76.6 %
MemAvail: 1813 M, 91.1 % | SwapFree: 561 M,  76.6 %
some avg10=41.53 avg60=10.19 avg300=2.24 total=7650134
PSI mem some avg: 54.31, PSI avg time: 4.4
PSI is OK  or  psi_min_sleep_time_after_action did not pass
MemAvail: 1813 M, 91.1 % | SwapFree: 561 M,  76.6 %
MemAvail: 1811 M, 91.0 % | SwapFree: 562 M,  76.7 %
MemAvail: 1805 M, 90.7 % | SwapFree: 568 M,  77.5 %
MemAvail: 1798 M, 90.4 % | SwapFree: 574 M,  78.4 %
MemAvail: 1796 M, 90.2 % | SwapFree: 576 M,  78.7 %
MemAvail: 1796 M, 90.2 % | SwapFree: 576 M,  78.7 %
MemAvail: 1796 M, 90.2 % | SwapFree: 576 M,  78.7 %
MemAvail: 1796 M, 90.2 % | SwapFree: 576 M,  78.7 %
MemAvail: 1796 M, 90.2 % | SwapFree: 576 M,  78.7 %
some avg10=39.81 avg60=13.12 avg300=3.04 total=9217960
PSI mem some avg: 35.45, PSI avg time: 4.4
PSI is OK  or  psi_min_sleep_time_after_action did not pass
MemAvail: 1796 M, 90.2 % | SwapFree: 576 M,  78.7 %
MemAvail: 1796 M, 90.2 % | SwapFree: 576 M,  78.7 %

Jan 02 '19 09:01 hakavlad

I saw false positives even if avg_min_time = 4 and psi_min_sleep_time_after_action = 20. IMHO avg10 is now the safest for default using. total need testing

Jan 02 '19 10:01 hakavlad

PSI mem some avg10:    0.0 | MemAvail:  235 M, 11.8 % | SwapFree: 731 M,  99.9 %
PSI mem some avg10:    0.0 | MemAvail:  165 M,  8.3 % | SwapFree: 731 M,  99.9 %
PSI mem some avg10:    0.0 | MemAvail:  101 M,  5.1 % | SwapFree: 731 M,  99.9 %
PSI mem some avg10:    0.0 | MemAvail:   49 M,  2.5 % | SwapFree: 731 M,  99.9 %
PSI mem some avg10:    0.0 | MemAvail:   17 M,  0.8 % | SwapFree: 731 M,  99.9 %
PSI mem some avg10:    0.0 | MemAvail:    1 M,  0.0 % | SwapFree: 727 M,  99.2 %
PSI mem some avg10:    0.0 | MemAvail:    0 M,  0.0 % | SwapFree: 717 M,  97.9 %
PSI mem some avg10:   1.99 | MemAvail:    0 M,  0.0 % | SwapFree: 713 M,  97.4 %
PSI mem some avg10:   1.99 | MemAvail:    0 M,  0.0 % | SwapFree: 708 M,  96.7 %
PSI mem some avg10:   1.99 | MemAvail:    0 M,  0.0 % | SwapFree: 703 M,  96.0 %
PSI mem some avg10:   1.99 | MemAvail:    7 M,  0.3 % | SwapFree: 695 M,  95.0 %
PSI mem some avg10:   1.99 | MemAvail:   13 M,  0.6 % | SwapFree: 688 M,  94.0 %
PSI mem some avg10:   1.99 | MemAvail:   19 M,  0.9 % | SwapFree: 681 M,  93.0 %
PSI mem some avg10:   1.99 | MemAvail:   27 M,  1.4 % | SwapFree: 673 M,  91.9 %
PSI mem some avg10:   1.99 | MemAvail:   32 M,  1.6 % | SwapFree: 668 M,  91.2 %
PSI mem some avg10:   1.99 | MemAvail:   34 M,  1.7 % | SwapFree: 664 M,  90.7 %
PSI mem some avg10:  10.32 | MemAvail:   39 M,  2.0 % | SwapFree: 659 M,  90.0 %
PSI mem some avg10:  10.32 | MemAvail:   45 M,  2.2 % | SwapFree: 653 M,  89.1 %
PSI mem some avg10:  10.32 | MemAvail:   49 M,  2.4 % | SwapFree: 648 M,  88.5 %
PSI mem some avg10:  10.32 | MemAvail:   55 M,  2.7 % | SwapFree: 641 M,  87.6 %
PSI mem some avg10:  10.32 | MemAvail:   61 M,  3.1 % | SwapFree: 634 M,  86.6 %
PSI mem some avg10:  10.32 | MemAvail:   64 M,  3.2 % | SwapFree: 632 M,  86.3 %
PSI mem some avg10:  10.32 | MemAvail:   51 M,  2.6 % | SwapFree: 630 M,  86.0 %
PSI mem some avg10:  10.32 | MemAvail:    4 M,  0.2 % | SwapFree: 630 M,  86.0 %
PSI mem some avg10:  10.32 | MemAvail:    0 M,  0.0 % | SwapFree: 621 M,  84.8 %
PSI mem some avg10:  20.05 | MemAvail:    0 M,  0.0 % | SwapFree: 612 M,  83.6 %
PSI mem some avg10:  20.05 | MemAvail:    0 M,  0.0 % | SwapFree: 611 M,  83.5 %
PSI mem some avg10:  20.05 | MemAvail:    0 M,  0.0 % | SwapFree: 607 M,  82.9 %
PSI mem some avg10:  20.05 | MemAvail:    0 M,  0.0 % | SwapFree: 600 M,  82.0 %
PSI mem some avg10:  20.05 | MemAvail:    0 M,  0.0 % | SwapFree: 600 M,  82.0 %
PSI mem some avg10:  20.05 | MemAvail:    1 M,  0.0 % | SwapFree: 591 M,  80.7 %
PSI mem some avg10:  20.05 | MemAvail:    0 M,  0.0 % | SwapFree: 583 M,  79.6 %
PSI mem some avg10:  20.05 | MemAvail:    0 M,  0.0 % | SwapFree: 583 M,  79.6 %
PSI mem some avg10:  20.05 | MemAvail:    0 M,  0.0 % | SwapFree: 582 M,  79.5 %
PSI mem some avg10:  20.05 | MemAvail:    0 M,  0.0 % | SwapFree: 578 M,  78.9 %
PSI mem some avg10:  31.09 | MemAvail:    0 M,  0.0 % | SwapFree: 572 M,  78.1 %
PSI mem some avg10:  31.09 | MemAvail:    0 M,  0.0 % | SwapFree: 567 M,  77.5 %
PSI mem some avg10:  31.09 | MemAvail:    0 M,  0.0 % | SwapFree: 563 M,  76.9 %
PSI mem some avg10:  31.09 | MemAvail:    0 M,  0.0 % | SwapFree: 557 M,  76.1 %
PSI mem some avg10:  31.09 | MemAvail:    0 M,  0.0 % | SwapFree: 556 M,  76.0 %
PSI mem some avg10:  31.09 | MemAvail:    0 M,  0.0 % | SwapFree: 555 M,  75.8 %
PSI mem some avg10:  31.09 | MemAvail:    0 M,  0.0 % | SwapFree: 550 M,  75.2 %
PSI mem some avg10:  31.09 | MemAvail:    0 M,  0.0 % | SwapFree: 548 M,  74.9 %
PSI mem some avg10:  31.09 | MemAvail:    0 M,  0.0 % | SwapFree: 541 M,  73.9 %
PSI mem some avg10:  31.09 | MemAvail:    0 M,  0.0 % | SwapFree: 535 M,  73.1 %
PSI mem some avg10:  31.09 | MemAvail:    0 M,  0.0 % | SwapFree: 532 M,  72.6 %
PSI mem some avg10:  40.85 | MemAvail:    0 M,  0.0 % | SwapFree: 527 M,  71.9 %
PSI mem some avg10:  40.85 | MemAvail:    0 M,  0.0 % | SwapFree: 527 M,  71.9 %
PSI mem some avg10:  40.85 | MemAvail:    0 M,  0.0 % | SwapFree: 527 M,  71.9 %
PSI mem some avg10:  40.85 | MemAvail:    0 M,  0.0 % | SwapFree: 518 M,  70.8 %
PSI mem some avg10:  40.85 | MemAvail:    0 M,  0.0 % | SwapFree: 514 M,  70.2 %
PSI mem some avg10:  40.85 | MemAvail:    0 M,  0.0 % | SwapFree: 507 M,  69.3 %
PSI mem some avg10:  40.85 | MemAvail:    0 M,  0.0 % | SwapFree: 506 M,  69.1 %
PSI mem some avg10:  40.85 | MemAvail:    0 M,  0.0 % | SwapFree: 506 M,  69.1 %
PSI mem some avg10:  40.85 | MemAvail:    0 M,  0.0 % | SwapFree: 506 M,  69.1 %
PSI mem some avg10:  40.85 | MemAvail:    0 M,  0.0 % | SwapFree: 501 M,  68.4 %
PSI mem some avg10:  40.85 | MemAvail:    0 M,  0.0 % | SwapFree: 498 M,  68.0 %
PSI mem some avg10:  50.66 | MemAvail:    0 M,  0.0 % | SwapFree: 490 M,  66.9 %
PSI mem some avg10:  50.66 | MemAvail:    0 M,  0.0 % | SwapFree: 485 M,  66.2 %
PSI mem some avg10:  50.66 | MemAvail:    0 M,  0.0 % | SwapFree: 481 M,  65.7 %
PSI mem some avg10:  50.66 | MemAvail:    0 M,  0.0 % | SwapFree: 472 M,  64.5 %
PSI mem some avg10:  50.66 | MemAvail:    0 M,  0.0 % | SwapFree: 466 M,  63.6 %
PSI mem some avg10:  50.66 | MemAvail:    0 M,  0.0 % | SwapFree: 460 M,  62.8 %
PSI mem some avg10:  50.66 | MemAvail:    0 M,  0.0 % | SwapFree: 456 M,  62.3 %
PSI mem some avg10:  50.66 | MemAvail:    0 M,  0.0 % | SwapFree: 449 M,  61.4 %
PSI mem some avg10:  50.66 | MemAvail:    0 M,  0.0 % | SwapFree: 444 M,  60.6 %
PSI mem some avg10:  50.66 | MemAvail:    0 M,  0.0 % | SwapFree: 438 M,  59.8 %
PSI mem some avg10:  50.66 | MemAvail:    0 M,  0.0 % | SwapFree: 437 M,  59.6 %
PSI mem some avg10:  50.66 | MemAvail:    0 M,  0.0 % | SwapFree: 433 M,  59.1 %
PSI mem some avg10:  50.66 | MemAvail:    0 M,  0.0 % | SwapFree: 424 M,  57.9 %
PSI mem some avg10:  55.25 | MemAvail:    0 M,  0.0 % | SwapFree: 424 M,  57.9 %
PSI mem some avg10:  55.25 | MemAvail:    0 M,  0.0 % | SwapFree: 424 M,  57.9 %
PSI mem some avg10:  55.25 | MemAvail:    0 M,  0.0 % | SwapFree: 424 M,  57.9 %
PSI mem some avg10:  55.25 | MemAvail:    0 M,  0.0 % | SwapFree: 421 M,  57.5 %
PSI mem some avg10:  55.25 | MemAvail:    0 M,  0.0 % | SwapFree: 418 M,  57.1 %
PSI mem some avg10:  55.25 | MemAvail:    0 M,  0.0 % | SwapFree: 412 M,  56.3 %
PSI mem some avg10:  55.25 | MemAvail:    0 M,  0.0 % | SwapFree: 409 M,  55.8 %
PSI mem some avg10:  55.25 | MemAvail:    0 M,  0.0 % | SwapFree: 404 M,  55.2 %
PSI mem some avg10:  55.25 | MemAvail:    0 M,  0.0 % | SwapFree: 399 M,  54.5 %
PSI mem some avg10:  55.25 | MemAvail:    0 M,  0.0 % | SwapFree: 398 M,  54.3 %
PSI mem some avg10:  55.25 | MemAvail:    0 M,  0.0 % | SwapFree: 398 M,  54.3 %
PSI mem some avg10:  55.25 | MemAvail:    0 M,  0.0 % | SwapFree: 393 M,  53.6 %
PSI mem some avg10:  55.25 | MemAvail:    0 M,  0.0 % | SwapFree: 391 M,  53.3 %
PSI mem some avg10:  55.25 | MemAvail:    0 M,  0.0 % | SwapFree: 386 M,  52.7 %
PSI mem some avg10:  55.25 | MemAvail:    0 M,  0.0 % | SwapFree: 384 M,  52.5 %
avg (62.99) > sigterm_psi (60.0)
Found the victim with highest badness:
  Name:    python3
  PID:     1486
  UID:     1000
  Badness: 621, oom_score: 621, oom_score_adj: 0
  VmSize:  1719 MiB
  VmRSS:   1624 MiB (Anon: 1618 MiB, File: 6 MiB, Shmem: 0 MiB)
  VmSwap:    66 MiB
  CmdLine: python3 /usr/bin/oom-trigger 
Implement corrective action:
  Sending SIGTERM to the victim; OK; response time: 70 ms

Duration of work: 1 min 0 sec; number of corrective actions:
  - Send SIGTERM to python3: 1
PSI mem some avg10:  62.99 | MemAvail: 1623 M, 81.5 % | SwapFree: 450 M,  61.4 %
PSI mem some avg10:  62.99 | MemAvail: 1663 M, 83.6 % | SwapFree: 446 M,  60.9 %
PSI mem some avg10:  62.99 | MemAvail: 1667 M, 83.7 % | SwapFree: 442 M,  60.4 %
PSI mem some avg10:  62.99 | MemAvail: 1667 M, 83.7 % | SwapFree: 442 M,  60.4 %
PSI mem some avg10:  68.43 | MemAvail: 1667 M, 83.7 % | SwapFree: 442 M,  60.4 %
PSI mem some avg10:  68.43 | MemAvail: 1666 M, 83.7 % | SwapFree: 442 M,  60.4 %
PSI mem some avg10:  68.43 | MemAvail: 1666 M, 83.7 % | SwapFree: 442 M,  60.4 %
PSI mem some avg10:  68.43 | MemAvail: 1665 M, 83.7 % | SwapFree: 443 M,  60.5 %
PSI mem some avg10:  68.43 | MemAvail: 1665 M, 83.6 % | SwapFree: 443 M,  60.5 %
PSI mem some avg10:  72.15 | MemAvail: 1664 M, 83.6 % | SwapFree: 443 M,  60.5 %
PSI mem some avg10:  72.15 | MemAvail: 1664 M, 83.6 % | SwapFree: 443 M,  60.5 %
PSI mem some avg10:  72.15 | MemAvail: 1663 M, 83.5 % | SwapFree: 443 M,  60.5 %
PSI mem some avg10:  72.15 | MemAvail: 1662 M, 83.5 % | SwapFree: 443 M,  60.5 %
PSI mem some avg10:  72.15 | MemAvail: 1661 M, 83.5 % | SwapFree: 443 M,  60.5 %
PSI mem some avg10:  71.94 | MemAvail: 1660 M, 83.4 % | SwapFree: 443 M,  60.5 %
PSI mem some avg10:  71.94 | MemAvail: 1660 M, 83.4 % | SwapFree: 443 M,  60.5 %
PSI mem some avg10:  71.94 | MemAvail: 1659 M, 83.3 % | SwapFree: 444 M,  60.6 %
PSI mem some avg10:  71.94 | MemAvail: 1658 M, 83.3 % | SwapFree: 444 M,  60.7 %
PSI mem some avg10:  71.94 | MemAvail: 1657 M, 83.2 % | SwapFree: 445 M,  60.8 %
PSI mem some avg10:  74.67 | MemAvail: 1655 M, 83.2 % | SwapFree: 446 M,  60.9 %
PSI mem some avg10:  74.67 | MemAvail: 1654 M, 83.1 % | SwapFree: 447 M,  61.0 %
PSI mem some avg10:  74.67 | MemAvail: 1653 M, 83.0 % | SwapFree: 447 M,  61.1 %
PSI mem some avg10:  74.67 | MemAvail: 1651 M, 82.9 % | SwapFree: 448 M,  61.2 %
PSI mem some avg10:  74.67 | MemAvail: 1649 M, 82.9 % | SwapFree: 450 M,  61.4 %
PSI mem some avg10:  78.35 | MemAvail: 1648 M, 82.8 % | SwapFree: 450 M,  61.5 %
PSI mem some avg10:  78.35 | MemAvail: 1648 M, 82.8 % | SwapFree: 450 M,  61.5 %
PSI mem some avg10:  78.35 | MemAvail: 1647 M, 82.7 % | SwapFree: 450 M,  61.5 %
PSI mem some avg10:  78.35 | MemAvail: 1647 M, 82.8 % | SwapFree: 450 M,  61.5 %
PSI mem some avg10:  79.56 | MemAvail: 1646 M, 82.7 % | SwapFree: 450 M,  61.5 %
PSI mem some avg10:  79.56 | MemAvail: 1646 M, 82.7 % | SwapFree: 451 M,  61.5 %
PSI mem some avg10:  79.56 | MemAvail: 1645 M, 82.7 % | SwapFree: 451 M,  61.6 %
PSI mem some avg10:  79.56 | MemAvail: 1644 M, 82.6 % | SwapFree: 452 M,  61.7 %
PSI mem some avg10:  79.56 | MemAvail: 1643 M, 82.5 % | SwapFree: 452 M,  61.8 %
PSI mem some avg10:   80.0 | MemAvail: 1642 M, 82.5 % | SwapFree: 453 M,  61.9 %
PSI mem some avg10:   80.0 | MemAvail: 1640 M, 82.4 % | SwapFree: 455 M,  62.1 %
PSI mem some avg10:   80.0 | MemAvail: 1640 M, 82.4 % | SwapFree: 455 M,  62.1 %
PSI mem some avg10:   80.0 | MemAvail: 1638 M, 82.3 % | SwapFree: 455 M,  62.1 %
PSI mem some avg10:   80.0 | MemAvail: 1637 M, 82.2 % | SwapFree: 455 M,  62.2 %
PSI mem some avg10:  79.09 | MemAvail: 1636 M, 82.2 % | SwapFree: 455 M,  62.2 %
PSI mem some avg10:  79.09 | MemAvail: 1637 M, 82.2 % | SwapFree: 455 M,  62.2 %
PSI mem some avg10:  79.09 | MemAvail: 1636 M, 82.2 % | SwapFree: 455 M,  62.2 %
PSI mem some avg10:  79.09 | MemAvail: 1635 M, 82.2 % | SwapFree: 455 M,  62.2 %
PSI mem some avg10:  79.09 | MemAvail: 1635 M, 82.1 % | SwapFree: 455 M,  62.2 %
PSI mem some avg10:  78.53 | MemAvail: 1635 M, 82.1 % | SwapFree: 455 M,  62.2 %
PSI mem some avg10:  78.53 | MemAvail: 1634 M, 82.1 % | SwapFree: 455 M,  62.2 %
PSI mem some avg10:  78.53 | MemAvail: 1634 M, 82.1 % | SwapFree: 455 M,  62.2 %
PSI mem some avg10:  78.53 | MemAvail: 1633 M, 82.1 % | SwapFree: 455 M,  62.2 %
PSI mem some avg10:  78.53 | MemAvail: 1633 M, 82.0 % | SwapFree: 455 M,  62.2 %
PSI mem some avg10:  77.71 | MemAvail: 1633 M, 82.0 % | SwapFree: 456 M,  62.2 %
PSI mem some avg10:  77.71 | MemAvail: 1632 M, 82.0 % | SwapFree: 456 M,  62.2 %
PSI mem some avg10:  77.71 | MemAvail: 1632 M, 82.0 % | SwapFree: 456 M,  62.2 %
PSI mem some avg10:  77.71 | MemAvail: 1632 M, 82.0 % | SwapFree: 456 M,  62.2 %
PSI mem some avg10:  78.67 | MemAvail: 1631 M, 82.0 % | SwapFree: 456 M,  62.2 %
PSI mem some avg10:  78.67 | MemAvail: 1631 M, 82.0 % | SwapFree: 456 M,  62.2 %
PSI mem some avg10:  78.67 | MemAvail: 1631 M, 81.9 % | SwapFree: 456 M,  62.2 %
PSI mem some avg10:  78.67 | MemAvail: 1631 M, 81.9 % | SwapFree: 456 M,  62.3 %
PSI mem some avg10:  78.67 | MemAvail: 1630 M, 81.9 % | SwapFree: 456 M,  62.3 %
PSI mem some avg10:  79.63 | MemAvail: 1630 M, 81.9 % | SwapFree: 456 M,  62.3 %
PSI mem some avg10:  79.63 | MemAvail: 1629 M, 81.9 % | SwapFree: 456 M,  62.3 %
PSI mem some avg10:  79.63 | MemAvail: 1629 M, 81.8 % | SwapFree: 456 M,  62.3 %
PSI mem some avg10:  79.63 | MemAvail: 1628 M, 81.8 % | SwapFree: 456 M,  62.3 %
PSI mem some avg10:  79.63 | MemAvail: 1628 M, 81.8 % | SwapFree: 456 M,  62.3 %
PSI mem some avg10:  81.15 | MemAvail: 1628 M, 81.8 % | SwapFree: 457 M,  62.4 %
PSI mem some avg10:  81.15 | MemAvail: 1627 M, 81.8 % | SwapFree: 457 M,  62.4 %
PSI mem some avg10:  81.15 | MemAvail: 1627 M, 81.7 % | SwapFree: 457 M,  62.4 %
PSI mem some avg10:  81.15 | MemAvail: 1626 M, 81.7 % | SwapFree: 457 M,  62.4 %
PSI mem some avg10:  81.15 | MemAvail: 1625 M, 81.7 % | SwapFree: 457 M,  62.4 %
PSI mem some avg10:  83.84 | MemAvail: 1625 M, 81.6 % | SwapFree: 457 M,  62.5 %
PSI mem some avg10:  83.84 | MemAvail: 1624 M, 81.6 % | SwapFree: 458 M,  62.5 %
PSI mem some avg10:  83.84 | MemAvail: 1624 M, 81.6 % | SwapFree: 458 M,  62.5 %
PSI mem some avg10:  83.84 | MemAvail: 1623 M, 81.5 % | SwapFree: 458 M,  62.5 %
PSI mem some avg10:  83.84 | MemAvail: 1622 M, 81.5 % | SwapFree: 458 M,  62.6 %
PSI mem some avg10:  84.77 | MemAvail: 1622 M, 81.5 % | SwapFree: 458 M,  62.6 %
PSI mem some avg10:  84.77 | MemAvail: 1621 M, 81.4 % | SwapFree: 458 M,  62.6 %
PSI mem some avg10:  84.77 | MemAvail: 1620 M, 81.4 % | SwapFree: 458 M,  62.6 %
PSI mem some avg10:  84.77 | MemAvail: 1620 M, 81.4 % | SwapFree: 459 M,  62.6 %
PSI mem some avg10:  86.26 | MemAvail: 1620 M, 81.4 % | SwapFree: 459 M,  62.6 %
PSI mem some avg10:  86.26 | MemAvail: 1619 M, 81.3 % | SwapFree: 459 M,  62.7 %
PSI mem some avg10:  86.26 | MemAvail: 1619 M, 81.3 % | SwapFree: 459 M,  62.7 %
PSI mem some avg10:  86.26 | MemAvail: 1618 M, 81.3 % | SwapFree: 459 M,  62.7 %
PSI mem some avg10:  86.26 | MemAvail: 1618 M, 81.3 % | SwapFree: 459 M,  62.7 %
PSI mem some avg10:  86.39 | MemAvail: 1618 M, 81.3 % | SwapFree: 459 M,  62.7 %
PSI mem some avg10:  86.39 | MemAvail: 1618 M, 81.3 % | SwapFree: 459 M,  62.7 %
PSI mem some avg10:  86.39 | MemAvail: 1617 M, 81.3 % | SwapFree: 459 M,  62.7 %
PSI mem some avg10:  86.39 | MemAvail: 1617 M, 81.2 % | SwapFree: 459 M,  62.7 %
PSI mem some avg10:  86.39 | MemAvail: 1617 M, 81.2 % | SwapFree: 459 M,  62.7 %
PSI mem some avg10:  86.68 | MemAvail: 1616 M, 81.2 % | SwapFree: 459 M,  62.7 %
PSI mem some avg10:  86.68 | MemAvail: 1616 M, 81.2 % | SwapFree: 459 M,  62.7 %
PSI mem some avg10:  86.68 | MemAvail: 1616 M, 81.2 % | SwapFree: 459 M,  62.7 %
PSI mem some avg10:  86.68 | MemAvail: 1615 M, 81.1 % | SwapFree: 460 M,  62.8 %
PSI mem some avg10:  86.68 | MemAvail: 1614 M, 81.1 % | SwapFree: 460 M,  62.8 %
PSI mem some avg10:  83.12 | MemAvail: 1614 M, 81.1 % | SwapFree: 460 M,  62.8 %
PSI mem some avg10:  83.12 | MemAvail: 1613 M, 81.0 % | SwapFree: 460 M,  62.8 %
PSI mem some avg10:  83.12 | MemAvail: 1613 M, 81.0 % | SwapFree: 460 M,  62.8 %
PSI mem some avg10:  83.12 | MemAvail: 1612 M, 81.0 % | SwapFree: 460 M,  62.8 %
PSI mem some avg10:  83.12 | MemAvail: 1611 M, 81.0 % | SwapFree: 460 M,  62.8 %
avg (79.47) > sigterm_psi (60.0)
Found the victim with highest badness:
  Name:    firefox
  PID:     1045
  UID:     1000
  Badness: 68, oom_score: 68, oom_score_adj: 0
  VmSize:  1843 MiB
  VmRSS:    115 MiB (Anon: 34 MiB, File: 22 MiB, Shmem: 58 MiB)
  VmSwap:    70 MiB
  CmdLine: /usr/lib/firefox/firefox 
Implement corrective action:
  Sending SIGTERM to the victim; OK; response time: 17 ms

Duration of work: 1 min 40 sec; number of corrective actions:
  - Send SIGTERM to python3: 1
  - Send SIGTERM to firefox: 1
PSI mem some avg10:  79.47 | MemAvail: 1611 M, 80.9 % | SwapFree: 460 M,  62.8 %
PSI mem some avg10:  79.47 | MemAvail: 1650 M, 82.9 % | SwapFree: 533 M,  72.7 %
PSI mem some avg10:  79.47 | MemAvail: 1649 M, 82.9 % | SwapFree: 533 M,  72.8 %
PSI mem some avg10:  79.47 | MemAvail: 1649 M, 82.8 % | SwapFree: 533 M,  72.8 %
PSI mem some avg10:  76.85 | MemAvail: 1649 M, 82.8 % | SwapFree: 533 M,  72.8 %
PSI mem some avg10:  76.85 | MemAvail: 1648 M, 82.8 % | SwapFree: 533 M,  72.8 %
PSI mem some avg10:  76.85 | MemAvail: 1647 M, 82.8 % | SwapFree: 534 M,  72.9 %
PSI mem some avg10:  76.85 | MemAvail: 1659 M, 83.4 % | SwapFree: 534 M,  72.9 %
PSI mem some avg10:  79.77 | MemAvail: 1659 M, 83.4 % | SwapFree: 534 M,  72.9 %
PSI mem some avg10:  79.77 | MemAvail: 1659 M, 83.4 % | SwapFree: 534 M,  73.0 %
PSI mem some avg10:  79.77 | MemAvail: 1658 M, 83.3 % | SwapFree: 534 M,  73.0 %
PSI mem some avg10:  79.77 | MemAvail: 1656 M, 83.2 % | SwapFree: 534 M,  73.0 %
PSI mem some avg10:  79.77 | MemAvail: 1656 M, 83.2 % | SwapFree: 534 M,  73.0 %
PSI mem some avg10:  75.65 | MemAvail: 1651 M, 82.9 % | SwapFree: 534 M,  73.0 %
PSI mem some avg10:  75.65 | MemAvail: 1650 M, 82.9 % | SwapFree: 534 M,  73.0 %
PSI mem some avg10:  75.65 | MemAvail: 1649 M, 82.9 % | SwapFree: 535 M,  73.0 %
PSI mem some avg10:  75.65 | MemAvail: 1646 M, 82.7 % | SwapFree: 535 M,  73.0 %
PSI mem some avg10:  68.28 | MemAvail: 1646 M, 82.7 % | SwapFree: 535 M,  73.1 %
PSI mem some avg10:  68.28 | MemAvail: 1645 M, 82.6 % | SwapFree: 535 M,  73.1 %
PSI mem some avg10:  68.28 | MemAvail: 1644 M, 82.6 % | SwapFree: 535 M,  73.1 %
PSI mem some avg10:  68.28 | MemAvail: 1642 M, 82.5 % | SwapFree: 536 M,  73.2 %
PSI mem some avg10:  68.28 | MemAvail: 1641 M, 82.4 % | SwapFree: 536 M,  73.2 %
PSI mem some avg10:  66.24 | MemAvail: 1637 M, 82.3 % | SwapFree: 536 M,  73.2 %
PSI mem some avg10:  66.24 | MemAvail: 1637 M, 82.3 % | SwapFree: 536 M,  73.2 %
PSI mem some avg10:  66.24 | MemAvail: 1633 M, 82.1 % | SwapFree: 536 M,  73.2 %
PSI mem some avg10:  66.24 | MemAvail: 1631 M, 82.0 % | SwapFree: 536 M,  73.2 %
PSI mem some avg10:  63.11 | MemAvail: 1630 M, 81.9 % | SwapFree: 536 M,  73.2 %
PSI mem some avg10:  63.11 | MemAvail: 1628 M, 81.8 % | SwapFree: 536 M,  73.2 %
PSI mem some avg10:  63.11 | MemAvail: 1625 M, 81.6 % | SwapFree: 536 M,  73.2 %
PSI mem some avg10:  63.11 | MemAvail: 1625 M, 81.6 % | SwapFree: 536 M,  73.2 %
PSI mem some avg10:  63.11 | MemAvail: 1623 M, 81.5 % | SwapFree: 536 M,  73.3 %
PSI mem some avg10:   62.0 | MemAvail: 1619 M, 81.4 % | SwapFree: 536 M,  73.3 %
PSI mem some avg10:   62.0 | MemAvail: 1615 M, 81.1 % | SwapFree: 536 M,  73.3 %
PSI mem some avg10:   62.0 | MemAvail: 1612 M, 81.0 % | SwapFree: 536 M,  73.3 %
PSI mem some avg10:   62.0 | MemAvail: 1611 M, 80.9 % | SwapFree: 536 M,  73.3 %
PSI mem some avg10:  56.39 | MemAvail: 1610 M, 80.9 % | SwapFree: 536 M,  73.3 %
PSI mem some avg10:  56.39 | MemAvail: 1605 M, 80.7 % | SwapFree: 536 M,  73.3 %
PSI mem some avg10:  56.39 | MemAvail: 1732 M, 87.0 % | SwapFree: 583 M,  79.6 %
PSI mem some avg10:  56.39 | MemAvail: 1732 M, 87.0 % | SwapFree: 583 M,  79.6 %
PSI mem some avg10:  56.39 | MemAvail: 1732 M, 87.0 % | SwapFree: 583 M,  79.6 %
PSI mem some avg10:  47.98 | MemAvail: 1732 M, 87.0 % | SwapFree: 583 M,  79.6 %
PSI mem some avg10:  47.98 | MemAvail: 1732 M, 87.0 % | SwapFree: 583 M,  79.7 %
PSI mem some avg10:  47.98 | MemAvail: 1732 M, 87.0 % | SwapFree: 583 M,  79.7 %
PSI mem some avg10:  47.98 | MemAvail: 1732 M, 87.0 % | SwapFree: 583 M,  79.7 %
PSI mem some avg10:  41.28 | MemAvail: 1732 M, 87.0 % | SwapFree: 583 M,  79.7 %
PSI mem some avg10:  41.28 | MemAvail: 1732 M, 87.0 % | SwapFree: 583 M,  79.7 %
PSI mem some avg10:  41.28 | MemAvail: 1732 M, 87.0 % | SwapFree: 583 M,  79.7 %
PSI mem some avg10:  41.28 | MemAvail: 1732 M, 87.0 % | SwapFree: 584 M,  79.7 %
PSI mem some avg10:  33.98 | MemAvail: 1732 M, 87.0 % | SwapFree: 584 M,  79.7 %
PSI mem some avg10:  33.98 | MemAvail: 1732 M, 87.0 % | SwapFree: 584 M,  79.8 %
PSI mem some avg10:  33.98 | MemAvail: 1732 M, 87.0 % | SwapFree: 584 M,  79.8 %
PSI mem some avg10:  33.98 | MemAvail: 1732 M, 87.0 % | SwapFree: 584 M,  79.8 %
PSI mem some avg10:  28.19 | MemAvail: 1732 M, 87.0 % | SwapFree: 584 M,  79.8 %
PSI mem some avg10:  28.19 | MemAvail: 1732 M, 87.0 % | SwapFree: 584 M,  79.8 %
PSI mem some avg10:  28.19 | MemAvail: 1732 M, 87.0 % | SwapFree: 584 M,  79.8 %
PSI mem some avg10:  28.19 | MemAvail: 1732 M, 87.0 % | SwapFree: 584 M,  79.8 %
PSI mem some avg10:  23.26 | MemAvail: 1732 M, 87.0 % | SwapFree: 584 M,  79.8 %
PSI mem some avg10:  23.26 | MemAvail: 1732 M, 87.0 % | SwapFree: 584 M,  79.8 %
PSI mem some avg10:  23.26 | MemAvail: 1732 M, 87.0 % | SwapFree: 584 M,  79.8 %
PSI mem some avg10:  23.26 | MemAvail: 1731 M, 87.0 % | SwapFree: 586 M,  80.1 %
PSI mem some avg10:  28.29 | MemAvail: 1730 M, 86.9 % | SwapFree: 587 M,  80.2 %
PSI mem some avg10:  28.29 | MemAvail: 1730 M, 86.9 % | SwapFree: 587 M,  80.2 %
PSI mem some avg10:  28.29 | MemAvail: 1730 M, 86.9 % | SwapFree: 587 M,  80.2 %
PSI mem some avg10:  28.29 | MemAvail: 1730 M, 86.9 % | SwapFree: 587 M,  80.2 %
PSI mem some avg10:  28.29 | MemAvail: 1730 M, 86.9 % | SwapFree: 587 M,  80.2 %
PSI mem some avg10:  23.16 | MemAvail: 1730 M, 86.9 % | SwapFree: 587 M,  80.2 %
PSI mem some avg10:  23.16 | MemAvail: 1730 M, 86.9 % | SwapFree: 587 M,  80.2 %
PSI mem some avg10:  23.16 | MemAvail: 1730 M, 86.9 % | SwapFree: 587 M,  80.2 %
PSI mem some avg10:  23.16 | MemAvail: 1730 M, 86.9 % | SwapFree: 587 M,  80.2 %
PSI mem some avg10:  18.96 | MemAvail: 1730 M, 86.9 % | SwapFree: 587 M,  80.2 %
PSI mem some avg10:  18.96 | MemAvail: 1730 M, 86.9 % | SwapFree: 587 M,  80.2 %
PSI mem some avg10:  18.96 | MemAvail: 1730 M, 86.9 % | SwapFree: 587 M,  80.2 %
PSI mem some avg10:  18.96 | MemAvail: 1730 M, 86.9 % | SwapFree: 587 M,  80.2 %
PSI mem some avg10:  15.71 | MemAvail: 1730 M, 86.9 % | SwapFree: 587 M,  80.2 %
PSI mem some avg10:  15.71 | MemAvail: 1730 M, 86.9 % | SwapFree: 587 M,  80.2 %

PSI is very slow to bounce back. FF is innocent victim in this output (I used psi_min_sleep_time_after_action = 40).

Jan 02 '19 15:01 hakavlad

Info: Switching off a swapspace (on HDD if it was not empty) can cause a significant increasing PSI value (I've seen some avg10 = 70 when ran swapoff -a).

Jun 04 '19 00:06 hakavlad

I have tested a few things with PSI, and it seems pretty difficult to do "the right thing".

The difficult example is this: https://github.com/rfjakob/earlyoom/blob/master/contrib/oomstat/loadshift.txt You have an app that uses 70% of your RAM but is idle, and you have enough swap free. Then another app wants to use 70% of your RAM. PSI goes crazy, but the situation calms down as the idle app is swapped out. The right thing here is not killing anything.

Jul 04 '19 19:07 rfjakob

The maximum PSI values depend on the type of swapspace. 99 for slow HDD, 60-70 for zram and SSD.

Not very long-term exceedances are normal.

Only long-term threshold exceedances are a clear pathology and require a reaction.

The optimal settings are very individual.

The right things are:

disable PSI monitoring by default
respond to only very long (>60 sec) thresholds exceeding.

Offer the following default settings:

psi_checking_enabled = False
psi_metrics = some_avg10
sigterm_psi_threshold = 60
sigkill_psi_threshold = 90
psi_excess_duration = 60 sec
psi_post_action_delay = 60 sec

Jul 14 '19 14:07 hakavlad

@rfjakob see https://www.kernel.org/doc/html/latest/accounting/psi.html#monitoring-for-pressure-thresholds and https://www.kernel.org/doc/html/latest/accounting/psi.html#userspace-monitor-usage-example (using poll() since Linux 5.2)

Users can register triggers and use poll() to be woken up when resource pressure exceeds certain thresholds

Aug 16 '19 12:08 hakavlad

I have read through https://github.com/hakavlad/nohang/issues/25, oh boy, PSI is such a headache. Although in this case it seems that muqss is interacting badly with PSI.

Aug 18 '19 20:08 rfjakob

PSI is such a headache

Only on desktops. But don't worry, rfjakob-sensei: PSI should be very useful for servers. Disable PSI by default, add warnings to the documentation, and PSI will no longer be a headache.

Aug 19 '19 12:08 hakavlad

muqss is interacting badly with PSI

No.

Do you use also btrfs? See https://bugzilla.kernel.org/show_bug.cgi?id=196729

https://github.com/hakavlad/nohang/issues/25#issuecomment-523226166

Yes, I do.

Seems like this problem was with swap on btrfs.

Sep 01 '19 10:09 hakavlad

Please note that MuQSS is a valid possibility still, the linked issue didn't verify that it wasn't. Nor does the linked bugreport by @hakavlad to claim BTRFS at fault have anything to do with MuQSS or PSI. MuQSS only has partial support for cgroups afaik, and PSI relies on cgroups support, so it's more likely due to MuQSS lacking proper support for cgroups that PSI requires.

While the user also uses BTRFS, there is no verification from that user by using a different filesystem to confirm that it was caused by BTRFS alone. The linked bug report also has users with other filesystems other than BTRFS claiming they are experiencing the same/similar problems.

Jan 04 '20 01:01 polarathene

users with other filesystems other than BTRFS claiming they are experiencing the same/similar problems.

Could you provide links, please?

Jan 04 '20 08:01 hakavlad

PSI relies on cgroups support

PSI is not directly dependent on сgroups. PSI provides metrics for each group with cgroup_v2. Without cgroup_v2 PSI works well and provides files only in /proc/pressure.

Jan 04 '20 09:01 hakavlad

The problem happen much more frequently when I used BtrFS. After switching to XFS, this happen less frequently (weekly instead of daily). - Comment 17

So, something was fixed in upstream, backported to LTS kernel 4.19 and to Ubuntu kernel. I don't know what. And that issue is 100% not in BTRFS but is another problem or another aspect of the problem. - Comment 54

I don't use btrfs, but only ext4 on an SSD. Since updating my system from kernel 5.1.7 to kernel 5.3.12 in Tumbleweed, I get regular ~1-2 second freezes - Comment 62

Probably the last one is unrelated though. Another comment(55 I think), pointed out their issue was related to CPU activity, which reminds me of another issue on github about BTRFS maintenance scripts which openSUSE at least shipped by default. As it's a non-traditional filesystem, and can require treating it differently along with maintenance, I could understand how it adds additional complications/complexities that can make issues more pronounced.

If you just want to find issues/reports of users with bad performance related to swap, there are plenty of those without being specific to BTRFS. This bug report has nothing to do with PSI or earlyoom/nohang? So it's relevance to identifying the problem a user experienced with PSI not working as expected is low.

The user would need to verify with a different filesystem that the symptoms were resolved, or better yet and easier verify that MuQSS kernel is not to blame by trying a kernel without MuQSS, just plain CFS and vanilla kernel that enables PSI.

Without cgroup_v2 PSI works well and provides files only in /proc/pressure.

Ah.. my mistake. But the log the user shared does seem to indicate that their system might indicate cgroup v2 support, but as MuQSS afaik only provides a partial implementation, what happens if PSI acknowledges cgroup v2 support and tries to use unimplemented features? It'd presumably break or behave incorrectly as the user indicates?

2019-08-14T21:15:47+0300 gentoo nohang[19609]:   CGroup_v2: /system.slice/system-beesd.slice/beesd@-----------------------------------------

That suggests that might be the case with that BTRFS daemon utility?

Jan 05 '20 04:01 polarathene

It's time to use PSI metrics to improve interactivity.

In this post I want to tell you about psi2log and psi-top scripts from nohang package. These simple tools can help you to conveniently measure and log various PSI metrics. See https://pagure.io/fedora-workstation/issue/98#comment-631355.

And PSI is not goes crazy, PSI metrics just provide stall information.

I have read through hakavlad/nohang#25, oh boy, PSI is such a headache.

PSI works incorrect with out-of-tree shedulers like MuQSS. But it is not a problem.

Mar 07 '20 18:03 hakavlad

I offer you to add optional PSI support.

For example, earlyoom --psi some,40. What this should mean? If -m threshold exceeded, we can shoot if memory some avg10 > 40. I offer to hardcode 15 sec reclaim time.

So, I offer hardcode avg10 metrics and 15 sec reclaim time. User can choose some/full and threshold. Checking -m should prevent false-positives with a lot of available memory. And of course, this shoulds be optional.

Mar 07 '20 18:03 hakavlad

nohang with PSI: no problem: https://www.youtube.com/watch?v=Y6GJqFE_ke4

Mar 07 '20 18:03 hakavlad

See also https://github.com/cdown/psi-notify

May 11 '20 18:05 hakavlad