crowdsec icon indicating copy to clipboard operation
crowdsec copied to clipboard

Add new optional scenario config overflowBehavior

Open janbaer opened this issue 3 years ago • 14 comments

If overflowBehavior becomes keep-alive the bucket will not die after the first overflow. Instead the next requests that is matching the filter will also cause an overflow and ban the IP address. If the leaky algorithm will drain the bucket, it will no longer overflow.

I also tried to create a test scenario for it, but I'm not sure if it's correct. I had problems with running the test, after installing all the requirements, the .environment.sh was still missing and I couldn't find any information about that file.

I created the PR after the discussion from here: https://discourse.crowdsec.net/t/ban-each-request-after-overflow/774/16 The requirement for this PR is that we need the ability to catch all requests from a specific country or provider and to show to all IPs a CAPTCHA after the bucket was overflown for the first time. Currently it's not working in the expected way, because the bucket dies after the first overflow happened.

I'm fully aware that the quality of my PR will not fulfill your requirements, but my knowledge about how everything is working in the Crowdsec is not good enough.

janbaer avatar May 20 '22 12:05 janbaer

Hello,

Thanks for the PR and I think it can benefit to other scenarios too, so let's work on this. I will have a proper look at it whenever I can squeeze a bit of time.

You told me that so far, your change doesn't work because you didn't manage to make the bucket not be destroyed when it overflows, is that correct ?

buixor avatar May 24 '22 15:05 buixor

@buixor Thank you for your offer to take a look on my PR. The problem with the PR is, that I couldn't run the tests at all locally on my machine, because of not really a good understanding of how to make it runnable. So to be honest I have no idea, how my change would affect behavior of the system. In best case, it shouldn't change anything, as long the overflowBehavior is not set. If it's set, it should no longer destroy the bucket after the first overflow.

janbaer avatar May 24 '22 17:05 janbaer

Hello @buixor I just wanted to ask about the progress with this PR and if you could find a bit of time to take a look on? If not, can you give an estimation about a time-frame when it could happen? I'm done with the integration of Crowdsec in our system, and it's working fine for detection Crawlers which are using a single IP address. But for the other use-case that a Crawler is using multiple IP addresses, Crowdsec is not working well. I also wonder, that others not also have this issue. Hackers are learning constantly and if they're learning that if they are using a single IP for crawling, they will be detected because of the amount of requests. That's why they are using sometimes multiple IPs at the same time.

Would it make sense if I also would create an issue in GitHub and then link the PR to the issue?

janbaer avatar Jun 02 '22 18:06 janbaer

Hello @janbaer !

Sorry for the delay, we're focusing on polishing 1.4, this should go next :)

buixor avatar Jun 30 '22 07:06 buixor

@Hello @buixor, after releasing 1.4, is there any chance to take a look on my PR and give me feedback about it? Thanks!

janbaer avatar Jul 25 '22 13:07 janbaer

Hello @janbaer ! I see you didn't forget us :D I didn't forget you either.

Plan to take some time to take a look at it this week and keep you posted :+1:

buixor avatar Jul 25 '22 13:07 buixor

Hey @janbaer ! I have updated your code to fix the logic : if the overflow_behavior is keep-alive : do not close the input channel when the bucket overflows and do not kill the leak routine when it overflows.

I did some very quick manual tests and it seems to have the expected behaviour.

Can you confirm this ?

buixor avatar Jul 26 '22 09:07 buixor

Codecov Report

Merging #1551 (a2c5716) into master (c78c833) will decrease coverage by 1.88%. The diff coverage is 78.57%.

@@            Coverage Diff             @@
##           master    #1551      +/-   ##
==========================================
- Coverage   52.20%   50.31%   -1.89%     
==========================================
  Files         136      116      -20     
  Lines       18586    16994    -1592     
==========================================
- Hits         9702     8551    -1151     
+ Misses       7830     7453     -377     
+ Partials     1054      990      -64     
Flag Coverage Δ
func-crowdsec 45.84% <ø> (+30.47%) :arrow_up:
func-cscli 44.55% <ø> (+<0.01%) :arrow_up:
unit-linux ?
unit-windows 53.62% <78.57%> (+0.02%) :arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
pkg/leakybucket/manager_load.go 69.55% <66.66%> (-0.07%) :arrow_down:
pkg/leakybucket/bucket.go 85.09% <87.50%> (+0.99%) :arrow_up:
pkg/apiclient/metrics.go 0.00% <0.00%> (-76.93%) :arrow_down:
pkg/acquisition/modules/journalctl/journalctl.go 0.00% <0.00%> (-71.74%) :arrow_down:
pkg/apiclient/signal.go 0.00% <0.00%> (-50.00%) :arrow_down:
pkg/acquisition/modules/kinesis/kinesis.go 0.00% <0.00%> (-34.24%) :arrow_down:
pkg/types/utils.go 0.00% <0.00%> (-32.65%) :arrow_down:
pkg/csplugin/watcher.go 69.30% <0.00%> (-25.75%) :arrow_down:
pkg/time/rate/rate.go 71.73% <0.00%> (-12.18%) :arrow_down:
pkg/csconfig/api.go 43.44% <0.00%> (-9.84%) :arrow_down:
... and 54 more

Help us with your feedback. Take ten seconds to tell us how you rate us.

codecov-commenter avatar Jul 26 '22 09:07 codecov-commenter

ps: giving it an extra shot as further tests confirm that it doesn't behave as intended.

buixor avatar Jul 27 '22 06:07 buixor

Hello @buixor thank you, for checking my PR and fixing it. I'll try to test it this evening and give you feedback.

janbaer avatar Jul 28 '22 06:07 janbaer

It is not working correctly yet. If you have a scenario like this :

# ssh bruteforce
type: leaky
overflow_behavior: keep-alive
debug: true
name: crowdsecurity/ssh-bf
description: "Detect ssh bruteforce"
filter: "evt.Meta.log_type == 'ssh_failed-auth'"
leakspeed: "10s"
references:
  - http://wikipedia.com/ssh-bf-is-bad
capacity: 5
groupby: evt.Meta.SourceRange
distinct: evt.Meta.source_ip
#blackhole: 1m
reprocess: true
labels:
 service: ssh
 type: bruteforce
 remediation: true

And we trigger it like this :

for i in `seq 1 7` ; do echo 'Sep 19 18:33:22 scw-d95986 sshd[24347]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=1.2.3.'${i} >> /tmp/ssh.log  ; done

When the bucket overflows (thanks to the debug added) the sources seem to be correctly collected :

DEBU[28-07-2022 09:29:49] gen APIAlerts -> 1.2.3.2                      bucket_id=empty-river capacity=5 cfg=shy-wave file=/home/bui/github/crowdsec/crowdsec-v1.4.1/tests/config/scenarios/ssh-bf.yaml name=crowdsecurity/ssh-bf partition=e483a32a86710e8f4e3fad62da60c55a8e9ba6f7
DEBU[28-07-2022 09:29:49] gen APIAlerts -> 1.2.3.3                      bucket_id=empty-river capacity=5 cfg=shy-wave file=/home/bui/github/crowdsec/crowdsec-v1.4.1/tests/config/scenarios/ssh-bf.yaml name=crowdsecurity/ssh-bf partition=e483a32a86710e8f4e3fad62da60c55a8e9ba6f7
DEBU[28-07-2022 09:29:49] gen APIAlerts -> 1.2.3.4                      bucket_id=empty-river capacity=5 cfg=shy-wave file=/home/bui/github/crowdsec/crowdsec-v1.4.1/tests/config/scenarios/ssh-bf.yaml name=crowdsecurity/ssh-bf partition=e483a32a86710e8f4e3fad62da60c55a8e9ba6f7
DEBU[28-07-2022 09:29:49] gen APIAlerts -> 1.2.3.5                      bucket_id=empty-river capacity=5 cfg=shy-wave file=/home/bui/github/crowdsec/crowdsec-v1.4.1/tests/config/scenarios/ssh-bf.yaml name=crowdsecurity/ssh-bf partition=e483a32a86710e8f4e3fad62da60c55a8e9ba6f7
DEBU[28-07-2022 09:29:49] gen APIAlerts -> 1.2.3.6                      bucket_id=empty-river capacity=5 cfg=shy-wave file=/home/bui/github/crowdsec/crowdsec-v1.4.1/tests/config/scenarios/ssh-bf.yaml name=crowdsecurity/ssh-bf partition=e483a32a86710e8f4e3fad62da60c55a8e9ba6f7
DEBU[28-07-2022 09:29:49] gen APIAlerts -> 1.2.3.7                      bucket_id=empty-river capacity=5 cfg=shy-wave file=/home/bui/github/crowdsec/crowdsec-v1.4.1/tests/config/scenarios/ssh-bf.yaml name=crowdsecurity/ssh-bf partition=e483a32a86710e8f4e3fad62da60c55a8e9ba6f7

But the alert received by the local API doesn't have the correct sources :

INFO[28-07-2022 09:29:49] processing alert : 1.2.3.1                   
INFO[28-07-2022 09:29:49] processing alert : 1.2.3.1                   
INFO[28-07-2022 09:29:49] processing alert : 1.2.3.1                   
INFO[28-07-2022 09:29:49] processing alert : 1.2.3.1                   
INFO[28-07-2022 09:29:49] processing alert : 1.2.3.1                   
INFO[28-07-2022 09:29:49] processing alert : 1.2.3.1                   
INFO[28-07-2022 09:29:49] processing alert : 1.2.3.7                   
INFO[28-07-2022 09:29:49] processing alert : 1.2.3.7                   
INFO[28-07-2022 09:29:49] processing alert : 1.2.3.7                   
INFO[28-07-2022 09:29:49] processing alert : 1.2.3.7                   
INFO[28-07-2022 09:29:49] processing alert : 1.2.3.7                   
INFO[28-07-2022 09:29:49] processing alert : 1.2.3.7                   

buixor avatar Jul 28 '22 07:07 buixor

I guess, you tested it in replay mode, isn't it? Maybe it is a side-effect of the parallelism and because all of the log entries had the same time-stamp. So I would expect here only two alerts because 5 requests are allowed. If the order would be correct, the IP addresses in the alerts should be 6 and 7.

janbaer avatar Jul 28 '22 15:07 janbaer

I just tested it today first with the replay mode. The behavior is not as expected. I had a logfile with multiple entries which all matched the filter, but should not be grouped by the IP. Instead it should be grouped by the IsoCode which is the same for all. So I expected one bucket which should overflow after the capacity of 2 were reached. So after running the replay with 3 different IP addresses, I got 3 alerts but all with the same IP (the last one) and one decision, as expected, but with the correct IP.

When I tests it with 4 different IPs, I got the same result and the last ip was just ignored

WARN[29-07-2022 20:20:31] Acquisition is finished, shutting down
DEBU[29-07-2022 20:20:31] source (0xc0002816c0): X.X.X.152        bucket_id=snowy-pond capacity=2 cfg=bitter-glitter file=/home/jan/Projects/check24/crowdsec-v1.4.1/tests/config/scenarios/bu-tariffs-by-country.yaml name=bu/tariffs-by-country partition=f2cb3acb8f1dd2fd2753deb9d421ab2c6dd8a548
DEBU[29-07-2022 20:20:31] APIAlerts -> X.X.X.152                  bucket_id=snowy-pond capacity=2 cfg=bitter-glitter file=/home/jan/Projects/check24/crowdsec-v1.4.1/tests/config/scenarios/bu-tariffs-by-country.yaml name=bu/tariffs-by-country partition=f2cb3acb8f1dd2fd2753deb9d421ab2c6dd8a548
DEBU[29-07-2022 20:20:31] source (0xc000281ce0): X.X.X.150        bucket_id=snowy-pond capacity=2 cfg=bitter-glitter file=/home/jan/Projects/check24/crowdsec-v1.4.1/tests/config/scenarios/bu-tariffs-by-country.yaml name=bu/tariffs-by-country partition=f2cb3acb8f1dd2fd2753deb9d421ab2c6dd8a548
DEBU[29-07-2022 20:20:31] APIAlerts -> X.X.X.150                  bucket_id=snowy-pond capacity=2 cfg=bitter-glitter file=/home/jan/Projects/check24/crowdsec-v1.4.1/tests/config/scenarios/bu-tariffs-by-country.yaml name=bu/tariffs-by-country partition=f2cb3acb8f1dd2fd2753deb9d421ab2c6dd8a548
DEBU[29-07-2022 20:20:31] source (0xc000281f10): X.X.X.151        bucket_id=snowy-pond capacity=2 cfg=bitter-glitter file=/home/jan/Projects/check24/crowdsec-v1.4.1/tests/config/scenarios/bu-tariffs-by-country.yaml name=bu/tariffs-by-country partition=f2cb3acb8f1dd2fd2753deb9d421ab2c6dd8a548
DEBU[29-07-2022 20:20:31] APIAlerts -> X.X.X.151                  bucket_id=snowy-pond capacity=2 cfg=bitter-glitter file=/home/jan/Projects/check24/crowdsec-v1.4.1/tests/config/scenarios/bu-tariffs-by-country.yaml name=bu/tariffs-by-country partition=f2cb3acb8f1dd2fd2753deb9d421ab2c6dd8a548
DEBU[29-07-2022 20:20:31] gen APIAlerts -> X.X.X.152              bucket_id=snowy-pond capacity=2 cfg=bitter-glitter file=/home/jan/Projects/check24/crowdsec-v1.4.1/tests/config/scenarios/bu-tariffs-by-country.yaml name=bu/tariffs-by-country partition=f2cb3acb8f1dd2fd2753deb9d421ab2c6dd8a548
DEBU[29-07-2022 20:20:31] gen APIAlerts -> X.X.X.150              bucket_id=snowy-pond capacity=2 cfg=bitter-glitter file=/home/jan/Projects/check24/crowdsec-v1.4.1/tests/config/scenarios/bu-tariffs-by-country.yaml name=bu/tariffs-by-country partition=f2cb3acb8f1dd2fd2753deb9d421ab2c6dd8a548
DEBU[29-07-2022 20:20:31] gen APIAlerts -> X.X.X.151              bucket_id=snowy-pond capacity=2 cfg=bitter-glitter file=/home/jan/Projects/check24/crowdsec-v1.4.1/tests/config/scenarios/bu-tariffs-by-country.yaml name=bu/tariffs-by-country partition=f2cb3acb8f1dd2fd2753deb9d421ab2c6dd8a548
DEBU[29-07-2022 20:20:31] Adding overflow to blackhole (2022-07-29 20:15:16 +0000 UTC)  bucket_id=snowy-pond capacity=2 cfg=bitter-glitter file=/home/jan/Projects/check24/crowdsec-v1.4.1/tests/config/scenarios/bu-tariffs-by-country.yaml name=bu/tariffs-by-country partition=f2cb3acb8f1dd2fd2753deb9d421ab2c6dd8a548
DEBU[29-07-2022 20:20:31] Overflow, keep-alive                          bucket_id=snowy-pond capacity=2 cfg=bitter-glitter file=/home/jan/Projects/check24/crowdsec-v1.4.1/tests/config/scenarios/bu-tariffs-by-country.yaml name=bu/tariffs-by-country partition=f2cb3acb8f1dd2fd2753deb9d421ab2c6dd8a548
INFO[29-07-2022 20:20:31] Killing parser routines
DEBU[29-07-2022 20:20:31] eval(evt.Meta.log_type == 'http_access-log' && evt.Enriched.IsoCode != 'DE' && evt.Parsed.request startsWith '/desktopapi/tariffs/') = TRUE  cfg=bitter-glitter file=/home/jan/Projects/check24/crowdsec-v1.4.1/tests/config/scenarios/bu-tariffs-by-country.yaml name=bu/tariffs-by-country
DEBU[29-07-2022 20:20:31] eval variables:                               cfg=bitter-glitter file=/home/jan/Projects/check24/crowdsec-v1.4.1/tests/config/scenarios/bu-tariffs-by-country.yaml name=bu/tariffs-by-country
DEBU[29-07-2022 20:20:31]        evt.Meta.log_type = 'http_access-log'  cfg=bitter-glitter file=/home/jan/Projects/check24/crowdsec-v1.4.1/tests/config/scenarios/bu-tariffs-by-country.yaml name=bu/tariffs-by-country
DEBU[29-07-2022 20:20:31]        evt.Enriched.IsoCode = 'AT'            cfg=bitter-glitter file=/home/jan/Projects/check24/crowdsec-v1.4.1/tests/config/scenarios/bu-tariffs-by-country.yaml name=bu/tariffs-by-country
DEBU[29-07-2022 20:20:31]        evt.Parsed.request = '/desktopapi/tariffs/insurances'  cfg=bitter-glitter file=/home/jan/Projects/check24/crowdsec-v1.4.1/tests/config/scenarios/bu-tariffs-by-country.yaml name=bu/tariffs-by-country
DEBU[29-07-2022 20:20:31] bucket 'bu/tariffs-by-country' is poured      cfg=bitter-glitter file=/home/jan/Projects/check24/crowdsec-v1.4.1/tests/config/scenarios/bu-tariffs-by-country.yaml name=bu/tariffs-by-country
DEBU[29-07-2022 20:20:31] Bucket overflow at 2022-07-29 20:15:19 +0000 UTC  bucket_id=snowy-pond capacity=2 cfg=bitter-glitter file=/home/jan/Projects/check24/crowdsec-v1.4.1/tests/config/scenarios/bu-tariffs-by-country.yaml name=bu/tariffs-by-country partition=f2cb3acb8f1dd2fd2753deb9d421ab2c6dd8a548
DEBU[29-07-2022 20:20:31] BucketConfig.OverflowBehavior will keep bucket alive.  bucket_id=snowy-pond capacity=2 cfg=bitter-glitter file=/home/jan/Projects/check24/crowdsec-v1.4.1/tests/config/scenarios/bu-tariffs-by-country.yaml name=bu/tariffs-by-country partition=f2cb3acb8f1dd2fd2753deb9d421ab2c6dd8a548
DEBU[29-07-2022 20:20:31] source (0xc000263c70): X.X.X.151        bucket_id=snowy-pond capacity=2 cfg=bitter-glitter file=/home/jan/Projects/check24/crowdsec-v1.4.1/tests/config/scenarios/bu-tariffs-by-country.yaml name=bu/tariffs-by-country partition=f2cb3acb8f1dd2fd2753deb9d421ab2c6dd8a548
DEBU[29-07-2022 20:20:31] APIAlerts -> X.X.X.151                  bucket_id=snowy-pond capacity=2 cfg=bitter-glitter file=/home/jan/Projects/check24/crowdsec-v1.4.1/tests/config/scenarios/bu-tariffs-by-country.yaml name=bu/tariffs-by-country partition=f2cb3acb8f1dd2fd2753deb9d421ab2c6dd8a548
DEBU[29-07-2022 20:20:31] source (0xc000263e30): X.X.X.152        bucket_id=snowy-pond capacity=2 cfg=bitter-glitter file=/home/jan/Projects/check24/crowdsec-v1.4.1/tests/config/scenarios/bu-tariffs-by-country.yaml name=bu/tariffs-by-country partition=f2cb3acb8f1dd2fd2753deb9d421ab2c6dd8a548
DEBU[29-07-2022 20:20:31] APIAlerts -> X.X.X.152                  bucket_id=snowy-pond capacity=2 cfg=bitter-glitter file=/home/jan/Projects/check24/crowdsec-v1.4.1/tests/config/scenarios/bu-tariffs-by-country.yaml name=bu/tariffs-by-country partition=f2cb3acb8f1dd2fd2753deb9d421ab2c6dd8a548
DEBU[29-07-2022 20:20:31] source (0xc000286000): X.X.X.153        bucket_id=snowy-pond capacity=2 cfg=bitter-glitter file=/home/jan/Projects/check24/crowdsec-v1.4.1/tests/config/scenarios/bu-tariffs-by-country.yaml name=bu/tariffs-by-country partition=f2cb3acb8f1dd2fd2753deb9d421ab2c6dd8a548
DEBU[29-07-2022 20:20:31] APIAlerts -> X.X.X.153                  bucket_id=snowy-pond capacity=2 cfg=bitter-glitter file=/home/jan/Projects/check24/crowdsec-v1.4.1/tests/config/scenarios/bu-tariffs-by-country.yaml name=bu/tariffs-by-country partition=f2cb3acb8f1dd2fd2753deb9d421ab2c6dd8a548
DEBU[29-07-2022 20:20:31] gen APIAlerts -> X.X.X.151              bucket_id=snowy-pond capacity=2 cfg=bitter-glitter file=/home/jan/Projects/check24/crowdsec-v1.4.1/tests/config/scenarios/bu-tariffs-by-country.yaml name=bu/tariffs-by-country partition=f2cb3acb8f1dd2fd2753deb9d421ab2c6dd8a548
DEBU[29-07-2022 20:20:31] gen APIAlerts -> X.X.X.152              bucket_id=snowy-pond capacity=2 cfg=bitter-glitter file=/home/jan/Projects/check24/crowdsec-v1.4.1/tests/config/scenarios/bu-tariffs-by-country.yaml name=bu/tariffs-by-country partition=f2cb3acb8f1dd2fd2753deb9d421ab2c6dd8a548
DEBU[29-07-2022 20:20:31] gen APIAlerts -> X.X.X.153              bucket_id=snowy-pond capacity=2 cfg=bitter-glitter file=/home/jan/Projects/check24/crowdsec-v1.4.1/tests/config/scenarios/bu-tariffs-by-country.yaml name=bu/tariffs-by-country partition=f2cb3acb8f1dd2fd2753deb9d421ab2c6dd8a548
DEBU[29-07-2022 20:20:31] Overflow discarded, still blackholed for 59s  bucket_id=snowy-pond capacity=2 cfg=bitter-glitter file=/home/jan/Projects/check24/crowdsec-v1.4.1/tests/config/scenarios/bu-tariffs-by-country.yaml name=bu/tariffs-by-country partition=f2cb3acb8f1dd2fd2753deb9d421ab2c6dd8a548
DEBU[29-07-2022 20:20:31] Overflow has been discarded (*leakybucket.Blackhole)  bucket_id=snowy-pond capacity=2 cfg=bitter-glitter file=/home/jan/Projects/check24/crowdsec-v1.4.1/tests/config/scenarios/bu-tariffs-by-country.yaml name=bu/tariffs-by-country partition=f2cb3acb8f1dd2fd2753deb9d421ab2c6dd8a548
INFO[29-07-2022 20:20:31] 127.0.0.1 - [Fri, 29 Jul 2022 20:20:31 CEST] "POST /v1/watchers/login HTTP/1.1 200 78.101878ms "crowdsec/-linux-" "
INFO[29-07-2022 20:20:31] 127.0.0.1 - [Fri, 29 Jul 2022 20:20:31 CEST] "POST /v1/watchers/login HTTP/1.1 200 94.732852ms "crowdsec/-linux-" "
INFO[29-07-2022 20:20:31] Ip 3 sources performed 'bu/tariffs-by-country' (3 events over 2s) at 2022-07-29 20:15:18 +0000 UTC
DEBU[29-07-2022 20:20:31] eval(evt.Meta.log_type == 'http_access-log' && evt.Enriched.IsoCode != 'DE' && evt.Parsed.request startsWith '/desktopapi/tariffs/') = FALSE  cfg=bitter-glitter file=/home/jan/Projects/check24/crowdsec-v1.4.1/tests/config/scenarios/bu-tariffs-by-country.yaml name=bu/tariffs-by-country
DEBU[29-07-2022 20:20:31] Overflow, keep-alive                          bucket_id=snowy-pond capacity=2 cfg=bitter-glitter file=/home/jan/Projects/check24/crowdsec-v1.4.1/tests/config/scenarios/bu-tariffs-by-country.yaml name=bu/tariffs-by-country partition=f2cb3acb8f1dd2fd2753deb9d421ab2c6dd8a548
DEBU[29-07-2022 20:20:31] eval variables:                               cfg=bitter-glitter file=/home/jan/Projects/check24/crowdsec-v1.4.1/tests/config/scenarios/bu-tariffs-by-country.yaml name=bu/tariffs-by-country
DEBU[29-07-2022 20:20:31]        evt.Meta.log_type = ''                 cfg=bitter-glitter file=/home/jan/Projects/check24/crowdsec-v1.4.1/tests/config/scenarios/bu-tariffs-by-country.yaml name=bu/tariffs-by-country
DEBU[29-07-2022 20:20:31]        evt.Enriched.IsoCode = ''              cfg=bitter-glitter file=/home/jan/Projects/check24/crowdsec-v1.4.1/tests/config/scenarios/bu-tariffs-by-country.yaml name=bu/tariffs-by-country
DEBU[29-07-2022 20:20:31]        evt.Parsed.request = ''                cfg=bitter-glitter file=/home/jan/Projects/check24/crowdsec-v1.4.1/tests/config/scenarios/bu-tariffs-by-country.yaml name=bu/tariffs-by-country
DEBU[29-07-2022 20:20:31] Event leaving node : ko (filter mismatch)     cfg=bitter-glitter file=/home/jan/Projects/check24/crowdsec-v1.4.1/tests/config/scenarios/bu-tariffs-by-country.yaml name=bu/tariffs-by-country
INFO[29-07-2022 20:20:32] processing alert : X.X.X.152
INFO[29-07-2022 20:20:32] processing alert : X.X.X.152
INFO[29-07-2022 20:20:32] processing alert : X.X.X.152
INFO[29-07-2022 20:20:32] (test/crowdsec) bu/tariffs-by-country by ip X.X.X.152 (AT/8447) : 5m captcha on Ip X.X.X.152

What I can see here, is, that at the end of the processing it's creating three alerts, but with the same IP. I would expect 2 alerts for IP 152 and 153. Finally there was one decision for a captcha for 152, which is correct, but I miss the second decision for 153.

janbaer avatar Jul 29 '22 18:07 janbaer

@buixor Do you see any way, to fix that problem? I guess it's not a fault of our change. It's something internal and how Crowdsec is treating the IP address that is causing the overflow. It seems that it is not expecting that the lifetime of a bucket can be longer than after the first overflow.

janbaer avatar Aug 03 '22 06:08 janbaer

@buixor I just wanted to ask if there is any chance to bring to feature in any of the next versions? We still have the issue, that a Crawler is using multiple IP addresses at the same time to scan our websites, but not reach any limit with a single IP address. Crowdsec can't help here at the moment, and we can't detect and block such IP addresses with using Crowdsec. I wonder, that no one else has the same issue.

janbaer avatar Jan 28 '23 10:01 janbaer