apache
help request: limit-req cannot limit access
Description
May I ask if the implementation principles of apisix limit req and nginx are the same? According to the official documentation, the leaky bucket algorithm is used. If the rate is 1000/s burst=0 nodelay, The number of access entries for nginx through access log grep 200 is less than the rate value, while apisix is much greater than the rate value
Referring to: https://github.com/alibaba/tengine/issues/855 https://github.com/openresty/lua-resty-limit-traffic/blob/master/lib/resty/limit/req.md https://groups.google.com/g/openresty/c/VY-LdQaEyDA/m/skf86NDHAAAJ?pli=1
route
{
"uri": "/rate",
"name": "rate-limit",
"priority": 100,
"host": "my-yace.com",
"plugins": {
"limit-req": {
"burst": 1,
"disable": false,
"nodelay": true,
"key": "server_addr",
"rate": 1000,
"rejected_code": 503
}
},
"upstream": {
"nodes": [
{
"host": "10.120.0.104",
"port": 80,
"weight": 1
}
],
"timeout": {
"connect": 6,
"send": 300,
"read": 300
},
"type": "roundrobin",
"scheme": "http",
"pass_host": "pass",
"keepalive_pool": {
"idle_timeout": 300,
"requests": 1000,
"size": 1000
}
},
"status": 1
}
1、Using WRK testing
Running 1m test @ http://10.120.0.113:9080/rate
2 threads and 2000 connections
Thread calibration: mean lat.: 5041.922ms, rate sampling interval: 21381ms
Thread calibration: mean lat.: 5870.648ms, rate sampling interval: 21889ms
Thread Stats Avg Stdev Max +/- Stdev
Latency 33.93s 12.77s 0.96m 57.86%
Req/Sec 10.22k 38.16 10.26k 50.00%
1250705 requests in 1.00m, 313.49MB read
Socket errors: connect 0, read 3, write 628, timeout 2567
Non-2xx or 3xx responses: 421753
Requests/sec: 20844.28
Transfer/sec: 5.22MB
2、Confirm that the request has arrived at Apisix
\* About to connect() to 10.120.0.113 port 9080 (#0)
\* Trying 10.120.0.113...
\* Connected to 10.120.0.113 (10.120.0.113) port 9080 (#0)
\> GET /rate HTTP/1.1
\> User-Agent: curl/7.29.0
\> Accept: */*
\> Host: my-yace.com
\>
< HTTP/1.1 200 OK
< Content-Type: text/plain; charset=utf-8
< Content-Length: 0
< Connection: keep-alive
< Date: Fri, 24 May 2024 06:48:09 GMT
< Server: APISIX/3.2.2
<
\* Connection #0 to host 10.120.0.113 left intact
3 Number of grep log entries
grep 'GET /rate HTTP/1.1' access.log | grep 200 | grep '14:4' | awk '{print $4}' | sort | uniq -c
9334 [24/May/2024:14:40:00
9833 [24/May/2024:14:40:01
14500 [24/May/2024:14:40:02
14340 [24/May/2024:14:40:03
14648 [24/May/2024:14:40:04
14445 [24/May/2024:14:40:05
14849 [24/May/2024:14:40:06
13449 [24/May/2024:14:40:07
13938 [24/May/2024:14:40:08
13796 [24/May/2024:14:40:09
14089 [24/May/2024:14:40:10
14876 [24/May/2024:14:40:11
14499 [24/May/2024:14:40:12
12827 [24/May/2024:14:40:13
14015 [24/May/2024:14:40:14
14330 [24/May/2024:14:40:15
12880 [24/May/2024:14:40:16
13783 [24/May/2024:14:40:17
13856 [24/May/2024:14:40:18
14909 [24/May/2024:14:40:19
10330 [24/May/2024:14:40:20
10198 [24/May/2024:14:40:21
13245 [24/May/2024:14:40:22
13757 [24/May/2024:14:40:23
13998 [24/May/2024:14:40:24
10176 [24/May/2024:14:40:25
1478 [24/May/2024:14:40:26
- Log fragments
4.1 access
10.120.0.104 - - [24/May/2024:14:39:34 +0800] my-yace.com "GET /rate HTTP/1.1" 503 269 0.000 "-" "-" - - - "http://my-yace.com"
10.120.0.104 - - [24/May/2024:14:39:34 +0800] my-yace.com "GET /rate HTTP/1.1" 200 0 0.027 "-" "-" 10.120.0.104:80 200 0.027 "http://my-yace.com"
10.120.0.104 - - [24/May/2024:14:39:34 +0800] my-yace.com "GET /rate HTTP/1.1" 503 269 0.000 "-" "-" - - - "http://my-yace.com"
10.120.0.104 - - [24/May/2024:14:39:34 +0800] my-yace.com "GET /rate HTTP/1.1" 503 269 0.000 "-" "-" - - - "http://my-yace.com"
10.120.0.104 - - [24/May/2024:14:39:34 +0800] my-yace.com "GET /rate HTTP/1.1" 503 269 0.000 "-" "-" - - - "http://my-yace.com"
10.120.0.104 - - [24/May/2024:14:39:34 +0800] my-yace.com "GET /rate HTTP/1.1" 200 0 0.027 "-" "-" 10.120.0.104:80 200 0.027 "http://my-yace.com"
10.120.0.104 - - [24/May/2024:14:39:34 +0800] my-yace.com "GET /rate HTTP/1.1" 200 0 0.027 "-" "-" 10.120.0.104:80 200 0.027 "http://my-yace.com"
10.120.0.104 - - [24/May/2024:14:39:34 +0800] my-yace.com "GET /rate HTTP/1.1" 200 0 0.027 "-" "-" 10.120.0.104:80 200 0.027 "http://my-yace.com"
10.120.0.104 - - [24/May/2024:14:39:34 +0800] my-yace.com "GET /rate HTTP/1.1" 200 0 0.027 "-" "-" 10.120.0.104:80 200 0.027 "http://my-yace.com"
10.120.0.104 - - [24/May/2024:14:39:34 +0800] my-yace.com "GET /rate HTTP/1.1" 200 0 0.027 "-" "-" 10.120.0.104:80 200 0.027 "http://my-yace.com"
4.2、 error
2024/05/24 14:40:26 [warn] 6913#6913: *83358 [lua] plugin.lua:1100: run_plugin(): limit-req exits with http status code 503, client: 10.120.0.104, server: _, request: "GET /rate HTTP/1.1", host: "my-yace.com"
2024/05/24 14:40:26 [warn] 6913#6913: *92234 [lua] plugin.lua:1100: run_plugin(): limit-req exits with http status code 503, client: 10.120.0.104, server: _, request: "GET /rate HTTP/1.1", host: "my-yace.com"
2024/05/24 14:40:26 [warn] 6913#6913: *80673 [lua] plugin.lua:1100: run_plugin(): limit-req exits with http status code 503, client: 10.120.0.104, server: _, request: "GET /rate HTTP/1.1", host: "my-yace.com"
2024/05/24 14:40:26 [warn] 6912#6912: *82365 [lua] plugin.lua:1100: run_plugin(): limit-req exits with http status code 503, client: 10.120.0.104, server: _, request: "GET /rate HTTP/1.1", host: "my-yace.com"
2024/05/24 14:40:26 [warn] 6913#6913: *86209 [lua] plugin.lua:1100: run_plugin(): limit-req exits with http status code 503, client: 10.120.0.104, server: _, request: "GET /rate HTTP/1.1", host: "my-yace.com"
2024/05/24 14:40:26 [warn] 6913#6913: *80674 [lua] plugin.lua:1100: run_plugin(): limit-req exits with http status code 503, client: 10.120.0.104, server: _, request: "GET /rate HTTP/1.1", host: "my-yace.com"
2024/05/24 14:40:26 [warn] 6913#6913: *83544 [lua] plugin.lua:1100: run_plugin(): limit-req exits with http status code 503, client: 10.120.0.104, server: _, request: "GET /rate HTTP/1.1", host: "my-yace.com"
2024/05/24 14:40:26 [warn] 6913#6913: *83847 [lua] plugin.lua:1100: run_plugin(): limit-req exits with http status code 503, client: 10.120.0.104, server: _, request: "GET /rate HTTP/1.1", host: "my-yace.com"
Environment
-
APISIX version (run
apisix version): 3.2.2 -
Operating system (run
uname -a): Linux nginx-gray02 3.10.0-1160.31.1.el7.x86_64 #1 SMP Thu Jun 10 13:32:12 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux -
OpenResty / Nginx version (run
openresty -Vornginx -V): -
nginx version: openresty/1.21.4.2 built by gcc 9.3.1 20200408 (Red Hat 9.3.1-2) (GCC) built with OpenSSL 1.1.1s 1 Nov 2022 (running with OpenSSL 1.1.1w 11 Sep 2023) TLS SNI support enabled configure arguments: --prefix=/usr/local/openresty/nginx --with-cc-opt='-O2 -DAPISIX_BASE_VER=1.21.4.2.2 -DNGX_GRPC_CLI_ENGINE_PATH=/usr/local/openresty/libgrpc_engine.so -DNGX_HTTP_GRPC_CLI_ENGINE_PATH=/usr/local/openresty/libgrpc_engine.so -DNGX_LUA_ABORT_AT_PANIC -I/usr/local/openresty/zlib/include -I/usr/local/openresty/pcre/include -I/usr/local/openresty/openssl111/include' --add-module=../ngx_devel_kit-0.3.2 --add-module=../echo-nginx-module-0.63 --add-module=../xss-nginx-module-0.06 --add-module=../ngx_coolkit-0.2 --add-module=../set-misc-nginx-module-0.33 --add-module=../form-input-nginx-module-0.12 --add-module=../encrypted-session-nginx-module-0.09 --add-module=../srcache-nginx-module-0.33 --add-module=../ngx_lua-0.10.25 --add-module=../ngx_lua_upstream-0.07 --add-module=../headers-more-nginx-module-0.34 --add-module=../array-var-nginx-module-0.06 --add-module=../memc-nginx-module-0.19 --add-module=../redis2-nginx-module-0.15 --add-module=../redis-nginx-module-0.3.9 --add-module=../ngx_stream_lua-0.0.13 --with-ld-opt='-Wl,-rpath,/usr/local/openresty/luajit/lib -Wl,-rpath,/usr/local/openresty/wasmtime-c-api/lib -L/usr/local/openresty/zlib/lib -L/usr/local/openresty/pcre/lib -L/usr/local/openresty/openssl111/lib -Wl,-rpath,/usr/local/openresty/zlib/lib:/usr/local/openresty/pcre/lib:/usr/local/openresty/openssl111/lib' --add-module=/tmp/tmp.2MSCN59HKX/openresty-1.21.4.2/../mod_dubbo-1.0.2 --add-module=/tmp/tmp.2MSCN59HKX/openresty-1.21.4.2/../ngx_multi_upstream_module-1.1.1 --add-module=/tmp/tmp.2MSCN59HKX/openresty-1.21.4.2/../apisix-nginx-module-1.15.0 --add-module=/tmp/tmp.2MSCN59HKX/openresty-1.21.4.2/../apisix-nginx-module-1.15.0/src/stream --add-module=/tmp/tmp.2MSCN59HKX/openresty-1.21.4.2/../apisix-nginx-module-1.15.0/src/meta --add-module=/tmp/tmp.2MSCN59HKX/openresty-1.21.4.2/../wasm-nginx-module-0.6.5 --add-module=/tmp/tmp.2MSCN59HKX/openresty-1.21.4.2/../lua-var-nginx-module-v0.5.3 --add-module=/tmp/tmp.2MSCN59HKX/openresty-1.21.4.2/../grpc-client-nginx-module-v0.4.4 --with-poll_module --with-pcre-jit --with-stream --with-stream_ssl_module --with-stream_ssl_preread_module --with-http_v2_module --without-mail_pop3_module --without-mail_imap_module --without-mail_smtp_module --with-http_stub_status_module --with-http_realip_module --with-http_addition_module --with-http_auth_request_module --with-http_secure_link_module --with-http_random_index_module --with-http_gzip_static_module --with-http_sub_module --with-http_dav_module --with-http_flv_module --with-http_mp4_module --with-http_gunzip_module --with-threads --with-compat --with-stream --with-http_ssl_module
-
etcd version, if relevant (run
curl http://127.0.0.1:9090/v1/server_info): {"id":"66356a45-bcc5-4a5a-9318-210342728ebd","boot_time":1716530994,"version":"3.2.2","etcd_version":"3.4.0","hostname":"nginx-gray02"} -
APISIX Dashboard version, if relevant:
-
Plugin runner version, for issues related to plugin runners:
-
LuaRocks version, for installation issues (run
luarocks --version):
By adding logs in/openresty/liulib/resty/limit/req.lua, the calculated elapsed has a complex number when the request is fast, After calculating the ABS function, Excess is 0
1 grep error log
When using OpenResty Limit Req, Set the rate to 200 Burst setting 1, after using WRK stress testing, I calculated that the number of 200 status entries in the access log is less than the rate （ see https://github.com/bingoohuang/blog/issues/209）
lua_shared_dict my_limit_req_store 100m;
server { listen 8700; default_type text/html;
location / {
return 200 OK;
}
location /limit {
access_by_lua_block {
local limit_req = require "resty.limit.req"
-- 限制每秒 200 个请求 (rate)，以及 100 的等待队列 （burst), 超过每次 300，直接拒绝
local rate = tonumber(ngx.var.arg_rate or 200)
local burst = tonumber(ngx.var.arg_burst or 100)
local lim, err = limit_req.new("my_limit_req_store", rate, burst)
if not lim then
ngx.log(ngx.ERR, "failed to instantiate a resty.limit.req object: ", err)
return ngx.exit(500)
end
-- 以远端IP为限制 key
local delay, excess = lim:incoming(ngx.var.binary_remote_addr, true)
if excess == "rejected" then
ngx.log(ngx.ERR, "rejected")
-- ngx.say, ngx.print 发送消息包，必然要先发送消息头，所以需要提前设置 response status （默认为200）
ngx.status = 503
ngx.say("XX delay:", delay, ", rate: ", rate, ", burst: ", burst)
return ngx.exit(503)
end
ngx.say("OK delay: ", delay, ", rate: ", rate, ", burst: ", burst, ", excess: ", excess)
return ngx.exit(200)
}
}
}
I have the same problem when using limit-conn plugin.
"limit-conn": {
"_meta": {
"disable": false
},
"allow_degradation": true,
"burst": 0,
"conn": 700,
"default_conn_delay": 0.01,
"key": "server_addr",
"key_type": "var",
"rejected_code": 429,
"rejected_msg": "{\"error_msg\":\"Requests are too frequent, please try again later.\"}"
},
I set conn to 700, but even as the single-node qps grew to more than 900, the limit-conn plugin still didn't work, I think it's a sample problem.
Can community help to check?
I have the same problem when using limit-conn plugin.
"limit-conn": { "_meta": { "disable": false }, "allow_degradation": true, "burst": 0, "conn": 700, "default_conn_delay": 0.01, "key": "server_addr", "key_type": "var", "rejected_code": 429, "rejected_msg": "{\"error_msg\":\"Requests are too frequent, please try again later.\"}" },I set conn to 700, but even as the single-node qps grew to more than 900, the limit-conn plugin still didn't work, I think it's a sample problem.
Can community help to check?
Ignore please. sry
Hi, any update ? I have the same problem where limit-req plugin in not working as intended and not able to block the request.
see https://github.com/openresty/lua-resty-limit-traffic/blob/master/lib/resty/limit/req.lua#L68C3-L68C4 @rahulracker7539 @dimon-v How many workers are running？
it was silly mistake from my side, had add my plugin in the plugin section of configmap so it can install and load to the apisix instance. I was just doing from apisix dashboard which not able to install automatically . thanks
i config rate 3000 burst 1000, then use wrk -t 1200 -c 3000 test,the qps cannot limited ,the real qps is 20000qps
see https://github.com/openresty/lua-resty-limit-traffic/blob/master/lib/resty/limit/req.lua#L68C3-L68C4 @rahulracker7539 @dimon-v How many workers are running？
Based on my recent tests and research using this feature, I found that when the concurrency of multiple workers gets high, there is a race condition in updating the rate limiting information, which ultimately leads to a significant margin of error in rate limiting. In light of this situation, are there any ideas for optimization?
