help request: high cache usage when using Kafka-logger, Prometheus plugins

[wolgod]

Description

My apisix is ​​deployed in a k8s, 12g of memory, rss uses about 40%, but the cache is about 10g, only the global promethues and kafka-logger plug-ins are enabled, after restarting, the cache curve slowly grows from 0 to about 10g Slowly and gradually, in theory, the cache will be reused, and it will not cause oom. I want to know what causes such a high cache, will it have any impact on apisix, is the parameter configuration incorrect, or kafka-logger, or other reasons ？

Environment

• APISIX version (run apisix version):2.9
• Operating system (run uname -a): el7.x86_64
• OpenResty / Nginx version (run openresty -V or nginx -V):openresty/1.19.9.1
• etcd version, if relevant (run curl http://127.0.0.1:9090/v1/server_info):3.5.0
• APISIX Dashboard version, if relevant:
• Plugin runner version, for issues related to plugin runners:
• LuaRocks version, for installation issues (run luarocks --version):

[spacewander]

Is it relative to https://github.com/apache/apisix/pull/5700? Would you try 2.15.0 to see if the problem still exists?

[wolgod]

Now it seems that it should be caused by writing the accesslog locally. Delete the accesslog log and the cache will be reduced.

[tzssangglass]

Now it seems that it should be caused by writing the accesslog locally. Delete the accesslog log and the cache will be reduced.

You can see the size of the access.log.

[wolgod]

Now it seems that it should be caused by writing the accesslog locally. Delete the accesslog log and the cache will be reduced.

You can see the size of the access.log.

Logrotate splits the log, one per hour, but the access-log is only about 2g, but the rest of the split access-log will also occupy the cache, unless which logs are manually deleted

[tzssangglass]

but the rest of the split access-log will also occupy the cache, unless which logs are manually deleted

we can use max_kept in log-rotate plugins to delete old logs files. ref: https://github.com/apache/apisix/blob/master/docs/en/latest/plugins/log-rotate.md

[wolgod]

but the rest of the split access-log will also occupy the cache, unless which logs are manually deleted

we can use max_kept in log-rotate plugins to delete old logs files. ref: https://github.com/apache/apisix/blob/master/docs/en/latest/plugins/log-rotate.md

yes, but I'm on 4 5g per hour, unless it's only reserved for an hour, the cache will still be high

[tzssangglass]

yes, but I'm on 4 5g per hour, unless it's only reserved for an hour, the cache will still be high

Very heavy usage, in which case I suggest you turn off access log (if feasible)

[wolgod]

yes,now i turn off access log, and use kafka-logger to send log