bug: Apaceh apisix taking more than 2 seconds for some of the requests

[hemakumar-ganumukkala]

Current Behavior

There are many requests are taking more than 50ms and some even seconds.

See below log information showing the apisix lua script took 2 sec .

2024/02/07 01:04:06 [info] 86#86: *33815 [lua] init.lua:636: http_access_phase(): 2024/02/07 01:04:08 [info] 86#86: *33815 [lua] client.lua:123: dns_parse():

Above error log is info log generated by the apache apisix , Log format can be configured only for the access log application request log which can print the in msec but on the error log . But it will not solve problem to debug which apisix lua script have taken the time in milli seconds . The latency which around 30ms - 1000ms are difficult to trace without milli sec being enabled

issue is with current apisix 3.8.0 version and even 3.7.0 too.

Expected Behavior

Apache apisix should not be having any performance issues , having more than 100 ms is a problem.

Error Logs

2024/02/07 01:04:06 [info] 86#86: *33815 [lua] init.lua:636: http_access_phase(): 2024/02/07 01:04:08 [info] 86#86: *33815 [lua] client.lua:123: dns_parse():

Steps to Reproduce

Configure a routes and run a load test of 100 req/sec run for 2 sec .

Environment

Apisix version 3.8 app.kubernetes.io/instance: apisix app.kubernetes.io/managed-by: Helm app.kubernetes.io/name: apisix app.kubernetes.io/version: 3.8.0 helm.sh/chart: apisix-2.6.0

[hanqingwu]

from you apisix log, can you check the dns resolve does work normal ? maybe the 2s time cost in dns resolved .

[juzhiyuan]

@hemakumar-ganumukkala check https://api7.ai/blog/api7-latency post please

[tiwarivikash]

We have observed similar behavior with Kong's DNS client library which APISIX uses (fork). Every once in a while DNS resolution takes up to 2-3 seconds. Reference - https://github.com/Kong/kong/pull/11869

[eduardoweiland]

I am currently testing apisix in a local Docker environment and was having this same issue. After some debugging and looking through the code, I found out that passing the option no_recurse = true to the resty-dns-client it works very well, at least for resolving Docker container names inside the same network.

Unfortunately, there is no way to specify this option in the config right now, as both DNS discovery and default DNS client pass a hardcoded options object for the client init. Maybe this option could be exposed as an apisix config?