performance: A large number of nodes in service discovery issue higher CPU usage

[yongboy]

Current Behavior

When your high-traffic services lie on serivce discovery in Apisix such as consul_kv or other, with hundreds and thousands upstream nodes, maybe issue the higher CPU usage twice or more than using nginx online.

The Apisix is 2.3 our using, with the help of CPU flame graph above, you can find the upstream_util.compare_upstream_node function covers a lot CPU.

When one request arrives, the route's upstream configed with service name, need to execute one comparison, as below.

    if up_conf.service_name then
        ......
        local dis = discovery[up_conf.discovery_type]
        ......
        local new_nodes = dis.nodes(up_conf.service_name)
        local same = upstream_util.compare_upstream_node(up_conf.nodes, new_nodes)
        if not same then
            ......
        end
    end

Our radical way, saves one comparison computation:

First, import a version for nodes():

function _M.nodes(service_name)
    ......
    local obj = applications[service_name]
    local resp_list = obj and obj.nodes
    ......
    local resp_ver = obj.ver or 0
    ......

    return resp_list, resp_ver
end

Second, compare the nodes version:

    if up_conf.service_name then
        ......
        local dis = discovery[up_conf.discovery_type]
        ......
        local new_nodes, nodes_ver = dis.nodes(up_conf.service_name)
        local same = up_conf.nodes_ver == nodes_ver
        if not same then
            up_conf.nodes_ver = nodes_ver
            ......
        end
    end

After appling these changes, the Apisix's CPU uage is close to the nginx now.

Expected Behavior

No response

Error Logs

No response

Steps to Reproduce

Run Apisix online

Environment

• APISIX version 2.*