carbonapi icon indicating copy to clipboard operation
carbonapi copied to clipboard
verified publisher icon go-graphite

[BUG] Send multiple targets in MultiFetchRequest

Open jonasbleyl opened this issue 4 years ago • 7 comments

Describe the bug When querying CarbonAPI with multiple targets it will sequentially send off a request for each target.

Can CarbonAPI be configured to simply send a single request containing an array of the targets in the MultiFetchRequest?

CarbonAPI Version main

CarbonAPI Configuration:

listen: "localhost:9090"
upstreams:
  backendsv2:
    backends:
      - protocol: "carbonapi_v3_pb"
        maxBatchSize: 0
        lbMethod: "rr"
        servers:
          - "http://localhost:5000"

Additional context

I've implemented the carbonapi_v3_pb protocol in a custom backend but MultiFetchRequest.Metrics will only ever have a size of 1 no matter how I configure CarbonAPI.

Alternatively, is there a way to query for the targets for a single request in parallel?

jonasbleyl avatar Jul 15 '21 15:07 jonasbleyl

Carbonapi should send multiple metrics in a request in following cases:

  1. When you specify multiple target= parameters
  2. When you have maxBatchSize != 0

As idea behind maxBatchSize=0 was not to expand globs on carbonapi side (carbonapi doesn't have knowledge about how to expand globs). If you can explain your use case in more details, maybe I will be able to suggest something or at least to understand better the requirements here (for FRs though I don't have any estimations when I'll have time to implement it, but I would gladly help to review a PR that would add an option that do what you want).

Civil avatar Jul 15 '21 16:07 Civil

Thanks @Civil

When you specify multiple target= parameters

This is what I assumed as well, but for some reason this is not the behaviour I'm seeing.

When I make a request with multiple targets e.g.

curl "localhost:9090/render?target=one1.two1.three1&target=one2.two2.three2&target=one3.two3.three3&from=1546299999&until=1546301700&format=json"

My backend receives 3 separate requests one after the other with just 1 target in the MultiFetchRequest.

Here are the debug logs when I make that request:

2021-07-15T17:51:07.405+0100    DEBUG   zipper  will try to fetch data  {"type": "broadcastGroup", "groupName": "root", "type": "fetch", "request": ["one1.two1.three1"], "carbonapi_uuid": "1bf17818-24b5-4494-b6db-599b5f400f38"}
2021-07-15T17:51:07.405+0100    DEBUG   zipper  single fetch    {"type": "broadcastGroup", "groupName": "root", "type": "fetch", "request": ["one1.two1.three1"], "carbonapi_uuid": "1bf17818-24b5-4494-b6db-599b5f400f38", "client": {}}
2021-07-15T17:51:07.405+0100    DEBUG   zipper  waiting for slot        {"type": "broadcastGroup", "groupName": "root", "type": "fetch", "request": ["one1.two1.three1"], "carbonapi_uuid": "1bf17818-24b5-4494-b6db-599b5f400f38", "multi_fetch": false, "backend_name": "", "max_connections": 0}
2021-07-15T17:51:07.405+0100    DEBUG   zipper  got slot        {"type": "broadcastGroup", "groupName": "root", "type": "fetch", "request": ["one1.two1.three1"], "carbonapi_uuid": "1bf17818-24b5-4494-b6db-599b5f400f38", "multi_fetch": false, "backend_name": ""}
2021-07-15T17:51:07.405+0100    DEBUG   zipper  sending request {"type": "broadcastGroup", "groupName": "root", "type": "fetch", "request": ["one1.two1.three1"], "carbonapi_uuid": "1bf17818-24b5-4494-b6db-599b5f400f38", "multi_fetch": false, "backend_name": ""}
2021-07-15T17:51:07.405+0100    DEBUG   zipper  will try to fetch data  {"type": "broadcastGroup", "groupName": "", "type": "fetch", "request": ["one1.two1.three1"], "carbonapi_uuid": "1bf17818-24b5-4494-b6db-599b5f400f38"}
2021-07-15T17:51:07.405+0100    DEBUG   zipper  single fetch    {"type": "broadcastGroup", "groupName": "", "type": "fetch", "request": ["one1.two1.three1"], "carbonapi_uuid": "1bf17818-24b5-4494-b6db-599b5f400f38", "client": {}}
2021-07-15T17:51:07.405+0100    DEBUG   zipper  waiting for slot        {"type": "broadcastGroup", "groupName": "", "type": "fetch", "request": ["one1.two1.three1"], "carbonapi_uuid": "1bf17818-24b5-4494-b6db-599b5f400f38", "multi_fetch": false, "backend_name": "http://localhost:5000", "max_connections": 0}
2021-07-15T17:51:07.405+0100    DEBUG   zipper  got slot        {"type": "broadcastGroup", "groupName": "", "type": "fetch", "request": ["one1.two1.three1"], "carbonapi_uuid": "1bf17818-24b5-4494-b6db-599b5f400f38", "multi_fetch": false, "backend_name": "http://localhost:5000"}
2021-07-15T17:51:07.405+0100    DEBUG   zipper  sending request {"type": "broadcastGroup", "groupName": "", "type": "fetch", "request": ["one1.two1.three1"], "carbonapi_uuid": "1bf17818-24b5-4494-b6db-599b5f400f38", "multi_fetch": false, "backend_name": "http://localhost:5000"}
2021-07-15T17:51:07.405+0100    DEBUG   zipper  trying to get slot      {"type": "protoV3Group", "name": "http://localhost:5000", "type": "fetch", "request": "&MultiFetchRequest{Metrics:[{one1.two1.three1 1546299999 1546301700 false one1.two1.three1 [] 0}],}", "function": "HttpQuery.doRequest", "server": "http://localhost:5000", "name": "http://localhost:5000", "uri": "http://localhost:5000/render/?format=carbonapi_v3_pb", "name": "http://localhost:5000"}
2021-07-15T17:51:07.405+0100    DEBUG   zipper  got slot for server     {"type": "protoV3Group", "name": "http://localhost:5000", "type": "fetch", "request": "&MultiFetchRequest{Metrics:[{one1.two1.three1 1546299999 1546301700 false one1.two1.three1 [] 0}],}", "function": "HttpQuery.doRequest", "server": "http://localhost:5000", "name": "http://localhost:5000", "uri": "http://localhost:5000/render/?format=carbonapi_v3_pb", "name": "http://localhost:5000"}
2021-07-15T17:51:09.413+0100    DEBUG   zipper  got response    {"type": "broadcastGroup", "groupName": "", "type": "fetch", "request": ["one1.two1.three1"], "carbonapi_uuid": "1bf17818-24b5-4494-b6db-599b5f400f38", "multi_fetch": false, "backend_name": "http://localhost:5000"}
2021-07-15T17:51:09.413+0100    DEBUG   zipper  got some fetch responses        {"type": "broadcastGroup", "groupName": "", "type": "fetch", "request": ["one1.two1.three1"], "carbonapi_uuid": "1bf17818-24b5-4494-b6db-599b5f400f38", "backends_count": 1, "response_count": 1, "have_errors": false, "errors": null, "response_count": 1}
2021-07-15T17:51:09.413+0100    DEBUG   zipper  got response    {"type": "broadcastGroup", "groupName": "root", "type": "fetch", "request": ["one1.two1.three1"], "carbonapi_uuid": "1bf17818-24b5-4494-b6db-599b5f400f38", "multi_fetch": false, "backend_name": ""}
2021-07-15T17:51:09.414+0100    DEBUG   zipper  got some fetch responses        {"type": "broadcastGroup", "groupName": "root", "type": "fetch", "request": ["one1.two1.three1"], "carbonapi_uuid": "1bf17818-24b5-4494-b6db-599b5f400f38", "backends_count": 1, "response_count": 1, "have_errors": false, "errors": null, "response_count": 1}
2021-07-15T17:51:09.414+0100    DEBUG   zipper  will try to fetch data  {"type": "broadcastGroup", "groupName": "root", "type": "fetch", "request": ["one2.two2.three2"], "carbonapi_uuid": "1bf17818-24b5-4494-b6db-599b5f400f38"}
2021-07-15T17:51:09.414+0100    DEBUG   zipper  single fetch    {"type": "broadcastGroup", "groupName": "root", "type": "fetch", "request": ["one2.two2.three2"], "carbonapi_uuid": "1bf17818-24b5-4494-b6db-599b5f400f38", "client": {}}
2021-07-15T17:51:09.414+0100    DEBUG   zipper  waiting for slot        {"type": "broadcastGroup", "groupName": "root", "type": "fetch", "request": ["one2.two2.three2"], "carbonapi_uuid": "1bf17818-24b5-4494-b6db-599b5f400f38", "multi_fetch": false, "backend_name": "", "max_connections": 0}
2021-07-15T17:51:09.414+0100    DEBUG   zipper  got slot        {"type": "broadcastGroup", "groupName": "root", "type": "fetch", "request": ["one2.two2.three2"], "carbonapi_uuid": "1bf17818-24b5-4494-b6db-599b5f400f38", "multi_fetch": false, "backend_name": ""}
2021-07-15T17:51:09.414+0100    DEBUG   zipper  sending request {"type": "broadcastGroup", "groupName": "root", "type": "fetch", "request": ["one2.two2.three2"], "carbonapi_uuid": "1bf17818-24b5-4494-b6db-599b5f400f38", "multi_fetch": false, "backend_name": ""}
2021-07-15T17:51:09.414+0100    DEBUG   zipper  will try to fetch data  {"type": "broadcastGroup", "groupName": "", "type": "fetch", "request": ["one2.two2.three2"], "carbonapi_uuid": "1bf17818-24b5-4494-b6db-599b5f400f38"}
2021-07-15T17:51:09.414+0100    DEBUG   zipper  single fetch    {"type": "broadcastGroup", "groupName": "", "type": "fetch", "request": ["one2.two2.three2"], "carbonapi_uuid": "1bf17818-24b5-4494-b6db-599b5f400f38", "client": {}}
2021-07-15T17:51:09.414+0100    DEBUG   zipper  waiting for slot        {"type": "broadcastGroup", "groupName": "", "type": "fetch", "request": ["one2.two2.three2"], "carbonapi_uuid": "1bf17818-24b5-4494-b6db-599b5f400f38", "multi_fetch": false, "backend_name": "http://localhost:5000", "max_connections": 0}
2021-07-15T17:51:09.414+0100    DEBUG   zipper  got slot        {"type": "broadcastGroup", "groupName": "", "type": "fetch", "request": ["one2.two2.three2"], "carbonapi_uuid": "1bf17818-24b5-4494-b6db-599b5f400f38", "multi_fetch": false, "backend_name": "http://localhost:5000"}
2021-07-15T17:51:09.414+0100    DEBUG   zipper  sending request {"type": "broadcastGroup", "groupName": "", "type": "fetch", "request": ["one2.two2.three2"], "carbonapi_uuid": "1bf17818-24b5-4494-b6db-599b5f400f38", "multi_fetch": false, "backend_name": "http://localhost:5000"}
2021-07-15T17:51:09.415+0100    DEBUG   zipper  trying to get slot      {"type": "protoV3Group", "name": "http://localhost:5000", "type": "fetch", "request": "&MultiFetchRequest{Metrics:[{one2.two2.three2 1546299999 1546301700 false one2.two2.three2 [] 0}],}", "function": "HttpQuery.doRequest", "server": "http://localhost:5000", "name": "http://localhost:5000", "uri": "http://localhost:5000/render/?format=carbonapi_v3_pb", "name": "http://localhost:5000"}
2021-07-15T17:51:09.415+0100    DEBUG   zipper  got slot for server     {"type": "protoV3Group", "name": "http://localhost:5000", "type": "fetch", "request": "&MultiFetchRequest{Metrics:[{one2.two2.three2 1546299999 1546301700 false one2.two2.three2 [] 0}],}", "function": "HttpQuery.doRequest", "server": "http://localhost:5000", "name": "http://localhost:5000", "uri": "http://localhost:5000/render/?format=carbonapi_v3_pb", "name": "http://localhost:5000"}
2021-07-15T17:51:11.419+0100    DEBUG   zipper  got response    {"type": "broadcastGroup", "groupName": "", "type": "fetch", "request": ["one2.two2.three2"], "carbonapi_uuid": "1bf17818-24b5-4494-b6db-599b5f400f38", "multi_fetch": false, "backend_name": "http://localhost:5000"}
2021-07-15T17:51:11.419+0100    DEBUG   zipper  got some fetch responses        {"type": "broadcastGroup", "groupName": "", "type": "fetch", "request": ["one2.two2.three2"], "carbonapi_uuid": "1bf17818-24b5-4494-b6db-599b5f400f38", "backends_count": 1, "response_count": 1, "have_errors": false, "errors": null, "response_count": 1}
2021-07-15T17:51:11.419+0100    DEBUG   zipper  got response    {"type": "broadcastGroup", "groupName": "root", "type": "fetch", "request": ["one2.two2.three2"], "carbonapi_uuid": "1bf17818-24b5-4494-b6db-599b5f400f38", "multi_fetch": false, "backend_name": ""}
2021-07-15T17:51:11.419+0100    DEBUG   zipper  got some fetch responses        {"type": "broadcastGroup", "groupName": "root", "type": "fetch", "request": ["one2.two2.three2"], "carbonapi_uuid": "1bf17818-24b5-4494-b6db-599b5f400f38", "backends_count": 1, "response_count": 1, "have_errors": false, "errors": null, "response_count": 1}
2021-07-15T17:51:11.419+0100    DEBUG   zipper  will try to fetch data  {"type": "broadcastGroup", "groupName": "root", "type": "fetch", "request": ["one3.two3.three3"], "carbonapi_uuid": "1bf17818-24b5-4494-b6db-599b5f400f38"}
2021-07-15T17:51:11.419+0100    DEBUG   zipper  single fetch    {"type": "broadcastGroup", "groupName": "root", "type": "fetch", "request": ["one3.two3.three3"], "carbonapi_uuid": "1bf17818-24b5-4494-b6db-599b5f400f38", "client": {}}
2021-07-15T17:51:11.419+0100    DEBUG   zipper  waiting for slot        {"type": "broadcastGroup", "groupName": "root", "type": "fetch", "request": ["one3.two3.three3"], "carbonapi_uuid": "1bf17818-24b5-4494-b6db-599b5f400f38", "multi_fetch": false, "backend_name": "", "max_connections": 0}
2021-07-15T17:51:11.419+0100    DEBUG   zipper  got slot        {"type": "broadcastGroup", "groupName": "root", "type": "fetch", "request": ["one3.two3.three3"], "carbonapi_uuid": "1bf17818-24b5-4494-b6db-599b5f400f38", "multi_fetch": false, "backend_name": ""}
2021-07-15T17:51:11.419+0100    DEBUG   zipper  sending request {"type": "broadcastGroup", "groupName": "root", "type": "fetch", "request": ["one3.two3.three3"], "carbonapi_uuid": "1bf17818-24b5-4494-b6db-599b5f400f38", "multi_fetch": false, "backend_name": ""}
2021-07-15T17:51:11.419+0100    DEBUG   zipper  will try to fetch data  {"type": "broadcastGroup", "groupName": "", "type": "fetch", "request": ["one3.two3.three3"], "carbonapi_uuid": "1bf17818-24b5-4494-b6db-599b5f400f38"}
2021-07-15T17:51:11.419+0100    DEBUG   zipper  single fetch    {"type": "broadcastGroup", "groupName": "", "type": "fetch", "request": ["one3.two3.three3"], "carbonapi_uuid": "1bf17818-24b5-4494-b6db-599b5f400f38", "client": {}}
2021-07-15T17:51:11.419+0100    DEBUG   zipper  waiting for slot        {"type": "broadcastGroup", "groupName": "", "type": "fetch", "request": ["one3.two3.three3"], "carbonapi_uuid": "1bf17818-24b5-4494-b6db-599b5f400f38", "multi_fetch": false, "backend_name": "http://localhost:5000", "max_connections": 0}
2021-07-15T17:51:11.419+0100    DEBUG   zipper  got slot        {"type": "broadcastGroup", "groupName": "", "type": "fetch", "request": ["one3.two3.three3"], "carbonapi_uuid": "1bf17818-24b5-4494-b6db-599b5f400f38", "multi_fetch": false, "backend_name": "http://localhost:5000"}
2021-07-15T17:51:11.419+0100    DEBUG   zipper  sending request {"type": "broadcastGroup", "groupName": "", "type": "fetch", "request": ["one3.two3.three3"], "carbonapi_uuid": "1bf17818-24b5-4494-b6db-599b5f400f38", "multi_fetch": false, "backend_name": "http://localhost:5000"}
2021-07-15T17:51:11.419+0100    DEBUG   zipper  trying to get slot      {"type": "protoV3Group", "name": "http://localhost:5000", "type": "fetch", "request": "&MultiFetchRequest{Metrics:[{one3.two3.three3 1546299999 1546301700 false one3.two3.three3 [] 0}],}", "function": "HttpQuery.doRequest", "server": "http://localhost:5000", "name": "http://localhost:5000", "uri": "http://localhost:5000/render/?format=carbonapi_v3_pb", "name": "http://localhost:5000"}
2021-07-15T17:51:11.420+0100    DEBUG   zipper  got slot for server     {"type": "protoV3Group", "name": "http://localhost:5000", "type": "fetch", "request": "&MultiFetchRequest{Metrics:[{one3.two3.three3 1546299999 1546301700 false one3.two3.three3 [] 0}],}", "function": "HttpQuery.doRequest", "server": "http://localhost:5000", "name": "http://localhost:5000", "uri": "http://localhost:5000/render/?format=carbonapi_v3_pb", "name": "http://localhost:5000"}
2021-07-15T17:51:13.427+0100    DEBUG   zipper  got response    {"type": "broadcastGroup", "groupName": "", "type": "fetch", "request": ["one3.two3.three3"], "carbonapi_uuid": "1bf17818-24b5-4494-b6db-599b5f400f38", "multi_fetch": false, "backend_name": "http://localhost:5000"}
2021-07-15T17:51:13.427+0100    DEBUG   zipper  got some fetch responses        {"type": "broadcastGroup", "groupName": "", "type": "fetch", "request": ["one3.two3.three3"], "carbonapi_uuid": "1bf17818-24b5-4494-b6db-599b5f400f38", "backends_count": 1, "response_count": 1, "have_errors": false, "errors": null, "response_count": 1}
2021-07-15T17:51:13.427+0100    DEBUG   zipper  got response    {"type": "broadcastGroup", "groupName": "root", "type": "fetch", "request": ["one3.two3.three3"], "carbonapi_uuid": "1bf17818-24b5-4494-b6db-599b5f400f38", "multi_fetch": false, "backend_name": ""}
2021-07-15T17:51:13.427+0100    DEBUG   zipper  got some fetch responses        {"type": "broadcastGroup", "groupName": "root", "type": "fetch", "request": ["one3.two3.three3"], "carbonapi_uuid": "1bf17818-24b5-4494-b6db-599b5f400f38", "backends_count": 1, "response_count": 1, "have_errors": false, "errors": null, "response_count": 1}
2021-07-15T17:51:13.428+0100    INFO    access  request served  {"data": {"handler":"render","carbonapi_uuid":"1bf17818-24b5-4494-b6db-599b5f400f38","url":"/render?target=one1.two1.three1&target=one2.two2.three2&target=one3.two3.three3&from=1546299999&until=1546301700&format=json","peer_ip":"127.0.0.1","peer_port":"62236","host":"localhost:9090","format":"json","use_cache":true,"targets":["one1.two1.three1","one2.two2.three2","one3.two3.three3"],"cache_timeout":60,"metrics":["one1.two1.three1","one2.two2.three2","one3.two3.three3"],"runtime":6.023283,"http_code":200,"carbonzipper_response_size_bytes":472,"carbonapi_response_size_bytes":817,"from":1546299999,"until":1546301700,"from_raw":"1546299999","until_raw":"1546301700","uri":"/render?target=one1.two1.three1&target=one2.two2.three2&target=one3.two3.three3&from=1546299999&until=1546301700&format=json","from_cache":false,"used_backend_cache":false,"request_headers":{}}}
2021-07-15T17:51:13.428+0100    WARN    slow    Slow Request    {"time": 6.0237565, "slowLogThreshold": 1, "url": "/render?target=one1.two1.three1&target=one2.two2.three2&target=one3.two3.three3&from=1546299999&until=1546301700&format=json", "referer": ""}

My use case is basically that I want to parallelise the requests to my DB when a Grafana panel has many queries. Grafana sends a single request with multiple targets but from there they are queried sequentially.

jonasbleyl avatar Jul 15 '21 16:07 jonasbleyl

What version of carbonapi are you using?

Also during the startup it prints config as parsed (or it also exported via expvars), could you please share that as well? (obviously feel free to redact any IP Addresses / Hostnames, etc you don't want to share)

Civil avatar Jul 16 '21 10:07 Civil

But tldr it seems that some options to split requests are enabled, while DoMultipleRequestsIfSplit is not

Civil avatar Jul 16 '21 10:07 Civil

I've built CarbonAPI from the latest commit on the main branch.

Here are the logs at startup:

2021-07-16T13:45:05.584+0100    INFO    main    will parse config as yaml       {"config_file": "config.yaml"}
2021-07-16T13:45:05.585+0100    INFO    functionInit    failed to read config file, using default      {"function": "timeShift", "error": "Config File \"config\" Not Found in \"[]\""}
2021-07-16T13:45:05.585+0100    WARN    functionInit    timeShift function in graphite-web have a default value for resetEnd set to true.carbonapi currently forces this to be false. This behavior will change in next major release (0.15)to be compatible with graphite-web. Please change your dashboards to explicitly pass resetEnd parameteror create a config file for this function that sets it to false.Please see https://github.com/go-graphite/carbonapi/blob/main/doc/configuration.md#example-for-timeshift     {"function": "timeShift"}
2021-07-16T13:45:05.586+0100    DEBUG   functionInit    no config file specified        {"function": "aliasByRedis", "message": "this function requrires config file to work properly"}
2021-07-16T13:45:05.586+0100    DEBUG   functionInit    no config file specified        {"function": "aliasByPostgres", "message": "this function requrires config file to work properly"}
2021-07-16T13:45:05.586+0100    DEBUG   functionInit    no config file specified        {"function": "graphiteWeb", "message": "this function requrires config file to work properly"}
2021-07-16T13:45:05.586+0100    INFO    main    cache: in-memory cache configured
2021-07-16T13:45:05.586+0100    INFO    main    starting carbonapi      {"build_version": "v0.15.4-16-g4d91", "config": "{\"ExtrapolateExperiment\":false,\"Logger\":[{\"logger\":\"\",\"file\":\"stdout\",\"level\":\"debug\",\"encoding\":\"console\",\"encoding-time\":\"iso8601\",\"encoding-duration\":\"seconds\",\"sample-tick\":\"\",\"sample-initial\":0,\"sample-thereafter\":0}],\"Listen\":\"localhost:9090\",\"Listeners\":[{\"Address\":\"localhost:9090\"}],\"Buckets\":10,\"Concurency\":20,\"ResponseCacheConfig\":{\"Type\":\"mem\",\"Size\":0,\"MemcachedServers\":[],\"DefaultTimeoutSec\":60},\"BackendCacheConfig\":{\"Type\":\"null\",\"Size\":0,\"MemcachedServers\":null,\"DefaultTimeoutSec\":0},\"Cpus\":0,\"TimezoneString\":\"\",\"UnicodeRangeTables\":null,\"Graphite\":{\"Pattern\":\"{prefix}.{fqdn}\",\"Host\":\"\",\"Interval\":60000000000,\"Prefix\":\"carbon.api\"},\"IdleConnections\":10,\"PidFile\":\"\",\"SendGlobsAsIs\":null,\"AlwaysSendGlobsAsIs\":null,\"MaxBatchSize\":100,\"Zipper\":\"\",\"Upstreams\":{\"Buckets\":10,\"SlowLogThreshold\":1000000000,\"ConcurrencyLimitPerServer\":0,\"MaxIdleConnsPerHost\":100,\"Backends\":null,\"BackendsV2\":{\"Backends\":[{\"GroupName\":\"\",\"Protocol\":\"carbonapi_v3_pb\",\"LBMethod\":\"broadcast\",\"Servers\":[\"http://localhost:5000\"],\"Timeouts\":{\"Find\":2000000000,\"Render\":10000000000000,\"Connect\":200000000},\"ConcurrencyLimit\":null,\"KeepAliveInterval\":null,\"MaxIdleConnsPerHost\":null,\"MaxTries\":null,\"MaxBatchSize\":0,\"BackendOptions\":null,\"ForceAttemptHTTP2\":false,\"DoMultipleRequestsIfSplit\":false}],\"MaxIdleConnsPerHost\":0,\"ConcurrencyLimitPerServer\":0,\"Timeouts\":{\"Find\":2000000000,\"Render\":10000000000000,\"Connect\":200000000},\"KeepAliveInterval\":0,\"MaxTries\":0,\"MaxBatchSize\":100},\"MaxBatchSize\":100,\"FallbackMaxBatchSize\":100,\"MaxTries\":0,\"DoMultipleRequestsIfSplit\":false,\"CarbonSearch\":{\"Backend\":\"\",\"Prefix\":\"virt.v1.*\"},\"CarbonSearchV2\":{\"Backends\":null,\"MaxIdleConnsPerHost\":0,\"ConcurrencyLimitPerServer\":0,\"Timeouts\":{\"Find\":0,\"Render\":0,\"Connect\":0},\"KeepAliveInterval\":0,\"MaxTries\":0,\"MaxBatchSize\":100,\"Prefix\":\"\"},\"ExpireDelaySec\":0,\"TLDCacheDisabled\":false,\"InternalRoutingCache\":600000000000,\"Timeouts\":{\"Find\":2000000000,\"Render\":10000000000000,\"Connect\":200000000},\"KeepAliveInterval\":30000000000,\"ScaleToCommonStep\":true},\"ExpireDelaySec\":600,\"GraphiteWeb09Compatibility\":false,\"IgnoreClientTimeout\":false,\"DefaultColors\":null,\"GraphTemplates\":\"\",\"FunctionsConfigs\":{},\"HeadersToPass\":null,\"HeadersToLog\":null,\"Define\":null,\"Prefix\":\"\",\"Expvar\":{\"Listen\":\"\",\"Enabled\":true,\"PProfEnabled\":false},\"NotFoundStatusCode\":200,\"HTTPResponseStackTrace\":true,\"UseCachingDNSResolver\":false,\"CachingDNSRefreshTime\":60000000000}"}
2021-07-16T13:45:05.587+0100    DEBUG   zipper  initializing zipper
2021-07-16T13:45:05.587+0100    DEBUG   zipper  creating lb group       {"name": "", "servers": ["http://localhost:5000"], "type": "broadcast"}
2021-07-16T13:45:05.587+0100    DEBUG   dns     no caching dns initialized, will return typical DialContext
2021-07-16T13:45:05.587+0100    DEBUG   zipper  zipper config   {"config": {"Buckets":10,"SlowLogThreshold":1000000000,"ConcurrencyLimitPerServer":0,"MaxIdleConnsPerHost":100,"Backends":null,"BackendsV2":{"Backends":[{"GroupName":"","Protocol":"carbonapi_v3_pb","LBMethod":"broadcast","Servers":["http://localhost:5000"],"Timeouts":{"Find":2000000000,"Render":10000000000000,"Connect":200000000},"ConcurrencyLimit":null,"KeepAliveInterval":null,"MaxIdleConnsPerHost":null,"MaxTries":null,"MaxBatchSize":0,"BackendOptions":null,"ForceAttemptHTTP2":false,"DoMultipleRequestsIfSplit":false}],"MaxIdleConnsPerHost":0,"ConcurrencyLimitPerServer":0,"Timeouts":{"Find":2000000000,"Render":10000000000000,"Connect":200000000},"KeepAliveInterval":0,"MaxTries":0,"MaxBatchSize":100},"MaxBatchSize":100,"FallbackMaxBatchSize":100,"MaxTries":0,"DoMultipleRequestsIfSplit":false,"CarbonSearch":{"Backend":"","Prefix":"virt.v1.*"},"CarbonSearchV2":{"Backends":null,"MaxIdleConnsPerHost":0,"ConcurrencyLimitPerServer":0,"Timeouts":{"Find":0,"Render":0,"Connect":0},"KeepAliveInterval":0,"MaxTries":0,"MaxBatchSize":100,"Prefix":""},"ExpireDelaySec":0,"TLDCacheDisabled":false,"InternalRoutingCache":600000000000,"Timeouts":{"Find":2000000000,"Render":10000000000000,"Connect":200000000},"KeepAliveInterval":30000000000,"ScaleToCommonStep":true}}
2021-07-16T13:45:05.587+0100    DEBUG   zipper  doing request   {"type": "protoV3Group", "name": "http://localhost:5000", "function": "prober", "request": ["*"]}
2021-07-16T13:45:05.587+0100    DEBUG   zipper  trying to get slot      {"type": "protoV3Group", "name": "http://localhost:5000", "type": "find", "request": ["*"], "function": "HttpQuery.doRequest", "server": "http://localhost:5000", "name": "http://localhost:5000", "uri": "http://localhost:5000/metrics/find/?format=carbonapi_v3_pb", "name": "http://localhost:5000"}
2021-07-16T13:45:05.587+0100    DEBUG   zipper  got slot for server     {"type": "protoV3Group", "name": "http://localhost:5000", "type": "find", "request": ["*"], "function": "HttpQuery.doRequest", "server": "http://localhost:5000", "name": "http://localhost:5000", "uri": "http://localhost:5000/metrics/find/?format=carbonapi_v3_pb", "name": "http://localhost:5000"}
2021-07-16T13:45:05.596+0100    DEBUG   zipper  will return data        {"type": "protoV3Group", "name": "http://localhost:5000", "function": "prober", "tlds": ["one1", "one2", "one3"]}

Formatted config from log:

{
   "config":{
      "Buckets":10,
      "SlowLogThreshold":1000000000,
      "ConcurrencyLimitPerServer":0,
      "MaxIdleConnsPerHost":100,
      "Backends":null,
      "BackendsV2":{
         "Backends":[
            {
               "GroupName":"",
               "Protocol":"carbonapi_v3_pb",
               "LBMethod":"broadcast",
               "Servers":[
                  "http://localhost:5000"
               ],
               "Timeouts":{
                  "Find":2000000000,
                  "Render":10000000000000,
                  "Connect":200000000
               },
               "ConcurrencyLimit":null,
               "KeepAliveInterval":null,
               "MaxIdleConnsPerHost":null,
               "MaxTries":null,
               "MaxBatchSize":0,
               "BackendOptions":null,
               "ForceAttemptHTTP2":false,
               "DoMultipleRequestsIfSplit":false
            }
         ],
         "MaxIdleConnsPerHost":0,
         "ConcurrencyLimitPerServer":0,
         "Timeouts":{
            "Find":2000000000,
            "Render":10000000000000,
            "Connect":200000000
         },
         "KeepAliveInterval":0,
         "MaxTries":0,
         "MaxBatchSize":100
      },
      "MaxBatchSize":100,
      "FallbackMaxBatchSize":100,
      "MaxTries":0,
      "DoMultipleRequestsIfSplit":false,
      "CarbonSearch":{
         "Backend":"",
         "Prefix":"virt.v1.*"
      },
      "CarbonSearchV2":{
         "Backends":null,
         "MaxIdleConnsPerHost":0,
         "ConcurrencyLimitPerServer":0,
         "Timeouts":{
            "Find":0,
            "Render":0,
            "Connect":0
         },
         "KeepAliveInterval":0,
         "MaxTries":0,
         "MaxBatchSize":100,
         "Prefix":""
      },
      "ExpireDelaySec":0,
      "TLDCacheDisabled":false,
      "InternalRoutingCache":600000000000,
      "Timeouts":{
         "Find":2000000000,
         "Render":10000000000000,
         "Connect":200000000
      },
      "KeepAliveInterval":30000000000,
      "ScaleToCommonStep":true
   }
}

I've tried enabling doMultipleRequestsIfSplit: true as well but that seems to have no effect in this scenario.

On the other end I'm just handling the graphite endpoints like /metrics/find/ and /render/. At the render endpoint I'm unmarshalling it as a MultiFetchRequest.

jonasbleyl avatar Jul 16 '21 12:07 jonasbleyl

Hey @Civil, I'd be happy to have a go fixing this but it'd be great if I could get a bit of direction.

I've had a look at the initial commit of cmd/http/render_handler.go file (commit 03b7a9d), it looks like the call to config.Config.ZipperInstance.Render is called for each target within a loop over the targets, rather than calling render with an array of the targets. This was then re-written in commit b99c09b where for each target FetchTargetExp is called, which in turn calls render for each target sequentially.

Where do you think would start be a good place to fix this by sending an array of targets to render? Or am I looking in the completely wrong places.

Thanks, Dan

dfairburn avatar Sep 02 '21 10:09 dfairburn

This is probably the cause of the bug in #509.

reyjrar avatar Dec 14 '21 18:12 reyjrar

I think #743 should fix the problem, once it's merged. It introduced a config option that will send multiple targets in a single request. It however might need some work before it's fully compatible (I have some concerns about fallbackSeries)

Civil avatar Dec 05 '22 01:12 Civil