Multiple queries return data in wrong order

[mvrhov]

I don't know how the phrase this properly. But we add multiple queries per dashboard. And it seems that the data for the queries gets messed up. I can refresh the dashboard multiple times and sometimes data from query A gets shown as data for query B and vice versa.

This is when the returned data is OK.

And this when the data is returned in the wrong order. The query is the same I just pressed refresh dashboard multiple times.

[enotspe]

I have noticed the same behavior

[kuzaxak]

https://github.com/quickwit-oss/quickwit-datasource/issues/143

[kuzaxak]

It is a bug on the QW side. I've added dubug logging to the plugin and can see that the same query got a response from /api/v1/_elastic/_msearch in a different order.

Query

{"ignore_unavailable":true,"index":["otel-traces-v*"]}
{"aggs":{"2":{"date_histogram":{"field":"span_start_timestamp_nanos","fixed_interval":"1m","min_doc_count":0,"extended_bounds":{"min":1743847765408,"max":1743847889194}}}},"query":{"bool":{"filter":[{"range":{"span_start_timestamp_nanos":{"gte":"2025-04-05T10:09:25.408Z","lte":"2025-04-05T10:11:29.194Z"}}},{"query_string":{"default_operator":"AND","query":"service_name:\"master-service\"\n -is_root:\"false\""}}]}},"size":0}
{"ignore_unavailable":true,"index":["otel-traces-v*"]}
{"aggs":{"2":{"date_histogram":{"field":"span_start_timestamp_nanos","fixed_interval":"5m","min_doc_count":0,"extended_bounds":{"min":1743847765408,"max":1743847889194}}}},"query":{"bool":{"filter":[{"range":{"span_start_timestamp_nanos":{"gte":"2025-04-05T10:09:25.408Z","lte":"2025-04-05T10:11:29.194Z"}}},{"query_string":{"default_operator":"AND","query":"service_name:\"ledger\"\n -is_root:\"false\""}}]}},"size":0}

Response 1 (correct):

{
  "responses": [
    {
      "status": 200,
      "took": 2,
      "timed_out": false,
      "_shards": {
        "total": 1,
        "successful": 1,
        "skipped": 0,
        "failed": 0
      },
      "hits": {
        "total": {
          "value": 41,
          "relation": "eq"
        },
        "hits": []
      },
      "aggregations": {
        "2": {
          "buckets": [
            {
              "doc_count": 0,
              "key": 1743847740000.0,
              "key_as_string": "2025-04-05T10:09:00Z"
            },
            {
              "doc_count": 26,
              "key": 1743847800000.0,
              "key_as_string": "2025-04-05T10:10:00Z"
            },
            {
              "doc_count": 15,
              "key": 1743847860000.0,
              "key_as_string": "2025-04-05T10:11:00Z"
            }
          ]
        }
      }
    },
    {
      "status": 200,
      "took": 2,
      "timed_out": false,
      "_shards": {
        "total": 1,
        "successful": 1,
        "skipped": 0,
        "failed": 0
      },
      "hits": {
        "total": {
          "value": 599,
          "relation": "eq"
        },
        "hits": []
      },
      "aggregations": {
        "2": {
          "buckets": [
            {
              "doc_count": 93,
              "key": 1743847500000.0,
              "key_as_string": "2025-04-05T10:05:00Z"
            },
            {
              "doc_count": 506,
              "key": 1743847800000.0,
              "key_as_string": "2025-04-05T10:10:00Z"
            }
          ]
        }
      }
    }
  ]
}

After few requests got Response 2 (flipped):

{
  "responses": [
    {
      "status": 200,
      "took": 2,
      "timed_out": false,
      "_shards": {
        "total": 1,
        "successful": 1,
        "skipped": 0,
        "failed": 0
      },
      "hits": {
        "total": {
          "value": 599,
          "relation": "eq"
        },
        "hits": []
      },
      "aggregations": {
        "2": {
          "buckets": [
            {
              "doc_count": 93,
              "key": 1743847500000.0,
              "key_as_string": "2025-04-05T10:05:00Z"
            },
            {
              "doc_count": 506,
              "key": 1743847800000.0,
              "key_as_string": "2025-04-05T10:10:00Z"
            }
          ]
        }
      }
    },
    {
      "status": 200,
      "took": 2,
      "timed_out": false,
      "_shards": {
        "total": 1,
        "successful": 1,
        "skipped": 0,
        "failed": 0
      },
      "hits": {
        "total": {
          "value": 41,
          "relation": "eq"
        },
        "hits": []
      },
      "aggregations": {
        "2": {
          "buckets": [
            {
              "doc_count": 0,
              "key": 1743847740000.0,
              "key_as_string": "2025-04-05T10:09:00Z"
            },
            {
              "doc_count": 26,
              "key": 1743847800000.0,
              "key_as_string": "2025-04-05T10:10:00Z"
            },
            {
              "doc_count": 15,
              "key": 1743847860000.0,
              "key_as_string": "2025-04-05T10:11:00Z"
            }
          ]
        }
      }
    }
  ]
}

[kuzaxak]

https://github.com/quickwit-oss/quickwit/issues/5731

[kuzaxak]

Fixed in https://github.com/quickwit-oss/quickwit/pull/5732