Dmitry Glushenok

Same problem on v1.98.0 - when you hit the limit once, shorting the time range will not help to avoid the error. This makes `search.maxSamplesPerQuery` unusable.

> What's the time range you query here, for old data like past month? I've noticed the problem browsing dashboards after shortening `search.maxSamplesPerQuery`. First time the error was generated on...

> Could you share the `vm_concurrent_queries` results on vmselect(both vmselect-native and vmselect-http) during that time? On the screenshot below, vmselect-multicloud-sts-npe is vmselect-http, vmselect-zonal-sts-prod is vmselect-native: > Is there any relevant...

> Does the first subcluster have twice the number of vmstorage nodes as the second? Nope. Both clusters has equal vmstorage nodes. > I see no `-search.maxQueryDuration` flag on vmselects...

> @dglushenok Could you help align the bove results timeline with `vm_vmselect_concurrent_requests_current `? I've multiplied `vm_vmselect_concurrent_requests_current` by 100 to be more visible: > And can you show those query results...

It looks like that hanging requests are not appearing in clusters without load. I've disabled incoming requests to one of the two clusters, working in parallel, and perform manual requests,...

> Interesting, perhaps there are already blocked requests on vmstorage before removing load, could you also query `rate(vm_vmselect_concurrent_requests_limit_reached_total[1m])` and `rate(vm_vmselect_concurrent_requests_limit_timeout_total[1m])` on that vmstorage node? > Does this situation always last...