Monai label server add API to return number of requests in the queue

Open AHarouni opened this issue 2 years ago • 1 comments

Please describe. My model is very large and takes up to 60 seconds on my gpu. I run with --workers=1 to avoid having 2 requests run on the same gpu and get into out of memory. Now when multiple users each run a request they are in a queue that is some times too long.

Describe the solution you'd like We should have an api to show how many requests are ahead of them in the queue and give an estimate on the wait time This should take into account if we have multiple gpus. Basically this allows the UI to set the users expectations on the wait time

Describe alternatives you've considered None

Jul 10 '23 20:07 AHarouni

API are invoked via ReST..

http/web container manages all the queue/threads/process.. in this case uvicorn..

API to know how many API requests are running can't be provided by the same application layer..

Jul 10 '23 20:07 SachidanandAlle