Add async/long-running processes to the webserver for computationally expensive tasks

[ThomasLaPiana]

Is your feature request related to a specific problem?

There is no way to run a blocking, long-running job async within the webserver

Describe the solution you'd like

Using something like Dask to handle multi-processing and spinning up additional processes to handle long-running jobs.

Describe alternatives you've considered, if any

An alternative like celery or rolling our own multi-processing implementation

Additional context

Add any other context or screenshots about the feature request here.

[sanders41]

FastAPI blocking functions are run in a threadpool so they don't block (more info here) and run asynchronously. There is also an option in FastAPI to run things as background tasks.

I'm not sure if either of these are solutions for what you have in mind, but wanted to mention them in case they might be easy solutions.

[ThomasLaPiana]

absolutely! I was thinking of using something like Dask but I much prefer native implementations. Thanks for the tip @sanders41 !

[ThomasLaPiana]

@seanpreston @pattisdr we're using Celery for this right? or in some cases using a local worker that is blocking?

[pattisdr]

Yes we use Celery, the latest round makes it so you don't have to run a separate worker by default , it runs in a subprocess https://github.com/ethyca/fides/pull/1552, although you can add a separate worker needed. I didn't think this was blocking.

[ThomasLaPiana]

I might've misremembered, just wanted to confirm we were good here (this was from pre-unified when ctl didn't have the concept of workers/subprocesses), thanks!