datafusion-ballista icon indicating copy to clipboard operation
datafusion-ballista copied to clipboard

Expose the lower level scheduler API

Open feribg opened this issue 2 years ago • 0 comments

I was reading through examples and source but there doesn't seem to be a very obvious way to interact with the scheduler.

Since currently there dont seem to be any good distributed computing libraries in the rust ecosystem and ray has no plans to offer official rust support, it would be a very nice addition to be able to interact with the underlying tasks in 2 main ways:

  • Submit tasks to the scheduler in the form of runnable rust code (even if for now the DAG is ignored)
  • be able to levelrage the data distribution to allow shared data to be handled by the tasks.

maybe there is already a straightforward way to do that in which case im happy to close this and contribute to the docs.

feribg avatar Oct 05 '23 16:10 feribg