databricks-sql-nodejs icon indicating copy to clipboard operation
databricks-sql-nodejs copied to clipboard
verified publisher icon databricks

Add support for results streaming

Open ShantiCumulio opened this issue 1 year ago • 2 comments

I've built a plugin for connecting Databricks to Luzmo. I would like to be able to process the results from a query as a readable stream, similar to how this library creates for Postgres to speed up query + data processing times.

ShantiCumulio avatar Apr 17 '24 18:04 ShantiCumulio

Hi @ShantiCumulio! This feature is already in our backlog, but, unfortunately, currently there are no estimates for when I can work on it. If you want to volunteer on implementing it - you're welcome 🙂

kravets-levko avatar Apr 17 '24 18:04 kravets-levko

A quick update on this issue. I was working on adding an iterable interface to operation (databricks/databricks-sql-nodejs#252) when I realized that Node has a built-in helper to convert iterators into readable streams. So expect streams support very soon

kravets-levko avatar Apr 19 '24 12:04 kravets-levko