databricks-sql-python icon indicating copy to clipboard operation
databricks-sql-python copied to clipboard

ExecuteMany performance is insanely bad

Open diwu-sf opened this issue 8 months ago • 1 comments

I know that the documentation makes it clear that executemany is a naive for loop:

No optimizations of the query (like batching) will be performed. 

But it's 2025 and please have a more optimized executemany that just issues a single SQL statement using VALUES(...) or something so this is actually usable in a data pipeline. Otherwise, trying to use the databricks SQL connector to write any non-trivial dataframe to a delta table is pointless.

Thanks!

diwu-sf avatar May 19 '25 00:05 diwu-sf