databricks-sql-python icon indicating copy to clipboard operation
databricks-sql-python copied to clipboard
verified publisher icon databricks

ENH: Add LOAD DATA LOCAL capability

Open dmoore247 opened this issue 2 years ago • 3 comments

Add a LOAD DATA LOCAL type capability to take files local to client, load to dbfs/volumes/bucket/storage account staging area, then COPY INTO a delta lake table, optionally remove data from staging area after COPY INTO. The benefit is to support bulk data loads from any client.

dmoore247 avatar May 26 '23 22:05 dmoore247

I have code for upload and COPY INTO, would like to figure out right design (if appropriate) to add to api.

dmoore247 avatar May 27 '23 03:05 dmoore247

Is this not already supported with the staging ingestion features? See the example here: https://github.com/databricks/databricks-sql-python/blob/main/examples/staging_ingestion.py

susodapop avatar May 27 '23 03:05 susodapop

@dmoore247 did you have a chance to try the staging ingestion feature? Does it solve your issue?

kravets-levko avatar Apr 17 '24 18:04 kravets-levko