Databricks.API.PowerShell icon indicating copy to clipboard operation
Databricks.API.PowerShell copied to clipboard
verified publisher icon gbrueckl

Export DBFS files

Open saldroubi opened this issue 3 years ago • 1 comments

I want to move DBFS files from one workspace to another. I am not able to export DBFS files?

I get this error message:

Export-DatabricksEnvironment -Artifacts @("DBFS") -LocalPath test

This feature is EXPERIMENTAL and still UNDER DEVELOPMENT! It is not possible to donwload the whole DBFS. This export will only download files from DBFS that already exist locally and overwrite them!

Also, what if the file is greater than 2 GB? What do you propose to make this happen? Could this feature be added to export a file/directory recursively and if file is > 2g to cut it into multiple files?

Thank you.

saldroubi avatar May 24 '22 22:05 saldroubi

the warning/error it is there to avoid downloading the whole DBFS as it could be hundreds of TBs feel free though to write your own recursive functions using the existing cmdlets that deal with the DBFS

regarding the 2 GB - I am/was not aware of any limit there. if it exists, it is probably enforced by the API and is nothing I could change. There is no filter/restriction applied within the cmdlets of this module

gbrueckl avatar May 30 '22 07:05 gbrueckl

Thank you.

saldroubi avatar Oct 20 '22 14:10 saldroubi