Workspace API removes extension from Python files

[fferegrino]

We are running into an issue when uploading Python files using the method import_workspace_dir in the WorkspaceApi class. However, these files are not treated as Python source files, but as Databricks notebooks.

The only reason I managed to pinpoint this issue is the fact that the API removes the extension and as such they are no longer interpreted as Python modules but as Databricks notebooks.

https://github.com/databricks/databricks-cli/blob/5c3879188322c0dfe94701db1ec8299bad7ccd5f/databricks_cli/workspace/api.py#L153

Is this intended behaviour? could this be changed by not removing the file extension, unless the file is actually a Databricks notebook?

[fferegrino]

Actually, the issue is not only with the extension but with the file format, according to the docs it is possible to upload arbitrary files (?) when the format is set to "AUTO", so that probably needs changing as well...

Now, it seems that the feature is in public preview, and as such this change has not made it to the CLI, rendering this and #618 a bit redundant.

[osuleiman]

I've just created a PR to help tackle this problem.

#643