databricks-sync icon indicating copy to clipboard operation
databricks-sync copied to clipboard
verified publisher icon databrickslabs

EXPORT failing due to temp files cleanup issue in Windows

Open ghost opened this issue 2 years ago • 2 comments

2023-02-13 18:20:49 [INFO] Run Id: cd40dac9-d9d9-4f12-b887-5078f4729f3c 2023-02-13 18:20:49 [INFO] Run Summary: 2023-02-13 18:20:49 [INFO] ============ 2023-02-13 18:20:49 [INFO] ╒════╤═════════════════════╤════════════════════╤═════════════════╤══════════════════════╤═══════════════════╕ │ │ object_type │ export_succeeded │ export_failed │ validate_succeeded │ validate_failed │ ╞════╪═════════════════════╪════════════════════╪═════════════════╪══════════════════════╪═══════════════════╡ │ 0 │ databricks_notebook │ 2 │ 0 │ 0 │ 2 │ ╘════╧═════════════════════╧════════════════════╧═════════════════╧══════════════════════╧═══════════════════╛ 2023-02-13 18:20:49 [INFO] 2023-02-13 18:20:49 [INFO] Run Errors Summary: 2023-02-13 18:20:49 [INFO] =================== 2023-02-13 18:20:49 [INFO] ╒════════════╤═══════╤═════════════╕ │ err_type │ msg │ err_count │ ╞════════════╪═══════╪═════════════╡ ╘════════════╧═══════╧═════════════╛ Traceback (most recent call last): File "D:\Users\username\AppData\Local\Programs\Python\Python39\lib\shutil.py", line 613, in _rmtree_unsafe os.unlink(fullname) PermissionError: [WinError 5] Access is denied: 'D:\Users\username\AppData\Local\Temp\tmpqdvu_qnu\.git\objects\pack\pack-a6e7ae45a0f17a8d8ca6af57587ce71016431626.idx' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "D:\Users\username\AppData\Local\Programs\Python\Python39\lib\tempfile.py", line 801, in onerror _os.unlink(path) PermissionError: [WinError 5] Access is denied: 'D:\Users\username\AppData\Local\Temp\tmpqdvu_qnu\.git\objects\pack\pack-a6e7ae45a0f17a8d8ca6af57587ce71016431626.idx' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "D:\Users\username\AppData\Local\Programs\Python\Python39\Scripts\databricks-sync-script.py", line 11, in load_entry_point('databricks-sync==0.3.2', 'console_scripts', 'databricks-sync')() File "D:\Users\username\AppData\Local\Programs\Python\Python39\lib\site-packages\click\core.py", line 1130, in call return self.main(*args, **kwargs) File "D:\Users\username\AppData\Local\Programs\Python\Python39\lib\site-packages\click\core.py", line 1055, in main rv = self.invoke(ctx) File "D:\Users\username\AppData\Local\Programs\Python\Python39\lib\site-packages\click\core.py", line 1657, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "D:\Users\username\AppData\Local\Programs\Python\Python39\lib\site-packages\click\core.py", line 1404, in invoke return ctx.invoke(self.callback, **ctx.params) File "D:\Users\username\AppData\Local\Programs\Python\Python39\lib\site-packages\click\core.py", line 760, in invoke return __callback(*args, **kwargs) File "D:\Users\username\AppData\Local\Programs\Python\Python39\lib\site-packages\databricks_cli\configure\config.py", line 55, in decorator return function(*args, **kwargs) File "D:\Users\username\AppData\Local\Programs\Python\Python39\lib\site-packages\databricks_sync\cmds\config.py", line 178, in modify_user_agent return function(*args, **kwargs) File "D:\Users\username\AppData\Local\Programs\Python\Python39\lib\site-packages\databricks_sync\cmds\config.py", line 161, in decorator return function(*args, **kwargs) File "D:\Users\username\AppData\Local\Programs\Python\Python39\lib\site-packages\click\decorators.py", line 26, in new_func return f(get_current_context(), *args, **kwargs) File "D:\Users\username\AppData\Local\Programs\Python\Python39\lib\site-packages\databricks_sync\cmds\export.py", line 33, in export_cli ExportCoordinator.export(api_client, Path(config_path), dask_mode=dask, dry_run=dry_run, git_ssh_url=git_ssh_url, File "D:\Users\username\AppData\Local\Programs\Python\Python39\lib\site-packages\databricks_sync\sdk\sync\export.py", line 119, in export tmp_dir.cleanup() File "D:\Users\username\AppData\Local\Programs\Python\Python39\lib\tempfile.py", line 830, in cleanup self._rmtree(self.name) File "D:\Users\username\AppData\Local\Programs\Python\Python39\lib\tempfile.py", line 812, in _rmtree _shutil.rmtree(name, onerror=onerror) File "D:\Users\username\AppData\Local\Programs\Python\Python39\lib\shutil.py", line 737, in rmtree return _rmtree_unsafe(path, onerror) File "D:\Users\username\AppData\Local\Programs\Python\Python39\lib\shutil.py", line 610, in _rmtree_unsafe _rmtree_unsafe(fullname, onerror) File "D:\Users\username\AppData\Local\Programs\Python\Python39\lib\shutil.py", line 610, in _rmtree_unsafe _rmtree_unsafe(fullname, onerror) File "D:\Users\username\AppData\Local\Programs\Python\Python39\lib\shutil.py", line 610, in _rmtree_unsafe _rmtree_unsafe(fullname, onerror) File "D:\Users\username\AppData\Local\Programs\Python\Python39\lib\shutil.py", line 615, in _rmtree_unsafe onerror(os.unlink, fullname, sys.exc_info()) File "D:\Users\username\AppData\Local\Programs\Python\Python39\lib\tempfile.py", line 804, in onerror cls._rmtree(path) File "D:\Users\username\AppData\Local\Programs\Python\Python39\lib\tempfile.py", line 812, in _rmtree _shutil.rmtree(name, onerror=onerror) File "D:\Users\username\AppData\Local\Programs\Python\Python39\lib\shutil.py", line 737, in rmtree return _rmtree_unsafe(path, onerror) File "D:\Users\username\AppData\Local\Programs\Python\Python39\lib\shutil.py", line 596, in _rmtree_unsafe onerror(os.scandir, path, sys.exc_info()) File "D:\Users\username\AppData\Local\Programs\Python\Python39\lib\shutil.py", line 593, in _rmtree_unsafe with os.scandir(path) as scandir_it: NotADirectoryError: [WinError 267] The directory name is invalid: 'D:\Users\username\AppData\Local\Temp\tmpqdvu_qnu\.git\objects\pack\pack-a6e7ae45a0f17a8d8ca6af57587ce71016431626.idx'

ghost avatar Feb 13 '23 13:02 ghost

Is is related to an open Issue? https://github.com/databrickslabs/databricks-sync/issues/122

and is this a valid fix? https://stackoverflow.com/questions/1889597/deleting-read-only-directory-in-python/1889686#1889686

ghost avatar Feb 13 '23 13:02 ghost

This is only happening with git push with local repo, the export is successful

ghost avatar Feb 14 '23 13:02 ghost