sa7mon
not actually dumping anything?
Download failed: Key: ****, Size: 0, LastModified: 2017-05-30 11:33:45+00:00 | [Errno 21] Is a directory: '***/item/20.Cf571aa6' -> '***/item/20'
It seems that when these errors occur, the program creates dummy files of the named directory. Then, it is unable to overwrite them to download actual files.
Download failed: Key: ***/css/style_back.css, Size: 12771, LastModified: 2018-01-11 05:28:03+00:00 | [Errno 17] File exists: '***/css'
is there an option to turn on overwriting as a quick dirty fix?
I wonder if adding a check for a zero size file in the download_file function will skip the directory objects. It seems like they will still be created by this line dest_file_path.parent.mkdir(parents=True, exist_ok=True)
Perhaps another if block after if not self.is_safe_file_to_download(obj.key, dest_directory):?
Something like if obj.size == 0: and skipping the download.
I'm a little confused about what the issue is you're running into. Does the bucket contain empty directories? If it's possible could you send me the name of the bucket for testing purposes? My email is listed on the contact page of my website if you don't want to post the bucket name here.
Sure, the bucket name is claycloud. Warning, it is around 28 GB.
At the end, I did get many files downloaded. The problem was that several directories (containing files) were not downloaded because empty dummy files were created earlier in the process. The second error had files in that directory skipped because that dummy file existed.
Thank you for your bug report. I have just merged the pull request for complete re-write of this tool in Go so this issue is likely no longer applicable.
I would encourage you to try out the new version either by checking out the main branch or downloading the new release once I tag it. If you encounter errors with the new version, please open a new issue.