Excluded files/folders are deleted on "total restore"

[solisinvictum]

Hi Guys,

i tried out BackInTime. Nice program. But only small problem:

The Restore option to delete everything what isnt in the snapshot.

It would be a great idea to simply write the excluded files and folder in the snapshot (as txt file or something), and to use this list to exclude what dont need to be deleted. Rsync supports that. Or to give the User option to set exclusions.

I implemented this in my script (for borg, but this script would work for backintime too with little change): https://github.com/solisinvictum/borgrestore

(support for borg would be great too. because borg can deduplicate and compress :P but i dont want to ask to much)

Thanks a lot.

[emtiu]

Hi, thanks for your report. Could you be more specific as to what you mean by: "everything what isnt in the snapshot"?

Did you restore a file or a directory? What did your (original and destination) paths look like?

It's important that we understand what got deleted against your expectation.

[buhtz]

@solisinvictum I also do not understand your Issue in details. Can you report back please.

After some experimenting I think this is a bug and @solisinvictum suggested a solution to it. The restore dialog offers the option Remove newer files in original folder.

Steps to reproduce:

1. Create a source folder with some files.
2. Explicit add a file exclude_me_file and a folder exclude_me_folder (including one extra file) to that source folder.
3. Create the backup profile and add the file and folder to the exclude list.
4. Run backups.
5. Restore using the option Remove newer files in original folder.

The main problem is that BIT/rsync do not take the exclude list into account. The excluded files/folders are not contained in the snapshot. That is why they are deleted in the source folder. It is IMHO a serious bug and I wonder why we do not hear about it more often.

building file list ... done
rsync: [generator] chgrp "/home" failed: Operation not permitted (1)
home/
deleting home/user/Downloads/foobar/exclude_me_folder/foobar
cannot delete non-empty directory: home/user/Downloads/foobar/exclude_me_folder
deleting home/user/Downloads/foobar/exclude_me_file
sent 267 bytes  received 204 bytes  942,00 bytes/sec
total size is 23  speedup is 0,05
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1338) [sender=3.2.7]
Restore permissions:
Restore permissions: Done

Looking into this (restore) snapshot log there is a secondary problem.

deleting home/user/Downloads/foobar/exclude_me_folder/foobar
cannot delete non-empty directory: home/user/Downloads/foobar/exclude_me_folder

The file exclude_me_folder/foobar is not really deleted but renamed to exclude_me_folder/foobar.backup.20231207 because of the first option in the restore dialog. So the folder exlude_me_folder is not empty and can not be deleted because of that.

[aryoda]

5. Restore using the option Remove newer files in original folder.

I see a usability and documentation issue here:

Newer files sounds different from "excluded" files to me (so I there is no indication what BiT will do with files with the existing ignored files in the source during restore).

How about mentioning that excluded files and folders will also be lost (as first "fix")?

The main problem is that BIT/rsync do not take the exclude list into account. The excluded files/folders are not contained in the snapshot. That is why they are deleted in the source folder.

The OP says rsync could handle this when we provide the exclude list during restore...

If this would be the solution I wonder if we should use the exclude list as it was when the snapshot was created or the most recent exclude list (which may be lost if the BiT profiles are missing eg. due to a hardware failure).

It is IMHO a serious bug and I wonder why we do not hear about it more often.

Restoring directly into the original folder is highly risky so perhaps this bug may not happen often (I remember only on case where the OP mentioned a loss of the original file by restoring directly into the original folder because of problems with the renaming of the original file).

The tooltip is already mentioning the problem:

https://github.com/bit-team/backintime/blob/59097f7a3c0ddd978e5a7a47b21909bed637a010/qt/app.py#L1346-L1354

How about adding/improving the "are you sure" question to mention the full impact?

I think we should not put too much effort in such risky restore operations (eg. other files neither contained in the snapshot nor in the exclude list may also get lost)...

[solisinvictum]

@solisinvictum I also do not understand your Issue in details. Can you report back please.

After some experimenting I think this is a bug and @solisinvictum suggested a solution to it. The restore dialog offers the option Remove newer files in original folder.

Steps to reproduce:

1. Create a source folder with some files.

2. Explicit add a file `exclude_me_file` and a folder `exclude_me_folder` (including one extra file) to that source folder.

3. Create the backup profile and add the file and folder to the exclude list.

4. Run backups.

5. Restore using the option _Remove newer files in original folder_.

The main problem is that BIT/rsync do not take the exclude list into account. The excluded files/folders are not contained in the snapshot. That is why they are deleted in the source folder. It is IMHO a serious bug and I wonder why we do not hear about it more often.

building file list ... done
rsync: [generator] chgrp "/home" failed: Operation not permitted (1)
home/
deleting home/user/Downloads/foobar/exclude_me_folder/foobar
cannot delete non-empty directory: home/user/Downloads/foobar/exclude_me_folder
deleting home/user/Downloads/foobar/exclude_me_file
sent 267 bytes  received 204 bytes  942,00 bytes/sec
total size is 23  speedup is 0,05
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1338) [sender=3.2.7]
Restore permissions:
Restore permissions: Done

Looking into this (restore) snapshot log there is a secondary problem.

deleting home/user/Downloads/foobar/exclude_me_folder/foobar
cannot delete non-empty directory: home/user/Downloads/foobar/exclude_me_folder

The file exclude_me_folder/foobar is not really deleted but renamed to exclude_me_folder/foobar.backup.20231207 because of the first option in the restore dialog. So the folder exlude_me_folder is not empty and can not be deleted because of that.

Uff its long ago. Yes, it seems now to be a bug. But so far i remember, 2021 this Option (delete newer files in original folder) didnt existed.

The Problem about BackInTime in 2021 for me was, that if you created a snapshot of a folder, and restored it, you had duplicates in the same path/folder. One old File/Version, and one the newer one.

But i wanted some Option, wich would allow me to make 1:1 snapshots/backups/images (whatever you want call it), and restore it exactly like the backup.

Something like Timeshift would be perfect, but due to an incomprehensible reason Timeshift dont allow to use a (NFS-) Networkshare as Destination (origin developer said in some issues/posts thats because of hardlinks and rsync (wich isnt true, because it works fine)). So i created my own "solution".

But thanks for you guys working on backintime! im eager to see for new improvements and bugfixes :)

Edit: If i had a free wish, i would wish BackInTime would ditch rsync/$what_ever_is_used and would use something like Borg or duplicati. Because than the user could choose any backend (wich can be mounted or used with ftp, webdav, what ever) as destination. And could use options like compression, encryption, dedoublication or a combination of them.

[aryoda]

The main problem is that BIT/rsync do not take the exclude list into account. The excluded files/folders are not contained in the snapshot. That is why they are deleted in the source folder.

I guess implementation of the "Remove newer elements in original folder" restore option can be fixed here:

https://github.com/bit-team/backintime/blob/726bffa624d9a6da4ebd13ceb40721e1290e1eaf/common/snapshots.py#L478-L482

This code in the snapshots.py#backup() function is called in a separated thread from the Restore Dialog:

https://github.com/bit-team/backintime/blob/726bffa624d9a6da4ebd13ceb40721e1290e1eaf/qt/restoredialog.py#L116-L118

Could this be "good first issue" for (not totally fresh) developers?

[aryoda]

@solisinvictum

Edit: If I had a free wish, i would wish BackInTime would ditch rsync/$what_ever_is_used and would use something like Borg or duplicati. Because then the user could choose any backend (wich can be mounted or used with ftp, webdav, what ever) as destination. And could use options like compression, encryption, dedublication or a combination of them.

Thanks a lot for your feedback!

I have opened a new feature request issue for this (knowing that this change would be huge work) and I would like to take the chance to understand and document your requirements in the new issue #1582

Would it be possible to answer my questions there? THX a lot :-)