rclone
bisync: add modtime support
What problem are you are trying to solve?
bisync delta engine and unit tests do not take backend time tolerance into account. this should be fixed.
modtime in unit tests
https://github.com/rclone/rclone/pull/5164#discussion_r634229878 (ivandeex)
the unit test runner is based on mangleListing which is a very far relative of fstest.CompareItems. It already has ability to ignore hashsums on demand. It still needs ability to compare times with per-backend tolerance or even ignore times if unsupported. Having that would let us run tests against backends without time support and probably lift the corresponding ban.
Tentatively this can be implemented as:
- parse out the time filed and convert to
time.Time - round time down to doubled tolerance
- put time field back into string
- feed the result to
diff.
Why not just parse and compare with tolerance?
Because mangleResult is two-fold. Besides comparing it can prepare a test state snapshot for storing in golden directory, which later can provide a bigger picture of what's going on.
https://github.com/rclone/rclone/pull/5164#issuecomment-843151817 (ivandeex)
With a feature must come a method of testing it.
Rclone has a comprehensive fstest framework dedicated to peculiarities of the plethora of supported cloud providers. Planning to work on this patch I had the heart to keep the project test engine's nature (while translating it from python to golang and refactoring it to work natively/unattended on Windows and with a wider set of backends) and propose it as a new kid on the set of rclone approaches (see my inline comment above), instead of rewriting it from the grounds up for fstest.
The bisync test suite lacks the ability to compare imprecise/absent timestamps as of the time of this writing (I hope I can say it, absent timestamps should be "comparable" too). It works correctly mostly because it compares logs from a particular filesystem with itself. Unless this ward is fixed, we can't run tests against backends without modtime and consequently can't declare them supported. I have an idea on how to fix that (see another inline comment), but the effort will need time.
modtime in delta engine
Try to extend operations.Equal into operations.Compare with more returned conditions like Newer, Older, ... (related to #4810) for direct use in bisync delta engine
https://github.com/rclone/rclone/pull/5164#issuecomment-842407758 (JWink3101)
The fact that this only supports a small subset of remotes is disappointing, esp. some popular ones not supporting mod time. Others, like S3, are way more performant if you disable it.
This is an issue with the overall algorithm of some sync tools but there is no reason you need modtime to decide what to sync. You may need it for conflicts but if you have past state, then you can tell if a changed file-size is a modification on one side only.
https://github.com/rclone/rclone/pull/5164#issuecomment-842512914 (ivandeex)
This is an issue with the overall algorithm of some sync tools but there is no reason you need modtime to decide what to sync.
Theoretically, no. The bisync change detection in its current state is all about bit masks (as opposed to "rclonic" Equal). But it needs more testing.
How to use GitHub
- Please use the 👍 reaction to show that you are affected by the same issue.
- Please don't comment if you have no relevant information to add. It's just extra noise for everyone subscribed to this issue.
- Subscribe to receive notifications on status change and new comments.
