nextcloud
[Bug]: Client randomly re-uploads already synced files to s3
⚠️ Before submitting, please verify the following: ⚠️
- [X] This is a bug, not a question or a configuration issue.
- [X] This issue is not already reported on Github (I've searched it).
- [X] Nextcloud Server and Desktop Client are up to date. See Server Maintenance and Release Schedule and Desktop Releases for supported versions.
- [X] I agree to follow Nextcloud's Code of Conduct
Bug description
Given, an external storage (Wasabi S3) of the user called "EigeneDateien" exists and contains more than 10 files (e.g. 30 files), and the user syncs 4 new files via the windows desktop client newer than 3.4.1, then not only the 4 new files but also more already synced files will be randomly picked and re-synced. So an upload of e.g. 4 files will lead to an upload of 20 files (4 new, 16 resynced). But not the whole folder content of 30 files. Over and over again.
This doesn't happen at local storage. Also doesn't happen, when the external storage has less than 10 files. This also doesn't happen with version 3.4.1, but with 3.5.x and newer. Client versions between them are untested.
Steps to reproduce
- Set up an user external storage (here: wasabi s3, check for changes -> never)
- Use Windows Desktop Client version 3.6.0, 3.5.xxxx
- Create a folder (here: "EigeneDateien")
- Upload 8 files -> 8 files will be synced correctly
- Upload additional 8 additional files -> 8 files will be synced correctly
- Upload further 8 files -> suddenly e.g. 12 files were uploaded -> the 8 new files and some already synced files.
- Uninstall everything, install client version 3.4.1 (I tested with)
- Repeat steps above and notice things working fine.
Expected behavior
Syncing to external storage should only sync modified or new files but not untouched, already synced files.
Which files are affected by this bug
All file-types are affected
Operating system
Windows
Which version of the operating system you are running.
Windows 10, Windows 11
Package
Other
Nextcloud Server version
24.0.5
Nextcloud Desktop Client version
3.6.0
Is this bug present after an update or on a fresh install?
Updated to a major version (ex. 3.3.6 to 3.4.0)
Are you using the Nextcloud Server Encryption module?
Encryption is Disabled
Are you using an external user-backend?
- [X] Default internal user-backend
- [ ] LDAP/ Active Directory
- [ ] SSO - SAML
- [ ] Other
Nextcloud Server logs
No response
Additional info
No response
This sounds a bit like https://github.com/nextcloud/server/issues/33785 On my side it is now that the etag of folder changes all the time I enter it, via web.
If the etag changes the client re-uploads it. Can you check this?
Hello,
today, I checked this. I can enter the folder online or via filesystem without triggering uploads. But I checked the database tables (oc_filecache) and saw, that each re-upload creates official "versions". So the images are stored two/three/four/five times with different etags and version entries. I have a folder (2022-08) with 600 smartphone photos. The oc_filecache has 800 entries for files beginning with "2022-08-". So some files already have versions. Now I added one jpg-file to that folder, resulting in an upload of 500+ files (not finished yet) and the database has now 1350+ entries of files beginning with "2022-08-".
+---------+------------+----------------------------------+-----------------------------------------------------+
| fileid | mtime | etag | name |
+---------+------------+----------------------------------+-----------------------------------------------------+
| 1808608 | 1664203847 | 6331bc07ae46f | 2022-08-28_18-13-10_20220828_181310.jpg |
| 1810782 | 1664203782 | a869b76b633cd8886c06663486ae97cc | 2022-08-28_18-13-10_20220828_181310.jpg.v1664197661 |
+---------+------------+----------------------------------+-----------------------------------------------------+
So the files are not overwritten, but new copies/versions. I didn't notice this before. I don't think that's the etag-problem because I can enter and leave the directory as usual but only adding files causes this behavior.
Thanks
Here are some debug news.....:
- The problem is definetly within the nextcloud client.
- Tested two instances => showing the same problem
- The problem occurs since version 3.4.4 (tried 3.4.1, 3.4.2, 3.4.3 successfully, 3.5.0 and 3.6.0 also have this error)
- When files are being (re-)uploaded, a new "version" will be saved (also physically) until my server space went full (even the storage path of the images are is on an external s3 storage, file versions will be stored locally).
- So regarding "file versions": Each re-upload is a version and also using/wasting local space.
The server instances are up to date (I tested two instances), both hosted under my control (docker env).
Hope that helps.
Hi - any news on this?
If required, I can create and provide some access keys/buckets for testing purposes on my s3 account. Just send me a private message if needed.
My offer is still valid. I can provide buckets to test with.
Or is this problem/bug solved? Or can't you reproduce it? Please, comment on this. If this is a real bug, it's a bad one.
thanks.