[BUG]: Media sync takes excessively long time

[Samda1]

Checked for duplicates?

• [X] This issue is not a duplicate

What are the steps to reproduce this bug?

I dowloaded Ankidroid 2.17alpha16 after experiencing "never-ending sync" from the latest Google Play release. While my collection syncronizes studied cards quickly in this new release, media of cards won't sync in a reasonable amount of time. I have kept my phone open for extended periods of time while waiting for media to sync, and it has never completed. This makes it difficult/impossible to study on the phone, as many/most cards lack images.

Expected behaviour

Media should download in a reasonable timeframe

Actual behaviour

Media download extremely slowly (functional never-ending media sync)

Debug info

AnkiDroid Version = 2.17alpha16 (37daacf8e6a9642b85135ce3c00fa44cb58ea061)

Backend Version = 0.1.34-anki23.12.1 (23.12.1 1a1d4d5419c6b57ef3baf99c9d2d9cf85d36ae0a)

Android Version = 11 (SDK 30)

ProductFlavor = full

Manufacturer = samsung

Model = SM-G973U

Hardware = qcom

Webview User Agent = Mozilla/5.0 (Linux; Android 11; SM-G973U Build/RP1A.200720.012; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/120.0.6099.230 Mobile Safari/537.36

ACRA UUID = a6ef61c9-0563-4547-9027-cedcc0fbc9de

Crash Reports Enabled = true

(Optional) Anything else you want to share?

No response

Research

• [X] I am reporting a bug specific to AnkiDroid (Android app)
• [X] I have checked the manual and the FAQ and could not find a solution to my issue
• [X] (Optional) I have confirmed the issue is not resolved in the latest alpha release (instructions)

[welcome[bot]]

Hello! 👋 Thanks for logging this issue. Please remember we are all volunteers here, so some patience may be required before we can get to the issue. Also remember that the fastest way to get resolution on an issue is to propose a change directly, https://github.com/ankidroid/Anki-Android/wiki/Contributing

[david-allison]

How much media do you have listed on https://ankiweb.net/account/media

[Samda1]

5726.33MB. I previously had closer to 8GB but emptied trash and synchronized my collection from my computer a few minutes ago. The sync is still excessively slow on Anki droid. Is this expected with a collection of this size?

[david-allison]

You're downloading 5GB, it'll take a while

[Samda1]

You're downloading 5GB, it'll take a while

Of course, but the time is excessive. On my computer this takes maybe 5-10. On Ankidroid, I can leave it open for 20+ minutes and not even come close to downloading everything.

[kieranclancy]

I have recently been trying to transfer all my phone's data over to a new phone, and the backup process was taking forever, it kept stalling or getting stuck and having to be restarted from the beginning.

When I investigated, I found the issue seemed to be that the Android/data/com.ichi2.anki/files/AnkiDroid1/collection.media contained over 22,000 files despite being just 450MB. So that's around 20KB per file on average (small images and audio files). So when trying to transfer files over either the network or over USB it was glacially slow, since each one took almost a second to transfer.

I wonder if this is the issue that @Samda1 has hit. Could a sqlite database be used instead of many small individual files? If others' collections are similar in the number of files to mine (or maybe 10 times larger for @Samda1) it could be adding a lot of overhead to transferring (not to mention potentially making the whole system slower when it has to index all these files). Maybe it would make sense to have a threshold where above e.g. 1MB the file is stored on the filesystem, and below 1MB it is stored in a database?

It looks like WebViewAssetLoader.PathHandler in Android supports intercepting any media URL, so it should be possible to register two PathHandlers, one that looks in the filesystem and one that looks in the SQLite database.

https://developer.android.com/reference/androidx/webkit/WebViewAssetLoader.PathHandler

[mikehardy]

Hey @kieranclancy - If the performance information wasn't obtained with the new 2.17betas (https://github.com/ankidroid/Anki-Android/releases) then I'm afraid it will be invalid as the sync code is vastly different now

We will most likely not change storage backend from files to database, that would be a very large change even if technically feasible so is simply unlikely.

We think sync in 2.17 should be faster though

[FubayashiKazue]

I installed Anki 2.17 through F Droid yesterday and started a full sync to get my decks and files from AnkiWeb. it's still excessively slow. My collection.media is roughly 100.000 files and 4GB, the media syncs at 3000 - 5000 files per hour and is still not done after 8 hours.

[david-allison]

Likely an Android 11 issue along with F-Droid using "all files access".

The filesystem was made up to 100x slower by Android around that time

See if you can change your AnkiDroid directory to /storage/emulated/0/Android/com.ichi2.anki/data/files/AnkiDroid

[FubayashiKazue]

I'm using Android 13 on a Fairphone 3. I previously used the Google playstore Anki where the media folder was at the com.ichi2.anki directory. But since Android13 I was not successful at accessing the files there (mainly to copy over fonts to use on cards). The Files app can see the folder, but won't load the contents). So I switched to the F Droid Anki which uses emulated/0/Ankidroid directory and I can access the media files there properly. So I really don't want to change back to the com.ichi2.anki directory.

I understand that there may be nothing you can do because of Android restrictions. I'm letting it download all media on the background currently. Just wanted to add that the problem with slow media sync still persists even with 2.17.

From what I understand, if I see the "sync successfull" message at the bottom of the start screen, I can still do my reviews without losing any progress, is that right? it's just that the media still missing?

[mikehardy]

Just wanted to add that the problem with slow media sync still persists even with 2.17.

This is a true statement, but only for people that persist in using the general/shared storage vs app-specific storage

I expect that percentage of people to be vanishingly small (though vocal) and as you state - there is nothing we can do about the performance in that folder owing to Android restrictions

So, from a performance perspective I don't think we can spend the time to work on performance that isn't app-private storage

We should have some way for people to load and manage fonts and other _ files though 🤔

[FubayashiKazue]

Thanks for your reply! I will just patiently wait for the sync to be finished. Thankfully it continues in the background now (previous anki versions were crashing when using other apps or turning off the screen).

I wouldn't mind using the com.ichi2.anki app specific storage if there was another way to manage fonts (or other _ files) for using them in cards. That would be so nice!