dio
dio copied to clipboard
cfug
[android] Download is slow due to buffer settings
Package
dio
Version
5.3.3
Operating-System
Android 13
Output of flutter doctor -v
Doctor summary (to see all details, run flutter doctor -v):
[!] Flutter (Channel [user-branch], 3.13.0-0.2.pre, on macOS 12.6.5 21G531 darwin-arm64, locale en-IT)
! Flutter version 3.13.0-0.2.pre on channel [user-branch] at /Users/xxxxxx/Development/flutter
Currently on an unknown channel. Run `flutter channel` to switch to an official channel.
If that doesn't fix the issue, reinstall Flutter by following instructions at https://flutter.dev/docs/get-started/install.
! Upstream repository unknown source is not a standard remote.
Set environment variable "FLUTTER_GIT_URL" to unknown source to dismiss this error.
[!] Android toolchain - develop for Android devices (Android SDK version 31.0.0)
✗ cmdline-tools component is missing
Run `path/to/sdkmanager --install "cmdline-tools;latest"`
See https://developer.android.com/studio/command-line for more details.
✗ Android license status unknown.
Run `flutter doctor --android-licenses` to accept the SDK licenses.
See https://flutter.dev/docs/get-started/install/macos#android-setup for more details.
[✓] Xcode - develop for iOS and macOS (Xcode 14.0.1)
[✓] Chrome - develop for the web
[✓] Android Studio (version 2022.2)
Dart Version
Dart 3.1.0 (build 3.1.0-262.2.beta) • DevTools 2.25.0
Steps to Reproduce
We currently use Dio to download files from google drive.
await dio.download(
'https://www.googleapis.com/drive/v3/files/$id',
path,
queryParameters: {
'alt': 'media',
},
options: Options(headers: {
HttpHeaders.acceptEncodingHeader: '*',
'Authorization': token,
},
responseType: ResponseType.stream,
),
onReceiveProgress: (count, total) =>
onDownloadProgress.call(count, total),
);
Code works but it is slow and we noticed it is related to buffer.
For example, it takes 2 secs to download a file of about 258KB with a 200Mps connection and progress prints all below values. The same thing on iOS, where we don't use dio, just print 1 value and completes in a few millisecs. Is it a bug? Or is it possible to instruct dio to download files in chunks and set chunk size to 1024 * 1024?
I/flutter (18530): CloudBackup:: Downloading sqlite db file.
I/flutter (18530): CloudBackup: downloading sqliteFile 762 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 3518 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 4896 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 6274 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 7652 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 9030 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 10408 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 11786 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 13164 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 14542 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 14992 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 16370 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 17748 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 19126 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 20504 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 21882 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 24638 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 26016 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 27394 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 28772 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 30150 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 31376 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 32754 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 34132 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 35510 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 36888 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 38266 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 39644 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 41022 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 42400 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 43778 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 45156 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 46534 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 47760 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 49138 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 50516 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 51894 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 53272 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 54650 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 56028 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 57406 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 58784 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 60162 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 61540 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 62918 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 64132 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 65510 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 66888 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 68266 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 69644 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 71022 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 72400 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 73778 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 75156 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 77912 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 79290 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 80516 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 81894 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 83272 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 84650 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 86028 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 87406 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 88784 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 90162 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 91540 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 92918 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 94296 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 95674 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 96900 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 98278 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 99656 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 101034 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 102412 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 103790 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 105168 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 106546 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 107924 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 109302 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 110680 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 112058 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 113284 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 114662 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 118796 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 120174 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 121552 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 122930 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 124308 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 125686 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 127064 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 128442 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 129668 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 131046 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 132424 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 133802 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 135180 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 136558 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 137936 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 139314 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 140692 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 142070 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 143448 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 144826 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 146052 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 147430 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 148808 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 150186 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 152942 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 154320 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 155698 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 157076 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 158454 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 159832 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 161210 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 162436 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 163814 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 165192 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 166570 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 167948 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 169326 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 170704 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 172082 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 173460 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 174838 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 176216 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 177594 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 178820 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 180198 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 181576 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 182954 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 184332 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 185710 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 187088 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 188466 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 189844 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 191222 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 192600 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 193978 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 195204 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 196582 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 197960 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 199338 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 200716 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 202094 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 203472 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 204850 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 206228 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 207606 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 210362 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 211588 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 212966 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 214344 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 215722 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 217100 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 218478 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 219856 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 221234 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 222612 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 223990 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 225368 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 226746 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 227972 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 229350 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 230728 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 232106 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 233484 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 234862 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 236240 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 237618 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 238996 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 240374 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 241752 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 243130 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 244356 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 245734 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 247112 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 248490 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 249868 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 251246 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 252624 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 254002 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 255380 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 256758 of 258048
Expected Result
File should be downloaded faster. This might be a log for a scenario with about 100 KB chunk size.
I/flutter (18530): CloudBackup:: Downloading sqlite db file.
I/flutter (18530): CloudBackup: downloading sqliteFile 105168 of 258048
I/flutter (18530): CloudBackup: downloading sqliteFile 256758 of 258048
Actual Result
As the above logs.
- We no longer support Dio v3.
- Please provide a minimal example of your comparison.
Hi Alex,
-
Sorry, pasted wrong number, using ^5.3.3 Issue is reproducible also with native_dio_adapter.dart. Strangely, same code doesn't work with http2 adaper.
-
The ios code uses a native plugin. But the sample provided should be good to reproduce issue. I didn't count the logs lines pasted above but I guess we print 100 lines of progress for a 200KB file. Progress depends much on system and configuration, but it is normal to expect zero or just a few lines of progress before a 200KB download completes.
Tested on macOS. It seems perfectly reached my maximum bandwidth. (https://hnd-jp-ping.vultr.com/vultr.com.100MB.bin)
Sorry, I missed to specify this occurs on android. We are testing with v13, maybe issue also occur with v12.
Sorry, I missed to specify this occurs on android. We are testing with v13, maybe issue also occur with v12.
What is v13?
I tried your file above.
Look at first logs: 7777, 15968, 15970, 24161, 32352, 32354 Is that like a buffer size issue?
CloudBackup: downloading 7777 of 104857600
I/flutter (21600): CloudBackup: downloading 15968 of 104857600
I/flutter (21600): CloudBackup: downloading 15970 of 104857600
I/flutter (21600): CloudBackup: downloading 24161 of 104857600
I/flutter (21600): CloudBackup: downloading 32352 of 104857600
I/flutter (21600): CloudBackup: downloading 32354 of 104857600
etc...
Tested on Android 11:
Launching lib/main.dart on ONEPLUS A6010 in debug mode...
Running Gradle task 'assembleDebug'...
✓ Built build/app/outputs/flutter-apk/app-debug.apk.
Installing build/app/outputs/flutter-apk/app-debug.apk...
Debug service listening on ws://127.0.0.1:50091/QhHYdlfBDfU=/ws
Syncing files to device ONEPLUS A6010...
I/flutter ( 2282): 0:00:24.937422, speed: 4266.666666666667KB/s
I/flutter ( 2282): 0:00:25.503617, speed: 4096.0KB/s
I/flutter ( 2282): 0:00:25.167354, speed: 4096.0KB/s
Tested on Android 13:
Launching lib/main.dart on PHB110 in debug mode...
Running Gradle task 'assembleDebug'...
✓ Built build/app/outputs/flutter-apk/app-debug.apk.
Installing build/app/outputs/flutter-apk/app-debug.apk...
Debug service listening on ws://127.0.0.1:50490/SY4CVCRm4_E=/ws
Syncing files to device PHB110...
I/flutter (22557): 0:00:25.496580, speed: 4096.0KB/s
I/flutter (22557): 0:00:25.543291, speed: 4096.0KB/s
I/flutter (22557): 0:00:25.027422, speed: 4096.0KB/s
I'm not sure about your statement regarding the buffer size. It seems we can get constant results between native platforms, regardless the implementation. The progress log only tells how many bytes are accepted in the current stream event. Other than that, if you dont throttle logs for it, it might cause performance regressions too.
I just tried printing %
onReceiveProgress: (received, total) {
if (total != -1) {
print('downloading: ${(received / total * 100).toStringAsFixed(0)} %');
}
I/flutter (10195): downloading: 2 %
I/flutter (10195): downloading: 10 %
I/flutter (10195): downloading: 14 %
I/flutter (10195): downloading: 18 %
I/flutter (10195): downloading: 22 %
I/flutter (10195): downloading: 26 %
I/flutter (10195): downloading: 30 %
I/flutter (10195): downloading: 34 %
I/flutter (10195): downloading: 38 %
I/flutter (10195): downloading: 42 %
I/flutter (10195): downloading: 43 %
I/flutter (10195): downloading: 47 %
I/flutter (10195): downloading: 51 %
I/flutter (10195): downloading: 55 %
I/flutter (10195): downloading: 59 %
I/flutter (10195): downloading: 63 %
I/flutter (10195): downloading: 67 %
I/flutter (10195): downloading: 71 %
I/flutter (10195): downloading: 75 %
I/flutter (10195): downloading: 79 %
I/flutter (10195): downloading: 83 %
I/flutter (10195): downloading: 87 %
I/flutter (10195): downloading: 94 %
I/flutter (10195): downloading: 98 %
I/flutter (10195): downloading: 100 %
And if I disable onReceiveProgress, download takes same amount of time.
To narrow the scope of the issue, could you remove the log in onReceiveProgress and leave it as a empty callback? Then we can determine whether it's a logging throttle issue or caused by the callback.
Logs removed
onReceiveProgress: (received, total) {
if (total != -1) {
// print('downloading: ${(received / total * 100).toStringAsFixed(0)} %');
}
It takes same amount of time. It indeed makes sense, coz considering % logs above, we can expect callback to be invoked 1 or 2 times (like 0% and soon after 100% coz download should complete in <1s).
Instead we see %: 2, 10, 14, 18, ... etc... 42, 43, ...etc ... which shouldn't occur for a 200KB file with a 200Mbps connection.
Mine, the same file (200KB) takes same amount of time to download, with or without printing log inside callback.
At first glance, I see two potential issues:
- As you mentioned, a callback issue. Dio invokes callback too often. Hence, for example, calling
onReceiveProgress30 times for a process that, if not stalled from callback, would complete in a few millisecs. - There might be a low level issue, coz if I disable
onReceiveProgress(not just leaving it empty but removing the callback itself), download stills take the same big amount of time to complete.
Using a small file (100-300KB) to test would make things more evident.
I added your code to also print speed, which in my case is InfinityKB/s
I/flutter (16212): downloading: 2 %
I/flutter (16212): downloading: 10 %
I/flutter (16212): downloading: 14 %
I/flutter (16212): downloading: 18 %
I/flutter (16212): downloading: 22 %
I/flutter (16212): downloading: 26 %
I/flutter (16212): downloading: 30 %
I/flutter (16212): downloading: 34 %
I/flutter (16212): downloading: 38 %
I/flutter (16212): downloading: 42 %
I/flutter (16212): downloading: 43 %
I/flutter (16212): downloading: 47 %
I/flutter (16212): downloading: 51 %
I/flutter (16212): downloading: 55 %
I/flutter (16212): downloading: 59 %
I/flutter (16212): downloading: 63 %
I/flutter (16212): downloading: 67 %
I/flutter (16212): downloading: 71 %
I/flutter (16212): downloading: 75 %
I/flutter (16212): downloading: 79 %
I/flutter (16212): downloading: 83 %
I/flutter (16212): downloading: 87 %
I/flutter (16212): downloading: 91 %
I/flutter (16212): downloading: 95 %
I/flutter (16212): downloading: 99 %
I/flutter (16212): downloading: 100 %
I/flutter (16212): 0:00:00.848312,speed: InfinityKB/s
I/flutter (16212): 0:00:00.687079,speed: InfinityKB/s
I/flutter (16212): 0:00:00.515818,speed: InfinityKB/s
Maybe this helps:
I tested the download by downloading in chunks (using your 100MB file) A)
I/flutter (16212): CloudBackup: downloading sqliteFile 1048576 of 104857600
I/flutter (16212): CloudBackup: downloading sqliteFile 2097152 of 104857600
I/flutter (16212): CloudBackup: downloading sqliteFile 3145728 of 104857600
I/flutter (16212): CloudBackup: downloading sqliteFile 4194304 of 104857600
I/flutter (16212): CloudBackup: downloading sqliteFile 5242880 of 104857600
Printed values are 1048576, 2097152, 3145728, 4194304, 5242880, etc
When using Dio with your 100MB file logs were: B)
I/flutter (21600): CloudBackup: downloading 15968 of 104857600
I/flutter (21600): CloudBackup: downloading 15970 of 104857600
I/flutter (21600): CloudBackup: downloading 24161 of 104857600
I/flutter (21600): CloudBackup: downloading 32352 of 104857600
I/flutter (21600): CloudBackup: downloading 32354 of 104857600
Printed values are 15968 , 15970 , 24161 , 32352 , 32354 , etc
So, with solution A we downloaded 1MB when DIO downloaded 15.9KB. And so on so forth.
Hence, we can assume that is a low level issue.
To test things yourselves, here is our chunkedDownloader:
library chunked_downloader;
/// Original from plugin https://pub.dev/packages/chunked_downloader
/// We just added minor changes to inject queryParameters
import 'dart:async';
import 'dart:io';
import 'dart:typed_data';
import 'package:async/async.dart';
import 'package:flutter/cupertino.dart';
import 'package:http/http.dart' as http;
import 'package:http/http.dart';
/// Progress Callback
/// [progress] is the current progress in bytes
/// [total] is the total size of the file in bytes
typedef ProgressCallback = void Function(int progress, int total, double speed);
/// On Done Callback
/// [file] is the downloaded file
typedef OnDoneCallback = void Function(File file);
/// On Error Callback
/// [error] is the error that occured
typedef OnErrorCallback = void Function(dynamic error);
/// Custom Downloader with ChunkSize
///
/// [chunkSize] is the size of each chunk in bytes
///
/// [onProgress] is the callback function that will be called when the download is in progress
///
/// [onDone] is the callback function that will be called when the download is done
///
/// [onError] is the callback function that will be called when the download is failed
///
/// [onCancel] is the callback function that will be called when the download is canceled
///
/// [onPause] is the callback function that will be called when the download is paused
///
/// [onResume] is the callback function that will be called when the download is resumed
///
class ChunkedDownloader {
final String url;
final String saveFilePath;
final int chunkSize;
final ProgressCallback? onProgress;
final OnDoneCallback? onDone;
final OnErrorCallback? onError;
final Function? onCancel;
final Function? onPause;
final Function? onResume;
StreamSubscription<StreamedResponse>? stream;
ChunkedStreamReader<int>? reader;
Map<String, String>? headers;
Map<String, String>? queryParameters;
double speed = 0;
bool paused = false;
bool done = false;
static const bool kDebugMode = false;
ChunkedDownloader({
required this.url,
required this.saveFilePath,
this.headers,
this.queryParameters,
this.chunkSize = 1024 * 1024, // 1 MB
this.onProgress,
this.onDone,
this.onError,
this.onCancel,
this.onPause,
this.onResume,
});
/// Start the download
/// @result {Future<ChunkedDownloader>} the current instance of the downloader
Future<ChunkedDownloader> start() async {
// Download file
try {
int offset = 0;
var httpClient = http.Client();
// Set queryParameters
var uri;
if (queryParameters != null) {
uri = Uri.parse(url).replace(queryParameters: queryParameters);
} else {
uri = Uri.parse(url);
}
// Uri uri = Uri.parse(url);
// var finalUri = uri.replace(queryParameters: queryParameters);
var request = http.Request('GET', uri);
// Set headers
if (headers != null) {
request.headers.addAll(headers!);
}
debugPrint('ChunkedDownloader: uri ${uri}');
var response = httpClient.send(request);
// Open file
File file = File('$saveFilePath');
stream = response.asStream().listen(null);
stream?.onData((http.StreamedResponse r) async {
// Get file size
int fileSize = int.tryParse(r.headers['content-length'] ?? '-1') ?? -1;
reader = ChunkedStreamReader(r.stream);
try {
Uint8List buffer;
do {
// TODO: better pausing
while (paused) {
await Future.delayed(const Duration(milliseconds: 500));
}
// Set start time for speed calculation
int startTime = DateTime.now().millisecondsSinceEpoch;
// Read chunk
buffer = await reader!.readBytes(chunkSize);
// Calculate speed
int endTime = DateTime.now().millisecondsSinceEpoch;
int timeDiff = endTime - startTime;
if (timeDiff > 0) {
speed = (buffer.length / timeDiff) * 1000;
}
// Add buffer to chunks list
offset += buffer.length;
if (kDebugMode) {
print('Downloading ${offset ~/ 1024 ~/ 1024}MB '
'Speed: ${speed ~/ 1024 ~/ 1024}MB/s');
}
if (onProgress != null) {
onProgress!(offset, fileSize, speed);
}
// Write buffer to disk
await file.writeAsBytes(buffer, mode: FileMode.append);
} while (buffer.length == chunkSize);
// Rename file from .tmp to non-tmp extension
//await file.rename(saveFilePath);
// Send done callback
done = true;
if (onDone != null) {
onDone!(file);
}
if (kDebugMode) {
print('Downloaded file.');
}
} catch (error) {
if (kDebugMode) {
print('Error downloading: $error');
}
if (onError != null) {
onError!(error);
}
} finally {
reader?.cancel();
stream?.cancel();
}
});
} catch (error) {
if (kDebugMode) {
print('Error downloading: $error');
}
if (onError != null) {
onError!(error);
}
}
return this;
}
/// Stop the download
void stop() {
stream?.cancel();
reader?.cancel();
if (onCancel != null) {
onCancel!();
}
}
/// Pause the download
void pause() {
paused = true;
if (onPause != null) {
onPause!();
}
}
/// Resume the download
void resume() {
paused = false;
if (onResume != null) {
onResume!();
}
}
}
And here is sample code:
final url = 'https://hnd-jp-ping.vultr.com/vultr.com.100MB.bin';
var chunkedDownloader = await ChunkedDownloader(
url: url,
saveFilePath: path,
chunkSize: 1024 * 1024, // Buffer set to 1MB
headers: {
'Content-Type': 'application/octet-stream',
'HttpHeaders.acceptEncodingHeader': '*',
},
onError: (error) {},
onProgress: (received, total, speed) {
print('downlaoding file $url : $received of $total');
},
onDone: (file) {})
.start();
Also tested this different approach with await sink.addStream(response.stream.map((e) {...}
Download is terribly slow as with Dio.
I might be wrong, but at this point I think it is an issue with flutter stream.
Currently tested only on android, it might also occur on iOS.
var httpClient = http.Client();
final url = YOUR_URL // Possibly try with a 200-300KB file to better notice perf. issue.
var request = new http.Request('GET', Uri.parse(url));
final response = await httpClient.send(request);
final file = File(path);
await downloadFile(
file,
response,
onProgress: (bytes, total) {
if (total != null) {
onDownloadProgress.call(bytes, total);
print('donwload file progress ${(100 * (bytes / total)).round()}%');
}
}
);
static Future<void> downloadFile(
File file,
http.StreamedResponse response, {
void Function(int, int?)? onProgress,
}) async {
final sink = file.openWrite();
if (onProgress == null) {
await sink.addStream(response.stream);
} else {
var bytes = 0;
await sink.addStream(response.stream.map((e) {
bytes += e.length;
onProgress(bytes, response.contentLength);
return e;
}));
}
await sink.flush();
await sink.close();
}
Tested with different files (all about 200-300KB) with 200Mbps connection. Download takes 3-5 secs.
I added your code to also print speed, which in my case is InfinityKB/s
I/flutter (16212): downloading: 2 % I/flutter (16212): downloading: 10 % I/flutter (16212): downloading: 14 % I/flutter (16212): downloading: 18 % I/flutter (16212): downloading: 22 % I/flutter (16212): downloading: 26 % I/flutter (16212): downloading: 30 % I/flutter (16212): downloading: 34 % I/flutter (16212): downloading: 38 % I/flutter (16212): downloading: 42 % I/flutter (16212): downloading: 43 % I/flutter (16212): downloading: 47 % I/flutter (16212): downloading: 51 % I/flutter (16212): downloading: 55 % I/flutter (16212): downloading: 59 % I/flutter (16212): downloading: 63 % I/flutter (16212): downloading: 67 % I/flutter (16212): downloading: 71 % I/flutter (16212): downloading: 75 % I/flutter (16212): downloading: 79 % I/flutter (16212): downloading: 83 % I/flutter (16212): downloading: 87 % I/flutter (16212): downloading: 91 % I/flutter (16212): downloading: 95 % I/flutter (16212): downloading: 99 % I/flutter (16212): downloading: 100 % I/flutter (16212): 0:00:00.848312,speed: InfinityKB/s I/flutter (16212): 0:00:00.687079,speed: InfinityKB/s I/flutter (16212): 0:00:00.515818,speed: InfinityKB/s
@AlexV525 this class is buggy and poorly coded, had to rework it.
But the concept is correct and I gain perf improvement this way without using Dio.
Idea is, buffer lenght is chunkSize, buffer is filled with bytes from reader, and we write file a buffer at time. This reduces the numbers of write required to create the file and speed things up. Ofc, I can do that because we know file size and pass it to construct (no need to fetch it from content-lenght, which is not supported in many cases).
Indeed that might be a feature to add to Dio: if fileSize is known, we can pass fileSize and chunkSize and download perf will be boosted.
However, the underneath problem related to poor performsnce of flutter stream remains unfixed.
This is the class, lmk if you see improvement
library chunked_downloader;
/// Inspired by plugin https://pub.dev/packages/chunked_downloader
/// It was reworked for the most part
import 'dart:async';
import 'dart:io';
import 'package:async/async.dart';
import 'package:flutter/cupertino.dart';
import 'package:http/http.dart' as http;
import 'package:http/http.dart';
/// Progress Callback
/// [progress] is the current progress in bytes
/// [total] is the total size of the file in bytes
typedef ProgressCallback = void Function(int progress, int total, double speed);
/// On Done Callback
/// [file] is the downloaded file
typedef OnDoneCallback = void Function(File file);
/// On Error Callback
/// [error] is the error that occured
typedef OnErrorCallback = void Function(dynamic error);
/// Custom Downloader with ChunkSize
///
/// [chunkSize] is the size of each chunk in bytes
///
/// [onProgress] is the callback function that will be called when the download is in progress
///
/// [onDone] is the callback function that will be called when the download is done
///
/// [onError] is the callback function that will be called when the download is failed
///
/// [onCancel] is the callback function that will be called when the download is canceled
///
/// [onPause] is the callback function that will be called when the download is paused
///
/// [onResume] is the callback function that will be called when the download is resumed
///
class ChunkedDownloader {
final String url;
final String saveFilePath;
final int chunkSize;
final ProgressCallback? onProgress;
final OnDoneCallback? onDone;
final OnErrorCallback? onError;
final Function? onCancel;
final Function? onPause;
final Function? onResume;
StreamSubscription<StreamedResponse>? stream;
ChunkedStreamReader<int>? reader;
Map<String, String>? headers;
Map<String, String>? queryParameters;
double speed = 0;
bool paused = false;
bool done = false;
int? fileSize;
static const bool kDebugMode = false;
static bool downloadCompleted = false;
static bool downloadError = false;
ChunkedDownloader({
required this.url,
required this.saveFilePath,
this.fileSize,
this.headers,
this.queryParameters,
this.chunkSize = 1024 * 1024, // default is 1 MB
this.onProgress,
this.onDone,
this.onError,
this.onCancel,
this.onPause,
this.onResume,
});
static const String _debugPrefix = 'ChunkedDownloader:';
/// Start the download
/// @result {Future<ChunkedDownloader>} the current instance of the downloader
Future<void> start() async {
// Download file
try {
int offset = 0;
var httpClient = http.Client();
// Set queryParameters
var uri;
if (queryParameters != null) {
uri = Uri.parse(url).replace(queryParameters: queryParameters);
} else {
uri = Uri.parse(url);
}
// Uri uri = Uri.parse(url);
// var finalUri = uri.replace(queryParameters: queryParameters);
var request = http.Request('GET', uri);
// Set headers
if (headers != null) {
request.headers.addAll(headers!);
}
//debugPrint('$_debugPrefix uri ${uri}');
var response = httpClient.send(request);
// Initialize file
File file = File(saveFilePath);
// Number of iterations of while/do loop
// while the file is downloaded in chunks
int cycles = 0;
// Buffer is the bytes readed from readerChunk
List<int> buffer = [];
// bytes if the total numbers od bytes at each cycle
List<int> bytes= [];
stream = response.asStream().listen(null);
stream?.onData((http.StreamedResponse r) async {
// Get file size. Many servers don't provide [content-length], so
// standard use of this class is to pass fileSize as parameter
// In case of Presence we can do it coz fileSize is known.
fileSize = fileSize ?? int.tryParse(r.headers['content-length'] ?? '-1') ?? -1;
reader = ChunkedStreamReader(r.stream);
try {
do {
// Wait indefinitely if paused
await waitWhile(() => (paused));
// Set start time for speed calculation
int startTime = DateTime.now().millisecondsSinceEpoch;
// Read chunk
buffer = (fileSize! - bytes.length >= chunkSize) ?
await reader!.readChunk(chunkSize)
: await reader!.readChunk(fileSize! - bytes.length);
bytes = bytes + buffer;
// Calculate speed
int endTime = DateTime.now().millisecondsSinceEpoch;
int timeDiff = endTime - startTime;
if (timeDiff > 0) {
speed = (buffer.length / timeDiff) * 1000;
}
// Add buffer to chunks list
offset += buffer.length;
if (kDebugMode) {
debugPrint('$_debugPrefix Downloading ${offset ~/ 1024 ~/ 1024}MB '
'Speed: ${speed ~/ 1024 ~/ 1024}MB/s');
}
if (onProgress != null) {
onProgress!(offset, fileSize!, speed);
}
// Write buffer to disk
// the first cycle we override the initial file
// coz flutter create new file with 1024 bytes
// default size. On subsequent cycle we append
// new bytes to previous ones.
if (cycles == 0) {
await file.writeAsBytes(bytes);
} else {
await file.writeAsBytes(buffer, mode: FileMode.append);
}
cycles++;
} while (bytes.length < fileSize!);
//debugPrint('$_debugPrefix downloaded file size ${file.lengthSync()}');
// Send done callback
done = true;
if (onDone != null) {
onDone!(file);
downloadCompleted = true;
}
if (kDebugMode) {
debugPrint('$_debugPrefix Downloaded file.');
}
} catch (error) {
downloadError = true;
bytes.clear();
buffer.clear();
if (kDebugMode) {
debugPrint('$_debugPrefix Error downloading: $error');
}
if (onError != null) {
onError!(error);
}
} finally {
reader?.cancel();
stream?.cancel();
bytes.clear();
buffer.clear();
}
});
} catch (error) {
if (kDebugMode) {
debugPrint('$_debugPrefix Error downloading: $error');
}
if (onError != null) {
onError!(error);
}
}
// Do not return until download is !completed and !failed
// Note: this is required because we cannot
// await stream?.onData((http.StreamedResponse r)
// and await won't work with this func
await waitWhile(() => (!downloadCompleted && !downloadError));
}
/// Stop the download
void stop() {
stream?.cancel();
reader?.cancel();
if (onCancel != null) {
onCancel!();
}
}
/// Pause the download
void pause() {
paused = true;
if (onPause != null) {
onPause!();
}
}
/// Resume the download
void resume() {
paused = false;
if (onResume != null) {
onResume!();
}
}
}
// This is used to wait until test condition is not false
Future waitWhile(bool condition(), [Duration pollInterval = Duration.zero]) {
var completer = new Completer();
check() {
if (!condition()) {
completer.complete();
} else {
new Timer(pollInterval, check);
}
}
check();
return completer.future;
}
Sample code to use it.
final url = 'https://freetestdata.com/wp-content/uploads/2022/02/Free_Test_Data_1MB_JPG.jpg';
var chunkedDownloader = ChunkedDownloader(
url: url,
saveFilePath: filePath,
fileSize: fileSize,
chunkSize: 1024 * 1024,
onError: (error) {},
onProgress: (received, total, speed) {
onDownloadProgress.call(received, total);
debugPrint('downloading file: $received of $total');
},
onDone: (file) async {
debugPrint('file $url downloaded');
});
await chunkedDownloader.start();
Hello @AlexV525 can you please run this demo on your side?
It downloads the same 1MB file with three different methods.
Dio takes between 250ms-1.000ms to download 1MB file (200Mps). Although it looks like the slowest one, we must consider also the write File time. However, values between 400ms-1.000ms look huge anyway.
Sample
// ignore_for_file: prefer_const_constructors
import 'dart:io';
import 'package:flutter/material.dart';
import 'package:http/http.dart' as http;
import 'package:path_provider/path_provider.dart';
import 'package:dio/dio.dart';
void main() {
runApp(const MyApp());
}
class MyApp extends StatelessWidget {
const MyApp({super.key});
@override
Widget build(BuildContext context) {
return MaterialApp(
title: 'Flutter Demo',
theme: ThemeData(
colorScheme: ColorScheme.fromSeed(seedColor: Colors.deepPurple),
useMaterial3: true,
),
home: const MyHomePage(title: 'Flutter Demo Home Page'),
);
}
}
class MyHomePage extends StatefulWidget {
const MyHomePage({super.key, required this.title});
final String title;
@override
State<MyHomePage> createState() => _MyHomePageState();
}
class _MyHomePageState extends State<MyHomePage> {
void activateDownload() async {
Directory appDocDir = await getApplicationDocumentsDirectory();
String path = appDocDir.path;
final file = File('$path/example.jpg');
final url =
'https://freetestdata.com/wp-content/uploads/2022/02/Free_Test_Data_1MB_JPG.jpg';
print('1) Download File $url');
await downloadFile(file.path, url, onProgress: (bytes, total) {
if (total != null) {
//print('donwload file progress ${(100 * (bytes / total)).round()}%');
}
});
print('\n2) Dio');
print('\nDownload File $url');
await downloadFileWithDio(url, file.path);
print('\n3) DownTest Code');
print('\nDownload File $url');
await downloadTest(url);
}
static Future<void> downloadFile(
String filePath,
url, {
void Function(int, int?)? onProgress,
}) async {
final sink = File(filePath).openWrite();
var httpClient = http.Client();
var request = http.Request('GET', Uri.parse(url));
final response = await httpClient.send(request);
var startTime = DateTime.timestamp();
if (onProgress == null) {
await sink.addStream(response.stream);
} else {
var bytes = 0;
await sink.addStream(response.stream.map((e) {
bytes += e.length;
onProgress(bytes, response.contentLength);
return e;
}));
print(
'Downloaded $bytes bytes in ${DateTime.timestamp().difference(startTime).inMilliseconds}ms');
}
await sink.flush();
await sink.close();
}
static Future<void> downloadFileWithDio(
url,
String filePath,
) async {
Dio dio = Dio();
var startTime = DateTime.timestamp();
await dio.download(url, filePath,
onReceiveProgress: (bytes, total) {
//int percentage = ((received / total) * 100).floor();
if (bytes == total) {
print(
'Downloaded $bytes bytes in '
'${DateTime.timestamp().difference(startTime).inMilliseconds}ms');
}
});
}
static Future<void> downloadTest(url) async {
var httpClient = http.Client();
var request = http.Request('GET', Uri.parse(url));
var response = await httpClient.send(request);
var totalLength = 0;
var startTime = DateTime.timestamp();
await response.stream.listen((data) {
totalLength += data.length;
}).asFuture();
print(
'Downloaded $totalLength bytes in ${DateTime.timestamp().difference(startTime).inMilliseconds}ms');
}
@override
Widget build(BuildContext context) {
return Scaffold(
appBar: AppBar(
backgroundColor: Theme.of(context).colorScheme.inversePrimary,
title: Text(widget.title),
),
body: const Center(
child: Column(
mainAxisAlignment: MainAxisAlignment.center,
children: <Widget>[
Text(
'You have pushed the button this many times:',
),
],
),
),
floatingActionButton: FloatingActionButton(
onPressed: () {activateDownload();},
tooltip: 'Increment',
child: const Icon(Icons.add),
), // This trailing comma makes auto-formatting nicer for build methods.
);
}
}
I just tested with your example, and there is nothing related to file writing. The performances are poor whether using GET+Stream or the download method, so it should be an adapter issue.
I also noticed downloads+write being slow. It roughly takes double the time of the download which seems too much. This is just a gut feeling tho.
This is hard to debug more than I thought. I tried to separate flows but all flows seem working well. The next step I may try is to rebuild all flows from the bottom to the top.
I think there is an issue with how TLS sockets are handled. I have noticed a similar issue on https://github.com/localsend/localsend/issues/384#issuecomment-1513917835
After some debugging, I have noticed, that HTTP is much faster than HTTPS. In my test case, HTTP reached 50 MB/s while HTTPS only reached 7 MB/s.
I am not sure how to fix this as I only use the Flutter API (both with
dioand withhttppackage).It seems that Android is the device causing slow transmission speeds.
Windows <-> macOSreached 60 MB/s with HTTPS.
You can try to download a large file via unencrypted HTTP - it should be much faster. ~~It seems that other platforms don't have this issue~~.
Edit: I think, the TLS socket is just too CPU demanding. I have tried out to run a test on an old iPhone 5S and it has a similar problem. Turning off encryption offers 2x download / upload speeds. I guess newer iPhones are just too performant to have this problem.
Edit: I think, the TLS socket is just too CPU demanding. I have tried out to run a test on an old iPhone 5S and it has a similar problem. Turning off encryption offers 2x download / upload speeds. I guess newer iPhones are just too performant to have this problem.
Thanks for the input, and sad that the issue didn't get any notice from the Dart team.
However during my tests, when I was using https://freetestdata.com/wp-content/uploads/2022/02/Free_Test_Data_1MB_JPG.jpg as the source, I can see that http and HttpClient had a more normal speed but our package doesn't.
Transferring a 24mb file from my laptop to my Samsung s22 is only 600KB/s. I turned off wifi on my phone and turned it back on and the speed went back to normal!
Testing in another network location, the download elapsed duration is slightly higher than the other libraries, but the regression seems not occurred.
I/flutter ( 6803): 1) http write sink https://freetestdata.com/wp-content/uploads/2021/09/png-5mb-1.png
I/flutter ( 6803): Downloaded 5244276 bytes in 4442ms
I/flutter ( 6803): 2) Dio Download https://freetestdata.com/wp-content/uploads/2021/09/png-5mb-1.png
I/flutter ( 6803): Downloaded 5244276 bytes in 4586ms
I/flutter ( 6803): 3) http Stream -> Future https://freetestdata.com/wp-content/uploads/2021/09/png-5mb-1.png
I/flutter ( 6803): Downloaded 5244276 bytes in 4448ms
I/flutter ( 6803): 1) http write sink https://freetestdata.com/wp-content/uploads/2022/02/Free_Test_Data_1MB_JPG.jpg
I/flutter ( 6803): Downloaded 1054294 bytes in 909ms
I/flutter ( 6803): 2) Dio Download https://freetestdata.com/wp-content/uploads/2022/02/Free_Test_Data_1MB_JPG.jpg
I/flutter ( 6803): Downloaded 1054294 bytes in 1039ms
I/flutter ( 6803): 3) http Stream -> Future https://freetestdata.com/wp-content/uploads/2022/02/Free_Test_Data_1MB_JPG.jpg
I/flutter ( 6803): Downloaded 1054294 bytes in 897ms
Flutter 3.16.9 • channel stable • https://github.com/AlexV525/flutter
Framework • revision 41456452f2 (6 weeks ago) • 2024-01-25 10:06:23 -0800
Engine • revision f40e976bed
Tools • Dart 3.2.6 • DevTools 2.28.5
@iosephmagno Would you be able to run the same code using the 3.16 or a newer version of Flutter?