nextcloud
Huge download: File gets completely read even after download is cancelled
Steps to reproduce
- Download a huge file using the webinterface
- Cancel that download
- Check Server IO on that file
Expected behaviour
After the user cancels the download of that file, the server should stop reading the contents of that file.
Actual behaviour
The server keeps reading the contents of that file until it's completely read.
Server configuration
Operating system:
Arch Linux, 4.14.15-1-ARCH
Web server:
Server version: Apache/2.4.29 (Unix)
Server built: Oct 21 2017 12:45:02
Database:
mysql Ver 15.1 Distrib 10.1.30-MariaDB, for Linux (x86_64) using readline 5.1
PHP version:
PHP 7.2.2 (cli) (built: Jan 30 2018 19:18:38) ( NTS )
Copyright (c) 1997-2018 The PHP Group
Zend Engine v3.2.0, Copyright (c) 1998-2018 Zend Technologies
with Zend OPcache v7.2.2, Copyright (c) 1999-2018, by Zend Technologies
Nextcloud version: (see Nextcloud admin page) 13.0.0 RC 4
Updated from an older Nextcloud/ownCloud or fresh install: Fresh setup on 12.x
Where did you install Nextcloud from: Official updater
Signing status:
Signing status
``` Login as admin user into your Nextcloud and access http://example.com/index.php/settings/integrity/failed paste the results here. ``` No errors have been found. ``` If you have access to your command line run e.g.: sudo -u www-data php occ app:list from within your Nextcloud installation folder ``` ``` Enabled: - activity: 2.6.1 - bruteforcesettings: 1.0.3 - calendar: 1.5.7 - comments: 1.3.0 - contacts: 2.0.1 - dav: 1.4.6 - federatedfilesharing: 1.3.1 - federation: 1.3.0 - files: 1.8.0 - files_external: 1.4.1 - files_pdfviewer: 1.2.0 - files_sharing: 1.5.0 - files_texteditor: 2.5.1 - files_trashbin: 1.3.0 - files_versions: 1.6.0 - files_videoplayer: 1.2.0 - firstrunwizard: 2.2.1 - gallery: 18.0.0 - logreader: 2.0.0 - lookup_server_connector: 1.1.0 - metadata: 0.6.0 - nextcloud_announcements: 1.2.0 - notifications: 2.1.2 - oauth2: 1.1.0 - password_policy: 1.3.0 - previewgenerator: 1.0.9 - provisioning_api: 1.3.0 - serverinfo: 1.3.0 - sharebymail: 1.3.0 - survey_client: 1.1.0 - systemtags: 1.3.0 - theming: 1.4.1 - twofactor_backupcodes: 1.2.3 - updatenotification: 1.3.0 - workflowengine: 1.3.0 Disabled: - admin_audit - encryption - user_external - user_ldap ```Nextcloud configuration:
Config report
If you have access to your command line run e.g.:
sudo -u www-data php occ config:list system
from within your Nextcloud installation folder
or
Insert your config.php content here.
Make sure to remove all sensitive content such as passwords. (e.g. database password, passwordsalt, secret, smtp password, …)
{
"system": {
"instanceid": "***REMOVED SENSITIVE VALUE***",
"passwordsalt": "***REMOVED SENSITIVE VALUE***",
"secret": "***REMOVED SENSITIVE VALUE***",
"trusted_domains": [
"192.168.2.101"
],
"datadirectory": "***REMOVED SENSITIVE VALUE***",
"overwrite.cli.url": "http:\/\/192.168.2.101\/nextcloud",
"dbtype": "mysql",
"version": "13.0.0.13",
"dbname": "***REMOVED SENSITIVE VALUE***",
"dbhost": "***REMOVED SENSITIVE VALUE***",
"dbport": "",
"dbtableprefix": "oc_",
"dbuser": "***REMOVED SENSITIVE VALUE***",
"dbpassword": "***REMOVED SENSITIVE VALUE***",
"installed": true,
"memcache.local": "\\OC\\Memcache\\APCu",
"mail_smtpmode": "smtp",
"mail_smtpauthtype": "LOGIN",
"mail_from_address": "***REMOVED SENSITIVE VALUE***",
"mail_domain": "***REMOVED SENSITIVE VALUE***",
"mail_smtpsecure": "tls",
"mail_smtpauth": 1,
"mail_smtphost": "***REMOVED SENSITIVE VALUE***",
"mail_smtpport": "587",
"mail_smtpname": "***REMOVED SENSITIVE VALUE***",
"mail_smtppassword": "***REMOVED SENSITIVE VALUE***",
"apps_paths": [
{
"path": "***REMOVED SENSITIVE VALUE***\/apps",
"url": "\/apps",
"writable": false
},
{
"path": "***REMOVED SENSITIVE VALUE***\/apps2",
"url": "\/apps2",
"writable": true
}
],
"loglevel": 2,
"maintenance": false,
"updater.release.channel": "beta",
"theme": ""
}
}
Are you using external storage, if yes which one: local
Are you using encryption: no
Are you using an external user-backend, if yes which one: None
Client configuration
Browser: Mozilla Firefox 58.0.1
Operating system:
Arch Linux 4.14.15-1-ARCH
Logs
Web server error log
Web server error log
No relevant log entries
Nextcloud log (data/nextcloud.log)
Nextcloud log
No relevant log entries
Browser log
Browser log
Insert your browser log here, this could for example include:
a) The javascript console log
b) The network log
c) ...
Request headers
Host: 192.168.2.101
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:58.0) Gecko/20100101 Firefox/58.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
Cookie: ***REMOVED SENSITIVE VALUE***
DNT: 1
Connection: keep-alive
Upgrade-Insecure-Requests: 1
Response Headers
Date: Fri, 02 Feb 2018 19:43:03 GMT
Server: Apache
Strict-Transport-Security: max-age=15552000; includeSubDomains
X-Powered-By: PHP/7.2.2
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate
Pragma: no-cache
Content-Security-Policy: default-src 'none';
X-Frame-Options: SAMEORIGIN
Set-Cookie: ocDownloadStarted=***REMOVED SENSITIVE VALUE***; expires=Fri, 02-Feb-2018 19:43:24 GMT; Max-Age=20; path=/
Last-Modified: Sat, 07 Oct 2017 15:30:29 GMT
ETag: ***REMOVED SENSITIVE VALUE***
Content-Length: 26874923280
Content-Disposition: attachment; filename*=UTF-8''IMG_2328.MOV; filename="IMG_2328.MOV"
OC-ETag: ***REMOVED SENSITIVE VALUE***
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
X-Robots-Tag: none
X-Download-Options: noopen
X-Permitted-Cross-Domain-Policies: none
Keep-Alive: timeout=5, max=99
Connection: Keep-Alive
Content-Type: video/quicktime
not really, but I guess we run a transaction/locking etc. so we can not simply quit out of the request. and the cancel command is not continued.
Thanks a lot for taking this serious :) Not all open source projects are open to improvement requests. Keep up the great work!
@kesselb Hi kesselb, as you replied in my previous issue https://github.com/nextcloud/server/issues/22334 , this weekend I did some testing.
Definitely video playback is using streaming because I saw clearly Range header and 206 status code in HTTP request, however there was still a high IO load on server side. The detailed status changing will be given below:
-
For NextCloud video playback
- At the beginning php-fpm takes all disk reading bandwidth, say 120MB/s
- After it is loaded, there is no reading load even I drag the progress bar, there is write load instead (20-30 MB/s)
-
For Nginx static file hosting
- There are only reading loads no more than 30MB/s anytime
Meanwhile, I noticed that web video player was using WebDAV protocol upon file request, I checked https://github.com/nextcloud/server/blob/master/remote.php in master branch, noticed there is a line require_once $file;, is it possible that file gets completely read caused by this line of code? Would like to know why there is a require_once $file; at the end of this function.
I also faced this issue.
On my Nextcloud server I have "not so much" space, so bigger files I want share I put into Nextcloud via "Local external storage" which points to a mounted NFS share from another server.
After a download of such a shared file gets canceled, the file continues to be transfered from the NFS share - so this also produces a lot of network overhead for canceled downloads...
Found that this problem was also already reported in #15055.
This still occurs in latest version, downloads are a completely broken. Experiencing:
- When you first download a file, it goes much slower than following downloads.
- Big delay for starting downloads.
- After downloads completes, server continues reading the HDD at full speed for minutes.
- If you cancel download, server continues reading files at full speed making server unusable for a while.
https://github.com/sabre-io/http/pull/207 This seems to fix download and streaming problems
sabre-io/http#207 This seems to fix download and streaming problems
Doesnt seem to fix the StreamerInstance? When downloading a folder all of the files will still be read after the download is cancelled?
This should be reopened, 27.0.1 did not fix the issue. Nextcloud keeps downloading full files even after canceling the file download. Just try to download a 100gb file and cancelling it immediately. You can execute this command to check it out:
sudo iotop -o -a
Updating the sabre/io lib only fixed part of the issue, as nextcloud uses directly the function stream_copy_to_stream and it has not been patched in any way.
Doesnt seem to fix the StreamerInstance?
I don't think so.
When downloading a folder all of the files will still be read after the download is cancelled?
You can try to patch https://github.com/nextcloud/server/blob/608ba174a3e3e3b73b2fb35bc05d7373072667c3/lib/private/Streamer.php#L125C30-L125C30 similarly. Inside the loop, check if the client is still there and stop the process if not. Happy hacking :construction_worker:
Doesnt seem to fix the StreamerInstance?
I don't think so.
When downloading a folder all of the files will still be read after the download is cancelled?
You can try to patch https://github.com/nextcloud/server/blob/608ba174a3e3e3b73b2fb35bc05d7373072667c3/lib/private/Streamer.php#L125C30-L125C30 similarly. Inside the loop, check if the client is still there and stop the process if not. Happy hacking construction_worker
I feel like thats a pretty bad way to fix it. Ideally i'd want to cancel reading as soon as i know the client is gone. And even though this is PHP (which tbh. makes things like this super annoying to deal with), i think that should be possible?
EDIT: I think i found a way. If we only read chunks, we can check after each chunk if the client is gone which would be much faster and consistent.
But it seems to be harder here as it uses the tool ZipStreamer.. but not even sure where i can find that one?
EDIT2: I added a PR for PHPZipStreamer: https://github.com/DeepDiver1975/PHPZipStreamer/pull/13
If that doesnt get merged, the alternative is to use a ProxyStream and check it there between read-requests.
I am thinking of replacing deepdiver/zipstreamer with maennchen/zipstream-php and therefore would prefer your ProxyStream approach.
https://github.com/icewind1991/Streams/blob/master/src/CallbackWrapper.php could be useful.
If you use the CallbackWrapper, please throw an exception from the read callback to handle the situation properly (e.g., close the input stream, bubble the error up, etc.).
Please don't hesitate to reach out via https://cloud.nextcloud.com/call/xs25tz5y if you need input or stuck.
This is rather problematic if you have a remote primary storage, e.g. S3, or simply download from an external storage. I canceled downloading a 80 GB folder immediately after starting it, and Nextcloud still read and compressed all objects from the bucket, taking quite a lot of time, bandwidth (which translates to cost, it is several dollars for such a big folder) and resources.
This is rather problematic if you have a remote primary storage, e.g. S3, or simply download from an external storage. I canceled downloading a 80 GB folder immediately after starting it, and Nextcloud still read and compressed all objects from the bucket, taking quite a lot of time, bandwidth (which translates to cost, it is several dollars for such a big folder) and resources.
Yes, very big issue, I confirm this still happens in nextcloud 28.0.0 and nothing changes. Surprising that this is not in top priority, main objective of nextcloud is download and sync files. But I noticed a download speed improvement, due to php performance improvements I guess.
It seems I was wrong, it works nicely in nextcloud 28.0.0, when you cancel a download it stops reading it from disk. It didn't work initialy after first very test after upgrading to 28.0.0 but its definetely working now after server restart:
- Single file download -> OK on user download cancel
- Video preview -> OK on user cancel
- Folder download -> Still an issue being addressed in #42352