plupload icon indicating copy to clipboard operation
plupload copied to clipboard

Corrupted files when uploading chunks with max_upload_slots > 1

Open Danoontjes opened this issue 7 years ago • 0 comments

I am trying to use plupload 3.1.2 to upload files an internal server (fast connection). When using a max_upload_slots setting more than 1 and combine_chunks_on_complete = true, the isLastChunk function from PluploadHandler.php returns positive for multiple upload streams. This results in multiple instances of upload.php starting to concatenate the uploaded chunks and a concatenated file that is much larger than the original.

I have looked at the code, but I cannot see a way to ensure the concatenating is only started once.

My workaround is to set combine_chunks_on_complete to false, and trigger my own concatenating code when the upload completes.

Debug log of an occurrence of the issue:

2018-04-05 09:42:11: x2goclient_4.1.1.0_setup2.exe.7z received, chunks enabled: 5 of 6
2018-04-05 09:42:11: standalone chunks being written to /lustre/dda/temp/x2goclient_4.1.1.0_setup2.exe.7z.dir.part
2018-04-05 09:42:11: /tmp/phpskCzu0 written to /lustre/dda/temp/x2goclient_4.1.1.0_setup2.exe.7z.dir.part/5.part
2018-04-05 09:42:11: filesize obtained via exec.
2018-04-05 09:42:11: x2goclient_4.1.1.0_setup2.exe.7z received, chunks enabled: 1 of 6
2018-04-05 09:42:11: standalone chunks being written to /lustre/dda/temp/x2goclient_4.1.1.0_setup2.exe.7z.dir.part
2018-04-05 09:42:11: /tmp/phpJuehIZ written to /lustre/dda/temp/x2goclient_4.1.1.0_setup2.exe.7z.dir.part/1.part
2018-04-05 09:42:11: filesize obtained via exec.
2018-04-05 09:42:12: x2goclient_4.1.1.0_setup2.exe.7z received, chunks enabled: 3 of 6
2018-04-05 09:42:12: standalone chunks being written to /lustre/dda/temp/x2goclient_4.1.1.0_setup2.exe.7z.dir.part
2018-04-05 09:42:12: x2goclient_4.1.1.0_setup2.exe.7z received, chunks enabled: 4 of 6
2018-04-05 09:42:12: standalone chunks being written to /lustre/dda/temp/x2goclient_4.1.1.0_setup2.exe.7z.dir.part
2018-04-05 09:42:12: x2goclient_4.1.1.0_setup2.exe.7z received, chunks enabled: 2 of 6
2018-04-05 09:42:12: standalone chunks being written to /lustre/dda/temp/x2goclient_4.1.1.0_setup2.exe.7z.dir.part
2018-04-05 09:42:12: x2goclient_4.1.1.0_setup2.exe.7z received, chunks enabled: 0 of 6
2018-04-05 09:42:12: standalone chunks being written to /lustre/dda/temp/x2goclient_4.1.1.0_setup2.exe.7z.dir.part
2018-04-05 09:42:12: /tmp/phpeQNUIZ written to /lustre/dda/temp/x2goclient_4.1.1.0_setup2.exe.7z.dir.part/3.part
2018-04-05 09:42:12: seems like last chunk (3), 'cause there are 6 out of 6 *.part files in /lustre/dda/temp/x2goclient_4.1.1.0_setup2.exe.7z.dir.part.
2018-04-05 09:42:12: /tmp/phpq6BYIZ written to /lustre/dda/temp/x2goclient_4.1.1.0_setup2.exe.7z.dir.part/4.part
2018-04-05 09:42:12: seems like last chunk (4), 'cause there are 6 out of 6 *.part files in /lustre/dda/temp/x2goclient_4.1.1.0_setup2.exe.7z.dir.part.
2018-04-05 09:42:12: /tmp/phpUvXZg0 written to /lustre/dda/temp/x2goclient_4.1.1.0_setup2.exe.7z.dir.part/0.part
2018-04-05 09:42:12: seems like last chunk (0), 'cause there are 6 out of 6 *.part files in /lustre/dda/temp/x2goclient_4.1.1.0_setup2.exe.7z.dir.part.
2018-04-05 09:42:12: /tmp/phpmNlwL8 written to /lustre/dda/temp/x2goclient_4.1.1.0_setup2.exe.7z.dir.part/2.part
2018-04-05 09:42:12: seems like last chunk (2), 'cause there are 6 out of 6 *.part files in /lustre/dda/temp/x2goclient_4.1.1.0_setup2.exe.7z.dir.part.
2018-04-05 09:42:12: /lustre/dda/temp/x2goclient_4.1.1.0_setup2.exe.7z.dir.part/0.part appended to /lustre/dda/temp/x2goclient_4.1.1.0_setup2.exe.7z.part
2018-04-05 09:42:12: /lustre/dda/temp/x2goclient_4.1.1.0_setup2.exe.7z.dir.part/0.part appended to /lustre/dda/temp/x2goclient_4.1.1.0_setup2.exe.7z.part
2018-04-05 09:42:12: /lustre/dda/temp/x2goclient_4.1.1.0_setup2.exe.7z.dir.part/0.part appended to /lustre/dda/temp/x2goclient_4.1.1.0_setup2.exe.7z.part
2018-04-05 09:42:12: /lustre/dda/temp/x2goclient_4.1.1.0_setup2.exe.7z.dir.part/0.part appended to /lustre/dda/temp/x2goclient_4.1.1.0_setup2.exe.7z.part
2018-04-05 09:42:12: /lustre/dda/temp/x2goclient_4.1.1.0_setup2.exe.7z.dir.part/1.part appended to /lustre/dda/temp/x2goclient_4.1.1.0_setup2.exe.7z.part
2018-04-05 09:42:13: /lustre/dda/temp/x2goclient_4.1.1.0_setup2.exe.7z.dir.part/1.part appended to /lustre/dda/temp/x2goclient_4.1.1.0_setup2.exe.7z.part
2018-04-05 09:42:13: /lustre/dda/temp/x2goclient_4.1.1.0_setup2.exe.7z.dir.part/1.part appended to /lustre/dda/temp/x2goclient_4.1.1.0_setup2.exe.7z.part
2018-04-05 09:42:13: /lustre/dda/temp/x2goclient_4.1.1.0_setup2.exe.7z.dir.part/1.part appended to /lustre/dda/temp/x2goclient_4.1.1.0_setup2.exe.7z.part
2018-04-05 09:42:13: /lustre/dda/temp/x2goclient_4.1.1.0_setup2.exe.7z.dir.part/2.part appended to /lustre/dda/temp/x2goclient_4.1.1.0_setup2.exe.7z.part
2018-04-05 09:42:13: /lustre/dda/temp/x2goclient_4.1.1.0_setup2.exe.7z.dir.part/2.part appended to /lustre/dda/temp/x2goclient_4.1.1.0_setup2.exe.7z.part
2018-04-05 09:42:13: /lustre/dda/temp/x2goclient_4.1.1.0_setup2.exe.7z.dir.part/2.part appended to /lustre/dda/temp/x2goclient_4.1.1.0_setup2.exe.7z.part
2018-04-05 09:42:13: /lustre/dda/temp/x2goclient_4.1.1.0_setup2.exe.7z.dir.part/2.part appended to /lustre/dda/temp/x2goclient_4.1.1.0_setup2.exe.7z.part
2018-04-05 09:42:13: /lustre/dda/temp/x2goclient_4.1.1.0_setup2.exe.7z.dir.part/3.part appended to /lustre/dda/temp/x2goclient_4.1.1.0_setup2.exe.7z.part
2018-04-05 09:42:13: /lustre/dda/temp/x2goclient_4.1.1.0_setup2.exe.7z.dir.part/3.part appended to /lustre/dda/temp/x2goclient_4.1.1.0_setup2.exe.7z.part
2018-04-05 09:42:13: /lustre/dda/temp/x2goclient_4.1.1.0_setup2.exe.7z.dir.part/3.part appended to /lustre/dda/temp/x2goclient_4.1.1.0_setup2.exe.7z.part
2018-04-05 09:42:14: /lustre/dda/temp/x2goclient_4.1.1.0_setup2.exe.7z.dir.part/3.part appended to /lustre/dda/temp/x2goclient_4.1.1.0_setup2.exe.7z.part
2018-04-05 09:42:14: /lustre/dda/temp/x2goclient_4.1.1.0_setup2.exe.7z.dir.part/4.part appended to /lustre/dda/temp/x2goclient_4.1.1.0_setup2.exe.7z.part
2018-04-05 09:42:14: /lustre/dda/temp/x2goclient_4.1.1.0_setup2.exe.7z.dir.part/5.part appended to /lustre/dda/temp/x2goclient_4.1.1.0_setup2.exe.7z.part
2018-04-05 09:42:14: /lustre/dda/temp/x2goclient_4.1.1.0_setup2.exe.7z.dir.part combined into /lustre/dda/temp/x2goclient_4.1.1.0_setup2.exe.7z.part
2018-04-05 09:42:14: /lustre/dda/temp/x2goclient_4.1.1.0_setup2.exe.7z.part successfully renamed to /lustre/dda/temp/x2goclient_4.1.1.0_setup2.exe.7z
2018-04-05 09:42:14: filesize obtained via exec.
2018-04-05 09:42:14: /lustre/dda/temp/x2goclient_4.1.1.0_setup2.exe.7z.dir.part/4.part appended to /lustre/dda/temp/x2goclient_4.1.1.0_setup2.exe.7z.part
2018-04-05 09:42:14: ERROR: Failed to open input stream.
2018-04-05 09:42:14: /lustre/dda/temp/x2goclient_4.1.1.0_setup2.exe.7z.dir.part/4.part appended to /lustre/dda/temp/x2goclient_4.1.1.0_setup2.exe.7z.part
2018-04-05 09:42:14: ERROR: Failed to open input stream.
2018-04-05 09:42:14: /lustre/dda/temp/x2goclient_4.1.1.0_setup2.exe.7z.dir.part/4.part appended to /lustre/dda/temp/x2goclient_4.1.1.0_setup2.exe.7z.part
2018-04-05 09:42:14: ERROR: Failed to open input stream.

Danoontjes avatar Apr 05 '18 08:04 Danoontjes