node-archiver
node-archiver copied to clipboard
archiverjs
Headers error when streaming file bigger than 4GB
Hi, I'm getting Headers Error when downloading files bigger than 4GB, with or without the 'forceZip64' option.
How zip file is created
// create zip archive
const zip = Archiver.create('zip', {
store: true
// forceZip64: contentLength > 4 * 1024 * 1024 * 1024
});
// pipe zip to response
zip.pipe(response);
// iterate targets download
for (const filePair of filePairs) {
const storage = await this.getFileStorage(filePair);
const stream = await storage.downloadFile(filePair.source); // returns Readable
zip.append(stream, { name: filePair.source.filename });
// await to download files one by one.
await new Promise((resolve, reject) => {
// handle success
stream.on('end', () => {
resolve();
});
stream.on('error', (err) => {
reject();
});
});
}
// finalize zip streaming
zip.finalize();
How zip file is unzipped
Using 7ZIP
- For example when I download a file bigger than 4GB and then I try to uncompress it using 7zip I will get following output:
7-Zip (a) [64] 16.02 : Copyright (c) 1999-2016 Igor Pavlov : 2016-05-21
p7zip Version 16.02 (locale=en_US.UTF-8,Utf16=on,HugeFiles=on,64 bits,8 CPUs Intel(R) Core(TM) i7-7700T CPU @ 2.90GHz (906E9),ASM,AES-NI)
Scanning the drive for archives:
1 file, 15587398396 bytes (15 GiB)
Extracting archive: files.zip
ERRORS:
Headers Error
--
Path = files.zip
Type = zip
ERRORS:
Headers Error
Physical Size = 15587398396
64-bit = +
Using Unzip (UnZip 6.00 of 20 April 2009, by Debian. Original by Info-ZIP)
- I will not get any error
Results
In both cases files of 15.6 GB are correctly unzipped and with the same MD5 checksum than the original, but it's confusing for the final user to have these errors and may result on some unzipper that does not work properly, for example Ubuntu default zip opener cannot list zip content.
