fs2
fs2 copied to clipboard
gzip compression not backwards compatible with gzip 1.3.12
Gzip compressed files produced with fs2.compression.gzip are not recognized by gzip 1.3.12. This is primarily an issue for users on CentOS 6 (CentOS 7 ships a newer version that handles this).
Here's how to reproduce:
Produce a compressed file:
import java.nio.file.Paths
import cats.effect.{Blocker, ExitCode, IO, IOApp}
import cats.syntax.all._
object WriteCompressedFile extends IOApp {
override def run(args: List[String]): IO[ExitCode] = Blocker[IO].use { b =>
fs2.Stream.emit("lorem ipsum")
.covary[IO]
.through(fs2.text.utf8Encode)
.through(fs2.compression.gzip())
.through(fs2.io.file.writeAll(Paths.get("/tmp/bar/foo.gz"), b)).compile.drain.as(ExitCode.Success)
}
}
Uncompress on CentOS 6
├─ /tmp/bar
╰ file foo.gz
foo.gz: gzip compressed data, has CRC, from MacOS, original size modulo 2^32 37
├─ /tmp/bar
╰ docker run --rm -it -v $(pwd):/foo centos:centos6 bash
[root@52c59137bb4b /]# cd foo
[root@52c59137bb4b foo]# ls -l
total 4
-rw-r--r-- 1 root root 57 Jun 5 08:21 foo.gz
[root@52c59137bb4b foo]# file foo.gz
foo.gz: gzip compressed data, has CRC, from MacOS
[root@52c59137bb4b foo]# gunzip foo.gz
gzip: foo.gz is a a multi-part gzip file -- not supported
[root@52c59137bb4b foo]# gzip -d foo.gz
gzip: foo.gz is a a multi-part gzip file -- not supported
[root@52c59137bb4b foo]#
Related issue on CentOS:
- https://serverfault.com/questions/727752/how-to-upgrade-gzip-from-1-3-12-to-1-6-on-centos-6
- https://forums.centos.org/viewtopic.php?p=260691#p260691
Doesn't seem likely to change there (snipped from last link):
What we ship is 1.3.12 and will be for the lifespan of the distro. CentOS 7 uses 1.5.8 and will do for its life too.
Not sure if it's feasible to be backwards compatible with gzip à 2007, but it's good to document incompatibilities anyway.
Feel free to close if this is out of scope