zed
zed copied to clipboard
ZNG write throughput drops by ~10x when compression is enabled
Compressing frames in parallel would probably fix that.
$ zq -version
Version: v1.1.0-49-g87a419e4
$ for i in 1 2 3; do /usr/bin/time zq -zngcompress=false wrccdc-year1.zng > /dev/null; done
2.26 real 8.41 user 0.61 sys
1.96 real 8.10 user 0.59 sys
2.23 real 8.34 user 0.60 sys
$ for i in 1 2 3; do /usr/bin/time zq -zngcompress=true wrccdc-year1.zng > /dev/null; done
22.95 real 28.93 user 0.68 sys
23.34 real 29.34 user 0.68 sys
23.03 real 28.93 user 0.73 sys