zed icon indicating copy to clipboard operation
zed copied to clipboard

ZNG write throughput drops by ~10x when compression is enabled

Open nwt opened this issue 3 years ago • 0 comments

Compressing frames in parallel would probably fix that.

$ zq -version
Version: v1.1.0-49-g87a419e4
$ for i in 1 2 3; do /usr/bin/time zq -zngcompress=false wrccdc-year1.zng > /dev/null; done
        2.26 real         8.41 user         0.61 sys
        1.96 real         8.10 user         0.59 sys
        2.23 real         8.34 user         0.60 sys
$ for i in 1 2 3; do /usr/bin/time zq -zngcompress=true wrccdc-year1.zng > /dev/null; done
       22.95 real        28.93 user         0.68 sys
       23.34 real        29.34 user         0.68 sys
       23.03 real        28.93 user         0.73 sys

nwt avatar Jul 07 '22 23:07 nwt