Performance?

Open chr4 opened this issue 2 years ago • 1 comments

I was super excited to see this! I'm currently looking for a fast compression alternative for zstd for compressing postgresql wal archives.

At least for this use-case, I wasn't able to reproduce the benchmarks you've provided.

(orz v1.6.2 installed using cargo install as described in the README, also tested with cargo build --release from current HEAD):

$ zstd 00000003000025EF0000007C
00000003000025EF0000007C : 50.55%   (  16.0 MiB =>   8.09 MiB, 00000003000025EF0000007C.zst)
'zstd 00000003000025EF0000007C' time: 0.064s, cpu: 104%

orz encode -l0 00000003000025EF0000007C 00000003000025EF0000007C.orz
[INFO] encode: 16777216 bytes => 8111757 bytes, 25.301MB/s
[INFO] statistics:
[INFO]   size:  16777216 bytes => 8111839 bytes
[INFO]   ratio: 48.35%
[INFO]   time:  0.669 sec

Which is factor ~10 slower than zstd :(

Platform: M1 Apple Silicon macOS (native), x86_64 Linux (musl cross-compiled)

Jun 18 '23 21:06 chr4

zstd's default compress level is much faster (and normally has worse compress ratio) than orz. however your input file seems not to have much redundant, so getting a similar output size. orz should perform better on data that have more redundant, like plain texts, logs, etc.

Jun 30 '23 03:06 richox