CodecZstd.jl
CodecZstd.jl copied to clipboard
Lots of allocations related to buffering
julia> data = rand(Float32, 1_000_000);
julia> io = open("foo", "w");
julia> zio = CodecZstd.ZstdCompressorStream(io);
julia> @time write(zio, data)
0.023578 seconds (60.39 k allocations: 2.623 MiB)
4000000
julia> @time write(zio, data)
0.024188 seconds (60.39 k allocations: 2.623 MiB)
4000000
julia> zio = CodecZstd.ZstdCompressorStream(io, bufsize=500_000);
julia> @time write(zio, data)
0.011804 seconds (1.02 k allocations: 44.641 KiB)
4000000
julia> @time write(zio, data)
0.013522 seconds (1.16 k allocations: 51.016 KiB)
4000000
julia> zio = CodecZstd.ZstdCompressorStream(io, bufsize=5_000_000);
julia> @time write(zio, data)
0.002285 seconds (1 allocation: 16 bytes)
4000000
julia> @time write(zio, data)
0.015165 seconds (146 allocations: 6.391 KiB)
4000000
julia> @time write(zio, data)
0.017407 seconds (146 allocations: 6.391 KiB)
4000000
This is a very simple MWE of a typical use case for CodecZstd - writing to a file. It seems to cause an unexpectedly large amount of allocations. They seem to be related to the buffer size, presumably a bug in the growth or re-use of buffers.