CodecZstd.jl icon indicating copy to clipboard operation
CodecZstd.jl copied to clipboard

Lots of allocations related to buffering

Open kcajf opened this issue 4 years ago • 0 comments

julia> data = rand(Float32, 1_000_000);

julia> io = open("foo", "w");

julia> zio = CodecZstd.ZstdCompressorStream(io);

julia> @time write(zio, data)
  0.023578 seconds (60.39 k allocations: 2.623 MiB)
4000000

julia> @time write(zio, data)
  0.024188 seconds (60.39 k allocations: 2.623 MiB)
4000000

julia> zio = CodecZstd.ZstdCompressorStream(io, bufsize=500_000);

julia> @time write(zio, data)
  0.011804 seconds (1.02 k allocations: 44.641 KiB)
4000000

julia> @time write(zio, data)
  0.013522 seconds (1.16 k allocations: 51.016 KiB)
4000000

julia> zio = CodecZstd.ZstdCompressorStream(io, bufsize=5_000_000);

julia> @time write(zio, data)
  0.002285 seconds (1 allocation: 16 bytes)
4000000

julia> @time write(zio, data)
  0.015165 seconds (146 allocations: 6.391 KiB)
4000000

julia> @time write(zio, data)
  0.017407 seconds (146 allocations: 6.391 KiB)
4000000

This is a very simple MWE of a typical use case for CodecZstd - writing to a file. It seems to cause an unexpectedly large amount of allocations. They seem to be related to the buffer size, presumably a bug in the growth or re-use of buffers.

kcajf avatar Sep 02 '21 13:09 kcajf