What about in-memory Compression?

Open JosephCottam opened this issue 12 years ago • 1 comments

What if we used an compression routine for the hbin format that was designed for in-memory representations? What changes?

Compression routines compared: http://terpconnect.umd.edu/~barua/matt-compress-tr.pdf

Oct 04 '13 13:10 JosephCottam

One approach would be to divide the input data into blocks and compress each block individually. Then they can be loaded/decompressed independently. This approach might also solve some of the IO-bound problems.

Dec 07 '13 05:12 JosephCottam