snappy-java icon indicating copy to clipboard operation
snappy-java copied to clipboard

How to make existed snappy compressed file compatible with hadoop?

Open darouwan opened this issue 7 years ago • 3 comments

I have output some snappy compressed file, and I need to read them via Spark. It seems the snappy file made by SnappyFramedOutputStream cannot be read by hadoop. So can I convert them to satisfied with the hadoop snappy codec?

darouwan avatar May 17 '18 07:05 darouwan

See the compatibility note here: https://github.com/xerial/snappy-java#compatibility-notes

We can use SnappyHadoopCompatibleOutputStream to generate Hadoop Snappy compatible compressed stream.

xerial avatar May 21 '18 18:05 xerial

Hi, thanks. But for existed snappy file generated by SnappyFramedOutputStream, how can I make it available for haddop?

darouwan avatar May 22 '18 01:05 darouwan

And I found another question, I cannot find isOpen() method in SnappyFramedOutputStream. Even though I can implement it in other ways, but it is better to include it in the SnappyFramedOutputStream itself.

darouwan avatar May 22 '18 08:05 darouwan