Chao Sun

icon indicating a website link http://sunchao.github.io icon indicating an email [email protected]

icon company @openai @apache icon location Sunnyvale icon location Member of @apache. Committer of Spark, Hadoop, Arrow, DataFusion, and Hive. Likes databases, distributed systems, and programming languages.

Results 91 comments of Chao Sun

Parquet Output Filesize

Sorry for the late reply. Have you resolved the issue? If not, can you share the code which does the writing? You should write multiple rows in each row group.

Parquet Output Filesize

Yes, it seems you are calling `close_column` and `close_row_group` for every row, which is not optimal. The latter will write the Parquet row group metadata to the file. Instead, you...

Parameterize reading of rows with a type parameter, checked against the schema, and used to specialize the deserialization of rows

Thanks @alecmocatta ! The performance improvement looks very impressive 👍 ! Looking forward to a PR on this 😄 .

Parameterize reading of rows with a type parameter, checked against the schema, and used to specialize the deserialization of rows

Hi @alecmocatta , just curious whether there's any update on this?

Parameterize reading of rows with a type parameter, checked against the schema, and used to specialize the deserialization of rows

Thanks. Yes, filing a JIRA against arrow is the right thing to do. Looking forward to it!

Parameterize reading of rows with a type parameter, checked against the schema, and used to specialize the deserialization of rows

Thanks @alecmocatta ! Could you open a pull request in arrow? it's a pretty big change and I'll take some time to look at it.

Support reading & writing Arrow in encoder/decoders.

> We have similar thing in record reader. Hmm... you mean `record/reader.rs`? I couldn't find anything related. This is on the encoding level though - so we'll need to add...

Support reading & writing Arrow in encoder/decoders.

The interface will be similar to [here](https://github.com/apache/arrow/blob/master/cpp/src/parquet/encoding.h#L51). The `valid_bits` will be computed from def/rep levels, and passed to the call. See [here](https://github.com/apache/arrow/blob/master/cpp/src/parquet/arrow/record_reader.cc#L419) for an example.

Add Arrow Support

Thanks @liurenjie1024 . Updated the description for some potential tasks.

Add Arrow Support

@sadikovi Thanks - added. @andygrove cool - will take a look.