arkouda icon indicating copy to clipboard operation
arkouda copied to clipboard

Closes #3219: Optimize old Parquet srting read code

Open bmcdonald3 opened this issue 1 year ago • 2 comments

After seeing more results gathered on different machines showing mixed results for the new Parquet string optimization, we have decided to make some changes and go back towards a simpler optimization that we are more confident in across the board optimizations, though, they may be more minor than the "best case" with the optimized version, but we shouldn't be seeing any cases get worse with this approach, which seems preferable.

bmcdonald3 avatar May 21 '24 20:05 bmcdonald3

@stress-tess I think this PR is finally ready to go. A similar pattern to this can be applied everywhere where we've been rolling back the batch optimization in favor of the single reads to accomodate null values.

I am still gathering some performance numbers on various machines, but if we could get this into the release this week, that'd be great.

bmcdonald3 avatar Jun 10 '24 16:06 bmcdonald3

This will pass CI after https://github.com/Bears-R-Us/arkouda/pull/3312 is merged (and we rebase on top of it)

bmcdonald3 avatar Jun 11 '24 01:06 bmcdonald3