SparkCyclone icon indicating copy to clipboard operation
SparkCyclone copied to clipboard

Replace JNA usage in ArrowTransferDefinitions with JavaCPP

Open EmergentOrder opened this issue 3 years ago • 5 comments

This needs to be reconciled with the addition of NullableScalarVec<T> before merging (transfer-definitions.cpp and parts of transfer-definitions.hpp can be removed when this is done).

Was passing all tests before the latest merge (with additions to NullableScalarVec<T>) , currently compiles but fails tests.

EmergentOrder avatar Feb 16 '22 19:02 EmergentOrder

Shows merge conflicts at the moment

image

wgip avatar Feb 20 '22 20:02 wgip

Working on the merge now.

EmergentOrder avatar Feb 20 '22 21:02 EmergentOrder

FYI there remain a few usages of JNA which need to be replaced, in CRunner.scala and CArrowNativeInterface.scala

EmergentOrder avatar Feb 20 '22 21:02 EmergentOrder

Unit test JVM looks good

com.nec.spark.cgescape.CodegenEscapeSpec fails but unrelated to this PR I believe

wgip avatar Feb 20 '22 22:02 wgip

CMake scope

image

If possible, might be appropriate load the lib programmatically rather than via SBT, then it should be more consistent across different scopes.

wgip avatar Feb 20 '22 22:02 wgip