spark-hats
spark-hats copied to clipboard
AbsaOSS
Nested array transformation helper extensions for Apache Spark
## Feature Create a method that given an input dataframe and the desired schema applies the schema to the dataframe as long as: * Same fields have same/compatible data types....
## Background If you have an array of arrays select doesn't work ## Feature Be able to select a column from array of arrays ## Example [Optional] ```scala scala> res0.printSchema...
Currently, when encountered a nested Map the following error is given: `java.lang.IllegalArgumentException: Field 'someNestedMap' is not a struct type or an array.` Would be a nice to have improvement. Are...
In the `NestedArrayTransformations.scala` UDF _"arrayDistinctErrors"_ is called but it's not registered anywhere in the library (To be precise it **is** registered but only the test part of the code.) Therefore...
## Feature Currying is preferred when a function is passed as a parameter. Switch the signature of nested transformation methods to use currying instead of a parameter list.
## Background Currently, the DataFrame extensions contain only basic nested routines. ## Feature Add routines that work with the error column as DataFrame extensions as well. Put these routines into...