spark-utils
spark-utils copied to clipboard
tupol
Basic framework utilities to quickly start writing production ready Apache Spark applications
## What changes were proposed in this pull request? (Please fill in changes proposed in this fix) ## How was this patch tested? (Please explain how this patch was tested....
Currently `spark-utils` is using the `scala-utils` configuration framework. This framework might be deprecated soon and replaced with [PureConfig](https://pureconfig.github.io/) Migrate the existing context factories to use [PureConfig](https://pureconfig.github.io/). The migration should be...
Hi, @tupol , I'd like to report a vulnerable dependency in **org.tupol:spark-utils-io_2.12:0.6.2**. ### Issue Description I noticed that **org.tupol:spark-utils-io_2.12:0.6.2** directly depends on **org.apache.spark:spark-core_2.12:3.0.1** in the [pom](https://repo1.maven.org/maven2/org/tupol/spark-utils-io_2.12/0.6.2/spark-utils-io_2.12-0.6.2.pom). However, as shown in...
In the early versions the `scala-utils` and `spark-utils` were `Try[T]` centric, meaning that everything that could fail returned a `Try[T]`. At some point, observing how some developers were using it,...
Currently the application configuration file name is limited to `application.conf`. We should be able to pass in an alternative application configuration file name as an application parameter, e.g. `-conf-file-name='my_app.conf'` and...
Starting Apache Spark 3.3.x there is a new `Trigger.AvailableNow`, which needs to be supported
Isolate the application implementation from the core module.