Release builds for newer versions of Scala and Spark
Is your feature request related to a problem? Please describe.
We currently only ship for Scala 2.11 and Spark 2.3.x.
Describe the solution you'd like
We should ship for newer versions in whatever pairs are appropriate.
Additional context
https://github.com/sbt/sbt-projectmatrix is probably the right tool for the job.
Minimal:
groups:
- spark:
- 2.3.4
scala:
- 2.11.12
Max:
groups:
- spark:
- 2.3.4
scala:
- 2.11.12
- spark:
- 2.4.8
- 3.0.3
- 3.1.3
scala:
- 2.12.15
- spark:
- 3.2.1
scala:
- 2.13.8
Spark versions: https://spark.apache.org/downloads.html
We should continue to build on JDK 1.8. Spark still supports that. However, Spark 3.0 allows JDK 1.11, so that could be another variable, but it'd have to be at the GHA level instead of sbt-projectmatrix.
I've got a start on this in branch gh-84-crossbuild. I've got builds for [Scala 2.11 + Spark 2.3.4] and [Scala 2.12 + Spark 2.4.8] working pretty easily but not Scala 2.13 yet. It'll need some refactoring and I've yet to address a spurious extra build, too.