data-validator icon indicating copy to clipboard operation
data-validator copied to clipboard

Release builds for newer versions of Scala and Spark

Open colindean opened this issue 3 years ago • 4 comments

Is your feature request related to a problem? Please describe.

We currently only ship for Scala 2.11 and Spark 2.3.x.

Describe the solution you'd like

We should ship for newer versions in whatever pairs are appropriate.

Additional context

https://github.com/sbt/sbt-projectmatrix is probably the right tool for the job.

colindean avatar Jun 02 '22 01:06 colindean

Minimal:

groups:
  - spark:
    - 2.3.4
    scala:
    - 2.11.12

Max:

groups:
  - spark:
    - 2.3.4
    scala:
    - 2.11.12
  - spark:
    - 2.4.8
    - 3.0.3
    - 3.1.3
    scala:
    - 2.12.15
  - spark:
    - 3.2.1
    scala:
    - 2.13.8

Spark versions: https://spark.apache.org/downloads.html

colindean avatar Jun 08 '22 22:06 colindean

We should continue to build on JDK 1.8. Spark still supports that. However, Spark 3.0 allows JDK 1.11, so that could be another variable, but it'd have to be at the GHA level instead of sbt-projectmatrix.

colindean avatar Jun 08 '22 22:06 colindean

I've got a start on this in branch gh-84-crossbuild. I've got builds for [Scala 2.11 + Spark 2.3.4] and [Scala 2.12 + Spark 2.4.8] working pretty easily but not Scala 2.13 yet. It'll need some refactoring and I've yet to address a spurious extra build, too.

colindean avatar Jun 13 '22 17:06 colindean