reference-apps icon indicating copy to clipboard operation
reference-apps copied to clipboard
verified publisher icon databricks

Spark 2 compatibility

Open tashoyan opened this issue 9 years ago • 1 comments

As Spark 2 is not a breaking news anymore, it's time to make reference apps working with it. I can see at least the following things to do:

  • Use SparkSession instead of SQLContext
  • Use Dataset API
  • Update JavaDStream.foreachRDD() invocations to new contract (returns nothing)
  • Update JavaPairDStream.updateStateByKey() to new contract (uses Spark implementation of Optional)

tashoyan avatar Oct 18 '16 10:10 tashoyan

I can handle Java part. The question: do we still need to maintain Java 6 compatible code? Can we exclude support for this outdated Java version?

tashoyan avatar Oct 18 '16 10:10 tashoyan