frameless icon indicating copy to clipboard operation
frameless copied to clipboard

implement the FlatMap typeclass on Job. This allows Job to be used as a Kleisli/Reader

Open gregnwosu opened this issue 7 years ago • 3 comments

this pull request allows us to make Job into cats Reader/Kleisli monads. It opens up the proper monad stack for Job and enables the potential to use Job in Monad Transformer stacks . Some example code will probably help explain this best.

import frameless.cats.jobber._
import frameless.{Job, TypedDataset}
import org.apache.spark.sql.{Dataset, SparkSession}
import cats.data.Kleisli

class Library extends App {
  val spark = SparkSession
    .builder()
    .appName("Spark SQL basic example")
    .config("spark.some.config.option", "some-value")
    .getOrCreate()
  import spark.implicits._
  implicit val sc = spark
  val ds1: Dataset[Int] = Seq(1, 2, 3, 4).toDS()
  val ds2: Dataset[Int] = TypedDataset.create(1 to 200).dataset

  def countAndTakeSumJob(sumFrom: Double): Job[Int] =
    for {
      count <- ds1.count()
      sample: Array[Int] = ds1.take(count)
    } yield sample.fold(sumFrom)(_ + _)

  def addAndConcat(delta: Int): Job[String] =
    for {
      sample <- ds2.map(_ + delta)
      result <- ds2.map((n: Int) => (n + sample).toString())
    } yield result
  val k1: Kleisli[Job, Double, Int] = Kleisli(countAndTakeSumJob _)
  val k2: Kleisli[Job, Int, String] = Kleisli(addAndConcat _)
  // This PR enables...
  //  I want to be able to be able to compose the Kleisli i.e.
  val composed: Kleisli[Job, Double, String] = k2.compose(k1)
  composed.run(5d)
}```

gregnwosu avatar Oct 02 '18 21:10 gregnwosu

Codecov Report

Merging #336 into master will decrease coverage by 0.25%. The diff coverage is 69.23%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #336      +/-   ##
==========================================
- Coverage   96.34%   96.08%   -0.26%     
==========================================
  Files          57       60       +3     
  Lines        1012     1048      +36     
  Branches       10        9       -1     
==========================================
+ Hits          975     1007      +32     
- Misses         37       41       +4
Impacted Files Coverage Δ
cats/src/main/scala/frameless/cats/implicits.scala 72.72% <69.23%> (-2.28%) :arrow_down:
...la/frameless/ml/internals/TreesInputsChecker.scala 100% <0%> (ø) :arrow_up:
...la/frameless/functions/NonAggregateFunctions.scala 100% <0%> (ø) :arrow_up:
...la/frameless/ml/internals/UnaryInputsChecker.scala 100% <0%> (ø) :arrow_up:
...in/scala/frameless/ml/clustering/TypedKMeans.scala 100% <0%> (ø)
...a/frameless/ml/internals/VectorInputsChecker.scala 100% <0%> (ø)
...frameless/ml/clustering/TypedBisectingKMeans.scala 100% <0%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 542c0c8...e0b1752. Read the comment docs.

codecov-io avatar Oct 02 '18 22:10 codecov-io

Hey @gregnwosu, do you want to keep this open? I would love to merge the PR if you can make the tests pass (adding a test may help with coverage). Let me know if you have any questions.

imarios avatar Nov 30 '18 19:11 imarios

Yes please I will write the tests this weekend

gregnwosu avatar Dec 01 '18 17:12 gregnwosu