implement the FlatMap typeclass on Job. This allows Job to be used as a Kleisli/Reader
this pull request allows us to make Job into cats Reader/Kleisli monads. It opens up the proper monad stack for Job and enables the potential to use Job in Monad Transformer stacks . Some example code will probably help explain this best.
import frameless.cats.jobber._
import frameless.{Job, TypedDataset}
import org.apache.spark.sql.{Dataset, SparkSession}
import cats.data.Kleisli
class Library extends App {
val spark = SparkSession
.builder()
.appName("Spark SQL basic example")
.config("spark.some.config.option", "some-value")
.getOrCreate()
import spark.implicits._
implicit val sc = spark
val ds1: Dataset[Int] = Seq(1, 2, 3, 4).toDS()
val ds2: Dataset[Int] = TypedDataset.create(1 to 200).dataset
def countAndTakeSumJob(sumFrom: Double): Job[Int] =
for {
count <- ds1.count()
sample: Array[Int] = ds1.take(count)
} yield sample.fold(sumFrom)(_ + _)
def addAndConcat(delta: Int): Job[String] =
for {
sample <- ds2.map(_ + delta)
result <- ds2.map((n: Int) => (n + sample).toString())
} yield result
val k1: Kleisli[Job, Double, Int] = Kleisli(countAndTakeSumJob _)
val k2: Kleisli[Job, Int, String] = Kleisli(addAndConcat _)
// This PR enables...
// I want to be able to be able to compose the Kleisli i.e.
val composed: Kleisli[Job, Double, String] = k2.compose(k1)
composed.run(5d)
}```
Codecov Report
Merging #336 into master will decrease coverage by
0.25%. The diff coverage is69.23%.
@@ Coverage Diff @@
## master #336 +/- ##
==========================================
- Coverage 96.34% 96.08% -0.26%
==========================================
Files 57 60 +3
Lines 1012 1048 +36
Branches 10 9 -1
==========================================
+ Hits 975 1007 +32
- Misses 37 41 +4
| Impacted Files | Coverage Δ | |
|---|---|---|
| cats/src/main/scala/frameless/cats/implicits.scala | 72.72% <69.23%> (-2.28%) |
:arrow_down: |
| ...la/frameless/ml/internals/TreesInputsChecker.scala | 100% <0%> (ø) |
:arrow_up: |
| ...la/frameless/functions/NonAggregateFunctions.scala | 100% <0%> (ø) |
:arrow_up: |
| ...la/frameless/ml/internals/UnaryInputsChecker.scala | 100% <0%> (ø) |
:arrow_up: |
| ...in/scala/frameless/ml/clustering/TypedKMeans.scala | 100% <0%> (ø) |
|
| ...a/frameless/ml/internals/VectorInputsChecker.scala | 100% <0%> (ø) |
|
| ...frameless/ml/clustering/TypedBisectingKMeans.scala | 100% <0%> (ø) |
Continue to review full report at Codecov.
Legend - Click here to learn more
Δ = absolute <relative> (impact),ø = not affected,? = missing dataPowered by Codecov. Last update 542c0c8...e0b1752. Read the comment docs.
Hey @gregnwosu, do you want to keep this open? I would love to merge the PR if you can make the tests pass (adding a test may help with coverage). Let me know if you have any questions.
Yes please I will write the tests this weekend