Mark Hamstra comments

Repositories
Issues
Comments

Results 13 comments of


                                            Mark Hamstra

Made combOp of aggregate() work as a reduce instead of an implicit fold

A fun little demo aggregating an `RDD[Char]` into a `String`: ``` MASTER=local[4] ./spark-shell . . . scala> val l = sc.parallelize("foobarbazquux".toList, 4) l: spark.RDD[Char] = ParallelCollectionRDD[0] at parallelize at :12...

Made combOp of aggregate() work as a reduce instead of an implicit fold

The zeroValue is not bound into combOp (which operation potentially needs a different zero than does seqOp), but the zeroElement is still part of the seqOp fold, so there won't...

Made combOp of aggregate() work as a reduce instead of an implicit fold

Yes, I'm not sure that there is a legitimate use case for seqOp and combOp that don't share the same zero. If we're willing to say that combOp must be...