Is there any plan to port TensorframeOnSpark(From yahoo) ?

[allwefantasy]

TensorframeOnSpark support the real distribute trainning on tensorflow . Maybe it's a good idea to port TensorframeOnSpark to spark deep learning project.

[sueann]

Thanks, @allwefantasy, that makes a lot of sense. In fact, we've been collaborating with the TensorFlowOnSpark (TFoS) team to see how we can bring all these ideas together...! They have implemented a Spark MLlib Estimator in the TFoS project (https://github.com/yahoo/TensorFlowOnSpark/blob/master/tensorflowonspark/pipeline.py#L214) which is very exciting. The two projects are quite complementary currently, and it could make sense to combine them together. What are your thoughts?

[allwefantasy]

Excellent, that's really what I want. Also TFEstimator implemented by TFoS proves it is easy to be a distributed runtime for spark-deep-learning. That means we can just treat TFoS as pip dependency.
I check the last PR in spark-deep-learning is KerasImageFileEstimator, we can add a new param eg. distribute_mode to make it run in TFoS.

class TFImageFileEstimator(Estimator, HasInputCol, HasInputImageNodeName,
                              HasOutputCol, HasOutputNodeName, HasLabelCol,
                              HasKerasModel, HasKerasOptimizer, HasKerasLoss,
                              CanLoadImage, HasOutputMode, DistributeMode="TFoS"):

spark-deep-learning depends on tensorframes which makes spark can use tf's power of numerical calculation and connects tf / spark strongly, and we can provide users with a perfect experience like MLlib. we don't encourage people to write complex TFcode . People should just define their network structure or tell us their model file and input/output,then everything should been done by spark-deep-learning.