Need help with the Mobius workflow understanding

[ramkumarravy]

Hi,

I'd like to contribute for Spark-Machine Learning in Mobius. I did some study on the Mobius workflow and with that I've also added support for a missing API "stat" in the Dataframe class and yes that worked well too.

For your reference, please find the sample code snippet from below.

DataFrame Stat - Correlation API call:

DataFrame Stat Initialization/:

DataFrameStatFunctions Class:

DataFrameStatFunctions Ipc Proxy:

IDataFrameStatFunctionsProxy interface:

When I went through the workflow, it seems like the use of the Proxy (with JVM call) made it easy to add support for an existing function. But, there was no luck, when I tried to create a new one for the Machine Learning (MLlib) on my own. I can understand that I am missing something with the SparkCLR - Scala project.

Can you please help me with this ? Also can you share some easy documentation for the better understanding ?

Thanks, Ramkumar

[skaarthik]

The implementation looks good to me on a quick look. What issue are you facing with this code?

[ramkumarravy]

Thanks for addressing my query. As I said earlier, the above code doesn't have any problem and in fact it worked well. All I wish to do is to add support for "MLlib" and for that when I tried the same way(as above), I couldn't achieve it.

When I went through the workflow, it seems like the use of the Proxy (with JVM call) made it easy to add support for an existing function. But, there was no luck, when I tried to create a new one for "MLlib" on my own. I can understand that I am missing something with the SparkCLR - Scala project.

Can you please help me with this ? Also can you share some easy documentation for the better understanding ?

Expecting a quick answer from your end... Thanks in advance :)

[skaarthik]

But, there was no luck, when I tried to create a new one for "MLlib" on my own

Can you share more details on this (error message, build error etc.)?

can you share some easy documentation for the better understanding

Unfortunately we do not have any documentation to explain the code components. You seem to be on the right track with the implementation for Corr. If you share details on the errors you are getting, one of the contributors might be able to help

[dwnichols]

@ramkumarravy : Feel free to open a pull request with your code. We can work with you to fix any issues you're experiencing and get it ready to be merged into master.

[ramkumarravy]

Hi,

I’ve added support for Spark - Machine Learning (MLlib) – Linear Regression Model class in Spark CLR project and have added a class “MLlib” with “LinearRegrssionModel()” API in Mobius CSharp Adapter to perform its function.

Please find the sample code snippets from below:

Linear Regression Model - API call:

Getting Spark Context Reference:

Linear Regression Model - JVM call:

Linear Regression Model – Scala implementation:

Linear Regression Mode – Result (temporary):

Temporarily, I’ve returned the key metrics of Linear Regression Model like MSE, RMSE, Intercept and Coefficients.

I’d like to know whether the flow which I’ve followed to add support for Spark-MLlib class is fine or not. So, please share me your comments and let me know if I went through a wrong way.

Thanks in advance :)

[dwnichols]

Hi @ramkumarravy, could you please create a pull request containing your changes. Based on the above code, I would create two separate pull requests, one for the stat API another for the linearRegrssionModel. Pull requests are specifically designed for the code review process.