microsoft
How to initialize sparkContext class in C#
Hi ,
Can you plz. someone send any sample to run simple word count in C# with Spark ,I have tried but getting error
No connection could be made because the target machine actively refused it
on declaration of SparkContext var sparkContext = new SparkContext()
It seems configuration issue ,Can you plz. suggest how to pass spark running instance , my running instance is http://localhost:4040
Thanks. Satish Jha
How you are trying to execute? From command line or from visual studio?
from visual studio,I have installed Spark & tested sample code on command prompt which is working fine. I have to use parallelism in C# with Spark.That's why I have installed Spark from nuget packages. Sample code: `using Microsoft.Spark.CSharp.Core; using System; using System.Collections.Generic; using System.Linq; using System.Text; using System.Threading.Tasks;
namespace ConsoleApp5 { class Program { public static void Main(string[] args) { string home = "http://127.0.0.1:4040";// "spark://hostname:port"); var sparkContext = new SparkContext(new SparkConf().SetAppName("test").SetMaster("Master").SetSparkHome(home));//error throwing here
try
{
const int slices = 3;
var numberOfItems = (int)Math.Min(100000L * slices, int.MaxValue);
var values = new List<int>(numberOfItems);
for (var i = 0; i <= numberOfItems; i++)
{
values.Add(i);
}
var rdd = sparkContext.Parallelize(values, slices);
CalculatePiUsingAnonymousMethod(numberOfItems, rdd);
}
catch (Exception ex)
{
}
sparkContext.Stop();
}
private static void CalculatePiUsingAnonymousMethod(int n, RDD<int> rdd)
{
var count = rdd
.Map(i =>
{
var random = new Random();
var x = random.NextDouble() * 2 - 1;
var y = random.NextDouble() * 2 - 1;
return (x * x + y * y) < 1 ? 1 : 0;
})
.Reduce((x, y) => x + y);
}
}
} ` Error: System.Net.Sockets.SocketException occurred HResult=0x80004005 Message=No connection could be made because the target machine actively refused it Source=System StackTrace: at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress) at System.Net.Sockets.Socket.Connect(EndPoint remoteEP) at Microsoft.Spark.CSharp.Network.DefaultSocketWrapper.Connect(IPAddress remoteaddr, Int32 port) at Microsoft.Spark.CSharp.Interop.Ipc.JvmBridge.GetConnection() at Microsoft.Spark.CSharp.Interop.Ipc.JvmBridge.CallJavaMethod(Boolean isStatic, Object classNameOrJvmObjectReference, String methodName, Object[] parameters) at Microsoft.Spark.CSharp.Interop.Ipc.JvmBridge.CallConstructor(String className, Object[] parameters) at Microsoft.Spark.CSharp.Proxy.Ipc.SparkCLRIpcProxy.CreateSparkConf(Boolean loadDefaults) at Microsoft.Spark.CSharp.Core.SparkConf..ctor(Boolean loadDefaults) at ConsoleApp5.Program.Main(String[] args) in C:\Users\satish.jha\documents\visual studio 2017\Projects\ConsoleApp5\ConsoleApp5\Program.cs:line 15
If you are trying to execute it from visual studio then you need to try in debug mode to initialise the spark context have you read this link : https://github.com/Microsoft/Mobius/blob/master/notes/running-mobius-app.md#running-mobius-examples-in-local-mode
Just read and set the environment accordingly you will able to execute
Hi jayprajapati,
I tried to run this command 'sparkclr-submit.cmd --exe SparkCLRSamples.exe C:\Users\satish.jha\Desktop\Mobius-master\csharp\Samples\Microsoft.Spark.CSharp\bin\Debug' on scala/spark command but getting error plz. suggest where I'm doing mistake. I have pasted my spark console below..plz. have a look .
Microsoft Windows [Version 10.0.10240] (c) 2015 Microsoft Corporation. All rights reserved.
C:\Users\satish.jha>cd c:\spark
c:\Spark>spark-shell Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). 17/11/04 12:13:51 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 17/11/04 12:14:02 WARN General: Plugin (Bundle) "org.datanucleus.store.rdbms" is already registered. Ensure you dont have multiple JAR versions of the same plugin in the classpath. The URL "file:/c:/Spark/bin/../jars/datanucleus-rdbms-3.2.9.jar" is already registered, and you are trying to register an identical plugin located at URL "file:/C:/Spark/jars/datanucleus-rdbms-3.2.9.jar." 17/11/04 12:14:02 WARN General: Plugin (Bundle) "org.datanucleus" is already registered. Ensure you dont have multiple JAR versions of the same plugin in the classpath. The URL "file:/c:/Spark/bin/../jars/datanucleus-core-3.2.10.jar" is already registered, and you are trying to register an identical plugin located at URL "file:/C:/Spark/jars/datanucleus-core-3.2.10.jar." 17/11/04 12:14:03 WARN General: Plugin (Bundle) "org.datanucleus.api.jdo" is already registered. Ensure you dont have multiple JAR versions of the same plugin in the classpath. The URL "file:/c:/Spark/bin/../jars/datanucleus-api-jdo-3.2.6.jar" is already registered, and you are trying to register an identical plugin located at URL "file:/C:/Spark/jars/datanucleus-api-jdo-3.2.6.jar." 17/11/04 12:14:12 WARN ObjectStore: Failed to get database global_temp, returning NoSuchObjectException Spark context Web UI available at http://192.168.22.103:4040 Spark context available as 'sc' (master = local[*], app id = local-1509777835771). Spark session available as 'spark'. Welcome to ____ __ / / ___ / / \ / _ / _ `/ __/ '/ // .__/_,// //_\ version 2.2.0 //
Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_144) Type in expressions to have them evaluated. Type :help for more information.
scala
sparkclr-submit.cmd --exe SparkCLRSamples.exe C:\Users\satish.jha\Desktop\Mobius-master\csharp\Samples\Microsoft.Spark.CSharp\bin\Debug
:1: error: ';' expected but '.' found. sparkclr-submit.cmd --exe SparkCLRSamples.exe C:\Users\satish.jha\Desktop\Mobius-master\csharp\Samples\Microsoft.Spark.CSharp\bin\Debug
^
Regards, Satish