Mobius
Mobius copied to clipboard
RegisterTempTable not found by compiler
After attempting to copy the sample code from the readme, I get a compilation error when trying to call the RegisterTempTable method on a data frame.
var sparkConf = new SparkConf();
sparkConf.SetMaster("local[2]"); //local standalone with 2 cores, ex:"spark://master:7077" to connect to other cluster
sparkConf.SetAppName("SparkCLRExample");
//sparkConf.SetSparkHome(""); //used to set the spark install location on worker nodes. Using default instead.
SparkContext sc = SparkContext.GetOrCreate(sparkConf);
var reqDataFrame = sc.TextFile(@"hdfs://path/to/requests.csv");
var metricDataFrame = sc.TextFile(@"hdfs://path/to/metrics.csv");
reqDataFrame.RegisterTempTable("requests");
metricDataFrame.RegisterTempTable("metrics");
// C0 - guid in requests DataFrame, C3 - guid in metrics DataFrame
var joinDataFrame = GetSqlContext().Sql(
"SELECT joinedtable.datacenter" +
", MAX(joinedtable.latency) maxlatency" +
", AVG(joinedtable.latency) avglatency " +
"FROM (" +
"SELECT a.C1 as datacenter, b.C6 as latency " +
"FROM requests a JOIN metrics b ON a.C0 = b.C3) joinedtable " +
"GROUP BY datacenter");
joinDataFrame.ShowSchema();
joinDataFrame.Show();
'RDD
' does not contain a definition for 'RegisterTempTable' and no extension method 'RegisterTempTable' accepting a first argument of type 'RDD ' could be found (press F4 to add a using directive or assembly reference)