graphx How to build project from https://github.com/apache/spark using sbt

can anyone mention the steps which i need to follow for building spark application also commands to test the scala testcase.referring from https://github.com/apache/spark

I m getting build error

Stacktrace

[ERROR] Failed to execute goal on project spark-core_2.10: Could not resolve dependencies for project org.apache.spark:spark-core_2.10:jar:1.1.0-SNAPSHOT: Failed to collect dependencies at org.easymoc k:easymockclassextension:jar:3.1: Failed to read artifact descriptor for org.easymock:easymockclassextension:jar:3.1: Could not transfer artifact org.easymock:easymockclassextension:pom:3.1 from/to ma ven-repo (http://repo.maven.apache.org/maven2): Access denied to: http://repo.maven.apache.org/maven2/org/easymock/easymockclassextension/3.1/easymockclassextension-3.1.pom , ReasonPhrase:Forbidden. -

[Help 1]

Thanx

Jul 17 '14 10:07 Pallavim54

Are you able to access the URL in the error message from a browser?

Jul 17 '14 19:07 ankurdave

i m not able to access this url http://repo.maven.apache.org/maven2/org/easymock/easymockclassextension/3.1/easymockclassextension-3.1.pom may be this is blocked by farewall.

and for (http://repo.maven.apache.org/maven2)

it showing this:

Browsing for this directory has been disabled. View this directory's contents on http://search.maven.org instead. Find out more about The Central Repository.

Jul 18 '14 04:07 Pallavim54

yes now i m able to access the url but i m not able to run graphx example under https://github.com/apache/spark/blob/master/examples/src/main/scala/org/apache/spark/examples/graphx/LiveJournalPageRank.scala

Jul 18 '14 10:07 Pallavim54

How are you invoking it? What is the error message?

Jul 18 '14 18:07 ankurdave

I m getting this message when i m trying to build using maven clean install -DskipTests=true

stacktrace:

java.lang.RuntimeException: No main class detected. at scala.sys.package$.error(package.scala:27) at sbt.Defaults$$anonfun$runTask$1$$anonfun$apply$36$$anonfun$apply$37$$ anonfun$26.apply(Defaults.scala:687) at sbt.Defaults$$anonfun$runTask$1$$anonfun$apply$36$$anonfun$apply$37$$ anonfun$26.apply(Defaults.scala:687) at scala.Option.getOrElse(Option.scala:120) at sbt.Defaults$$anonfun$runTask$1$$anonfun$apply$36$$anonfun$apply$37.a pply(Defaults.scala:687) at sbt.Defaults$$anonfun$runTask$1$$anonfun$apply$36$$anonfun$apply$37.a pply(Defaults.scala:686) at scala.Function1$$anonfun$compose$1.apply(Function1.scala:47) at sbt.$tilde$greater$$anonfun$$u2219$1.apply(TypeFunctions.scala:42) at sbt.std.Transform$$anon$4.work(System.scala:64) at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:23 7) at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:23 7) at sbt.ErrorHandling$.wideConvert(ErrorHandling.scala:18) at sbt.Execute.work(Execute.scala:244) at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:237) at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:237) at sbt.ConcurrentRestrictions$$anon$4$$anonfun$1.apply(ConcurrentRestric tions.scala:160) at sbt.CompletionService$$anon$2.call(CompletionService.scala:30) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:47 1) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor. java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor .java:615) at java.lang.Thread.run(Thread.java:724) error No main class detected. [error] Total time: 1 s, completed 21 Jul, 2014 10:46:11 AM

Jul 21 '14 05:07 Pallavim54

if i used sbt/sbt assembly then i m geeting the below error

stactrace

java.lang.RuntimeException: Nonzero exit code (1): git clone https://github.com/ScrapCodes/sbt-pom-reader.git C:\Users\pmukherjee.sbt\0.13\staging\ec3aa8f39111944cc5f at scala.sys.package$.error(package.scala:27) at sbt.Resolvers$.run(Resolvers.scala:134) at sbt.Resolvers$.run(Resolvers.scala:123) at sbt.Resolvers$$anon$2.clone(Resolvers.scala:78) at sbt.Resolvers$DistributedVCS$$anonfun$toResolver$1$$anonfun$apply$12$$anonfun$apply$6.apply$mcV$sp(Resolvers.scala:108) at sbt.Resolvers$.creates(Resolvers.scala:141) at sbt.Resolvers$DistributedVCS$$anonfun$toResolver$1$$anonfun$apply$12.apply(Resolvers.scala:108) at sbt.Resolvers$DistributedVCS$$anonfun$toResolver$1$$anonfun$apply$12.apply(Resolvers.scala:108) at sbt.BuildLoader$$anonfun$componentLoader$1$$anonfun$apply$3.apply(BuildLoader.scala:90) at sbt.BuildLoader$$anonfun$componentLoader$1$$anonfun$apply$3.apply(BuildLoader.scala:89) at scala.Option.map(Option.scala:145) at sbt.BuildLoader$$anonfun$componentLoader$1.apply(BuildLoader.scala:89) at sbt.BuildLoader$$anonfun$componentLoader$1.apply(BuildLoader.scala:85) at sbt.MultiHandler.apply(BuildLoader.scala:16) at sbt.BuildLoader.apply(BuildLoader.scala:142) at sbt.Load$.loadAll(Load.scala:312) at sbt.Load$.loadURI(Load.scala:264) at sbt.Load$.load(Load.scala:260) at sbt.Load$.load(Load.scala:251) at sbt.Load$.apply(Load.scala:134) at sbt.Load$.buildPluginDefinition(Load.scala:613) at sbt.Load$.buildPlugins(Load.scala:579) at sbt.Load$.plugins(Load.scala:567) at sbt.Load$.loadUnit(Load.scala:411) at sbt.Load$$anonfun$15$$anonfun$apply$11.apply(Load.scala:256) at sbt.Load$$anonfun$15$$anonfun$apply$11.apply(Load.scala:256) at sbt.BuildLoader$$anonfun$componentLoader$1$$anonfun$apply$4$$anonfun$apply$5$$anonfun$apply$6.apply(BuildLoader.scala:93) at sbt.BuildLoader$$anonfun$componentLoader$1$$anonfun$apply$4$$anonfun$apply$5$$anonfun$apply$6.apply(BuildLoader.scala:92) at sbt.BuildLoader.apply(BuildLoader.scala:143) at sbt.Load$.loadAll(Load.scala:312) at sbt.Load$.loadURI(Load.scala:264) at sbt.Load$.load(Load.scala:260) at sbt.Load$.load(Load.scala:251) at sbt.Load$.apply(Load.scala:134) at sbt.Load$.defaultLoad(Load.scala:37) at sbt.BuiltinCommands$.doLoadProject(Main.scala:473) at sbt.BuiltinCommands$$anonfun$loadProjectImpl$2.apply(Main.scala:467) at sbt.BuiltinCommands$$anonfun$loadProjectImpl$2.apply(Main.scala:467) at sbt.Command$$anonfun$applyEffect$1$$anonfun$apply$2.apply(Command.scala:60) at sbt.Command$$anonfun$applyEffect$1$$anonfun$apply$2.apply(Command.scala:60) at sbt.Command$$anonfun$applyEffect$2$$anonfun$apply$3.apply(Command.scala:62) at sbt.Command$$anonfun$applyEffect$2$$anonfun$apply$3.apply(Command.scala:62) at sbt.Command$.process(Command.scala:95) at sbt.MainLoop$$anonfun$1$$anonfun$apply$1.apply(MainLoop.scala:100) at sbt.MainLoop$$anonfun$1$$anonfun$apply$1.apply(MainLoop.scala:100) at sbt.State$$anon$1.process(State.scala:179) at sbt.MainLoop$$anonfun$1.apply(MainLoop.scala:100) at sbt.MainLoop$$anonfun$1.apply(MainLoop.scala:100) at sbt.ErrorHandling$.wideConvert(ErrorHandling.scala:18) at sbt.MainLoop$.next(MainLoop.scala:100) at sbt.MainLoop$.run(MainLoop.scala:93) at sbt.MainLoop$$anonfun$runWithNewLog$1.apply(MainLoop.scala:71) at sbt.MainLoop$$anonfun$runWithNewLog$1.apply(MainLoop.scala:66) at sbt.Using.apply(Using.scala:25) at sbt.MainLoop$.runWithNewLog(MainLoop.scala:66) at sbt.MainLoop$.runAndClearLast(MainLoop.scala:49) at sbt.MainLoop$.runLoggedLoop(MainLoop.scala:33) at sbt.MainLoop$.runLogged(MainLoop.scala:25) at sbt.StandardMain$.runManaged(Main.scala:57) at sbt.xMain.run(Main.scala:29) at xsbt.boot.Launch$$anonfun$run$1.apply(Launch.scala:109) at xsbt.boot.Launch$.withContextLoader(Launch.scala:129) at xsbt.boot.Launch$.run(Launch.scala:109) at xsbt.boot.Launch$$anonfun$apply$1.apply(Launch.scala:36) at xsbt.boot.Launch$.launch(Launch.scala:117) at xsbt.boot.Launch$.apply(Launch.scala:19) at xsbt.boot.Boot$.runImpl(Boot.scala:44) at xsbt.boot.Boot$.main(Boot.scala:20) at xsbt.boot.Boot.main(Boot.scala) [error] Nonzero exit code (1): git clone https://github.com/ScrapCodes/sbt-pom-reader.git C:\Users\pmukherjee.sbt\0.13\staging\ec3aa8f39111944cc5f2\sbt-pom-reader

Jul 21 '14 05:07 Pallavim54

can you please mention in steps that how to build the whole project and run the examples from https://github.com/apache/spark

Jul 21 '14 05:07 Pallavim54

Here's how I build and run the project. I run OS X and use SBT, not Maven.

Make sure Java and Git are installed:

~$ java -version
java version "1.7.0_60"
Java(TM) SE Runtime Environment (build 1.7.0_60-b19)
Java HotSpot(TM) 64-Bit Server VM (build 24.60-b09, mixed mode)
~$ git --version
git version 2.0.1

Clone the Spark repo:

~$ git clone https://github.com/apache/spark
~$ cd spark

Build an assembly JAR using SBT:
```
~/spark$ sbt/sbt assembly
```
Run tests:
```
~/spark$ sbt/sbt test
```
Run examples:
```
~/spark$ bin/run-example SparkPi
```

Jul 21 '14 06:07 ankurdave

From your sbt assembly stack trace, it looks like Git is failing when running git clone https://github.com/ScrapCodes/sbt-pom-reader.git. What happens when you run that command?

Jul 21 '14 06:07 ankurdave

i m able to clone the repository by using git clone https://github.com/ScrapCodes/sbt-pom-reader.git

Jul 21 '14 06:07 Pallavim54

now i m able to build an assembly JAR using SBT: also i m able to run example using bin/run-example SparkPi. but when i m doing ~/spark$ sbt/sbt test using SBT getting error in testcases

stacktrace:

error sbt.TestsFailedException: Tests unsuccessful error sbt.TestsFailedException: Tests unsuccessful error sbt.TestsFailedException: Tests unsuccessful error sbt.TestsFailedException: Tests unsuccessful error sbt.TestsFailedException: Tests unsuccessful error sbt.TestsFailedException: Tests unsuccessful error sbt.TestsFailedException: Tests unsuccessful error sbt.TestsFailedException: Tests unsuccessful error sbt.TestsFailedException: Tests unsuccessful error sbt.TestsFailedException: Tests unsuccessful error sbt.TestsFailedException: Tests unsuccessful error sbt.TestsFailedException: Tests unsuccessful error sbt.TestsFailedException: Tests unsuccessful error sbt.TestsFailedException: Tests unsuccessful [error] Total time: 292 s, completed 21 Jul, 2014 3:03:12 PM

Jul 21 '14 09:07 Pallavim54

Can u plz tell me the syntax of input for running this graphx test https://github.com/apache/spark/blob/master/examples/src/main/scala/org/apache/spark/examples/graphx/LiveJournalPageRank.scala .

i m using ~/spark$ bin/run-example graphx.LiveJournalPageRank soc-LiveJournal1.txt --numEPart=6

which is throwing eror ;

Exception in thread "main" java.lang.IllegalArgumentException: Invalid argument: --numEPart at org.apache.spark.graphx.lib.Analytics$$anonfun$2.apply(Analytics.scala:43) at org.apache.spark.graphx.lib.Analytics$$anonfun$2.apply(Analytics.scala:40) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33) at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108) at scala.collection.TraversableLike$class.map(TraversableLike.scala:244) at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:108) at org.apache.spark.graphx.lib.Analytics$.main(Analytics.scala:40) at org.apache.spark.examples.graphx.LiveJournalPageRank$.main(LiveJournalPageRank.scala: at org.apache.spark.examples.graphx.LiveJournalPageRank.main(LiveJournalPageRank.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:304) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:69) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

I have also tried with this run-example graphx.LiveJournalPageRank soc-LiveJournal1.txt --tol=0.001 --output=D:/output.txt --numEPart=50 --partStrategy=RandomVertexCut

but no success

Jul 21 '14 10:07 Pallavim54

That command works for me:

$ ./bin/run-example graphx.LiveJournalPageRank ~/Downloads/web-Google.txt --numEPart=8
======================================
|             PageRank               |
======================================
[...]

Is it possible you used a space instead of an equals sign, as in --numEPart 8? The error occurs if I do that.

Jul 21 '14 10:07 ankurdave

Geeting error with both cases:

$ ./bin/run-example graphx.LiveJournalPageRank ~/Downloads/web-Google/web-Google.txt --numEPart 8 or --numEPart=8 Exception in thread "main" java.lang.IllegalArgumentException: Invalid argument: --numEPart at org.apache.spark.graphx.lib.Analytics$$anonfun$2.apply(Analytics.scala:43) at org.apache.spark.graphx.lib.Analytics$$anonfun$2.apply(Analytics.scala:40) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33) at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108) at scala.collection.TraversableLike$class.map(TraversableLike.scala:244) at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:108) at org.apache.spark.graphx.lib.Analytics$.main(Analytics.scala:40) at org.apache.spark.examples.graphx.LiveJournalPageRank$.main(LiveJournalPageRank.scala:47) at org.apache.spark.examples.graphx.LiveJournalPageRank.main(LiveJournalPageRank.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:304) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:69) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

Jul 22 '14 04:07 Pallavim54

can u plz let me know about this command bcoz i m not able to run this example

Jul 23 '14 09:07 Pallavim54