twitter
Problem with installing scalding and running REPL
Hi!
My goal is to practice with scalding. Firstly, in local mode using REPL.
I cloned last version of scalding (v0.16.0-RC6-17-g0302603) to my local directory (C:\scalding).
Then changed scalaVersion := "2.11.7" to "2.10.5", as I use this version of scala, in build.sbt.
Then run sbt "scalding-repl/run --local". And it failed with following:
C:\scalding>sbt "scalding-repl/run --local"
[info] Loading project definition from C:\scalding\project
[info] Set current project to scalding (in build file:/C:/scalding/)
[info] Running com.twitter.scalding.ScaldingShell --local
log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.Shell)
.
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more in
fo.
#### Scalding mode: Local(true)
#### User home: C:\scalding
[error] (run-main-0) java.lang.NullPointerException
java.lang.NullPointerException
at java.io.File.<init>(File.java:277)
at com.twitter.scalding.ScaldingILoop$$anonfun$3.apply(ScaldingILoop.sca
la:36)
at com.twitter.scalding.ScaldingILoop$$anonfun$3.apply(ScaldingILoop.sca
la:32)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
at scala.collection.Iterator$class.foreach(Iterator.scala:727)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:
48)
at scala.collection.mutable.ListBuffer.$plus$plus$eq(ListBuffer.scala:17
6)
at scala.collection.mutable.ListBuffer.$plus$plus$eq(ListBuffer.scala:45
)
at scala.collection.TraversableOnce$class.to(TraversableOnce.scala:273)
at scala.collection.AbstractIterator.to(Iterator.scala:1157)
at scala.collection.TraversableOnce$class.toList(TraversableOnce.scala:2
57)
at scala.collection.AbstractIterator.toList(Iterator.scala:1157)
at com.twitter.scalding.ScaldingILoop$.findAllUpPath(ScaldingILoop.scala
:45)
at com.twitter.scalding.ScaldingILoop$$anonfun$createInterpreter$1.apply
$mcV$sp(ScaldingILoop.scala:116)
at com.twitter.scalding.ScaldingILoop$$anonfun$createInterpreter$1.apply
(ScaldingILoop.scala:109)
at com.twitter.scalding.ScaldingILoop$$anonfun$createInterpreter$1.apply
(ScaldingILoop.scala:109)
at scala.tools.nsc.interpreter.IMain.beQuietDuring(IMain.scala:232)
at com.twitter.scalding.ScaldingILoop.createInterpreter(ScaldingILoop.sc
ala:109)
at scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply$mcZ$sp(ILo
op.scala:839)
at scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply(ILoop.scal
a:837)
at scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply(ILoop.scal
a:837)
at scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClass
Loader.scala:135)
at scala.tools.nsc.interpreter.ILoop.process(ILoop.scala:837)
at com.twitter.scalding.BaseScaldingShell$class.process(ScaldingShell.sc
ala:99)
at com.twitter.scalding.ScaldingShell$.process(ScaldingShell.scala:196)
at com.twitter.scalding.BaseScaldingShell$class.main(ScaldingShell.scala
:127)
at com.twitter.scalding.ScaldingShell$.main(ScaldingShell.scala:196)
at com.twitter.scalding.ScaldingShell.main(ScaldingShell.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
sorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
[trace] Stack trace suppressed: run last scalding-repl/unprovided:run for the fu
ll output.
java.lang.RuntimeException: Nonzero exit code: 1
at scala.sys.package$.error(package.scala:27)
[trace] Stack trace suppressed: run last scalding-repl/*:run for the full output
.
[error] (scalding-repl/*:run) Nonzero exit code: 1
[error] Total time: 24 s, completed Apr 28, 2016 9:45:42 AM
Also I tried to run consequentially "sbt update", "sbt test", "sbt assembly" as it mentioned in documentation, for running my scala code using sripts/sclad.rb. "sbt update" finished with success, but "sbt test" failed. If trace is helpful I will provide it.
Only command works fine is "sbt scalding-repl/console". It opens scala console (scala>, not scalding>) and there I can run import com.twitter.scalding._ for example, but It's not exactly what I'm looking for.
My settings: Windows 7 Enetrprise, JDK 1.7, Scala 2.10.5, sbt 0.13.11.
What else I tried: use other version of scalding repo (0.15.0, 0.14.0), use Scala 2.11.8, 2.11.7, use JDK 1.8.
Please help to resolve this.
Hi @yavladys, thanks for reporting this issue. I think we primarily test Scalding in Mac OSX/Linux environments, so this might be a case of how the JVM reports Windows files.
A workaround may be to create an empty .scalding_repl file in the directory you're running the REPL from. Sorry you're having these problems :)
Hello again!
It didn't help btw and only way I found how to run scalding REPL is run it under linux virtual machine :)
But in the same time there is another approach for testing scalding jobs under Windows locally. I run this in project directory:
sbt "run-main com.twitter.scalding.Tool your.package.scaldingJob --local --input1 arg1 --input2 arg2"
In fact, I run scalding job, not REPL, but this is really fast way to debug your code locally as all dependencies are loading only once even if you make changes in your scaldingJob. So, thanks to this you can edit your scalding code and quickly test it locally before moving to hdfs,
Hey @yavladys, there are some great advantages to using a REPL. I think people like using it because it's interactive.
Did you receive the same stacktrace when adding .scalding_repl in the directory you are running from?
I think it's fair to say we've never really tried to support windows. We definitely don't run any automated tests for windows. I think we'd probably need to have a discussion around whether or not windows support is something we want to commit to as well.
