GoSpark testing problem

Open pbandyop opened this issue 10 years ago • 1 comments

Hi, I have trying to run this project successfully in my local computer - using both the scheduler and workers in same machine but using different terminal and ports of localhost. Terminal - using scheduler go test -run MasterMRLineCount -debug runs fine and I get the following output:

Test: Master MapReduce Line Count... master ip localhost port :4567 2015/03/30 11:22:10 method AssignJob has wrong number of ins: 5 2015/03/30 11:22:10 method KillWorkers has wrong number of ins: 1 2015/03/30 11:22:10 method Shutdown has wrong number of ins: 1 2015/03/30 11:22:10 method StartRegistrationServer has wrong number of ins: 1 2015/03/30 11:22:10 method WorkersAvailable has wrong number of ins: 1 2015/03/30 11:22:10 RegistrationServer: ready

but if I run the other command: go test -run KMeans -timeout 1h then I get the following errors: `In NewScheduler: master ip localhost port :4567 2015/03/30 11:21:23 method AssignJob has wrong number of ins: 5 2015/03/30 11:21:23 method KillWorkers has wrong number of ins: 1 2015/03/30 11:21:23 method Shutdown has wrong number of ins: 1 2015/03/30 11:21:23 method StartRegistrationServer has wrong number of ins: 1 2015/03/30 11:21:23 method WorkersAvailable has wrong number of ins: 1 2015/03/30 11:21:23 Context [kmeans] is started. x1: Iter: 0 --- FAIL: TestKMeans (0.81s) panic: runtime error: index out of range [recovered] panic: runtime error: index out of range

goroutine 7 [running]: testing.func·006() /usr/local/go/src/testing/testing.go:441 +0x181 spark.(*RDD).Collect(0xc20805a800, 0x0, 0x0, 0x0) /path_to_project/GoSpark/Spark/src/spark/rdd.go:107 +0x2a8 spark.TestKMeans(0xc20803c090) /path_to_project/GoSpark/Spark/src/spark/kmeans_test.go:149 +0x8b6 testing.tRunner(0xc20803c090, 0x650cb8) /usr/local/go/src/testing/testing.go:447 +0xbf created by testing.RunTests /usr/local/go/src/testing/testing.go:555 +0xa8b

goroutine 1 [chan receive]: testing.RunTests(0x50fc98, 0x650ca0, 0xb, 0xb, 0xa88773a4036b1b01) /usr/local/go/src/testing/testing.go:556 +0xad6 testing.(*M).Run(0xc208030910, 0x65e240) /usr/local/go/src/testing/testing.go:485 +0x6c main.main() spark/_test/_testmain.go:72 +0x1d5

goroutine 8 [IO wait]: net.(_pollDesc).Wait(0xc208010370, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(_pollDesc).WaitRead(0xc208010370, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(_netFD).accept(0xc208010310, 0x0, 0x724ff0, 0xc20802af30) /usr/local/go/src/net/fd_unix.go:419 +0x40b net.(_TCPListener).AcceptTCP(0xc20802e0c8, 0x713000, 0x0, 0x0) /usr/local/go/src/net/tcpsock_posix.go:234 +0x4e net.(_TCPListener).Accept(0xc20802e0c8, 0x0, 0x0, 0x0, 0x0) /usr/local/go/src/net/tcpsock_posix.go:244 +0x4c spark.func·003() /path_to_project/GoSpark/Spark/src/spark/master.go:121 +0x92 created by spark.(_Master).StartRegistrationServer /path_to_project/GoSpark/Spark/src/spark/master.go:133 +0x451

goroutine 9 [IO wait]: net.(_pollDesc).Wait(0xc2080101b0, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47 net.(_pollDesc).WaitRead(0xc2080101b0, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43 net.(_netFD).accept(0xc208010150, 0x0, 0x724ff0, 0xc20802af70) /usr/local/go/src/net/fd_unix.go:419 +0x40b net.(_TCPListener).AcceptTCP(0xc20802e0e0, 0x1abf94, 0x0, 0x0) /usr/local/go/src/net/tcpsock_posix.go:234 +0x4e net/http.tcpKeepAliveListener.Accept(0xc20802e0e0, 0x0, 0x0, 0x0, 0x0) /usr/local/go/src/net/http/server.go:1976 +0x4c net/http.(_Server).Serve(0xc208032300, 0x7277c0, 0xc20802e0e0, 0x0, 0x0) /usr/local/go/src/net/http/server.go:1728 +0x92 net/http.(_Server).ListenAndServe(0xc208032300, 0x0, 0x0) /usr/local/go/src/net/http/server.go:1718 +0x154 net/http.ListenAndServe(0x42deb0, 0x6, 0x726618, 0xc20802af50, 0x0, 0x0) /usr/local/go/src/net/http/server.go:1808 +0xba spark.func·001() /path_to_project/GoSpark/Spark/src/spark/master.go:59 +0xc1 created by spark.MakeMaster /path_to_project/GoSpark/Spark/src/spark/master.go:63 +0x300 exit status 2 FAIL spark 0.830s`

Also, if I run the worker commands then I get register RPC error. If I give the following command: go test -run BasicWorker -timeout 1h Test: Basic Worker... master ip localhost port :4567 worker ip localhost port :5347 Register: RPC localhost:4567 register error Register: RPC localhost:4567 register error Register: RPC localhost:4567 register error Register: RPC localhost:4567 register error Register: RPC localhost:4567 register error

Also, if I give the following command: go test -run WorkerSuperUnrel -timeout 1h -debug I get the same error:

Test: Worker with Unreliable RPC and Crash Reboot ... master ip localhost port :4567 worker ip localhost port :5911 2015/03/30 11:23:01 RunWorker localhost:5911 2015/03/30 11:23:01 worker: ready Register: RPC localhost:4567 register error Register: RPC localhost:4567 register error Register: RPC localhost:4567 register error Register: RPC localhost:4567 register error

Any idea if I am missing something or why am I getting these errors?

Mar 30 '15 09:03 pbandyop

Hi pbandyop, sorry for the bugs. This project was a course project, so we may not have tested well enough. And now we don't have much spare cycles to take care of this project. So you may need to debug by yourself by tracing the Go script. Feel free to fork it and patch the bugs you find. Thanks!

Mar 30 '15 12:03 peterkty