Franky Romero
Franky Romero
Ok, I created the function `spark-lambda` through the AWS console which has the code from `spark-lambda-os.py`, and this function is in the same VPC and subnet as the EC2 instance...
Awesome, yes! Adding `--master lambda://` fixes that problem. One more issue (I hope): when it finally tries to launch a job, I get the following error: `WARN TaskSchedulerImpl: Initial job...
Unfortunately, there still aren't any invocations of `spark-lambda`, so there are no CloudWatch logs available to check. I also noticed that even though it says it's created directories in the...
Glad you figured it out, @habemusne. For your second point, I think the combination of [this section](https://github.com/faromero/spark-on-lambda#setting-up-the-ec2-and-network-environment) and [this section](https://github.com/faromero/spark-on-lambda#setting-up-the-lambda-function) in my documentation should cover it as well. Also, regarding...
I am also having the same issue (also tried adding hadoop-lzo dependency manually to pom.xml with no success). Have there been any updates on resolving this issue?
I believe I have found a solution: In `spark-on-lambda/common/network-common/pom.xml`, add the following dependency (as suggested previously): ``` com.hadoop.gplcompression hadoop-lzo 0.4.19 ``` Then, in `spark-on-lambda/pom.xml`, add the following repository (which "houses"...
Thanks for working to update it! We are working on a research project associated with using Lambda for what we call "interactive massively parallel" applications, and wanted to compare Spark-on-Lambda...
You can view our work [here](https://github.com/StanfordSNR/gg): we call it *gg*, and while it was originally intended for compilation, it now supports general purpose applications (as simple as sorting and as...