Roger Giuffrè

this is the main error in cloudwatch Caused by: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid directory for output-  18:48:23 at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext$DirSelector.getPathForWrite(LocalDirAllocator.java:541)  18:48:23 at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:627)  18:48:23 at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.createTmpFileForWrite(LocalDirAllocator.java:640)...

if I provide the configuration information I entered but followed the documentation

I had already tried these parameters without success. Now out of desperation I was thinking of bypassing the hadoop interface and managing the stream directly. Is your email on linkedin...

ll try again and let you know

if i want to recompile it you suggest to use your hadoop version 2.6.0-qds-0.4.13 but not the reference to your repository. Can you suggest something about version 2.8 for example?

recompiling as you say I have the following error: Exception in thread "dag-scheduler-event-loop" java.lang.NoSuchMethodError: com.amazonaws.http.AmazonHttpClient.disableStrictHostnameVerification()

I have a repository of a docker image: https://github.com/webroboteu/sparklambdadriver I'm using hadoop version 2.7 and its dependencies.

With hadoop 2.9, referring to bundle 1.11.199 with these docker lines there is progress but I still have to confirm that it works on lambda context RUN wget http://central.maven.org/maven2/com/amazonaws/aws-java-sdk-bundle/1.11.199/aws-java-sdk-bundle-1.11.199.jar RUN...

with local execution now i have this problem: java.lang.NullPointerException at org.apache.spark.util.Utils$.localFileToS3(Utils.scala:2517) at org.apache.spark.shuffle.S3ShuffleBlockResolver.writeIndexFileAndCommit(S3ShuffleBlockResolver.scala:177) at org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:158) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53) at org.apache.spark.scheduler.Task.run(Task.scala:99) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:282) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:748) I...

I'm in the right direction since I can now recompile it correctly. For some strange reason try to load the data from the same executorId 4775351731 java.io.FileNotFoundException: No such file...