linkedin
example failure on run
(1) $hadoop version Hadoop 0.20.203.0
$hadoop jar mlease-1.0-jar-with-dependencies.jar com.linkedin.mlease.regression.jobs.Regression sample-config.job
Exception in thread "main" java.lang.NoSuchMethodError: org.codehaus.jackson.JsonFactory.enable(Lorg/codehaus/jackson/JsonParser$Feature;)Lorg/codehaus/jackson/JsonFactory;
at org.apache.avro.Schema.
(2) $hadoop version Hadoop 2.4.0
$hadoop jar mlease-1.0-jar-with-dependencies.jar com.linkedin.mlease.regression.jobs.Regression sample-config.job 14/10/11 09:58:40 INFO Configuration.deprecation: mapred.job.queue.name is deprecated. Instead, use mapreduce.job.queuename 14/10/11 09:58:40 INFO jobs.RegressionPrepare: Running the preparation job of admm with map.key = and num.blocks=20 14/10/11 09:58:40 INFO mapred.AvroUtils: Running hadoop job with input paths: 14/10/11 09:58:40 INFO mapred.AvroUtils: hdfs://ns1/user/junhui1/ml-ease/sample-data.avro 14/10/11 09:58:40 INFO mapred.AvroUtils: Output path=hdfs://ns1/user/junhui1/ml-ease/sample-out/tmp-data 14/10/11 09:58:40 INFO client.RMProxy: Connecting to ResourceManager at yz2135.hadoop.data.sina.com.cn/10.39.2.135:9031 14/10/11 09:58:41 WARN mapreduce.JobSubmitter: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this. 14/10/11 09:58:43 INFO mapred.FileInputFormat: Total input paths to process : 1 14/10/11 09:58:44 INFO mapreduce.JobSubmitter: number of splits:2 14/10/11 09:58:44 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1406191544821_301459 14/10/11 09:58:45 INFO impl.YarnClientImpl: Submitted application application_1406191544821_301459 14/10/11 09:58:45 INFO mapreduce.Job: The url to track the job: http://yz2135.hadoop.data.sina.com.cn:9008/proxy/application_1406191544821_301459/ 14/10/11 09:58:45 INFO mapreduce.Job: Running job: job_1406191544821_301459 14/10/11 09:58:52 INFO mapreduce.Job: Job job_1406191544821_301459 running in uber mode : false 14/10/11 09:58:52 INFO mapreduce.Job: map 0% reduce 0% 14/10/11 09:58:56 INFO mapreduce.Job: Task Id : attempt_1406191544821_301459_m_000000_0, Status : FAILED Error: org.apache.avro.generic.GenericData.createDatumWriter(Lorg/apache/avro/Schema;)Lorg/apache/avro/io/DatumWriter; 14/10/11 09:58:57 INFO mapreduce.Job: Task Id : attempt_1406191544821_301459_m_000001_0, Status : FAILED Error: org.apache.avro.generic.GenericData.createDatumWriter(Lorg/apache/avro/Schema;)Lorg/apache/avro/io/DatumWriter; 14/10/11 09:59:03 INFO mapreduce.Job: Task Id : attempt_1406191544821_301459_m_000001_1, Status : FAILED Error: org.apache.avro.generic.GenericData.createDatumWriter(Lorg/apache/avro/Schema;)Lorg/apache/avro/io/DatumWriter; 14/10/11 09:59:03 INFO mapreduce.Job: Task Id : attempt_1406191544821_301459_m_000000_1, Status : FAILED Error: org.apache.avro.generic.GenericData.createDatumWriter(Lorg/apache/avro/Schema;)Lorg/apache/avro/io/DatumWriter; 14/10/11 09:59:08 INFO mapreduce.Job: Task Id : attempt_1406191544821_301459_m_000000_2, Status : FAILED Error: org.apache.avro.generic.GenericData.createDatumWriter(Lorg/apache/avro/Schema;)Lorg/apache/avro/io/DatumWriter; 14/10/11 09:59:09 INFO mapreduce.Job: Task Id : attempt_1406191544821_301459_m_000001_2, Status : FAILED Error: org.apache.avro.generic.GenericData.createDatumWriter(Lorg/apache/avro/Schema;)Lorg/apache/avro/io/DatumWriter; 14/10/11 09:59:15 INFO mapreduce.Job: map 100% reduce 0% 14/10/11 09:59:15 INFO mapreduce.Job: Job job_1406191544821_301459 failed with state FAILED due to: Task failed task_1406191544821_301459_m_000000 Job failed as tasks failed. failedMaps:1 failedReduces:0
14/10/11 09:59:15 INFO mapreduce.Job: Counters: 13 Job Counters Failed map tasks=7 Killed map tasks=1 Launched map tasks=8 Other local map tasks=6 Rack-local map tasks=2 Total time spent by all maps in occupied slots (ms)=68644 Total time spent by all reduces in occupied slots (ms)=0 Total time spent by all map tasks (ms)=34322 Total vcore-seconds taken by all map tasks=34322 Total megabyte-seconds taken by all map tasks=52718592 Map-Reduce Framework CPU time spent (ms)=0 Physical memory (bytes) snapshot=0 Virtual memory (bytes) snapshot=0 14/10/11 09:59:15 INFO jobs.RegressionAdmmTrain: Now running Regression Train using ADMM... Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 0 at com.linkedin.mapred.AvroUtils.getAvroInputSchema(AvroUtils.java:204) at com.linkedin.mapred.AbstractAvroJob.createJobConf(AbstractAvroJob.java:283) at com.linkedin.mlease.regression.jobs.RegressionAdmmTrain.run(RegressionAdmmTrain.java:135) at com.linkedin.mlease.regression.jobs.Regression.run(Regression.java:60) at com.linkedin.mlease.regression.jobs.Regression.main(Regression.java:97) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
Excuse me for the late reply. Currently ml-ease only works for Hadoop 1.x. We are fixing issues with Hadoop 2.
