Angel-ML
求助 Failed to run job : Application application_1591113812497_103797 failed 2 times 作业运行失败
[2020-06-25 22:33:23.064]Container exited with a non-zero exit code 1. Error file: prelaunch.err. Last 4096 bytes of prelaunch.err : Last 4096 bytes of stderr : Java HotSpot(TM) 64-Bit Server VM warning: ignoring option PermSize=100M; support was removed in 8.0 Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=200M; support was removed in 8.0
[2020-06-25 22:33:23.065]Container exited with a non-zero exit code 1. Error file: prelaunch.err. Last 4096 bytes of prelaunch.err : Last 4096 bytes of stderr : Java HotSpot(TM) 64-Bit Server VM warning: ignoring option PermSize=100M; support was removed in 8.0 Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=200M; support was removed in 8.0
For more detailed output, check the application tracking page: http://ecs-hn1b-bd-cdp-edg-2:8188/applicationhistory/app/application_1591113812497_103797 Then click on links to logs of each attempt. . Failing the application. at com.tencent.angel.client.yarn.AngelYarnClient.updateMaster(AngelYarnClient.java:517) at com.tencent.angel.client.yarn.AngelYarnClient.startPSServer(AngelYarnClient.java:170) at com.tencent.angel.client.AngelPSClient.startPS(AngelPSClient.java:115) at com.tencent.angel.sona.core.DriverContext.startAngelAndPSAgent(DriverContext.scala:97) at com.tencent.angel.sona.examples.JsonRunnerExamples$.main(JsonRunnerExamples.scala:69) at com.tencent.angel.sona.examples.JsonRunnerExamples.main(JsonRunnerExamples.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52) at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:851) at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167) at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195) at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86) at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:926) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:935) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) Exception in thread "main" com.tencent.angel.exception.AngelException: java.io.IOException: Failed to run job : Application application_1591113812497_103797 failed 2times (global limit =3; local limit is =2) due to AM Container for appattempt_1591113812497_103797_000002 exited with exitCode: 1 Failing this attempt.Diagnostics: [2020-06-25 22:33:23.063]Exception from container-launch. Container id: container_e74_1591113812497_103797_02_000001 Exit code: 1
[2020-06-25 22:33:23.064]Container exited with a non-zero exit code 1. Error file: prelaunch.err. Last 4096 bytes of prelaunch.err : Last 4096 bytes of stderr : Java HotSpot(TM) 64-Bit Server VM warning: ignoring option PermSize=100M; support was removed in 8.0 Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=200M; support was removed in 8.0
[2020-06-25 22:33:23.065]Container exited with a non-zero exit code 1. Error file: prelaunch.err. Last 4096 bytes of prelaunch.err : Last 4096 bytes of stderr : Java HotSpot(TM) 64-Bit Server VM warning: ignoring option PermSize=100M; support was removed in 8.0 Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=200M; support was removed in 8.0
For more detailed output, check the application tracking page: http://ecs-hn1b-bd-cdp-edg-2:8188/applicationhistory/app/application_1591113812497_103797 Then click on links to logs of each attempt. . Failing the application. at com.tencent.angel.client.yarn.AngelYarnClient.startPSServer(AngelYarnClient.java:176) at com.tencent.angel.client.AngelPSClient.startPS(AngelPSClient.java:115) at com.tencent.angel.sona.core.DriverContext.startAngelAndPSAgent(DriverContext.scala:97) at com.tencent.angel.sona.examples.JsonRunnerExamples$.main(JsonRunnerExamples.scala:69) at com.tencent.angel.sona.examples.JsonRunnerExamples.main(JsonRunnerExamples.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52) at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:851) at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167) at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195) at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86) at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:926) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:935) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) Caused by: java.io.IOException: Failed to run job : Application application_1591113812497_103797 failed 2 times (global limit =3; local limit is =2) due to AM Container for appattempt_1591113812497_103797_000002 exited with exitCode: 1 Failing this attempt.Diagnostics: [2020-06-25 22:33:23.063]Exception from container-launch. Container id: container_e74_1591113812497_103797_02_000001 Exit code: 1
[2020-06-25 22:33:23.064]Container exited with a non-zero exit code 1. Error file: prelaunch.err. Last 4096 bytes of prelaunch.err : Last 4096 bytes of stderr : Java HotSpot(TM) 64-Bit Server VM warning: ignoring option PermSize=100M; support was removed in 8.0 Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=200M; support was removed in 8.0
[2020-06-25 22:33:23.065]Container exited with a non-zero exit code 1. Error file: prelaunch.err. Last 4096 bytes of prelaunch.err : Last 4096 bytes of stderr : Java HotSpot(TM) 64-Bit Server VM warning: ignoring option PermSize=100M; support was removed in 8.0 Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=200M; support was removed in 8.0
For more detailed output, check the application tracking page: http://ecs-hn1b-bd-cdp-edg-2:8188/applicationhistory/app/application_1591113812497_103797 Then click on links to logs of each attempt. . Failing the application. at com.tencent.angel.client.yarn.AngelYarnClient.updateMaster(AngelYarnClient.java:517) at com.tencent.angel.client.yarn.AngelYarnClient.startPSServer(AngelYarnClient.java:170) ... 16 more 20/06/25 22:33:15 INFO spark.SparkContext: Invoking stop() from shutdown hook 20/06/25 22:33:15 INFO server.AbstractConnector: Stopped Spark@7a389761{HTTP/1.1,[http/1.1]}{0.0.0.0:4041} 20/06/25 22:33:15 INFO ui.SparkUI: Stopped Spark web UI at http://ecs-hn1a-xng-alg-rcmd-edg-1:4041 20/06/25 22:33:15 INFO cluster.YarnClientSchedulerBackend: Interrupting monitor thread 20/06/25 22:33:15 INFO cluster.YarnClientSchedulerBackend: Shutting down all executors 20/06/25 22:33:15 INFO cluster.YarnSchedulerBackend$YarnDriverEndpoint: Asking each executor to shut down 20/06/25 22:33:15 INFO cluster.SchedulerExtensionServices: Stopping SchedulerExtensionServices (serviceOption=None, services=List(), started=false) 20/06/25 22:33:15 INFO cluster.YarnClientSchedulerBackend: Stopped 20/06/25 22:33:15 INFO spark.MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped! 20/06/25 22:33:15 INFO memory.MemoryStore: MemoryStore cleared 20/06/25 22:33:15 INFO storage.BlockManager: BlockManager stopped 20/06/25 22:33:15 INFO storage.BlockManagerMaster: BlockManagerMaster stopped 20/06/25 22:33:15 INFO scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped! 20/06/25 22:33:15 INFO spark.SparkContext: Successfully stopped SparkContext 20/06/25 22:33:15 INFO util.ShutdownHookManager: Shutdown hook called 20/06/25 22:33:15 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-a689fb57-937d-4d2b-bf50-f59d08e9b6b8 20/06/25 22:33:15 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-2d255dca-b9e1-45ad-ba9d-6b8beae996e3 20/06/25 22:33:15 INFO client.AngelClient: stop the application 20/06/25 22:33:15 INFO client.AngelClient: master is null, just kill the application 20/06/25 22:33:15 INFO impl.YarnClientImpl: Killed application application_1591113812497_103797
