环境变量超过最大长度该如何处理?
报错如下:
19/02/21 17:42:52 INFO XLearningContainer: Cluster def is: {"ps":["node38.hadoop.com:28249"],"worker":["node36.hadoop.com:24129","node5.hadoop.com:25401","node11.hadoop.com:26075","node18.hadoop.com:21271","node62.hadoop.com:29775"]}
19/02/21 17:42:52 WARN XLearningContainer: Current container environments length 576369 exceed the configuration xlearning.env.maxlength 102400
19/02/21 17:42:52 WARN XLearningContainer: InputFile list had written to local file: inputFileList.txt !!
19/02/21 17:42:52 INFO XLearningContainer: Executing command:bash -x dist_train.sh 194063 ./tfmodel
19/02/21 17:42:52 ERROR XLearningContainer: Some errors has occurred during container running!
java.io.IOException: Cannot run program "bash": error=7, Argument list too long
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
at java.lang.Runtime.exec(Runtime.java:620)
at java.lang.Runtime.exec(Runtime.java:450)
at java.lang.Runtime.exec(Runtime.java:388)
at net.qihoo.xlearning.container.XLearningContainer.run(XLearningContainer.java:673)
at net.qihoo.xlearning.container.XLearningContainer.main(XLearningContainer.java:983)
Caused by: java.io.IOException: error=7, Argument list too long
at java.lang.UNIXProcess.forkAndExec(Native Method)
at java.lang.UNIXProcess.
修改了xlearning.env.maxlength,但是仍然会报Cannot run program "bash": error=7, Argument list too long的错误
这个受系统环境的限制,已经有了解决办法,最近先会提交个PR