[Question]: Configuring Spark Executor for GC Logging in the Same Directory as Standard Output

[huangzhir]

Contact Details

No response

What would you like to ask or discuss?

I have a question regarding memory waste diagnosis, which relies on the executor's GC logs. However, by default, Spark's executor logs do not include this. How should I add the parameter "spark.executor.extraJavaOptions=-verbose:gc -XX:+PrintGCDetails -Xloggc:gc.log"? How can I specify that the gc.log should be in the same directory as the standard output? This is necessary so that the YARN history server can collect the logs.

[nilnon]

We can add this parameter -Xloggc:<LOG_DIR>/gc.log.

[huangzhir]

Thank you for the suggestion. This approach is indeed viable.