spark
spark copied to clipboard
apache
[SPARK-38910][YARN] Clean spark staging before unregister
What changes were proposed in this pull request?
After discussing about https://github.com/apache/spark/pull/36207 and re-check the whole logic, we should revert https://github.com/apache/spark/pull/36207 and do some change
- No matter whether it's client or cluster mode if it's the last attempt, anyway yarn won't rerun the job, we can clean staging dir first then we can avoid remaining staging dir if unregister failed.
- If it's cluster or client mode, and it's not the last attempt and the final status is SUCCESS, if unregister failed, YARN will rerun the job again, we can't clean the staging dir before unregistering success because if we clean the staging dir before rerunning, yarn can't download the related files and fail.
- If it's cluster unmanaged mode, if it failed, we can first delete the staging dir since it won't rerun.
Why are the changes needed?
Revert change and make it more accurate
Does this PR introduce any user-facing change?
No
How was this patch tested?
items 2 and 3 in the description, is one of those supposed to be client mode? Otherwise they are the same.
ping @dongjoon-hyun The latest GA failed caused by
* DONE (miniUI)
ERROR: dependency ‘pkgdown’ is not available for package ‘devtools’
* removing ‘/usr/local/lib/R/site-library/devtools’
The downloaded source packages are in
‘/tmp/RtmpTvMfJ6/downloaded_packages’
Warning messages:
1: In install.packages(c("devtools"), repos = "https://cloud.r-project.org/") :
installation of package ‘systemfonts’ had non-zero exit status
2: In install.packages(c("devtools"), repos = "https://cloud.r-project.org/") :
installation of package ‘textshaping’ had non-zero exit status
3: In install.packages(c("devtools"), repos = "https://cloud.r-project.org/") :
installation of package ‘ragg’ had non-zero exit status
4: In install.packages(c("devtools"), repos = "https://cloud.r-project.org/") :
installation of package ‘pkgdown’ had non-zero exit status
5: In install.packages(c("devtools"), repos = "https://cloud.r-project.org/") :
installation of package ‘devtools’ had non-zero exit status
Error in loadNamespace(x) : there is no package called ‘devtools’
Calls: loadNamespace -> withRestarts -> withOneRestart -> doWithOneRestart
Execution halted
Error: Process completed with exit code 1.
Any advise?