spark icon indicating copy to clipboard operation
spark copied to clipboard

[SPARK-39080][SHUFFLE]Optimize shuffle error handler

Open wankunde opened this issue 3 years ago • 4 comments

What changes were proposed in this pull request?

Optimize ErrorHandler code.

Why are the changes needed?

Shuffle ErrorHandler only has two stateless method, so we do not need create a ErrorHandler instance for each shuffle fetcher or shuffle pusher.

Does this PR introduce any user-facing change?

No

How was this patch tested?

Exists UT

wankunde avatar Apr 30 '22 15:04 wankunde

As a followup of SPARK-38965, if there are no BLOCK_APPEND_COLLISION exception, shuffle fetcher and shuffle pusher do not handle the exception returned from server, and RetryingBlockTransferor will retry if there is a IOException and errorHandler.shouldRetryError(e) is true.

wankunde avatar Apr 30 '22 15:04 wankunde

Can one of the admins verify this patch?

AmplabJenkins avatar May 01 '22 06:05 AmplabJenkins

I have not looked at the PR in detail, but we have to be careful to preserve backwards compatible and not make breaking changes.

mridulm avatar May 01 '22 07:05 mridulm

We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable. If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!

github-actions[bot] avatar Aug 10 '22 00:08 github-actions[bot]