Ye Zhou comments

Results 8 comments of


                                            Ye Zhou

[SPARK-36620][SHUFFLE] Add Push Based Shuffle client side metrics

@thejdeep This PR should have missed the changes required in ShuffleBlockFetchIterator which update the ShuffleMetrics while shuffle fetching is on-going.

[SPARK-36620][SHUFFLE] Add Push Based Shuffle client side metrics

@thejdeep Thanks for updating the PR. Yet to review it, but several UTs within HistoryServerSuite failed, can you help double check?

Spark 2.3 is now out... Will Dr.elephant support it out of the box?

SPARK-18085 brings the LevelDB storage for Spark History Server(SHS), this would help Dr. Elephant to gather metrics as it improves SHS overall performance. If you have some former versions of...

Spark 2.3 is now out... Will Dr.elephant support it out of the box?

> @zhouyejoe does your team/Linkedin have any plans to open source the Spark 2.3/4 patches you mentioned above? Those PRs has been merged into Spark trunk, please take a look...

[SPARK-33573][SHUFFLE][YARN] Shuffle server side metrics for Push-based shuffle

There were some test failures in YarnShuffleServiceSuite from some of the UTs adde by me in the NodeManager work preserving restart PR. Reran the PR test job to see whether...

[SPARK-36620][SHUFFLE] Add Push Based Shuffle client side metrics

Left some minor comments last week, others mostly LGTM. @thejdeep

[SPARK-36620][SHUFFLE] Add Push Based Shuffle client side metrics

There are also UT failures related to this PR @thejdeep

[SPARK-36620][SHUFFLE] Add Push Based Shuffle client side metrics

LGTM. Thanks for working on this PR. @thejdeep