Colin
Colin
@summaryzb For this optimization, I think there has once more `System.arraycopy` to add record, and it will impact performance a lot with previous test. For this PR, I think it...
> > @summaryzb For this optimization, I think there has once more `System.arraycopy` to add record, and it will impact performance a lot with previous test. For this PR, I...
In spark client, all memory are requested from executor, so there shouldn't have critical problem, eg, memory leak, oom, etc. Can you show the case how this PR improve the...
I think AE is an experience feature in Spark 2.4, and it was officially announced in Spark 3. So it isn't planned to be supported in Spark 2.4 in short...
> I found if use `MEMORY_LOCALFILE`, `finishShuffle` will not be called, and buffer in server side may not flush in time, and than reader will fail because read index file...
> We had set `spark.rss.data.replica.write=2` and `spark.rss.data.replica=3`.But we found all shuffle server of a partition have not flush in time today and we have found in two applications. It may...
@frankliee can you do more clarification about how to config `spark.rss.data.replica.write` & `spark.rss.data.replica.read` ?
@zuston You can refer this [blog](https://cloud.tencent.com/developer/article/1943179) for the benchmark related.
@zuston The benchmark of blog is based on Spark 2.4.6. If there has no random disk IO problem with ESS, Uniffle is expected has **poor performance** than ESS
I think @xianjingfeng is right, with current implementation, OOM will happen if `requireBufferId` was expired in Shuffle Server already, this maybe caused by GC, network problem, high workload in shuffle...