celeborn

celeborn copied to clipboard

[CELEBORN-1337] Optimize WorkerInfo in WorkerStatusTracker by removing unused fields

11

### What changes were proposed in this pull request? WorkerSummary is added as parent of WorkerInfo to be used in WorkerStatusTracker to reduce memory footprint ### Why are the changes...

ErikFang

[CELEBORN-1340] Congestion control slow start logic should make all interval can be customized and make it smaller

3

### What changes were proposed in this pull request? In this pr we do two things: 1. Current sleep time is too long for small task, will cause task delay...

AngersZhuuuu

[CELEBORN-1361] MaxInFlightPerWorker should use the value provided by PushStrategy

1

### What changes were proposed in this pull request? The data push thread should first send requests to workers that are not under pressure. Use PushStrategy's `currentMaxReqsInFlight` to better filter...

mcdull-zhang

[CELEBORN-1378] Call PartitionDataWriter#flush() before submitting to commitThreadPool

2

### What changes were proposed in this pull request? There are high commit files latency in some production environments. This PR triggers flush before calling close to avoid waiting in...

waitinfuture

[CELEBORN-914] Support memory file storage

2

### What changes were proposed in this pull request? To support memory file storage. ### Why are the changes needed? To improve shuffle performance for small shuffle files. Design doc:...

FMX

[CELEBORN-1402] SparkShuffleManager print warning log for spark.executor.userClassPathFirst=true with ShuffleManager defined in user jar

16

### What changes were proposed in this pull request? `SparkShuffleManager` print warning log for `spark.executor.userClassPathFirst=true` with `ShuffleManager` defined in user jar via `--jar` or `spark.jars`. ### Why are the changes...

SteNicholas

### What changes were proposed in this pull request? When build the open stream request, additionally check whether the host is equivalent to judge whether to read locally. ### Why...

onebox-li

### What changes were proposed in this pull request? For spark3 client, use a long variable to help to count written records instead of a `tmpRecords` array. ### Why are...

onebox-li

### What changes were proposed in this pull request? 1. Use ConcurrentSet to replace ArrayList for workers. 2. Remove unnecessary sync and snapshot when get workers/lostworkers/shutdownWorkers ### Why are the...

RexXiong

[CELEBORN-1400] Bump Ratis version from 2.5.1 to 3.0.1

7

### What changes were proposed in this pull request? Bump Ratis version from 2.5.1 to 3.0.1. Address incompatible changes: - RATIS-589. Eliminate buffer copying in SegmentedRaftLogOutputStream.(https://github.com/apache/ratis/pull/964) - RATIS-1677. Do not...

SteNicholas