tez icon indicating copy to clipboard operation
tez copied to clipboard

TEZ-4450: Shuffle data fetch fails when shuffle data is transferred via CompositeRoutedDataMovementEvent

Open ganeshashree opened this issue 3 years ago • 5 comments

Fixed failure in fetching shuffle data when it's transferred via CompositeRoutedDataMovementEvent. Added unit test to test the failure scenario.

Jira ticket: https://issues.apache.org/jira/browse/TEZ-4450

ganeshashree avatar Oct 11 '22 07:10 ganeshashree

thanks for the patch @ganeshashree, could you please open a jira ticket for this?

abstractdog avatar Oct 11 '22 07:10 abstractdog

@abstractdog Jira ticket is https://issues.apache.org/jira/browse/TEZ-4450. Please review.

ganeshashree avatar Oct 11 '22 07:10 ganeshashree

:broken_heart: -1 overall

Vote Subsystem Runtime Comment
+0 :ok: reexec 33m 46s Docker mode activated.
_ Prechecks _
+1 :green_heart: dupname 0m 0s No case conflicting files found.
+1 :green_heart: @author 0m 0s The patch does not contain any @author tags.
+1 :green_heart: test4tests 0m 0s The patch appears to include 2 new or modified test files.
_ master Compile Tests _
+1 :green_heart: mvninstall 16m 3s master passed
+1 :green_heart: compile 0m 50s master passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 :green_heart: compile 0m 43s master passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 :green_heart: checkstyle 1m 14s master passed
+1 :green_heart: javadoc 0m 51s master passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 :green_heart: javadoc 0m 40s master passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+0 :ok: spotbugs 1m 47s Used deprecated FindBugs config; considering switching to SpotBugs.
+1 :green_heart: findbugs 1m 44s master passed
_ Patch Compile Tests _
+1 :green_heart: mvninstall 0m 27s the patch passed
+1 :green_heart: compile 0m 31s the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 :green_heart: javac 0m 31s the patch passed
+1 :green_heart: compile 0m 26s the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 :green_heart: javac 0m 26s the patch passed
-0 :warning: checkstyle 0m 18s tez-runtime-library: The patch generated 5 new + 23 unchanged - 1 fixed = 28 total (was 24)
+1 :green_heart: whitespace 0m 0s The patch has no whitespace issues.
+1 :green_heart: javadoc 0m 24s the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 :green_heart: javadoc 0m 22s the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 :green_heart: findbugs 1m 18s the patch passed
_ Other Tests _
+1 :green_heart: unit 5m 49s tez-runtime-library in the patch passed.
-1 :x: asflicense 0m 15s The patch generated 1 ASF License warnings.
66m 41s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-243/2/artifact/out/Dockerfile
GITHUB PR https://github.com/apache/tez/pull/243
JIRA Issue TEZ-4450
Optional Tests dupname asflicense javac javadoc unit spotbugs findbugs checkstyle compile
uname Linux a01bf78e9ea3 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/tez.sh
git revision master / 921de531f
Default Java Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
checkstyle https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-243/2/artifact/out/diff-checkstyle-tez-runtime-library.txt
Test Results https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-243/2/testReport/
asflicense https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-243/2/artifact/out/patch-asflicense-problems.txt
Max. process+thread count 1090 (vs. ulimit of 5500)
modules C: tez-runtime-library U: tez-runtime-library
Console output https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-243/2/console
versions git=2.25.1 maven=3.6.3 findbugs=3.0.1
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

tez-yetus avatar Oct 11 '22 09:10 tez-yetus

:broken_heart: -1 overall

Vote Subsystem Runtime Comment
+0 :ok: reexec 39m 16s Docker mode activated.
_ Prechecks _
+1 :green_heart: dupname 0m 0s No case conflicting files found.
+1 :green_heart: @author 0m 0s The patch does not contain any @author tags.
+1 :green_heart: test4tests 0m 0s The patch appears to include 2 new or modified test files.
_ master Compile Tests _
+1 :green_heart: mvninstall 16m 4s master passed
+1 :green_heart: compile 0m 51s master passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 :green_heart: compile 0m 45s master passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 :green_heart: checkstyle 1m 15s master passed
+1 :green_heart: javadoc 0m 51s master passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 :green_heart: javadoc 0m 40s master passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+0 :ok: spotbugs 1m 49s Used deprecated FindBugs config; considering switching to SpotBugs.
+1 :green_heart: findbugs 1m 46s master passed
_ Patch Compile Tests _
+1 :green_heart: mvninstall 0m 26s the patch passed
+1 :green_heart: compile 0m 32s the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 :green_heart: javac 0m 32s the patch passed
+1 :green_heart: compile 0m 25s the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 :green_heart: javac 0m 25s the patch passed
-0 :warning: checkstyle 0m 18s tez-runtime-library: The patch generated 5 new + 23 unchanged - 1 fixed = 28 total (was 24)
+1 :green_heart: whitespace 0m 0s The patch has no whitespace issues.
+1 :green_heart: javadoc 0m 24s the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 :green_heart: javadoc 0m 22s the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 :green_heart: findbugs 1m 22s the patch passed
_ Other Tests _
+1 :green_heart: unit 5m 47s tez-runtime-library in the patch passed.
-1 :x: asflicense 0m 17s The patch generated 1 ASF License warnings.
72m 19s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-243/1/artifact/out/Dockerfile
GITHUB PR https://github.com/apache/tez/pull/243
JIRA Issue TEZ-4450
Optional Tests dupname asflicense javac javadoc unit spotbugs findbugs checkstyle compile
uname Linux fd1842961ee8 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/tez.sh
git revision master / 921de531f
Default Java Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
checkstyle https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-243/1/artifact/out/diff-checkstyle-tez-runtime-library.txt
Test Results https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-243/1/testReport/
asflicense https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-243/1/artifact/out/patch-asflicense-problems.txt
Max. process+thread count 1090 (vs. ulimit of 5500)
modules C: tez-runtime-library U: tez-runtime-library
Console output https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-243/1/console
versions git=2.25.1 maven=3.6.3 findbugs=3.0.1
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

tez-yetus avatar Oct 11 '22 09:10 tez-yetus

:confetti_ball: +1 overall

Vote Subsystem Runtime Comment
+0 :ok: reexec 1m 3s Docker mode activated.
_ Prechecks _
+1 :green_heart: dupname 0m 0s No case conflicting files found.
+1 :green_heart: @author 0m 0s The patch does not contain any @author tags.
+1 :green_heart: test4tests 0m 0s The patch appears to include 2 new or modified test files.
_ master Compile Tests _
+1 :green_heart: mvninstall 16m 2s master passed
+1 :green_heart: compile 0m 44s master passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 :green_heart: compile 0m 39s master passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 :green_heart: checkstyle 1m 12s master passed
+1 :green_heart: javadoc 0m 48s master passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 :green_heart: javadoc 0m 35s master passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+0 :ok: spotbugs 1m 31s Used deprecated FindBugs config; considering switching to SpotBugs.
+1 :green_heart: findbugs 1m 30s master passed
_ Patch Compile Tests _
+1 :green_heart: mvninstall 0m 23s the patch passed
+1 :green_heart: compile 0m 26s the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 :green_heart: javac 0m 26s the patch passed
+1 :green_heart: compile 0m 21s the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 :green_heart: javac 0m 21s the patch passed
+1 :green_heart: checkstyle 0m 16s tez-runtime-library: The patch generated 0 new + 23 unchanged - 1 fixed = 23 total (was 24)
+1 :green_heart: whitespace 0m 0s The patch has no whitespace issues.
+1 :green_heart: javadoc 0m 20s the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
+1 :green_heart: javadoc 0m 19s the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
+1 :green_heart: findbugs 1m 0s the patch passed
_ Other Tests _
+1 :green_heart: unit 5m 32s tez-runtime-library in the patch passed.
+1 :green_heart: asflicense 0m 15s The patch does not generate ASF License warnings.
32m 17s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-243/3/artifact/out/Dockerfile
GITHUB PR https://github.com/apache/tez/pull/243
JIRA Issue TEZ-4450
Optional Tests dupname asflicense javac javadoc unit spotbugs findbugs checkstyle compile
uname Linux c7cd89b43789 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/tez.sh
git revision master / 921de531f
Default Java Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
Test Results https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-243/3/testReport/
Max. process+thread count 2091 (vs. ulimit of 5500)
modules C: tez-runtime-library U: tez-runtime-library
Console output https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-243/3/console
versions git=2.25.1 maven=3.6.3 findbugs=3.0.1
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

tez-yetus avatar Oct 11 '22 11:10 tez-yetus