hbase icon indicating copy to clipboard operation
hbase copied to clipboard

HBASE-27264 Add options to consider compressed size when delimiting blocks during hfile writes

Open wchevreuil opened this issue 3 years ago • 24 comments

Here we propose two additional properties,"hbase.block.size.limit.compressed" and "hbase.block.size.max.compressed" that would allow for consider the compressed size (if compression is in use) for delimiting blocks during hfile writing. When compression is enabled, certain datasets can have very high compression efficiency, so that the default 64KB block size and 10GB max file size can lead to hfiles with very large number of blocks.

In this proposal, "hbase.block.size.limit.compressed" is a boolean flag that switches to compressed size for delimiting blocks, and "hbase.block.size.max.compressed" is an int with the limit, in bytes for the compressed block size, in order to avoid very large uncompressed blocks (defaulting to 320KB).

wchevreuil avatar Aug 02 '22 11:08 wchevreuil

:broken_heart: -1 overall

Vote Subsystem Runtime Comment
+0 :ok: reexec 1m 5s Docker mode activated.
_ Prechecks _
+1 :green_heart: dupname 0m 0s No case conflicting files found.
+1 :green_heart: hbaseanti 0m 0s Patch does not have any anti-patterns.
+1 :green_heart: @author 0m 0s The patch does not contain any @author tags.
_ master Compile Tests _
+1 :green_heart: mvninstall 2m 28s master passed
+1 :green_heart: compile 2m 15s master passed
+1 :green_heart: checkstyle 0m 33s master passed
+1 :green_heart: spotless 0m 44s branch has no errors when running spotless:check.
+1 :green_heart: spotbugs 1m 19s master passed
_ Patch Compile Tests _
+1 :green_heart: mvninstall 2m 9s the patch passed
+1 :green_heart: compile 2m 11s the patch passed
+1 :green_heart: javac 2m 11s the patch passed
-0 :warning: checkstyle 0m 31s hbase-server: The patch generated 2 new + 1 unchanged - 0 fixed = 3 total (was 1)
+1 :green_heart: whitespace 0m 0s The patch has no whitespace issues.
+1 :green_heart: hadoopcheck 11m 32s Patch does not cause any errors with Hadoop 3.1.2 3.2.2 3.3.1.
-1 :x: spotless 0m 36s patch has 65 errors when running spotless:check, run spotless:apply to fix.
+1 :green_heart: spotbugs 1m 24s the patch passed
_ Other Tests _
+1 :green_heart: asflicense 0m 8s The patch does not generate ASF License warnings.
32m 7s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4675/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR https://github.com/apache/hbase/pull/4675
Optional Tests dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
uname Linux 08dc61e24ed0 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / e8c14ee308
Default Java AdoptOpenJDK-1.8.0_282-b08
checkstyle https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4675/1/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt
spotless https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4675/1/artifact/yetus-general-check/output/patch-spotless.txt
Max. process+thread count 60 (vs. ulimit of 30000)
modules C: hbase-server U: hbase-server
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4675/1/console
versions git=2.17.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase avatar Aug 02 '22 12:08 Apache-HBase

:broken_heart: -1 overall

Vote Subsystem Runtime Comment
+0 :ok: reexec 3m 36s Docker mode activated.
-0 :warning: yetus 0m 2s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 :green_heart: mvninstall 2m 30s master passed
+1 :green_heart: compile 0m 35s master passed
+1 :green_heart: shadedjars 4m 0s branch has no errors when building our shaded downstream artifacts.
+1 :green_heart: javadoc 0m 23s master passed
_ Patch Compile Tests _
+1 :green_heart: mvninstall 2m 14s the patch passed
+1 :green_heart: compile 0m 35s the patch passed
+1 :green_heart: javac 0m 35s the patch passed
+1 :green_heart: shadedjars 3m 59s patch has no errors when building our shaded downstream artifacts.
+1 :green_heart: javadoc 0m 22s the patch passed
_ Other Tests _
-1 :x: unit 205m 7s hbase-server in the patch failed.
225m 35s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4675/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR https://github.com/apache/hbase/pull/4675
Optional Tests javac javadoc unit shadedjars compile
uname Linux beda996b50e9 5.4.0-1081-aws #88~18.04.1-Ubuntu SMP Thu Jun 23 16:29:17 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / e8c14ee308
Default Java AdoptOpenJDK-1.8.0_282-b08
unit https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4675/1/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt
Test Results https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4675/1/testReport/
Max. process+thread count 2680 (vs. ulimit of 30000)
modules C: hbase-server U: hbase-server
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4675/1/console
versions git=2.17.1 maven=3.6.3
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase avatar Aug 02 '22 15:08 Apache-HBase

:broken_heart: -1 overall

Vote Subsystem Runtime Comment
+0 :ok: reexec 6m 1s Docker mode activated.
-0 :warning: yetus 0m 2s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 :green_heart: mvninstall 2m 49s master passed
+1 :green_heart: compile 0m 47s master passed
+1 :green_heart: shadedjars 3m 48s branch has no errors when building our shaded downstream artifacts.
+1 :green_heart: javadoc 0m 28s master passed
_ Patch Compile Tests _
+1 :green_heart: mvninstall 2m 38s the patch passed
+1 :green_heart: compile 0m 46s the patch passed
+1 :green_heart: javac 0m 46s the patch passed
+1 :green_heart: shadedjars 3m 45s patch has no errors when building our shaded downstream artifacts.
+1 :green_heart: javadoc 0m 26s the patch passed
_ Other Tests _
-1 :x: unit 206m 18s hbase-server in the patch failed.
229m 28s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4675/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR https://github.com/apache/hbase/pull/4675
Optional Tests javac javadoc unit shadedjars compile
uname Linux 0e15a50216e9 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / e8c14ee308
Default Java AdoptOpenJDK-11.0.10+9
unit https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4675/1/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt
Test Results https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4675/1/testReport/
Max. process+thread count 2688 (vs. ulimit of 30000)
modules C: hbase-server U: hbase-server
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4675/1/console
versions git=2.17.1 maven=3.6.3
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase avatar Aug 02 '22 15:08 Apache-HBase

:confetti_ball: +1 overall

Vote Subsystem Runtime Comment
+0 :ok: reexec 1m 14s Docker mode activated.
_ Prechecks _
+1 :green_heart: dupname 0m 1s No case conflicting files found.
+1 :green_heart: hbaseanti 0m 0s Patch does not have any anti-patterns.
+1 :green_heart: @author 0m 0s The patch does not contain any @author tags.
_ master Compile Tests _
+1 :green_heart: mvninstall 2m 22s master passed
+1 :green_heart: compile 2m 13s master passed
+1 :green_heart: checkstyle 0m 31s master passed
+1 :green_heart: spotless 0m 44s branch has no errors when running spotless:check.
+1 :green_heart: spotbugs 1m 22s master passed
_ Patch Compile Tests _
+1 :green_heart: mvninstall 2m 20s the patch passed
+1 :green_heart: compile 2m 12s the patch passed
+1 :green_heart: javac 2m 12s the patch passed
+1 :green_heart: checkstyle 0m 31s the patch passed
+1 :green_heart: whitespace 0m 0s The patch has no whitespace issues.
+1 :green_heart: hadoopcheck 11m 31s Patch does not cause any errors with Hadoop 3.1.2 3.2.2 3.3.1.
+1 :green_heart: spotless 0m 44s patch has no errors when running spotless:check.
+1 :green_heart: spotbugs 1m 22s the patch passed
_ Other Tests _
+1 :green_heart: asflicense 0m 10s The patch does not generate ASF License warnings.
32m 9s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4675/2/artifact/yetus-general-check/output/Dockerfile
GITHUB PR https://github.com/apache/hbase/pull/4675
Optional Tests dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
uname Linux 44bb01a13bd5 5.4.0-122-generic #138-Ubuntu SMP Wed Jun 22 15:00:31 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / d734acc00e
Default Java AdoptOpenJDK-1.8.0_282-b08
Max. process+thread count 64 (vs. ulimit of 30000)
modules C: hbase-server U: hbase-server
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4675/2/console
versions git=2.17.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase avatar Aug 02 '22 17:08 Apache-HBase

:confetti_ball: +1 overall

Vote Subsystem Runtime Comment
+0 :ok: reexec 0m 46s Docker mode activated.
-0 :warning: yetus 0m 3s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 :green_heart: mvninstall 2m 20s master passed
+1 :green_heart: compile 0m 34s master passed
+1 :green_heart: shadedjars 3m 59s branch has no errors when building our shaded downstream artifacts.
+1 :green_heart: javadoc 0m 22s master passed
_ Patch Compile Tests _
+1 :green_heart: mvninstall 2m 11s the patch passed
+1 :green_heart: compile 0m 35s the patch passed
+1 :green_heart: javac 0m 35s the patch passed
+1 :green_heart: shadedjars 4m 1s patch has no errors when building our shaded downstream artifacts.
+1 :green_heart: javadoc 0m 21s the patch passed
_ Other Tests _
+1 :green_heart: unit 202m 37s hbase-server in the patch passed.
219m 38s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4675/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR https://github.com/apache/hbase/pull/4675
Optional Tests javac javadoc unit shadedjars compile
uname Linux f0e617a00992 5.4.0-1081-aws #88~18.04.1-Ubuntu SMP Thu Jun 23 16:29:17 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / d734acc00e
Default Java AdoptOpenJDK-1.8.0_282-b08
Test Results https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4675/2/testReport/
Max. process+thread count 2644 (vs. ulimit of 30000)
modules C: hbase-server U: hbase-server
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4675/2/console
versions git=2.17.1 maven=3.6.3
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase avatar Aug 02 '22 20:08 Apache-HBase

:confetti_ball: +1 overall

Vote Subsystem Runtime Comment
+0 :ok: reexec 1m 3s Docker mode activated.
-0 :warning: yetus 0m 2s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 :green_heart: mvninstall 2m 29s master passed
+1 :green_heart: compile 0m 49s master passed
+1 :green_heart: shadedjars 3m 44s branch has no errors when building our shaded downstream artifacts.
+1 :green_heart: javadoc 0m 26s master passed
_ Patch Compile Tests _
+1 :green_heart: mvninstall 2m 38s the patch passed
+1 :green_heart: compile 0m 50s the patch passed
+1 :green_heart: javac 0m 50s the patch passed
+1 :green_heart: shadedjars 3m 44s patch has no errors when building our shaded downstream artifacts.
+1 :green_heart: javadoc 0m 26s the patch passed
_ Other Tests _
+1 :green_heart: unit 204m 28s hbase-server in the patch passed.
221m 41s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4675/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR https://github.com/apache/hbase/pull/4675
Optional Tests javac javadoc unit shadedjars compile
uname Linux 7a7fd5073ed1 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / d734acc00e
Default Java AdoptOpenJDK-11.0.10+9
Test Results https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4675/2/testReport/
Max. process+thread count 2672 (vs. ulimit of 30000)
modules C: hbase-server U: hbase-server
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4675/2/console
versions git=2.17.1 maven=3.6.3
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase avatar Aug 02 '22 20:08 Apache-HBase

:confetti_ball: +1 overall

Vote Subsystem Runtime Comment
+0 :ok: reexec 1m 40s Docker mode activated.
_ Prechecks _
+1 :green_heart: dupname 0m 0s No case conflicting files found.
+1 :green_heart: hbaseanti 0m 0s Patch does not have any anti-patterns.
+1 :green_heart: @author 0m 0s The patch does not contain any @author tags.
_ master Compile Tests _
+1 :green_heart: mvninstall 3m 18s master passed
+1 :green_heart: compile 3m 25s master passed
+1 :green_heart: checkstyle 0m 47s master passed
+1 :green_heart: spotless 1m 15s branch has no errors when running spotless:check.
+1 :green_heart: spotbugs 2m 18s master passed
_ Patch Compile Tests _
+1 :green_heart: mvninstall 3m 22s the patch passed
+1 :green_heart: compile 3m 19s the patch passed
+1 :green_heart: javac 3m 19s the patch passed
+1 :green_heart: checkstyle 0m 43s the patch passed
+1 :green_heart: whitespace 0m 0s The patch has no whitespace issues.
+1 :green_heart: hadoopcheck 15m 39s Patch does not cause any errors with Hadoop 3.1.2 3.2.2 3.3.1.
+1 :green_heart: spotless 0m 56s patch has no errors when running spotless:check.
+1 :green_heart: spotbugs 1m 48s the patch passed
_ Other Tests _
+1 :green_heart: asflicense 0m 8s The patch does not generate ASF License warnings.
45m 47s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4675/3/artifact/yetus-general-check/output/Dockerfile
GITHUB PR https://github.com/apache/hbase/pull/4675
Optional Tests dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
uname Linux 81e0073683d3 5.4.0-122-generic #138-Ubuntu SMP Wed Jun 22 15:00:31 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 54f2106f33
Default Java AdoptOpenJDK-1.8.0_282-b08
Max. process+thread count 72 (vs. ulimit of 30000)
modules C: hbase-server U: hbase-server
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4675/3/console
versions git=2.17.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase avatar Aug 08 '22 16:08 Apache-HBase

:confetti_ball: +1 overall

Vote Subsystem Runtime Comment
+0 :ok: reexec 0m 48s Docker mode activated.
-0 :warning: yetus 0m 2s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 :green_heart: mvninstall 2m 53s master passed
+1 :green_heart: compile 0m 45s master passed
+1 :green_heart: shadedjars 4m 30s branch has no errors when building our shaded downstream artifacts.
+1 :green_heart: javadoc 0m 31s master passed
_ Patch Compile Tests _
+1 :green_heart: mvninstall 2m 33s the patch passed
+1 :green_heart: compile 0m 43s the patch passed
+1 :green_heart: javac 0m 43s the patch passed
+1 :green_heart: shadedjars 4m 23s patch has no errors when building our shaded downstream artifacts.
+1 :green_heart: javadoc 0m 31s the patch passed
_ Other Tests _
+1 :green_heart: unit 200m 58s hbase-server in the patch passed.
221m 28s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4675/3/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR https://github.com/apache/hbase/pull/4675
Optional Tests javac javadoc unit shadedjars compile
uname Linux dd0c49080f0a 5.4.0-1081-aws #88~18.04.1-Ubuntu SMP Thu Jun 23 16:29:17 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 54f2106f33
Default Java AdoptOpenJDK-1.8.0_282-b08
Test Results https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4675/3/testReport/
Max. process+thread count 3904 (vs. ulimit of 30000)
modules C: hbase-server U: hbase-server
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4675/3/console
versions git=2.17.1 maven=3.6.3
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase avatar Aug 08 '22 19:08 Apache-HBase

:confetti_ball: +1 overall

Vote Subsystem Runtime Comment
+0 :ok: reexec 1m 0s Docker mode activated.
-0 :warning: yetus 0m 3s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 :green_heart: mvninstall 2m 36s master passed
+1 :green_heart: compile 0m 47s master passed
+1 :green_heart: shadedjars 3m 47s branch has no errors when building our shaded downstream artifacts.
+1 :green_heart: javadoc 0m 26s master passed
_ Patch Compile Tests _
+1 :green_heart: mvninstall 2m 38s the patch passed
+1 :green_heart: compile 0m 47s the patch passed
+1 :green_heart: javac 0m 47s the patch passed
+1 :green_heart: shadedjars 3m 46s patch has no errors when building our shaded downstream artifacts.
+1 :green_heart: javadoc 0m 27s the patch passed
_ Other Tests _
+1 :green_heart: unit 203m 55s hbase-server in the patch passed.
221m 50s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4675/3/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR https://github.com/apache/hbase/pull/4675
Optional Tests javac javadoc unit shadedjars compile
uname Linux ca492d459a6e 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 54f2106f33
Default Java AdoptOpenJDK-11.0.10+9
Test Results https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4675/3/testReport/
Max. process+thread count 2500 (vs. ulimit of 30000)
modules C: hbase-server U: hbase-server
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4675/3/console
versions git=2.17.1 maven=3.6.3
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase avatar Aug 08 '22 19:08 Apache-HBase

I think this is a useful feature. When implementing an in-house LSM tree based storage system, I used to make use of the compression rates for the already written blocks to predicate the compression rate of the next block, to determine whether we should finish a block.

I think the approach here is also OK, compressing once when we reach the default block size to predicate the compression rate.

Maybe we could introduce something like a plugin to predicate the compression rate? The default implementation just always returns 1, and we could introduce different algorithms to predicate the compression rate of the current block size.

What do you guys think? For me, the option names are a bit confusing... What does 'size.limit.compressed' mean?

Thanks.

Apache9 avatar Aug 09 '22 10:08 Apache9

:confetti_ball: +1 overall

Vote Subsystem Runtime Comment
+0 :ok: reexec 1m 35s Docker mode activated.
_ Prechecks _
+1 :green_heart: dupname 0m 0s No case conflicting files found.
+1 :green_heart: hbaseanti 0m 0s Patch does not have any anti-patterns.
+1 :green_heart: @author 0m 0s The patch does not contain any @author tags.
_ master Compile Tests _
+1 :green_heart: mvninstall 3m 20s master passed
+1 :green_heart: compile 3m 18s master passed
+1 :green_heart: checkstyle 0m 40s master passed
+1 :green_heart: spotless 1m 1s branch has no errors when running spotless:check.
+1 :green_heart: spotbugs 1m 50s master passed
_ Patch Compile Tests _
+1 :green_heart: mvninstall 3m 14s the patch passed
+1 :green_heart: compile 3m 22s the patch passed
+1 :green_heart: javac 3m 23s the patch passed
+1 :green_heart: checkstyle 0m 50s the patch passed
+1 :green_heart: whitespace 0m 0s The patch has no whitespace issues.
+1 :green_heart: hadoopcheck 16m 29s Patch does not cause any errors with Hadoop 3.1.2 3.2.2 3.3.1.
+1 :green_heart: spotless 0m 57s patch has no errors when running spotless:check.
+1 :green_heart: spotbugs 2m 1s the patch passed
_ Other Tests _
+1 :green_heart: asflicense 0m 9s The patch does not generate ASF License warnings.
45m 39s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4675/4/artifact/yetus-general-check/output/Dockerfile
GITHUB PR https://github.com/apache/hbase/pull/4675
Optional Tests dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
uname Linux 9e5b624931ae 5.4.0-122-generic #138-Ubuntu SMP Wed Jun 22 15:00:31 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 5919b30b6d
Default Java AdoptOpenJDK-1.8.0_282-b08
Max. process+thread count 64 (vs. ulimit of 30000)
modules C: hbase-server U: hbase-server
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4675/4/console
versions git=2.17.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase avatar Aug 09 '22 11:08 Apache-HBase

Maybe we could introduce something like a plugin to predicate the compression rate? The default implementation just always returns 1, and we could introduce different algorithms to predicate the compression rate of the current block size.

I thought about it originally, but then concluded this feature made more sense as an on/off (off by default) behaviour. Do you see other variations in between?

What do you guys think? For me, the option names are a bit confusing... What does 'size.limit.compressed' mean?

Thanks.

The properties were a bit confusing indeed. I tried to rename those now, let me know if those are more intuitive.

wchevreuil avatar Aug 09 '22 13:08 wchevreuil

Maybe we could introduce something like a plugin to predicate the compression rate? The default implementation just always returns 1, and we could introduce different algorithms to predicate the compression rate of the current block size.

I thought about it originally, but then concluded this feature made more sense as an on/off (off by default) behaviour. Do you see other variations in between? At least I've proposed another way to predicate the compression rate right? But I do not think there is a always win algorithm here, so a pluggable algorithm will be more appropriate here?

What do you guys think? For me, the option names are a bit confusing... What does 'size.limit.compressed' mean? Thanks.

The properties were a bit confusing indeed. I tried to rename those now, let me know if those are more intuitive.

Apache9 avatar Aug 09 '22 13:08 Apache9

:broken_heart: -1 overall

Vote Subsystem Runtime Comment
+0 :ok: reexec 0m 41s Docker mode activated.
-0 :warning: yetus 0m 3s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 :green_heart: mvninstall 2m 26s master passed
+1 :green_heart: compile 0m 35s master passed
+1 :green_heart: shadedjars 4m 1s branch has no errors when building our shaded downstream artifacts.
+1 :green_heart: javadoc 0m 24s master passed
_ Patch Compile Tests _
+1 :green_heart: mvninstall 2m 11s the patch passed
+1 :green_heart: compile 0m 34s the patch passed
+1 :green_heart: javac 0m 34s the patch passed
+1 :green_heart: shadedjars 3m 57s patch has no errors when building our shaded downstream artifacts.
+1 :green_heart: javadoc 0m 21s the patch passed
_ Other Tests _
-1 :x: unit 202m 38s hbase-server in the patch failed.
219m 36s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4675/4/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR https://github.com/apache/hbase/pull/4675
Optional Tests javac javadoc unit shadedjars compile
uname Linux 15e1602e8994 5.4.0-1081-aws #88~18.04.1-Ubuntu SMP Thu Jun 23 16:29:17 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 5919b30b6d
Default Java AdoptOpenJDK-1.8.0_282-b08
unit https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4675/4/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt
Test Results https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4675/4/testReport/
Max. process+thread count 3669 (vs. ulimit of 30000)
modules C: hbase-server U: hbase-server
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4675/4/console
versions git=2.17.1 maven=3.6.3
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase avatar Aug 09 '22 14:08 Apache-HBase

:confetti_ball: +1 overall

Vote Subsystem Runtime Comment
+0 :ok: reexec 1m 6s Docker mode activated.
-0 :warning: yetus 0m 3s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 :green_heart: mvninstall 2m 35s master passed
+1 :green_heart: compile 0m 47s master passed
+1 :green_heart: shadedjars 3m 51s branch has no errors when building our shaded downstream artifacts.
+1 :green_heart: javadoc 0m 26s master passed
_ Patch Compile Tests _
+1 :green_heart: mvninstall 2m 37s the patch passed
+1 :green_heart: compile 0m 47s the patch passed
+1 :green_heart: javac 0m 47s the patch passed
+1 :green_heart: shadedjars 3m 47s patch has no errors when building our shaded downstream artifacts.
+1 :green_heart: javadoc 0m 27s the patch passed
_ Other Tests _
+1 :green_heart: unit 203m 35s hbase-server in the patch passed.
221m 16s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4675/4/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR https://github.com/apache/hbase/pull/4675
Optional Tests javac javadoc unit shadedjars compile
uname Linux 035c2843d744 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 5919b30b6d
Default Java AdoptOpenJDK-11.0.10+9
Test Results https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4675/4/testReport/
Max. process+thread count 2667 (vs. ulimit of 30000)
modules C: hbase-server U: hbase-server
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4675/4/console
versions git=2.17.1 maven=3.6.3
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase avatar Aug 09 '22 14:08 Apache-HBase

:confetti_ball: +1 overall

Vote Subsystem Runtime Comment
+0 :ok: reexec 1m 27s Docker mode activated.
_ Prechecks _
+1 :green_heart: dupname 0m 1s No case conflicting files found.
+1 :green_heart: hbaseanti 0m 0s Patch does not have any anti-patterns.
+1 :green_heart: @author 0m 0s The patch does not contain any @author tags.
_ master Compile Tests _
+1 :green_heart: mvninstall 3m 19s master passed
+1 :green_heart: compile 3m 3s master passed
+1 :green_heart: checkstyle 0m 36s master passed
+1 :green_heart: spotless 1m 3s branch has no errors when running spotless:check.
+1 :green_heart: spotbugs 2m 1s master passed
_ Patch Compile Tests _
+1 :green_heart: mvninstall 3m 16s the patch passed
+1 :green_heart: compile 3m 29s the patch passed
+1 :green_heart: javac 3m 29s the patch passed
+1 :green_heart: checkstyle 0m 41s the patch passed
+1 :green_heart: whitespace 0m 0s The patch has no whitespace issues.
+1 :green_heart: hadoopcheck 15m 49s Patch does not cause any errors with Hadoop 3.1.2 3.2.2 3.3.1.
+1 :green_heart: spotless 1m 13s patch has no errors when running spotless:check.
+1 :green_heart: spotbugs 2m 3s the patch passed
_ Other Tests _
+1 :green_heart: asflicense 0m 13s The patch does not generate ASF License warnings.
44m 57s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4675/5/artifact/yetus-general-check/output/Dockerfile
GITHUB PR https://github.com/apache/hbase/pull/4675
Optional Tests dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
uname Linux 797f7c91315e 5.4.0-122-generic #138-Ubuntu SMP Wed Jun 22 15:00:31 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 5919b30b6d
Default Java AdoptOpenJDK-1.8.0_282-b08
Max. process+thread count 70 (vs. ulimit of 30000)
modules C: hbase-server U: hbase-server
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4675/5/console
versions git=2.17.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase avatar Aug 09 '22 14:08 Apache-HBase

:confetti_ball: +1 overall

Vote Subsystem Runtime Comment
+0 :ok: reexec 4m 2s Docker mode activated.
-0 :warning: yetus 0m 3s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 :green_heart: mvninstall 2m 6s master passed
+1 :green_heart: compile 0m 34s master passed
+1 :green_heart: shadedjars 4m 0s branch has no errors when building our shaded downstream artifacts.
+1 :green_heart: javadoc 0m 21s master passed
_ Patch Compile Tests _
+1 :green_heart: mvninstall 2m 7s the patch passed
+1 :green_heart: compile 0m 35s the patch passed
+1 :green_heart: javac 0m 35s the patch passed
+1 :green_heart: shadedjars 4m 0s patch has no errors when building our shaded downstream artifacts.
+1 :green_heart: javadoc 0m 22s the patch passed
_ Other Tests _
+1 :green_heart: unit 200m 14s hbase-server in the patch passed.
220m 17s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4675/5/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR https://github.com/apache/hbase/pull/4675
Optional Tests javac javadoc unit shadedjars compile
uname Linux d6b31a11b2b6 5.4.0-1081-aws #88~18.04.1-Ubuntu SMP Thu Jun 23 16:29:17 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 5919b30b6d
Default Java AdoptOpenJDK-1.8.0_282-b08
Test Results https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4675/5/testReport/
Max. process+thread count 3029 (vs. ulimit of 30000)
modules C: hbase-server U: hbase-server
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4675/5/console
versions git=2.17.1 maven=3.6.3
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase avatar Aug 09 '22 17:08 Apache-HBase

:confetti_ball: +1 overall

Vote Subsystem Runtime Comment
+0 :ok: reexec 6m 33s Docker mode activated.
-0 :warning: yetus 0m 3s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 :green_heart: mvninstall 2m 37s master passed
+1 :green_heart: compile 0m 47s master passed
+1 :green_heart: shadedjars 3m 44s branch has no errors when building our shaded downstream artifacts.
+1 :green_heart: javadoc 0m 26s master passed
_ Patch Compile Tests _
+1 :green_heart: mvninstall 2m 37s the patch passed
+1 :green_heart: compile 0m 47s the patch passed
+1 :green_heart: javac 0m 47s the patch passed
+1 :green_heart: shadedjars 3m 48s patch has no errors when building our shaded downstream artifacts.
+1 :green_heart: javadoc 0m 27s the patch passed
_ Other Tests _
+1 :green_heart: unit 203m 21s hbase-server in the patch passed.
226m 27s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4675/5/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR https://github.com/apache/hbase/pull/4675
Optional Tests javac javadoc unit shadedjars compile
uname Linux fe0213d6333e 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 5919b30b6d
Default Java AdoptOpenJDK-11.0.10+9
Test Results https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4675/5/testReport/
Max. process+thread count 2882 (vs. ulimit of 30000)
modules C: hbase-server U: hbase-server
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4675/5/console
versions git=2.17.1 maven=3.6.3
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase avatar Aug 09 '22 17:08 Apache-HBase

At least I've proposed another way to predicate the compression rate right? But I do not think there is a always win algorithm here, so a pluggable algorithm will be more appropriate here?

Sorry, I think I misunderstood it, previously. What if we leave this current PR as is, for an initial approach on using compressed sizes to determine blocks, then work on the pluggable solution on a separate ticket?

wchevreuil avatar Aug 09 '22 20:08 wchevreuil

At least I've proposed another way to predicate the compression rate right? But I do not think there is a always win algorithm here, so a pluggable algorithm will be more appropriate here?

Sorry, I think I misunderstood it, previously. What if we leave this current PR as is, for an initial approach on using compressed sizes to determine blocks, then work on the pluggable solution on a separate ticket?

If we expose the configs out then it will be a pain to switch to a pluggable solution because we can not simply remove these configs without a deprecation cycle...

I still prefer we make a pluggable solution first, and then the first approach could be the algorithm in this PR.

Apache9 avatar Aug 11 '22 02:08 Apache9

If we expose the configs out then it will be a pain to switch to a pluggable solution because we can not simply remove these configs without a deprecation cycle...

I still prefer we make a pluggable solution first, and then the first approach could be the algorithm in this PR.

Ok, had just pushed a new commit where the algorithm for defining the compression rate and adjust the block size limit is pluggable.

wchevreuil avatar Aug 11 '22 16:08 wchevreuil

:broken_heart: -1 overall

Vote Subsystem Runtime Comment
+0 :ok: reexec 1m 31s Docker mode activated.
_ Prechecks _
+1 :green_heart: dupname 0m 0s No case conflicting files found.
+1 :green_heart: hbaseanti 0m 0s Patch does not have any anti-patterns.
+1 :green_heart: @author 0m 0s The patch does not contain any @author tags.
_ master Compile Tests _
+1 :green_heart: mvninstall 3m 9s master passed
+1 :green_heart: compile 2m 50s master passed
+1 :green_heart: checkstyle 0m 32s master passed
+1 :green_heart: spotless 0m 52s branch has no errors when running spotless:check.
+1 :green_heart: spotbugs 1m 38s master passed
-0 :warning: patch 1m 44s Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
_ Patch Compile Tests _
+1 :green_heart: mvninstall 2m 59s the patch passed
+1 :green_heart: compile 2m 58s the patch passed
-0 :warning: javac 2m 58s hbase-server generated 1 new + 192 unchanged - 1 fixed = 193 total (was 193)
-0 :warning: checkstyle 0m 38s hbase-server: The patch generated 9 new + 1 unchanged - 0 fixed = 10 total (was 1)
-0 :warning: whitespace 0m 0s The patch has 3 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
+1 :green_heart: hadoopcheck 13m 4s Patch does not cause any errors with Hadoop 3.1.2 3.2.2 3.3.1.
-1 :x: spotless 0m 41s patch has 69 errors when running spotless:check, run spotless:apply to fix.
+1 :green_heart: spotbugs 1m 36s the patch passed
_ Other Tests _
+1 :green_heart: asflicense 0m 7s The patch does not generate ASF License warnings.
38m 7s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4675/7/artifact/yetus-general-check/output/Dockerfile
GITHUB PR https://github.com/apache/hbase/pull/4675
Optional Tests dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
uname Linux 4a5d9dce3542 5.4.0-122-generic #138-Ubuntu SMP Wed Jun 22 15:00:31 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 39b496e191
Default Java AdoptOpenJDK-1.8.0_282-b08
javac https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4675/7/artifact/yetus-general-check/output/diff-compile-javac-hbase-server.txt
checkstyle https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4675/7/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt
whitespace https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4675/7/artifact/yetus-general-check/output/whitespace-eol.txt
spotless https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4675/7/artifact/yetus-general-check/output/patch-spotless.txt
Max. process+thread count 64 (vs. ulimit of 30000)
modules C: hbase-server U: hbase-server
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4675/7/console
versions git=2.17.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase avatar Aug 11 '22 21:08 Apache-HBase

:broken_heart: -1 overall

Vote Subsystem Runtime Comment
+0 :ok: reexec 0m 43s Docker mode activated.
-0 :warning: yetus 0m 3s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 :green_heart: mvninstall 2m 17s master passed
+1 :green_heart: compile 0m 35s master passed
+1 :green_heart: shadedjars 4m 0s branch has no errors when building our shaded downstream artifacts.
+1 :green_heart: javadoc 0m 21s master passed
-0 :warning: patch 4m 28s Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
_ Patch Compile Tests _
+1 :green_heart: mvninstall 2m 10s the patch passed
+1 :green_heart: compile 0m 33s the patch passed
+1 :green_heart: javac 0m 33s the patch passed
+1 :green_heart: shadedjars 3m 59s patch has no errors when building our shaded downstream artifacts.
+1 :green_heart: javadoc 0m 21s the patch passed
_ Other Tests _
-1 :x: unit 199m 33s hbase-server in the patch failed.
216m 58s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4675/7/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR https://github.com/apache/hbase/pull/4675
Optional Tests javac javadoc unit shadedjars compile
uname Linux c53ce3e6bdd2 5.4.0-1081-aws #88~18.04.1-Ubuntu SMP Thu Jun 23 16:29:17 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 39b496e191
Default Java AdoptOpenJDK-1.8.0_282-b08
unit https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4675/7/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt
Test Results https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4675/7/testReport/
Max. process+thread count 2628 (vs. ulimit of 30000)
modules C: hbase-server U: hbase-server
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4675/7/console
versions git=2.17.1 maven=3.6.3
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase avatar Aug 12 '22 00:08 Apache-HBase

:broken_heart: -1 overall

Vote Subsystem Runtime Comment
+0 :ok: reexec 1m 12s Docker mode activated.
-0 :warning: yetus 0m 3s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 :green_heart: mvninstall 2m 57s master passed
+1 :green_heart: compile 0m 48s master passed
+1 :green_heart: shadedjars 3m 47s branch has no errors when building our shaded downstream artifacts.
+1 :green_heart: javadoc 0m 29s master passed
-0 :warning: patch 4m 25s Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
_ Patch Compile Tests _
+1 :green_heart: mvninstall 2m 35s the patch passed
+1 :green_heart: compile 0m 46s the patch passed
+1 :green_heart: javac 0m 46s the patch passed
+1 :green_heart: shadedjars 3m 46s patch has no errors when building our shaded downstream artifacts.
+1 :green_heart: javadoc 0m 26s the patch passed
_ Other Tests _
-1 :x: unit 202m 37s hbase-server in the patch failed.
220m 41s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4675/7/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR https://github.com/apache/hbase/pull/4675
Optional Tests javac javadoc unit shadedjars compile
uname Linux a695044eedf7 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 39b496e191
Default Java AdoptOpenJDK-11.0.10+9
unit https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4675/7/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt
Test Results https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4675/7/testReport/
Max. process+thread count 2566 (vs. ulimit of 30000)
modules C: hbase-server U: hbase-server
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4675/7/console
versions git=2.17.1 maven=3.6.3
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase avatar Aug 12 '22 00:08 Apache-HBase

:broken_heart: -1 overall

Vote Subsystem Runtime Comment
+0 :ok: reexec 0m 42s Docker mode activated.
-0 :warning: yetus 0m 2s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 :green_heart: mvninstall 2m 12s master passed
+1 :green_heart: compile 0m 35s master passed
+1 :green_heart: shadedjars 3m 59s branch has no errors when building our shaded downstream artifacts.
+1 :green_heart: javadoc 0m 21s master passed
-0 :warning: patch 4m 26s Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
_ Patch Compile Tests _
+1 :green_heart: mvninstall 2m 11s the patch passed
+1 :green_heart: compile 0m 34s the patch passed
+1 :green_heart: javac 0m 34s the patch passed
+1 :green_heart: shadedjars 4m 0s patch has no errors when building our shaded downstream artifacts.
+1 :green_heart: javadoc 0m 21s the patch passed
_ Other Tests _
-1 :x: unit 8m 8s hbase-server in the patch failed.
23m 53s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4675/8/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR https://github.com/apache/hbase/pull/4675
Optional Tests javac javadoc unit shadedjars compile
uname Linux a369d6f85cc9 5.4.0-1081-aws #88~18.04.1-Ubuntu SMP Thu Jun 23 16:29:17 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 73759be396
Default Java AdoptOpenJDK-1.8.0_282-b08
unit https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4675/8/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt
Test Results https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4675/8/testReport/
Max. process+thread count 562 (vs. ulimit of 30000)
modules C: hbase-server U: hbase-server
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4675/8/console
versions git=2.17.1 maven=3.6.3
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase avatar Aug 13 '22 23:08 Apache-HBase

:broken_heart: -1 overall

Vote Subsystem Runtime Comment
+0 :ok: reexec 1m 2s Docker mode activated.
-0 :warning: yetus 0m 2s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 :green_heart: mvninstall 2m 41s master passed
+1 :green_heart: compile 0m 48s master passed
+1 :green_heart: shadedjars 3m 47s branch has no errors when building our shaded downstream artifacts.
+1 :green_heart: javadoc 0m 28s master passed
-0 :warning: patch 4m 24s Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
_ Patch Compile Tests _
+1 :green_heart: mvninstall 2m 32s the patch passed
+1 :green_heart: compile 0m 47s the patch passed
+1 :green_heart: javac 0m 47s the patch passed
+1 :green_heart: shadedjars 3m 45s patch has no errors when building our shaded downstream artifacts.
+1 :green_heart: javadoc 0m 25s the patch passed
_ Other Tests _
-1 :x: unit 10m 40s hbase-server in the patch failed.
27m 53s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4675/8/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR https://github.com/apache/hbase/pull/4675
Optional Tests javac javadoc unit shadedjars compile
uname Linux 9f6f29948798 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 73759be396
Default Java AdoptOpenJDK-11.0.10+9
unit https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4675/8/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt
Test Results https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4675/8/testReport/
Max. process+thread count 1019 (vs. ulimit of 30000)
modules C: hbase-server U: hbase-server
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4675/8/console
versions git=2.17.1 maven=3.6.3
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase avatar Aug 13 '22 23:08 Apache-HBase

:broken_heart: -1 overall

Vote Subsystem Runtime Comment
+0 :ok: reexec 1m 6s Docker mode activated.
_ Prechecks _
+1 :green_heart: dupname 0m 0s No case conflicting files found.
+1 :green_heart: hbaseanti 0m 0s Patch does not have any anti-patterns.
+1 :green_heart: @author 0m 0s The patch does not contain any @author tags.
_ master Compile Tests _
+1 :green_heart: mvninstall 2m 15s master passed
+1 :green_heart: compile 2m 12s master passed
+1 :green_heart: checkstyle 0m 31s master passed
+1 :green_heart: spotless 0m 43s branch has no errors when running spotless:check.
+1 :green_heart: spotbugs 1m 16s master passed
-0 :warning: patch 1m 23s Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
_ Patch Compile Tests _
+1 :green_heart: mvninstall 2m 18s the patch passed
+1 :green_heart: compile 2m 13s the patch passed
-0 :warning: javac 2m 13s hbase-server generated 1 new + 192 unchanged - 1 fixed = 193 total (was 193)
-0 :warning: checkstyle 0m 31s hbase-server: The patch generated 10 new + 1 unchanged - 0 fixed = 11 total (was 1)
-0 :warning: whitespace 0m 0s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
+1 :green_heart: hadoopcheck 11m 26s Patch does not cause any errors with Hadoop 3.1.2 3.2.2 3.3.1.
-1 :x: spotless 0m 37s patch has 69 errors when running spotless:check, run spotless:apply to fix.
+1 :green_heart: spotbugs 1m 23s the patch passed
_ Other Tests _
+1 :green_heart: asflicense 0m 9s The patch does not generate ASF License warnings.
31m 27s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4675/8/artifact/yetus-general-check/output/Dockerfile
GITHUB PR https://github.com/apache/hbase/pull/4675
Optional Tests dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
uname Linux 88a1ccee7a70 5.4.0-122-generic #138-Ubuntu SMP Wed Jun 22 15:00:31 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 73759be396
Default Java AdoptOpenJDK-1.8.0_282-b08
javac https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4675/8/artifact/yetus-general-check/output/diff-compile-javac-hbase-server.txt
checkstyle https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4675/8/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt
whitespace https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4675/8/artifact/yetus-general-check/output/whitespace-eol.txt
spotless https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4675/8/artifact/yetus-general-check/output/patch-spotless.txt
Max. process+thread count 60 (vs. ulimit of 30000)
modules C: hbase-server U: hbase-server
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4675/8/console
versions git=2.17.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase avatar Aug 13 '22 23:08 Apache-HBase

:broken_heart: -1 overall

Vote Subsystem Runtime Comment
+0 :ok: reexec 0m 50s Docker mode activated.
-0 :warning: yetus 0m 2s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 :green_heart: mvninstall 2m 23s master passed
+1 :green_heart: compile 0m 34s master passed
+1 :green_heart: shadedjars 4m 2s branch has no errors when building our shaded downstream artifacts.
+1 :green_heart: javadoc 0m 24s master passed
-0 :warning: patch 4m 32s Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
_ Patch Compile Tests _
-1 :x: mvninstall 1m 9s root in the patch failed.
-1 :x: compile 0m 34s hbase-server in the patch failed.
-0 :warning: javac 0m 34s hbase-server in the patch failed.
-1 :x: shadedjars 3m 6s patch has 14 errors when building our shaded downstream artifacts.
+1 :green_heart: javadoc 0m 22s the patch passed
_ Other Tests _
-1 :x: unit 0m 35s hbase-server in the patch failed.
14m 42s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4675/9/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR https://github.com/apache/hbase/pull/4675
Optional Tests javac javadoc unit shadedjars compile
uname Linux 1621206a977c 5.4.0-1081-aws #88~18.04.1-Ubuntu SMP Thu Jun 23 16:29:17 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 9215066e2b
Default Java AdoptOpenJDK-1.8.0_282-b08
mvninstall https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4675/9/artifact/yetus-jdk8-hadoop3-check/output/patch-mvninstall-root.txt
compile https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4675/9/artifact/yetus-jdk8-hadoop3-check/output/patch-compile-hbase-server.txt
javac https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4675/9/artifact/yetus-jdk8-hadoop3-check/output/patch-compile-hbase-server.txt
shadedjars https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4675/9/artifact/yetus-jdk8-hadoop3-check/output/patch-shadedjars.txt
unit https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4675/9/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt
Test Results https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4675/9/testReport/
Max. process+thread count 64 (vs. ulimit of 30000)
modules C: hbase-server U: hbase-server
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4675/9/console
versions git=2.17.1 maven=3.6.3
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase avatar Aug 15 '22 14:08 Apache-HBase

:broken_heart: -1 overall

Vote Subsystem Runtime Comment
+0 :ok: reexec 1m 12s Docker mode activated.
-0 :warning: yetus 0m 2s Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 :green_heart: mvninstall 2m 50s master passed
+1 :green_heart: compile 0m 47s master passed
+1 :green_heart: shadedjars 3m 51s branch has no errors when building our shaded downstream artifacts.
+1 :green_heart: javadoc 0m 27s master passed
-0 :warning: patch 4m 26s Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
_ Patch Compile Tests _
-1 :x: mvninstall 1m 35s root in the patch failed.
-1 :x: compile 0m 46s hbase-server in the patch failed.
-0 :warning: javac 0m 46s hbase-server in the patch failed.
-1 :x: shadedjars 3m 7s patch has 14 errors when building our shaded downstream artifacts.
+1 :green_heart: javadoc 0m 27s the patch passed
_ Other Tests _
-1 :x: unit 0m 49s hbase-server in the patch failed.
16m 45s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4675/9/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR https://github.com/apache/hbase/pull/4675
Optional Tests javac javadoc unit shadedjars compile
uname Linux ad054304246c 5.4.0-122-generic #138-Ubuntu SMP Wed Jun 22 15:00:31 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 9215066e2b
Default Java AdoptOpenJDK-11.0.10+9
mvninstall https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4675/9/artifact/yetus-jdk11-hadoop3-check/output/patch-mvninstall-root.txt
compile https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4675/9/artifact/yetus-jdk11-hadoop3-check/output/patch-compile-hbase-server.txt
javac https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4675/9/artifact/yetus-jdk11-hadoop3-check/output/patch-compile-hbase-server.txt
shadedjars https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4675/9/artifact/yetus-jdk11-hadoop3-check/output/patch-shadedjars.txt
unit https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4675/9/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt
Test Results https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4675/9/testReport/
Max. process+thread count 78 (vs. ulimit of 30000)
modules C: hbase-server U: hbase-server
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4675/9/console
versions git=2.17.1 maven=3.6.3
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase avatar Aug 15 '22 14:08 Apache-HBase

:broken_heart: -1 overall

Vote Subsystem Runtime Comment
+0 :ok: reexec 0m 59s Docker mode activated.
_ Prechecks _
+1 :green_heart: dupname 0m 0s No case conflicting files found.
+1 :green_heart: hbaseanti 0m 0s Patch does not have any anti-patterns.
+1 :green_heart: @author 0m 0s The patch does not contain any @author tags.
_ master Compile Tests _
+1 :green_heart: mvninstall 2m 14s master passed
+1 :green_heart: compile 2m 12s master passed
+1 :green_heart: checkstyle 0m 30s master passed
+1 :green_heart: spotless 0m 39s branch has no errors when running spotless:check.
+1 :green_heart: spotbugs 1m 15s master passed
-0 :warning: patch 1m 21s Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
_ Patch Compile Tests _
-1 :x: mvninstall 1m 14s root in the patch failed.
-1 :x: compile 1m 15s hbase-server in the patch failed.
-0 :warning: javac 1m 15s hbase-server in the patch failed.
+1 :green_heart: checkstyle 0m 29s the patch passed
+1 :green_heart: whitespace 0m 0s The patch has no whitespace issues.
-1 :x: hadoopcheck 1m 21s The patch causes 14 errors with Hadoop v3.1.2.
-1 :x: hadoopcheck 2m 43s The patch causes 14 errors with Hadoop v3.2.2.
-1 :x: hadoopcheck 4m 6s The patch causes 14 errors with Hadoop v3.3.1.
+1 :green_heart: spotless 0m 36s patch has no errors when running spotless:check.
-1 :x: spotbugs 0m 27s hbase-server in the patch failed.
_ Other Tests _
+1 :green_heart: asflicense 0m 8s The patch does not generate ASF License warnings.
17m 8s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4675/9/artifact/yetus-general-check/output/Dockerfile
GITHUB PR https://github.com/apache/hbase/pull/4675
Optional Tests dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
uname Linux 9ae751a1263a 5.4.0-122-generic #138-Ubuntu SMP Wed Jun 22 15:00:31 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/hbase-personality.sh
git revision master / 9215066e2b
Default Java AdoptOpenJDK-1.8.0_282-b08
mvninstall https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4675/9/artifact/yetus-general-check/output/patch-mvninstall-root.txt
compile https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4675/9/artifact/yetus-general-check/output/patch-compile-hbase-server.txt
javac https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4675/9/artifact/yetus-general-check/output/patch-compile-hbase-server.txt
hadoopcheck https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4675/9/artifact/yetus-general-check/output/patch-javac-3.1.2.txt
hadoopcheck https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4675/9/artifact/yetus-general-check/output/patch-javac-3.2.2.txt
hadoopcheck https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4675/9/artifact/yetus-general-check/output/patch-javac-3.3.1.txt
spotbugs https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4675/9/artifact/yetus-general-check/output/patch-spotbugs-hbase-server.txt
Max. process+thread count 64 (vs. ulimit of 30000)
modules C: hbase-server U: hbase-server
Console output https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4675/9/console
versions git=2.17.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase avatar Aug 15 '22 14:08 Apache-HBase