orc icon indicating copy to clipboard operation
orc copied to clipboard
verified publisher icon apache

[JAVA] mvn package fails if test compiling was skipped

Open zjx20 opened this issue 3 years ago • 6 comments

I would like to run mvn -Dmaven.test.skip=true clean package, but maven-dependency-plugin complains Unused declared dependencies for some libraries used by the test code, which breaks the compilation. Please check the attached logs.

I'm not a java expert, but I'm guessing the cause of the problem is the misuse of analyze-only in the package phase. According to the documentation, the analyze-only goal is meant to be used during the test-compile phase. In our case, the test class was not compiled, so the dependency analyzer treated some libraries as unused. I don't know the right way to fix it though.

The setting of maven-dependency-plugin: https://github.com/apache/orc/blob/8cf1047f9ace3799df12f24d2a5096b17a9a6ed0/java/pom.xml#L373-L388

Logs:

$ cd orc/java

$ mvn -Dmaven.test.skip=true clean package
[INFO] Scanning for projects...
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Build Order:
[INFO]
[INFO] Apache ORC                                                         [pom]
[INFO] ORC Shims                                                          [jar]
[INFO] ORC Core                                                           [jar]
[INFO] ORC MapReduce                                                      [jar]
[INFO] ORC Tools                                                          [jar]
[INFO] ORC Examples                                                       [jar]
[INFO]
[INFO] -------------------------< org.apache.orc:orc >-------------------------
[INFO] Building Apache ORC 1.9.0-SNAPSHOT                                 [1/6]
[INFO] --------------------------------[ pom ]---------------------------------
[INFO]
[INFO] --- maven-clean-plugin:3.2.0:clean (default-clean) @ orc ---
[INFO] Deleting /Users/x/Documents/playground/orc/java/target
[INFO]
[INFO] --- maven-enforcer-plugin:3.1.0:enforce (enforce-maven-version) @ orc ---
[INFO]
[INFO] --- maven-enforcer-plugin:3.1.0:enforce (enforce-java-version) @ orc ---
[INFO]
[INFO] --- maven-enforcer-plugin:3.1.0:enforce (enforce-maven) @ orc ---
[INFO]
[INFO] --- maven-remote-resources-plugin:1.7.0:process (process-resource-bundles) @ orc ---
[INFO] Preparing remote bundle org.apache:apache-jar-resource-bundle:1.4
[INFO] Copying 3 resources from 1 bundle.
[INFO]
[INFO] --- maven-antrun-plugin:3.1.0:run (setup-test-dirs) @ orc ---
[INFO] Executing tasks
[INFO]     [mkdir] Created dir: /Users/x/Documents/playground/orc/java/target/testing-tmp
[INFO] Executed tasks
[INFO]
[INFO] --- maven-site-plugin:3.12.0:attach-descriptor (attach-descriptor) @ orc ---
[INFO] No site descriptor found: nothing to attach.
[INFO]
[INFO] --- maven-source-plugin:3.2.1:jar-no-fork (create-source-jar) @ orc ---
[INFO]
[INFO] --- maven-source-plugin:3.2.1:test-jar-no-fork (create-source-jar) @ orc ---
[INFO]
[INFO] --- reproducible-build-maven-plugin:0.15:strip-jar (default) @ orc ---
[INFO]
[INFO] ----------------------< org.apache.orc:orc-shims >----------------------
[INFO] Building ORC Shims 1.9.0-SNAPSHOT                                  [2/6]
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- maven-clean-plugin:3.2.0:clean (default-clean) @ orc-shims ---
[INFO] Deleting /Users/x/Documents/playground/orc/java/shims/target
[INFO]
[INFO] --- maven-enforcer-plugin:3.1.0:enforce (enforce-maven-version) @ orc-shims ---
[INFO]
[INFO] --- maven-enforcer-plugin:3.1.0:enforce (enforce-java-version) @ orc-shims ---
[INFO]
[INFO] --- maven-enforcer-plugin:3.1.0:enforce (enforce-maven) @ orc-shims ---
[INFO]
[INFO] --- build-helper-maven-plugin:3.3.0:add-source (add-source) @ orc-shims ---
[INFO] Source directory: /Users/x/Documents/playground/orc/java/shims/target/generated-sources added.
[INFO]
[INFO] --- maven-remote-resources-plugin:1.7.0:process (process-resource-bundles) @ orc-shims ---
[INFO] Preparing remote bundle org.apache:apache-jar-resource-bundle:1.4
[INFO] Copying 3 resources from 1 bundle.
[INFO]
[INFO] --- maven-resources-plugin:3.2.0:resources (default-resources) @ orc-shims ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Using 'UTF-8' encoding to copy filtered properties files.
[INFO] skip non existing resourceDirectory /Users/x/Documents/playground/orc/java/shims/src/main/resources
[INFO] Copying 3 resources
[INFO]
[INFO] --- maven-compiler-plugin:3.10.1:compile (default-compile) @ orc-shims ---
[INFO] Compiling 13 source files to /Users/x/Documents/playground/orc/java/shims/target/classes
[INFO]
[INFO] --- maven-resources-plugin:3.2.0:testResources (default-testResources) @ orc-shims ---
[INFO] Not copying test resources
[INFO]
[INFO] --- maven-antrun-plugin:3.1.0:run (setup-test-dirs) @ orc-shims ---
[INFO] Executing tasks
[INFO]     [mkdir] Created dir: /Users/x/Documents/playground/orc/java/shims/target/testing-tmp
[INFO] Executed tasks
[INFO]
[INFO] --- maven-compiler-plugin:3.10.1:testCompile (default-testCompile) @ orc-shims ---
[INFO] Not compiling test sources
[INFO]
[INFO] --- maven-surefire-plugin:3.0.0-M5:test (default-test) @ orc-shims ---
[INFO] Tests are skipped.
[INFO]
[INFO] --- maven-jar-plugin:3.3.0:jar (default-jar) @ orc-shims ---
[INFO] Building jar: /Users/x/Documents/playground/orc/java/shims/target/orc-shims-1.9.0-SNAPSHOT.jar
[INFO]
[INFO] --- maven-site-plugin:3.12.0:attach-descriptor (attach-descriptor) @ orc-shims ---
[INFO] Skipping because packaging 'jar' is not pom.
[INFO]
[INFO] --- maven-source-plugin:3.2.1:jar-no-fork (create-source-jar) @ orc-shims ---
[INFO] Building jar: /Users/x/Documents/playground/orc/java/shims/target/orc-shims-1.9.0-SNAPSHOT-sources.jar
[INFO]
[INFO] --- maven-source-plugin:3.2.1:test-jar-no-fork (create-source-jar) @ orc-shims ---
[INFO] Building jar: /Users/x/Documents/playground/orc/java/shims/target/orc-shims-1.9.0-SNAPSHOT-test-sources.jar
[INFO]
[INFO] --- reproducible-build-maven-plugin:0.15:strip-jar (default) @ orc-shims ---
[INFO] Stripping /Users/x/Documents/playground/orc/java/shims/target/orc-shims-1.9.0-SNAPSHOT-test-sources.jar
[INFO] Stripping /Users/x/Documents/playground/orc/java/shims/target/orc-shims-1.9.0-SNAPSHOT-sources.jar
[INFO] Stripping /Users/x/Documents/playground/orc/java/shims/target/orc-shims-1.9.0-SNAPSHOT.jar
[INFO]
[INFO] --- maven-dependency-plugin:3.1.2:analyze-only (default) @ orc-shims ---
[WARNING] Unused declared dependencies found:
[WARNING]    org.junit.jupiter:junit-jupiter-api:jar:5.9.0:test
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Apache ORC 1.9.0-SNAPSHOT:
[INFO]
[INFO] Apache ORC ......................................... SUCCESS [  4.108 s]
[INFO] ORC Shims .......................................... FAILURE [  4.817 s]
[INFO] ORC Core ........................................... SKIPPED
[INFO] ORC MapReduce ...................................... SKIPPED
[INFO] ORC Tools .......................................... SKIPPED
[INFO] ORC Examples ....................................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  9.189 s
[INFO] Finished at: 2022-10-13T15:07:06+08:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-dependency-plugin:3.1.2:analyze-only (default) on project orc-shims: Dependency problems found -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <args> -rf :orc-shims

zjx20 avatar Oct 13 '22 07:10 zjx20

Thank you for reporting, @zjx20 .

dongjoon-hyun avatar Oct 14 '22 00:10 dongjoon-hyun

Why don't use the following? It doesn't cost you much.

mvn -DskipTests clean package

dongjoon-hyun avatar Oct 14 '22 04:10 dongjoon-hyun

Thanks for the quick reply. You're right, that's a simple workaround indeed. But the previous one is also a common practice, people may get confused as to why this doesn't work. That's my opinion.

zjx20 avatar Oct 14 '22 06:10 zjx20

Ya, I agree that could be considered in that way.

dongjoon-hyun avatar Oct 14 '22 16:10 dongjoon-hyun

@zjx20 Wanted to understand the use case. Is your requirement that you want to build and not generate any test artifacts (also includes skipping the tests)?

The steps given by @dongjoon-hyun skips the tests but generates the test artifacts.

pavibhai avatar Oct 14 '22 16:10 pavibhai

@zjx20 Wanted to understand the use case. Is your requirement that you want to build and not generate any test artifacts (also includes skipping the tests)?

The steps given by @dongjoon-hyun skips the tests but generates the test artifacts.

It's definitely a solution for me, but my point is:

  • People would encounter the same problem as I did and take some time to figure out what the error is and how to solve it. They would finally find the solution from this post, but it is just not feeling good.
  • The usage of analyze-only is likely incorrect.

Besides that, I think it should be fine to put analyze-only in the test phase. I'm assuming that tests are run by some sort of CI/CD pipeline or run manually before a release. In this case, dependency problems can still be exposed.

zjx20 avatar Oct 15 '22 02:10 zjx20

Did you try that? Could you make a PR for your proposal, @zjx20 ?

I think it should be fine to put analyze-only in the test phase.

dongjoon-hyun avatar Oct 16 '22 22:10 dongjoon-hyun

@dongjoon-hyun I have created a POC branch, and you can reproduce the test by yourself. If you are fine with this approach, then I will make a PR. (not work)

Only one commit was added to the branch https://github.com/zjx20/orc/commit/2610434d50d0ca40df1a07e5327d9889049f921c. In this commit, I changed the execute phase of dependency-plugin from package to test-compile, and also added two dummy dependencies for the orc-core module.

Then run mvn clean package (or mvn clean test), and the same error should occur.

$ mvn clean package
[INFO] Scanning for projects...
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Build Order:
[INFO]
[INFO] Apache ORC                                                         [pom]
[INFO] ORC Shims                                                          [jar]
[INFO] ORC Core                                                           [jar]
[INFO] ORC MapReduce                                                      [jar]
[INFO] ORC Tools                                                          [jar]
[INFO] ORC Examples                                                       [jar]
[INFO]
[INFO] -------------------------< org.apache.orc:orc >-------------------------
[INFO] Building Apache ORC 1.9.0-SNAPSHOT                                 [1/6]
[INFO] --------------------------------[ pom ]---------------------------------
[INFO]
[INFO] --- maven-clean-plugin:3.2.0:clean (default-clean) @ orc ---
[INFO] Deleting /Users/x/Documents/playground/orc/java/target
[INFO]
[INFO] --- maven-enforcer-plugin:3.1.0:enforce (enforce-maven-version) @ orc ---
[INFO]
[INFO] --- maven-enforcer-plugin:3.1.0:enforce (enforce-java-version) @ orc ---
[INFO]
[INFO] --- maven-enforcer-plugin:3.1.0:enforce (enforce-maven) @ orc ---
[INFO]
[INFO] --- maven-remote-resources-plugin:1.7.0:process (process-resource-bundles) @ orc ---
[INFO] Preparing remote bundle org.apache:apache-jar-resource-bundle:1.4
[INFO] Copying 3 resources from 1 bundle.
[INFO]
[INFO] --- maven-antrun-plugin:3.1.0:run (setup-test-dirs) @ orc ---
[INFO] Executing tasks
[INFO]     [mkdir] Created dir: /Users/x/Documents/playground/orc/java/target/testing-tmp
[INFO] Executed tasks
[INFO]
[INFO] --- maven-site-plugin:3.12.0:attach-descriptor (attach-descriptor) @ orc ---
[INFO] No site descriptor found: nothing to attach.
[INFO]
[INFO] --- maven-source-plugin:3.2.1:jar-no-fork (create-source-jar) @ orc ---
[INFO]
[INFO] --- maven-source-plugin:3.2.1:test-jar-no-fork (create-source-jar) @ orc ---
[INFO]
[INFO] --- reproducible-build-maven-plugin:0.15:strip-jar (default) @ orc ---
[INFO]
[INFO] ----------------------< org.apache.orc:orc-shims >----------------------
[INFO] Building ORC Shims 1.9.0-SNAPSHOT                                  [2/6]
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- maven-clean-plugin:3.2.0:clean (default-clean) @ orc-shims ---
[INFO] Deleting /Users/x/Documents/playground/orc/java/shims/target
[INFO]
[INFO] --- maven-enforcer-plugin:3.1.0:enforce (enforce-maven-version) @ orc-shims ---
[INFO]
[INFO] --- maven-enforcer-plugin:3.1.0:enforce (enforce-java-version) @ orc-shims ---
[INFO]
[INFO] --- maven-enforcer-plugin:3.1.0:enforce (enforce-maven) @ orc-shims ---
[INFO]
[INFO] --- build-helper-maven-plugin:3.3.0:add-source (add-source) @ orc-shims ---
[INFO] Source directory: /Users/x/Documents/playground/orc/java/shims/target/generated-sources added.
[INFO]
[INFO] --- maven-remote-resources-plugin:1.7.0:process (process-resource-bundles) @ orc-shims ---
[INFO] Preparing remote bundle org.apache:apache-jar-resource-bundle:1.4
[INFO] Copying 3 resources from 1 bundle.
[INFO]
[INFO] --- maven-resources-plugin:3.2.0:resources (default-resources) @ orc-shims ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Using 'UTF-8' encoding to copy filtered properties files.
[INFO] skip non existing resourceDirectory /Users/x/Documents/playground/orc/java/shims/src/main/resources
[INFO] Copying 3 resources
[INFO]
[INFO] --- maven-compiler-plugin:3.10.1:compile (default-compile) @ orc-shims ---
[INFO] Compiling 13 source files to /Users/x/Documents/playground/orc/java/shims/target/classes
[INFO]
[INFO] --- maven-resources-plugin:3.2.0:testResources (default-testResources) @ orc-shims ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Using 'UTF-8' encoding to copy filtered properties files.
[INFO] Copying 1 resource
[INFO] Copying 3 resources
[INFO]
[INFO] --- maven-antrun-plugin:3.1.0:run (setup-test-dirs) @ orc-shims ---
[INFO] Executing tasks
[INFO]     [mkdir] Created dir: /Users/x/Documents/playground/orc/java/shims/target/testing-tmp
[INFO] Executed tasks
[INFO]
[INFO] --- maven-compiler-plugin:3.10.1:testCompile (default-testCompile) @ orc-shims ---
[INFO] Compiling 1 source file to /Users/x/Documents/playground/orc/java/shims/target/test-classes
[INFO]
[INFO] --- maven-dependency-plugin:3.1.2:analyze-only (default) @ orc-shims ---
[INFO] No dependency problems found
[INFO]
[INFO] --- maven-surefire-plugin:3.0.0-M5:test (default-test) @ orc-shims ---
[INFO]
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.apache.orc.impl.TestHadoopShimsPre2_7
SLF4J: No SLF4J providers were found.
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See https://www.slf4j.org/codes.html#noProviders for further details.
SLF4J: Class path contains SLF4J bindings targeting slf4j-api versions 1.7.x or earlier.
SLF4J: Ignoring binding found at [jar:file:/Users/x/.m2/repository/org/slf4j/slf4j-log4j12/1.7.10/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Ignoring binding found at [jar:file:/Users/x/.m2/repository/ch/qos/logback/logback-classic/1.2.10/logback-classic-1.2.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See https://www.slf4j.org/codes.html#ignoredBindings for an explanation.
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.113 s - in org.apache.orc.impl.TestHadoopShimsPre2_7
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO]
[INFO] --- maven-jar-plugin:3.3.0:jar (default-jar) @ orc-shims ---
[INFO] Building jar: /Users/x/Documents/playground/orc/java/shims/target/orc-shims-1.9.0-SNAPSHOT.jar
[INFO]
[INFO] --- maven-site-plugin:3.12.0:attach-descriptor (attach-descriptor) @ orc-shims ---
[INFO] Skipping because packaging 'jar' is not pom.
[INFO]
[INFO] --- maven-source-plugin:3.2.1:jar-no-fork (create-source-jar) @ orc-shims ---
[INFO] Building jar: /Users/x/Documents/playground/orc/java/shims/target/orc-shims-1.9.0-SNAPSHOT-sources.jar
[INFO]
[INFO] --- maven-source-plugin:3.2.1:test-jar-no-fork (create-source-jar) @ orc-shims ---
[INFO] Building jar: /Users/x/Documents/playground/orc/java/shims/target/orc-shims-1.9.0-SNAPSHOT-test-sources.jar
[INFO]
[INFO] --- reproducible-build-maven-plugin:0.15:strip-jar (default) @ orc-shims ---
[INFO] Stripping /Users/x/Documents/playground/orc/java/shims/target/orc-shims-1.9.0-SNAPSHOT-test-sources.jar
[INFO] Stripping /Users/x/Documents/playground/orc/java/shims/target/orc-shims-1.9.0-SNAPSHOT-sources.jar
[INFO] Stripping /Users/x/Documents/playground/orc/java/shims/target/orc-shims-1.9.0-SNAPSHOT.jar
[INFO]
[INFO] --- maven-jar-plugin:3.3.0:test-jar (default) @ orc-shims ---
[INFO] Building jar: /Users/x/Documents/playground/orc/java/shims/target/orc-shims-1.9.0-SNAPSHOT-tests.jar
[INFO]
[INFO] ----------------------< org.apache.orc:orc-core >-----------------------
[INFO] Building ORC Core 1.9.0-SNAPSHOT                                   [3/6]
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- maven-clean-plugin:3.2.0:clean (default-clean) @ orc-core ---
[INFO] Deleting /Users/x/Documents/playground/orc/java/core/target
[INFO]
[INFO] --- maven-enforcer-plugin:3.1.0:enforce (enforce-maven-version) @ orc-core ---
[INFO]
[INFO] --- maven-enforcer-plugin:3.1.0:enforce (enforce-java-version) @ orc-core ---
[INFO]
[INFO] --- maven-enforcer-plugin:3.1.0:enforce (enforce-maven) @ orc-core ---
[INFO]
[INFO] --- protoc-jar-maven-plugin:3.11.4:run (default) @ orc-core ---
[INFO] Protoc version: 3.17.3
protoc-jar: protoc version: 3.17.3, detected platform: osx-x86_64 (mac os x/x86_64)
protoc-jar: cached: /var/folders/tb/f6vv2x291jdf0v5nb3hs4k0h0000gn/T/protocjar.webcache/com/google/protobuf/protoc/maven-metadata.xml
protoc-jar: cached: /var/folders/tb/f6vv2x291jdf0v5nb3hs4k0h0000gn/T/protocjar.webcache/com/google/protobuf/protoc/3.17.3/protoc-3.17.3-osx-x86_64.exe
protoc-jar: executing: [/var/folders/tb/f6vv2x291jdf0v5nb3hs4k0h0000gn/T/protocjar8908455585973251406/bin/protoc.exe, --version]
libprotoc 3.17.3
[INFO] Protoc command: /var/folders/tb/f6vv2x291jdf0v5nb3hs4k0h0000gn/T/protocjar8908455585973251406/bin/protoc.exe
[INFO] Input directories:
[INFO]     /Users/x/Documents/playground/orc/proto
[INFO] Include directories:
[INFO]     /Users/x/Documents/playground/orc/proto
[INFO] Output targets:
[INFO]     java: /Users/x/Documents/playground/orc/java/core/target/generated-sources (add: none, clean: false, plugin: null, outputOptions: null)
[INFO] /Users/x/Documents/playground/orc/java/core/target/generated-sources does not exist. Creating...
[INFO]     Processing (java): orc_proto.proto
protoc-jar: executing: [/var/folders/tb/f6vv2x291jdf0v5nb3hs4k0h0000gn/T/protocjar8908455585973251406/bin/protoc.exe, -I/Users/x/Documents/playground/orc/proto, -I/Users/x/Documents/playground/orc/proto, --java_out=/Users/x/Documents/playground/orc/java/core/target/generated-sources, /Users/x/Documents/playground/orc/proto/orc_proto.proto]
[INFO]
[INFO] --- build-helper-maven-plugin:3.3.0:add-source (add-source) @ orc-core ---
[INFO] Source directory: /Users/x/Documents/playground/orc/java/core/target/generated-sources added.
[INFO]
[INFO] --- maven-remote-resources-plugin:1.7.0:process (process-resource-bundles) @ orc-core ---
[INFO] Preparing remote bundle org.apache:apache-jar-resource-bundle:1.4
[INFO] Copying 3 resources from 1 bundle.
[INFO]
[INFO] --- maven-resources-plugin:3.2.0:resources (default-resources) @ orc-core ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Using 'UTF-8' encoding to copy filtered properties files.
[INFO] Copying 2 resources
[INFO] Copying 3 resources
[INFO]
[INFO] --- maven-compiler-plugin:3.10.1:compile (default-compile) @ orc-core ---
[INFO] Compiling 166 source files to /Users/x/Documents/playground/orc/java/core/target/classes
[INFO] /Users/x/Documents/playground/orc/java/core/src/java/org/apache/orc/impl/ReaderImpl.java: 某些输入文件使用或覆盖了已过时的 API。
[INFO] /Users/x/Documents/playground/orc/java/core/src/java/org/apache/orc/impl/ReaderImpl.java: 有关详细信息, 请使用 -Xlint:deprecation 重新编译。
[INFO] /Users/x/Documents/playground/orc/java/core/src/java/org/apache/orc/OrcFile.java: 某些输入文件使用了未经检查或不安全的操作。
[INFO] /Users/x/Documents/playground/orc/java/core/src/java/org/apache/orc/OrcFile.java: 有关详细信息, 请使用 -Xlint:unchecked 重新编译。
[INFO]
[INFO] --- maven-resources-plugin:3.2.0:testResources (default-testResources) @ orc-core ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Using 'UTF-8' encoding to copy filtered properties files.
[INFO] Copying 6 resources
[INFO] Copying 3 resources
[INFO]
[INFO] --- maven-antrun-plugin:3.1.0:run (setup-test-dirs) @ orc-core ---
[INFO] Executing tasks
[INFO]     [mkdir] Created dir: /Users/x/Documents/playground/orc/java/core/target/testing-tmp
[INFO] Executed tasks
[INFO]
[INFO] --- maven-compiler-plugin:3.10.1:testCompile (default-testCompile) @ orc-core ---
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- maven-dependency-plugin:3.1.2:analyze-only (default) @ orc-core ---
[WARNING] Unused declared dependencies found:
[WARNING]    net.sourceforge.javacsv:javacsv:jar:2.0:compile
[WARNING]    org.openjdk.jmh:jmh-core:jar:1.35:test
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Apache ORC 1.9.0-SNAPSHOT:
[INFO]
[INFO] Apache ORC ......................................... SUCCESS [  4.215 s]
[INFO] ORC Shims .......................................... SUCCESS [ 11.100 s]
[INFO] ORC Core ........................................... FAILURE [ 11.781 s]
[INFO] ORC MapReduce ...................................... SKIPPED
[INFO] ORC Tools .......................................... SKIPPED
[INFO] ORC Examples ....................................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  27.362 s
[INFO] Finished at: 2022-10-17T10:35:19+08:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-dependency-plugin:3.1.2:analyze-only (default) on project orc-core: Dependency problems found -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <args> -rf :orc-core

zjx20 avatar Oct 17 '22 02:10 zjx20

@zjx20 I asked you to make a PR(pull request) to Apache ORC repo in order to accept and merge after testing on the PR. :)

dongjoon-hyun avatar Oct 17 '22 03:10 dongjoon-hyun

BTW, I followed your branch like the following. Your branch still seems to fail to achieve your goal, doesn't it?

$ git diff
diff --git a/java/pom.xml b/java/pom.xml
index 1f4ae275..a0084d50 100644
--- a/java/pom.xml
+++ b/java/pom.xml
@@ -376,7 +376,7 @@
           <version>3.1.2</version>
           <executions>
             <execution>
-              <phase>package</phase>
+              <phase>test-compile</phase>
               <goals>
                 <goal>analyze-only</goal>
               </goals>

$ mvn -Dmaven.test.skip=true clean package
Running `/Users/dongjoon/APACHE/orc-merge/java/mvnw`...
Using `mvn` from path: /opt/homebrew/bin/mvn
[INFO] Scanning for projects...
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Build Order:
[INFO]
[INFO] Apache ORC                                                         [pom]
[INFO] ORC Shims                                                          [jar]
[INFO] ORC Core                                                           [jar]
[INFO] ORC MapReduce                                                      [jar]
[INFO] ORC Tools                                                          [jar]
[INFO] ORC Examples                                                       [jar]
[INFO]
[INFO] -------------------------< org.apache.orc:orc >-------------------------
[INFO] Building Apache ORC 1.9.0-SNAPSHOT                                 [1/6]
[INFO] --------------------------------[ pom ]---------------------------------
[INFO]
[INFO] --- maven-clean-plugin:3.2.0:clean (default-clean) @ orc ---
[INFO] Deleting /Users/dongjoon/APACHE/orc-merge/java/target
[INFO]
[INFO] --- maven-enforcer-plugin:3.1.0:enforce (enforce-maven-version) @ orc ---
[INFO]
[INFO] --- maven-enforcer-plugin:3.1.0:enforce (enforce-java-version) @ orc ---
[INFO]
[INFO] --- maven-enforcer-plugin:3.1.0:enforce (enforce-maven) @ orc ---
[INFO]
[INFO] --- maven-remote-resources-plugin:1.7.0:process (process-resource-bundles) @ orc ---
[INFO] Preparing remote bundle org.apache:apache-jar-resource-bundle:1.4
[INFO] Copying 3 resources from 1 bundle.
[INFO]
[INFO] --- maven-antrun-plugin:3.1.0:run (setup-test-dirs) @ orc ---
[INFO] Executing tasks
[INFO]     [mkdir] Created dir: /Users/dongjoon/APACHE/orc-merge/java/target/testing-tmp
[INFO] Executed tasks
[INFO]
[INFO] --- maven-site-plugin:3.12.0:attach-descriptor (attach-descriptor) @ orc ---
[INFO] No site descriptor found: nothing to attach.
[INFO]
[INFO] --- maven-source-plugin:3.2.1:jar-no-fork (create-source-jar) @ orc ---
[INFO]
[INFO] --- maven-source-plugin:3.2.1:test-jar-no-fork (create-source-jar) @ orc ---
[INFO]
[INFO] --- reproducible-build-maven-plugin:0.15:strip-jar (default) @ orc ---
[INFO]
[INFO] ----------------------< org.apache.orc:orc-shims >----------------------
[INFO] Building ORC Shims 1.9.0-SNAPSHOT                                  [2/6]
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- maven-clean-plugin:3.2.0:clean (default-clean) @ orc-shims ---
[INFO] Deleting /Users/dongjoon/APACHE/orc-merge/java/shims/target
[INFO]
[INFO] --- maven-enforcer-plugin:3.1.0:enforce (enforce-maven-version) @ orc-shims ---
[INFO]
[INFO] --- maven-enforcer-plugin:3.1.0:enforce (enforce-java-version) @ orc-shims ---
[INFO]
[INFO] --- maven-enforcer-plugin:3.1.0:enforce (enforce-maven) @ orc-shims ---
[INFO]
[INFO] --- build-helper-maven-plugin:3.3.0:add-source (add-source) @ orc-shims ---
[INFO] Source directory: /Users/dongjoon/APACHE/orc-merge/java/shims/target/generated-sources added.
[INFO]
[INFO] --- maven-remote-resources-plugin:1.7.0:process (process-resource-bundles) @ orc-shims ---
[INFO] Preparing remote bundle org.apache:apache-jar-resource-bundle:1.4
[INFO] Copying 3 resources from 1 bundle.
[INFO]
[INFO] --- maven-resources-plugin:3.2.0:resources (default-resources) @ orc-shims ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Using 'UTF-8' encoding to copy filtered properties files.
[INFO] skip non existing resourceDirectory /Users/dongjoon/APACHE/orc-merge/java/shims/src/main/resources
[INFO] Copying 3 resources
[INFO]
[INFO] --- maven-compiler-plugin:3.10.1:compile (default-compile) @ orc-shims ---
[INFO] Compiling 13 source files to /Users/dongjoon/APACHE/orc-merge/java/shims/target/classes
[INFO]
[INFO] --- maven-resources-plugin:3.2.0:testResources (default-testResources) @ orc-shims ---
[INFO] Not copying test resources
[INFO]
[INFO] --- maven-antrun-plugin:3.1.0:run (setup-test-dirs) @ orc-shims ---
[INFO] Executing tasks
[INFO]     [mkdir] Created dir: /Users/dongjoon/APACHE/orc-merge/java/shims/target/testing-tmp
[INFO] Executed tasks
[INFO]
[INFO] --- maven-compiler-plugin:3.10.1:testCompile (default-testCompile) @ orc-shims ---
[INFO] Not compiling test sources
[INFO]
[INFO] --- maven-dependency-plugin:3.1.2:analyze-only (default) @ orc-shims ---
[WARNING] Unused declared dependencies found:
[WARNING]    org.junit.jupiter:junit-jupiter-api:jar:5.9.0:test
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Apache ORC 1.9.0-SNAPSHOT:
[INFO]
[INFO] Apache ORC ......................................... SUCCESS [  0.730 s]
[INFO] ORC Shims .......................................... FAILURE [  1.134 s]
[INFO] ORC Core ........................................... SKIPPED
[INFO] ORC MapReduce ...................................... SKIPPED
[INFO] ORC Tools .......................................... SKIPPED
[INFO] ORC Examples ....................................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  1.924 s
[INFO] Finished at: 2022-10-16T21:03:42-07:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-dependency-plugin:3.1.2:analyze-only (default) on project orc-shims: Dependency problems found -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <args> -rf :orc-shims

dongjoon-hyun avatar Oct 17 '22 04:10 dongjoon-hyun

Sorry about that, I didn't test that and thought it should just work. It seems that mvn -Dmaven.test.skip=true is not executed the way I thought it would. The approach is not working and I have no more ideas.

zjx20 avatar Oct 17 '22 04:10 zjx20

Thank you for the reply. Let me think about this more.

dongjoon-hyun avatar Oct 17 '22 16:10 dongjoon-hyun

The root cause of the difference between these two commands is that -Dmaven.test.skip=true does not compile the test classes. Apache Maven Dependency Plugin will throw an error if a dependency is not used. We can use ignoreNonCompile defined in https://maven.apache.org/plugins/maven-dependency-plugin/analyze-only-mojo.html#ignoreNonCompile to avoid the problem.

After adding this tag, I found that we have some classes such as slf4j-api in MapReduce module that only appear in test but scope is defined as compile. Maybe we can fix the problem.

What do you think? @zjx20 @dongjoon-hyun

deshanxiao avatar Oct 20 '22 03:10 deshanxiao

It's great for discovering new dependency problems in this way. But the flag seems to entirely skip analyzing for the test scope (precisely non-compile scopes), so we cannot find unused test-only dependencies too. I don't know if it's acceptable.

zjx20 avatar Oct 20 '22 04:10 zjx20

Ah, I deleted my previous post. Sorry for that. It seems that the drawback (Ignoring all Runtime/Provided/Test/System scope) is too big.

dongjoon-hyun avatar Oct 20 '22 05:10 dongjoon-hyun

Agree. I think the purpose of introducing Apache Maven Dependency Plugin is to reduce some unused dependencies into the package. For Runtime/Provided/Test/System scope, it will not be packed into the final package. So the cost of introducing this tag looks like acceptable.

Parquet-MR: https://github.com/apache/parquet-mr/blob/62b774cd0f0c60cfbe540bbfa60bee15929af5d4/pom.xml#L554

deshanxiao avatar Oct 20 '22 05:10 deshanxiao

Good point, I vote for it.

zjx20 avatar Oct 20 '22 05:10 zjx20

Okay. Please make a PR. :)

dongjoon-hyun avatar Oct 20 '22 10:10 dongjoon-hyun

Closed by https://github.com/apache/orc/commit/4160e81714d2ccf2a20f79816c64f63eee5c85f9. Thank you guys!

zjx20 avatar Oct 25 '22 02:10 zjx20

Thank you for closing, @zjx20 . :)

dongjoon-hyun avatar Oct 25 '22 02:10 dongjoon-hyun