Fat Jar doesn't seem to work as expected
I'm submitting a…
- [X] bug report
- [ ] feature request
- [ ] other
Short description of the issue/suggestion: When following your example of creating a package using a fat jar, JavaPackager seems to not actually use that jar when it builds the package.
So here are my POM settings:
<plugin>
<groupId>io.github.fvarrui</groupId>
<artifactId>javapackager</artifactId>
<version>${javapackager}</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>package</goal>
</goals>
<configuration>
<mainClass>${mainClass}</mainClass>
<bundleJre>true</bundleJre>
<runnableJar>${build.directory}/${artifactId}-${project.version}.jar-with-dependencies.jar</runnableJar>
<copyDependencies>false</copyDependencies>
<platform>linux</platform>
<name>iget</name>
<vmArgs>
<arg>--enable-preview</arg>
</vmArgs>
<linuxConfig>
<generateAppImage>false</generateAppImage>
<generateDeb>true</generateDeb>
<generateRpm>false</generateRpm>
<wrapJar>true</wrapJar>
</linuxConfig>
</configuration>
</execution>
</executions>
</plugin>
The fat jar is created with maven-assembly plugin.
The build completes without any errors. But when I run the program and it gets to where it is needing to access a dependency, it throws an error saying it cannot find the dependency. HOWEVER, when I run the fat jar with java -jar, everything works fine.
So I looked more closely at the feedback during the build process and I noticed right after the fat jar is built, JavaPackager takes over, but it doesn't actually use the fat jar, rather, it makes its own jar instead.
In this text from the build feedback, notice that the fat jar is indeed listed in the runnableJar setting:
[INFO] --- assembly:3.7.1:single (make-assembly) @ iGet ---
[INFO] Building jar: /home/michael/java/iGet/target/iGet-2.0.0-jar-with-dependencies.jar
[INFO]
[INFO] --- javapackager:1.7.5:package (default) @ iGet ---
[INFO] Using packager io.github.fvarrui.javapackager.packagers.LinuxPackager
[INFO] Creating app ...
[INFO] Initializing packager ...
[INFO] PackagerSettings [
outputDirectory=/home/michael/java/iGet/target,
licenseFile=null,
iconFile=null,
generateInstaller=true,
forceInstaller=false,
mainClass=com.simtechdata.Main,
name=iget,
displayName=iGet,
version=2.0.0,
description=A program that downloads Instagram Reels and Youtube videos,
url=null,
administratorRequired=false,
organizationName=ACME,
organizationUrl=,
organizationEmail=null,
bundleJre=true,
customizedJre=true,
jrePath=null,
jdkPath=/home/michael/.sdkman/candidates/java/22.0.1-graal,
additionalResources=[],
modules=[],
additionalModules=[],
platform=linux,
envPath=null,
vmArgs=[--enable-preview],
runnableJar=/home/michael/java/iGet/target/iGet-2.0.0.jar-with-dependencies.jar,
^ This is correct and the file exists
copyDependencies=false,
jreDirectoryName=jre,
winConfig=null,
linuxConfig=LinuxConfig [categories=[Utility],
generateDeb=true,
generateRpm=false,
generateAppImage=false,
pngFile=null,
wrapJar=true,
installationPath=/opt],
macConfig=null,
createTarball=false,
tarballName=null,
createZipball=false,
zipballName=null,
extra=null,
useResourcesAsWorkingDir=true,
assetsDir=/home/michael/java/iGet/assets,
classpath=null,
jreMinVersion=null,
manifest=null,
additionalModulePaths=[],
fileAssociations=[],
packagingJdk=/home/michael/.sdkman/candidates/java/22.0.1-graal,
scripts=Scripts [bootstrap=null,
preInstall=null,
postInstall=null],
arch=x64,
templates=[Template [name=windows/iss.vtl,
bom=true]]]
But then what comes next, is that it shows adding assets and resources, then it goes straight into building it's own jar called iGet-2.0.0-runnable.jar where I would have assumed that it would just use the fat jar that I specified. But the jar that it builds doesn't have the dependencies of course because you said to disable that when using a fat jar.
[INFO] Packager initialized!
[INFO]
[INFO] Creating app structure ...
[INFO] App folder created: /home/michael/java/iGet/target/iget
[INFO] Assets folder created: /home/michael/java/iGet/target/assets
[INFO] App structure created!
[INFO]
[INFO] Resolving resources ...
[INFO] Trying to resolve license from POM ...
[INFO] License not resolved!
[INFO]
[WARNING] No license file specified
[INFO] Copying resource [/linux/default-icon.png] to file [/home/michael/java/iGet/target/assets/iget.png]
[INFO] Icon file resolved: /home/michael/java/iGet/target/assets/iget.png
[INFO] Effective additional resources [/home/michael/java/iGet/target/assets/iget.png]
[INFO] Resources resolved!
[INFO]
[INFO] Copying additional resources
[INFO] Copying file [/home/michael/java/iGet/target/assets/iget.png] to folder [/home/michael/java/iGet/target/iget]
[INFO] Executing command: /bin/sh -c cd '/home/michael/java/iGet/.' && 'cp' /home/michael/java/iGet/target/assets/iget.png /home/michael/java/iGet/target/iget/iget.png
[INFO] All additional resources copied!
[INFO]
[INFO] Copying all dependencies ...
[INFO] Dependencies copied to null!
[INFO]
[INFO] Creating runnable JAR...
[INFO] Building jar: /home/michael/java/iGet/target/iGet-2.0.0-runnable.jar
[INFO] Runnable jar created in /home/michael/java/iGet/target/iGet-2.0.0-runnable.jar!
All throughout the rest of the build, it only ever uses the jar that it built and the only mention at all of the fat jar is in the config that it spits out at the top.
Notice its using the runnable jar when it creates the final program and not the jar-with-dependencies as it should be using.
[INFO] Creating GNU/Linux executable ...
[INFO] Rendering desktop file to /home/michael/java/iGet/target/assets/iget.desktop
[INFO] Startup script generated in /home/michael/java/iGet/target/assets/startup.sh
[INFO] Concatenating files [/home/michael/java/iGet/target/assets/startup.sh,/home/michael/java/iGet/target/iGet-2.0.0-runnable.jar] into file [/home/michael/java/iGet/target/iget/iget]
[INFO] GNU/Linux executable created in /home/michael/java/iGet/target/iget/iget!
I've managed to get what I needed by just letting JP build the package without the fat jar, but I wanted to bring this up because it might be something you want to address or explain to me what I did wrong?
Please tell us about your environment:
- JavaPackager version: 1.7.5
- OS version: Ubuntu 22.0.4
- JDK version: 22
- Build tool:
- [X] Maven
- [ ] Gradle
Hi @EasyG0ing1! Wow! It's a bit weird ... I've just generated an app and it worked as expected:
[INFO] Using runnable JAR: /mnt/c/Users/fvarrui/GitHub/HelloWorldMaven/target/HelloWorldMaven-1.0.0-jar-with-dependencies.jar
[...]
[INFO] Creating GNU/Linux executable ...
[INFO] Rendering desktop file to /mnt/c/Users/fvarrui/GitHub/HelloWorldMaven/target/assets/HelloWorldMaven.desktop
[INFO] Rendering mime.xml file to /mnt/c/Users/fvarrui/GitHub/HelloWorldMaven/target/assets/HelloWorldMaven.xml
[INFO] Startup script generated in /mnt/c/Users/fvarrui/GitHub/HelloWorldMaven/target/assets/startup.sh
[INFO] Concatenating files [/mnt/c/Users/fvarrui/GitHub/HelloWorldMaven/target/assets/startup.sh,/mnt/c/Users/fvarrui/GitHub/HelloWorldMaven/target/HelloWorldMaven-1.0.0-jar-with-dependencies.jar] into file [/mnt/c/Users/fvarrui/GitHub/HelloWorldMaven/target/HelloWorldMaven/HelloWorldMaven]
[INFO] GNU/Linux executable created in /mnt/c/Users/fvarrui/GitHub/HelloWorldMaven/target/HelloWorldMaven/HelloWorldMaven!
Plugins config here:
<!-- creates runnable fat jar: HelloWorldMaven-1.0.0-SNAPSHOT-jar-with-dependencies.jar -->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-assembly-plugin</artifactId>
<version>3.3.0</version>
<configuration>
<archive>
<manifest>
<mainClass>${exec.mainClass}</mainClass>
</manifest>
</archive>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
</configuration>
<executions>
<execution>
<id>make-assembly</id>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>io.github.fvarrui</groupId>
<artifactId>javapackager</artifactId>
<version>1.7.5</version>
<!--
<version>1.7.6-20240225.211252-1</version>
-->
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>package</goal>
</goals>
<configuration>
<platform>linux</platform>
<bundleJre>true</bundleJre>
<customizedJre>true</customizedJre>
<generateInstaller>false</generateInstaller>
<administratorRequired>false</administratorRequired>
<!--
<jdkPath>C:\Users\fvarrui\jdks\mac\aarch64\jdk-21.0.2+13\Contents\Home</jdkPath>
-->
<additionalResources>
<additionalResource>src/main/resources/info.txt</additionalResource>
<additionalResource>HelloWorldMaven.l4j.ini</additionalResource>
</additionalResources>
<runnableJar>${project.build.directory}/${project.artifactId}-${project.version}-jar-with-dependencies.jar</runnableJar>
<copyDependencies>false</copyDependencies>
<vmArgs>
<vmArg>-Dcustom.variable="Hi!"</vmArg>
<vmArg>-Dother.custom.variable="Bye!"</vmArg>
</vmArgs>
<fileAssociations>
<fileAssociation>
<description>HelloWorld File</description>
<extension>hello</extension>
<mimeType>application/hello</mimeType>
</fileAssociation>
</fileAssociations>
<!--
<jrePath>C:\Users\fvarrui\jdks\windows\jdk-11.0.22+7-jre</jrePath>
-->
<arch>x64</arch>
<winConfig>
<headerType>gui</headerType>
<exeCreationTool>launch4j</exeCreationTool>
<icoFile>src/main/resources/HelloWorldMaven.ico</icoFile>
<generateSetup>true</generateSetup>
<generateMsi>false</generateMsi>
</winConfig>
<createZipball>false</createZipball>
<createTarball>false</createTarball>
</configuration>
</execution>
</executions>
</plugin>
I think I know what's happening .... have a look!!! 😄
Your generated fat jar is: iGet-2.0.0-jar-with-dependencies.jar with a hyphen between 2.0.0 and jar-with...
[INFO] --- assembly:3.7.1:single (make-assembly) @ iGet ---
[INFO] Building jar: /home/michael/java/iGet/target/iGet-2.0.0-jar-with-dependencies.jar
and the jar which JP is referencing is iGet-2.0.0.jar-with-dependencies.jar with a dot between 2.0.0 and jar-with...
<runnableJar>${build.directory}/${artifactId}-${project.version}.jar-with-dependencies.jar</runnableJar>
runnableJar=/home/michael/java/iGet/target/iGet-2.0.0.jar-with-dependencies.jar,
^ This is "correct and the file exists" ;-D
But I realized that JP should warn about this situation and/or stop the building process if runnableJar doesn't exist.
Here is the problem: https://github.com/fvarrui/JavaPackager/blob/f01e4f0d5e89e70c3a31181cc1a9aeee445a0919/src/main/java/io/github/fvarrui/javapackager/packagers/Packager.java#L402-L410 It creates a runnableJar if it's null or doesn't exist (without any warning) ... just continues.
I've just fixed and now the building process is stopped if it's not able to find the specified runnable jar ... snapshot version 1.7.6-20240430.234925-8 with this patch:
https://github.com/fvarrui/JavaPackager/blob/7de5a568e26b1b4cf77284c01e30627a8771269e/src/main/java/io/github/fvarrui/javapackager/packagers/Packager.java#L397-L407
Please, try it and give me some feedback. Thanks!!
@fvarrui I see it and it makes sense. Yeah had it told me that it couldn't find my jar, I would have looked into that deeper, but it seemed to me that everything was in order so I didn't catch the slight mis-name in the jar file.
I'll give the fix a try and let you know how it works out.
@fvarrui I fixed the name of the fat jar in the POM file and re-ran the package with 1.7.5 which worked fine and it did use the fat jar. But then when I simply changed the version of JP to 1.7.6-SNAPSHOT, it errors out. This zip file has the output from each version for your review. Output.zip
@fvarrui Also, when I say that it worked fine with 1.7.5, I meant there were no errors making the .deb file. However, when I installed the program and tried to run it, it wouldn't run:
michael@ubuntudt:~/Downloads$ sudo dpkg -i iget_2.0.0.deb
Selecting previously unselected package iget.
(Reading database ... 211479 files and directories currently installed.)
Preparing to unpack iget_2.0.0.deb ...
Unpacking iget (2.0.0) ...
Setting up iget (2.0.0) ...
Processing triggers for mailcap (3.70+nmu1ubuntu1) ...
Processing triggers for gnome-menus (3.36.0-1ubuntu3) ...
Processing triggers for desktop-file-utils (0.26-1ubuntu3) ...
michael@ubuntudt:~/Downloads$ iget
Error: Unable to initialize main class com.simtechdata.Main
Caused by: java.lang.NoClassDefFoundError: java/sql/SQLException
But when I run the jar file by itself, it works fine:
michael@ubuntudt:~/java/iGet$ java --enable-preview -jar target/iGet-jar-with-dependencies.jar
Add URLs to the list simply by typing:
iget http://www.someserver.com/some/link - Add a link to the que
iget get http://www.someserver.com/some/link - Download link immediately
(http must begin the link to be recognized as a link)
Options:
get - Download links in que
get <url> - download one URL right now
watch - Watch mode looks for links to show up in clipboard then downloads them
setFolder - Set the folder where downloads get stored
[...]
@fvarrui And some more details that might help...
When I configure JP so that it handles the jar packaging on its own, the resulting .deb file is created without error and when I install the program using that deb file, the program runs without any problems at all.
Here is the pom setup for that scenario:
<configuration>
<mainClass>${mainClass}</mainClass>
<bundleJre>true</bundleJre>
<customizedJre>false</customizedJre>
<copyDependencies>true</copyDependencies>
<platform>linux</platform>
<name>iget</name>
<vmArgs>
<arg>--enable-preview</arg>
</vmArgs>
<linuxConfig>
<generateAppImage>false</generateAppImage>
<generateDeb>true</generateDeb>
<generateRpm>false</generateRpm>
<wrapJar>true</wrapJar>
</linuxConfig>
</configuration>
However, when I configure JP so that it uses the FAT JAR, it will package the deb file without any errors, but then when I install the program using the deb file, the program does not run and it gives me this error:
Error: Unable to initialize main class com.simtechdata.Main
Caused by: java.lang.NoClassDefFoundError: java/sql/SQLException
But when I run the fat jar like this:
java --enable-preview -jar target/iGet-jar-with-dependencies.jar
It runs fine.
So it apparently is not executing the fat jar properly or something else is going on that I don't understand.
And here is the POM config that gives that behavior:
<configuration>
<mainClass>${mainClass}</mainClass>
<bundleJre>true</bundleJre>
<runnableJar>${build.directory}/${artifactId}-jar-with-dependencies.jar</runnableJar>
<copyDependencies>false</copyDependencies>
<platform>linux</platform>
<name>iget</name>
<vmArgs>
<arg>--enable-preview</arg>
</vmArgs>
<linuxConfig>
<generateAppImage>false</generateAppImage>
<generateDeb>true</generateDeb>
<generateRpm>false</generateRpm>
<wrapJar>true</wrapJar>
</linuxConfig>
</configuration>
And all of that is with version 1.7.5
As soon as I switch to 1.7.6-SNAPSHOT, it won't even compile the deb file and it throws the error I gave you in the zip file in my previous message.
@fvarrui I fixed the name of the fat jar in the POM file and re-ran the package with 1.7.5 which worked fine and it did use the fat jar. But then when I simply changed the version of JP to 1.7.6-SNAPSHOT, it errors out. This zip file has the output from each version for your review. Output.zip
You should use the specific snapshot version: 1.7.6-20240430.234925-8 ... this version contains your patch, don't use 1.7.6-SNAPSHOT because it's the first 1.7.6 snapshot released, which has a known and already fixed error.
However, when I configure JP so that it uses the FAT JAR, it will package the
debfile without any errors, but then when I install the program using thedebfile, the program does not run and it gives me this error:Error: Unable to initialize main class com.simtechdata.Main Caused by: java.lang.NoClassDefFoundError: java/sql/SQLExceptionBut when I run the fat jar like this:
java --enable-preview -jar target/iGet-jar-with-dependencies.jar
Yes, this error is caused by an incomplete generated JRE. jdeps can't deal fine with far jars, I don't know why. A work around is bundle a full jre. Please, read the https://github.com/fvarrui/JavaPackager/issues/399#issuecomment-2059905949
@fvarrui And I'm going to assume for the moment, that jdeps is used to compile Java code whether or not the project has any module files? Cause mine does not have a module-info.java file (it is not modular).
Let me explain:
- Java Platform Module System (JPMS), aka Java modules, was introduced in Java 9, so since this version all JDKs now include two new tools:
jdepsandjlink. - Now all Java core classes have been organized internally into modules, so each module contains a set of classes
- The JDK includes all those core modules, and if you run next command you can get a full list:
C:\Users\fvarrui>java --list-modules
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
-
jdepstries to find out those core modules needed by your code, analyzing your classes.
For example, you can see here my jar, containing my compiled classes:
C:\Users\fvarrui\GitHub\HelloWorldMaven\target\HelloWorldMaven-1.0.0-runnable.jar
and its dependencies, all copied into libs folder:
C:\Users\fvarrui\GitHub\HelloWorldMaven\target\HelloWorldMaven\libs\commons-io-2.7.jar
C:\Users\fvarrui\GitHub\HelloWorldMaven\target\HelloWorldMaven\libs\jna-5.13.0.jar
C:\Users\fvarrui\GitHub\HelloWorldMaven\target\HelloWorldMaven\libs\poi-ooxml-5.2.3.jar
C:\Users\fvarrui\GitHub\HelloWorldMaven\target\HelloWorldMaven\libs\poi-5.2.3.jar
C:\Users\fvarrui\GitHub\HelloWorldMaven\target\HelloWorldMaven\libs\commons-codec-1.15.jar
C:\Users\fvarrui\GitHub\HelloWorldMaven\target\HelloWorldMaven\libs\commons-math3-3.6.1.jar
C:\Users\fvarrui\GitHub\HelloWorldMaven\target\HelloWorldMaven\libs\SparseBitSet-1.2.jar
C:\Users\fvarrui\GitHub\HelloWorldMaven\target\HelloWorldMaven\libs\poi-ooxml-lite-5.2.3.jar
C:\Users\fvarrui\GitHub\HelloWorldMaven\target\HelloWorldMaven\libs\xmlbeans-5.1.1.jar
C:\Users\fvarrui\GitHub\HelloWorldMaven\target\HelloWorldMaven\libs\commons-compress-1.21.jar
C:\Users\fvarrui\GitHub\HelloWorldMaven\target\HelloWorldMaven\libs\curvesapi-1.07.jar
C:\Users\fvarrui\GitHub\HelloWorldMaven\target\HelloWorldMaven\libs\log4j-api-2.18.0.jar
C:\Users\fvarrui\GitHub\HelloWorldMaven\target\HelloWorldMaven\libs\commons-collections4-4.4.jar
then you run jdeps this way:
C:\Users\fvarrui\GitHub\HelloWorldMaven>jdeps -q --multi-release 21 --ignore-missing-deps --print-module-deps --add-modules=ALL-MODULE-PATH --module-path=C:\Users\fvarrui\GitHub\HelloWorldMaven\target\HelloWorldMaven-1.0.0-runnable.jar;C:\Users\fvarrui\GitHub\HelloWorldMaven\target\HelloWorldMaven\libs
java.base,java.desktop,java.logging,java.security.jgss,java.xml.crypto
and it'll find all necessary modules: java.basejava.desktop,java.logging,java.security.jgss,java.xml.crypto
-
jlinkis able to generate a Java runtime (JRE) including only specific modules (so, a reduced set of classes), what is the same as a reduced JRE (less megabytes to distribute with your app).
Therefore, now you can pass the modules found by jdeps to jlink using the --add-modules argument:
C:\Users\fvarrui\GitHub\HelloWorldMaven>jlink "--module-path=C:\Program Files\GraalVM\graalvm-community-openjdk-21.0.2+13.1\jmods" --add-modules java.base,java.desktop,java.logging,java.security.jgss,java.xml.crypto --output C:\Users\fvarrui\GitHub\HelloWorldMaven\target\HelloWorldMaven\jre --no-header-files --no-man-pages --strip-debug
Finally, to check that all worked fine, you can run java --list-modules from your brand new generated JRE:
C:\Users\fvarrui\GitHub\HelloWorldMaven>C:\Users\fvarrui\GitHub\HelloWorldMaven\target\HelloWorldMaven\jre\bin\java --list-modules
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
And voila! Your JRE only includes those specific modules and the modules they depend on.
I think that if you include a module-info in your jar it helps to jdeps ... but, what happen with fat jars if a jar only can have one module-info? do you have to generate a new module-info mixing all jar's module-infos? Not sure how to answer these questions.
I hope this helps!
@fvarrui So this is probably where my confusion comes in with Java modularity ... has Java replaced the concept of dependencies and libraries with the module concept? Can you think of dependencies as modules now? I was under the impression that modules were essentially sections of code that are defined by the package structure in any given Java project.
@fvarrui So this is probably where my confusion comes in with Java modularity ... has Java replaced the concept of dependencies and libraries with the module concept? Can you think of dependencies as modules now? I was under the impression that modules were essentially sections of code that are defined by the package structure in any given Java project.
I think it's not a replacement, but a encapsulation improvement for better practices. A JAR can be modularized just including a module-info, basically specifying which other modules are required and what your module offers to other modules. You still can use the old way, JPMS is not mandatory. Even a non-modular JAR can be used as a modular one, it depends how it's specified to Java at runtime (not a good practice in my opinion, but sometimes you can't control this as with dependencies).
Sorry for my late reply!