AppImage
No AppRun extracted when using --appimage-extract on a type 2 AppImage
Using Ubuntu 16.04 running in the WSL (on Windows 10 Fall Creators Update), passing --appimage-extract on a type-2 AppImage generated using linuxdeployqt extracts all files except the AppRun file.
The AppImage used in the log below can be found inside this attachment: Godot-Linux-x86_64.zip
squashfs-root/.DirIcon
squashfs-root/AppRun
TODO: Implement inode.base.inode_type 10
squashfs-root/godot.desktop
squashfs-root/godot.svg
squashfs-root/usr
squashfs-root/usr/bin
squashfs-root/usr/bin/godot
squashfs-root/usr/lib
squashfs-root/usr/lib/libFLAC.so.8
squashfs-root/usr/lib/libX11-xcb.so.1
squashfs-root/usr/lib/libXau.so.6
squashfs-root/usr/lib/libXcursor.so.1
squashfs-root/usr/lib/libXdamage.so.1
squashfs-root/usr/lib/libXdmcp.so.6
squashfs-root/usr/lib/libXext.so.6
squashfs-root/usr/lib/libXfixes.so.3
squashfs-root/usr/lib/libXinerama.so.1
squashfs-root/usr/lib/libXrandr.so.2
squashfs-root/usr/lib/libXrender.so.1
squashfs-root/usr/lib/libXxf86vm.so.1
squashfs-root/usr/lib/libasyncns.so.0
squashfs-root/usr/lib/libcrypto.so.1.0.0
squashfs-root/usr/lib/libdbus-1.so.3
squashfs-root/usr/lib/libfreetype.so.6
squashfs-root/usr/lib/libglapi.so.0
squashfs-root/usr/lib/libjson-c.so.2
squashfs-root/usr/lib/libnsl.so.1
squashfs-root/usr/lib/libogg.so.0
squashfs-root/usr/lib/libpng12.so.0
squashfs-root/usr/lib/libpulse-simple.so.0
squashfs-root/usr/lib/libpulse.so.0
squashfs-root/usr/lib/libpulsecommon-4.0.so
squashfs-root/usr/lib/libsndfile.so.1
squashfs-root/usr/lib/libssl.so.1.0.0
squashfs-root/usr/lib/libvorbis.so.0
squashfs-root/usr/lib/libvorbisenc.so.2
squashfs-root/usr/lib/libwrap.so.0
squashfs-root/usr/lib/libxcb-dri2.so.0
squashfs-root/usr/lib/libxcb-dri3.so.0
squashfs-root/usr/lib/libxcb-glx.so.0
squashfs-root/usr/lib/libxcb-present.so.0
squashfs-root/usr/lib/libxcb-sync.so.1
squashfs-root/usr/lib/libxshmfence.so.1
squashfs-root/usr/share
squashfs-root/usr/share/icons
squashfs-root/usr/share/icons/godot.svg
Please provide a link to your build recipe, and ideally also a build log.
The build recipe can be found here.
Build log:
Desktop file as first argument: "appdir/godot.desktop"
desktopExecEntry: "godot"
desktopIconEntry: "godot"
Found binary from desktop file: "/builds/Calinou/godot-builds/godot/appdir/usr/bin/godot"
FHS-like mode with PREFIX, fhsPrefix: "/builds/Calinou/godot-builds/godot/appdir/usr"
app-binary: "/builds/Calinou/godot-builds/godot/appdir/usr/bin/godot"
appDirPath: "/builds/Calinou/godot-builds/godot/appdir"
relativeBinPath: "usr/bin/godot"
Found icons from desktop file: ("/builds/Calinou/godot-builds/godot/appdir/usr/share/icons/godot.svg")
iconToBeUsed: "/builds/Calinou/godot-builds/godot/appdir/usr/share/icons/godot.svg"
Copied "/builds/Calinou/godot-builds/godot/appdir/usr/share/icons/godot.svg" to "/builds/Calinou/godot-builds/godot/appdir/godot.svg"
WARNING: gpg2 is missing, please install it if you want to create digital signatures
Name: Godot Engine
Icon: godot
Exec: godot -pm
Comment: Multi-platform 2D and 3D game engine with a feature rich editor
Type: Application
Categories: Development;IDE;
File used for determining architecture: /builds/Calinou/godot-builds/godot/appdir/usr/lib/libjson-c.so.2
Arch: 86_64
App name for filename: Godot_Engine
dest_path: Godot_Engine-x86_64.AppImage
Generating squashfs...
Parallel mksquashfs: Using 2 processors
Creating 4.0 filesystem on Godot_Engine-x86_64.AppImage, block size 131072.
[===============================================================|] 405/405 100%
Exportable Squashfs 4.0 filesystem, gzip compressed, data block size 131072
compressed data, compressed metadata, compressed fragments, compressed xattrs
duplicates are removed
Filesystem size 18518.11 Kbytes (18.08 Mbytes)
38.49% of uncompressed filesystem size (48112.35 Kbytes)
Inode table size 1697 bytes (1.66 Kbytes)
42.02% of uncompressed inode table size (4039 bytes)
Directory table size 587 bytes (0.57 Kbytes)
57.05% of uncompressed directory table size (1029 bytes)
Xattr table size 55 bytes (0.05 Kbytes)
100.00% of uncompressed xattr table size (55 bytes)
Number of duplicate files found 2
Number of inodes 47
Number of files 40
Number of fragments 7
Number of symbolic links 1
Number of device nodes 0
Number of fifo nodes 0
Number of socket nodes 0
Number of directories 6
Number of ids (unique uids + gids) 1
Number of uids 1
root (0)
Number of gids 1
root (0)
Embedding ELF...
Marking the AppImage as executable...
Success
Desktop file: /builds/Calinou/godot-builds/godot/appdir/godot.desktop
/builds/Calinou/godot-builds/godot/appdir should be packaged as Godot_Engine-x86_64.AppImage
Size of the embedded runtime: 97112 bytes
Edit: I built an AppImage in an Ubuntu 14.04 VM using the same script, and the AppRun is extracted in this one: Godot_Engine-x86_64.zip
This is really strange, since the GitLab CI instance uses Ubuntu 14.04 as well.
Could this be related to the fact that AppRun is a symlink in this specific AppImage? Just guessing here.
@Calinou can you please verify whether
- ...symlinks generally work on WSL
- ...AppImages that do not contain a symlink as their AppRun file work, e.g., https://bintray.com/probono/AppImages/Leafpad/0.8.18.1.glibc2.4#files
- ...it works on WSL if you extract the non-working AppImage, and manually make a symlink from the main executable to AppRun in the squashfs-root directory
We need to find out whether there is an issue with symlinks on WSL in general, or whether the way we extract symlinks has issues on WSL.
Which filesystem are you using on Windows?
...symlinks generally work on WSL
Symlinks work in the WSL, I just tested by symlinking the Leafpad binary to a different name.
...AppImages that do not contain a symlink as their AppRun file work, e.g., https://bintray.com/probono/AppImages/Leafpad/0.8.18.1.glibc2.4#files
The AppImage works once it's extracted (the WSL does not support FUSE, so it cannot be run directly), by running the AppRun file directly. It crashes on start (since it cannot find a display), but I don't have a setup which lets me run graphical applications in the WSL (which isn't supported by Microsoft, even though it can work), so that's not relevant to this issue.
Which filesystem are you using on Windows?
I'm using NTFS.
Thanks @Calinou for testing. You can install an X server on Windows and use (extracted) GUI AppImages this way. I had done it once.
Actually I now see TODO: Implement inode.base.inode_type 10 in your output. This is a dead giveaway that we must implement this!
Actually, I get the same behavior on Linux:
me@host:~$ Downloads/Godot-Linux-x86_64.AppImage --appimage-extract AppRun
squashfs-root/AppRun
TODO: Implement inode.base.inode_type 10
me@host:~$ find squashfs-root/
squashfs-root/
So let's see what we have there:
me@host:~$ Downloads/Godot-Linux-x86_64.AppImage --appimage-mount
/tmp/.mount_Godot-b9mCtL
me@host:~$ ls -lh /tmp/.mount_Godot-b9mCtL/AppRun
lrwxrwxrwx. 1 root root 13 Oct 22 15:28 /tmp/.mount_Godot-b9mCtL/AppRun -> usr/bin/godot
So, question, what is an inode type 10... why is one in this particular AppImage, and what do we need to implement to support it. cc @TheAssassin
The same as type 3, a symlink? https://github.com/wereHamster/squashfs/blob/master/squashfs-tools/unsquashfs.c#L85
As squashfs is indeed a real file system and not "just" an archive standard, it needs to implement unix/posix filesystem entities like inodes etc. https://github.com/ddollar/squashr/tree/master/ext/squashfs3.4 gives an introduction on the different types.
The same as type 3, a symlink? https://github.com/wereHamster/squashfs/blob/master/squashfs-tools/unsquashfs.c#L85
I assume this type is the "extended" link inode type, considering the document I've linked to in my last comment. I'll have to dive into the topic a bit in order to tell what needs to be done.
@probonopd what version of squashfs are we using at the moment?
Thanks for the pointer @TheAssassin but I am not sure what to make of this:
The uid and gid indexes are 4 bits in length. Ordinarily, this will allow 16
unique indexes into the uid table. To minimise overhead, the uid index is
used in conjunction with the spare bit in the file type to form a 48 entry
index as follows:
inode type 1 - 5: uid index = uid
inode type 5 -10: uid index = 16 + uid
inode type 11 - 15: uid index = 32 + uid
@probonopd what version of squashfs are we using at the moment?
The current one, 4.3 (I think)
I still can't explain what difference it could've made when @Calinou's re-packaged on another system, because as he said, the one made in a VM and not on the CI worked fine for him. I guess we'll have to ldd our binaries and see whether some dependency that is not bundled could have made a difference.
I was able to re-package this on Ubuntu, and the re-packaged version can extract the AppRun symlink just fine:
rm -rf squashfs-root/
Downloads/Godot-Linux-x86_64.AppImage --appimage-extract
cd squashfs-root/
ln -s usr/bin/godot AppRun
cd ..
./appimagetool-x86_64.AppImage squashfs-root/
rm -rf squashfs-root/
./Godot_Engine-x86_64.AppImage --appimage-extract AppRun
ls -lh squashfs-root/AppRun
# Result:
# lrwxrwxrwx 1 me me 13 Oct 29 20:04 squashfs-root/AppRun -> usr/bin/godot
So something inside the Godot AppImage seemingly produced a "strange" (extended? whatever that is) symlink.
We should avoid trying to find a fix for that CI however, and rather attempt to implement type 10 inodes for squashfs, that'll be the cleaner solution.
There may well be other "strange" inode types that I had not implemented so far (e.g., things like fifos, hardlinks come to mind immediately).
I'd say a symlink (even if it's some weird one as here) is more common than any other kind of usual inode, such as named pipes etc.
The list here is not too long either, we should just check the unsquashfs.c we use and maybe port the feature or include a newer version.
I can reproduce this: AppImages generated using linuxdeployqt on GitLab CI in a ubuntu:trusty container say
me@host:~/Downloads$ '/home/me/Downloads/Inkscape-x86_64.AppImage' --appimage-extract
squashfs-root/.DirIcon
squashfs-root/AppRun
TODO: Implement inode.base.inode_type 10
(...)
AppRun is a symlink to the main binary, and it does not get extracted.
So it looks like we at least need to implement the extraction of inode type 10, which seem to be symlinks.
@probonopd inode type 10 is not about symlinks. If that was the case, any linuxdeployqt or linuxdeploy AppImage would yield this error.
We still haven't found out why mksquashfs sometimes creates such inodes in the image. I'd rebuild the AppImage with appimagetool and check whether the issue persists.
The AppImage in question is built using linuxdeployqt which uses appimagetool internally. What I suspect is that it depends on the filesystem on the machine on which appimagetool is running using which inode type symlinks end up in the squashfs image. But that's just speculation right now.
@probonopd that might be, yeah. Can you provide references regarding inode type 10?
If --appimage-extract doesn't work, try mount as the follows that worked for me.
mkdir /tmp/mountpoint
sudo mount -o loop your.AppImage /tmp/mountpoint
As another workaround, what happens if you get the offset with --appimage-offset and then run unsquashfs --offset ...? Possibly the squashfs implementation in libappimage is still not 100% working. This would be a good test.