CPack: Refactor AppImage and Apple DMG Generation

[tresf]

Features:

• Simplifies Mac and Linux packaging command to simply: make package

  - make install && make appimage
  - make install && make dmg
  + make package
  # -- AppImage create: /__w/lmms/lmms/build/lmms-1.3.0-alpha.1.638+pr7252.gaaa9b4816-linux-x86_64.AppImage
  

• Adds new WANT_DEBUG_CPACK flag to easily show detailed console messages about packaging

  cmake .. -DWANT_DEBUG_CPACK=true
  

    Developer options
    -----------------------------------------
    * Debug FP exceptions               : Disabled
    * Debug using AddressSanitizer      : Disabled
    * Debug using ThreadSanitizer       : Disabled
    * Debug using MemorySanitizer       : Disabled
    * Debug using UBSanitizer           : Disabled
  + * Debug packaging commands          : Enabled
  

  • Preserves old CPack TGZ behavior through a hidden flag, WANT_CPACK_TARBALL. Does anyone use this?
• Adds new CPACK_TOOL parameter to switch between .AppImage (default) and .run (experimental).

TODO:

• [x] Investigate possibly using the provided Mac DMG generators.
  • [x] Port the install_apple.sh script to CMake.
    It's called MacDeployQt.cmake now.
• [x] Investigate possibility using CPack for creating Linux AppImages
  • [x] Port the package_linux.sh script to Cmake
    It's called LinuxDeploy.cmake now.
• [x] Upgrade CMake for Linux runners

BUGS:

• [x] Fix background missing for DMG
• [x] Fix AppImages not building for Linux
• [x] WANT_DEBUG_CPACK is incorrectly toggled in CI. https://github.com/LMMS/lmms/actions/runs/9131345271/job/25110338288. Closed via https://github.com/LMMS/lmms/pull/7252/commits/82a3aa6370b434c445bc048d101b57ac7b1d6aea.

STRETCH GOALS:

• [x] Port project from linuxdeployqt to linuxdeploy
  Examples:
  • pbek/QOwnNotes (thank you!)
  • https://github.com/wireshark/wireshark/commit/34ce99f32e3d478ebf2d4c9e24a10b3fe5ef4ef6 (thank you!)
• [x] Remove all references to linuxdeployqt (in favor of linuxdeploy)
• [ ] Add ARM64 AppImage
  • https://github.com/linuxdeploy/linuxdeploy/issues/258

Background:

Click to expand

• The AppImage and Apple DMG installers historically (confusingly) require make install to be run prior to making packages. This PR is a proposal to remove that by switching to CPack.
  • Utilizes make package now, just like Windows does
  • Mac utilizes CPACK_GENERATOR of ~~DragNDrop~~ Bundle which will create a DMG without the need for node's appdmg package (still requires macdeployqt)
  • Linux will utilize CPACK_GENERATOR of "External", which will use custom CMake commands to replace package_linux.sh (still requires linuxdeployqt).

[tresf]

Rule out possibly using the provided Mac DMG generators.

I was able to get the CPack DMG generator to work with macdeployqt however in order for us to brand our own DMG, we either need to leverage AppleScript (which is much slower and requires user interaction over appdmg), or we need to provide a premade .DS_Store file to bundle.

Edit: For simplicity, I've checked in the .DS_Store from our current DMG. Since it will be unobvious how to update this file in the event of a rebranding, I've kept basic support for appdmg for recreating it, although it's not used or needed unless we rebrand the installer.

[tresf]

Hmm... the AppImage is missing... investigating...

[tresf]

The AppImage is missing because the CPack feature for running custom scripts through variable CPACK_PRE_BUILD_SCRIPTS was not available in Ubuntu 20.04. There doesn't seem to be a replacement for this feature, which is alarming, I wonder how people used "External" packaging techniques prior to CMake 3.19. Hmm...

[midi-pascal]

Perhaps I have a hint for this:

I use Ubuntu 20.04 that comes with cmake version 3.16 BUT Qt (5.15.2 installed from Qt site) comes with cmake 3.27 in its tools directory. This is the one I use.

[tresf]

I use Ubuntu 20.04 that comes with cmake version 3.16 BUT Qt (5.15.2 installed from Qt site) comes with cmake 3.27 in its tools directory. This is the one I use.

Thanks! Any recommendation for getting this into a CI (command line?)

[midi-pascal]

Not sure of what I will suggest below since I build using Qt Creator(so all is set automagically) but there are environment variables related to cmake:

CMAKE_PREFIX_PATH

CMAKE_PREFIX_PATH Semicolon-separated list of directories specifying installation prefixes to be searched by the find_package(), find_program(), find_library(), find_file(), and find_path() commands. Each command will add appropriate subdirectories (like bin, lib, or include) as specified in its own documentation. By default this is empty. It is intended to be set by the project. There is also an environment variable CMAKE_PREFIX_PATH, which is used as an additional list of search prefixes. See also CMAKE_SYSTEM_PREFIX_PATH, CMAKE_INCLUDE_PATH, CMAKE_LIBRARY_PATH, CMAKE_PROGRAM_PATH, and CMAKE_IGNORE_PATH.

Hope this can help :-)

[PhysSong]

FYI, CMake homepage lists the official PPA and pip package as alternative installation methods.

[messmerd]

@tresf I could update the Linux image with a newer CMake using the official PPA if you want. That might be easiest and also won't impact build times

[tresf]

@tresf I could update the Linux image with a newer CMake using the official PPA if you want. That might be easiest and also won't impact build times

If parties agree on this CPack strategy, yes please (or from snap).

Tested on Ubuntu 20.04:

sudo snap install cmake --classic
alias cmake='snap run cmake'
# cmake --version
# cmake version 3.29.3
#
# CMake suite maintained and supported by Kitware (kitware.com/cmake).

... I had reservations about whether or not the snap version of cmake would play nicely... and it seems to work just fine.

[tresf]

@tresf I could update the Linux image with a newer CMake using the official PPA if you want. That might be easiest and also won't impact build times

@messmerd yeah, I'll take that offer. 🤣

[Rossmaxx]

can we use the PPA for mingw ci image too?

[messmerd]

yeah, I'll take that offer. 🤣

I'll try to do it sometime today

can we use the PPA for mingw ci image too?

Yes, I think so, though I don't think it will be needed for this PR

[tresf]

Yes, I think so, though I don't think it will be needed for this PR

Actually, we can justify it here since bad79a8.

[tresf]

I've marked this ready for review because I think it's in pretty good shape. (artifacts should start generating once cmake is updated).

https://github.com/LMMS/lmms/assets/6345473/de5565a8-776a-4264-8cfd-8fa825054034

[tresf]

Runners are fixed.

As it turns out, the CIs don't have FUSE installed. This wasn't obvious at first because on master, we're side-stepping this dependency by extracting linuxdeployqt before using it to expose appimagetool(blindly copied into this PR):

https://github.com/LMMS/lmms/blob/3bf662b230f3414db22bfd07e358ee810cf5e7f9/cmake/linux/LinuxDeployQt.cmake#L47-L51

... however, this isn't needed for the new linuxdeploy (without the qt) tool; I really feel this is overkill for the average Desktop user, so I've done my best to detect if FUSE is installed in the kernel 8dfb3a9. (well, technically if one of the common FUSE tools is installed).

Some initial feedback from @ok cheez on Discord about this build (edited/formatted for readability)

Here's some good news so far:

• it runs on Puppy Linux with the --allowroot ting. [...]
• 32-bit VST support seems stable thus far. Tested with the latest version of the GUI-less 1bitstudio plugins.
• MeowSynth works somewhat.
• 32-bit VSTi support seems solid so far
• 32-bit VST effects are OK.
• Pi 400 works [running the ARM64 AppImage]

I think this PR is ready for nitpicking (flow, comments, etc). Any and all feedback is greatly appreciated.

[tresf]

Basic .run (via makeself) support has been added via 177cd21. This is experimental and should not be offered to our downloads page until it has been curated, but this allows a user-wide or system-wide installation (unlike the AppImage, which uses a FUSE filesystem each time the application is run). .run installers also are much more likely to work on limited systems (e.g. those without FUSE available).

Currently installs to ~/opt/lmms or /opt/lmms if run as ordinary user or root user, respectively, but this can be changed.

[tresf]

~~After rebasing, I'm having trouble with the ARM64 DMGs. They're much harder to get working using CPack's built-in tools (versus node's appdmg utility which is working fine on master branch). I may have to revert to appdmg. If so, MacDeployQt will need a bit of refactoring.~~

Edit: DMGs on ARM64 are fixed, but the background image is missing again, likely due to the .DS_Store file :/

[tresf]

Yeah, I can't seem to get the background image to work with CMake's DMG generator. I tried to copy the .DS_Store from a working DMG and the exact ./background/background.tiff file but it's always a blank background. Mozilla has a nice write-up on the .DS_Store file format here: https://wiki.mozilla.org/DS_Store_File_Format, but nothing stood out to me as easily patchable. I can confirm with vbindiff that there are slight differences between the working .DS_Store and the non-working .DS_Store, but I think fixing this on a binary level is reinventing the wheel that appdmg has already invented.

I'll have to revert the MacDeployQt script back to using appdmg.

[tresf]

Basic .run (via makeself) support has been added via 177cd21.

Testing on Discord shows the following error, investigating...

- ./lmms-1.3.0.run: /home/ubuntu/lmms2/build/cmake/linux/setup.sh: inaccessible or not found

[tresf]

I'll have to revert the MacDeployQt script back to using appdmg.

Done via ddca4e7.

[tresf]

Due to unreliability with FUSE (availability, compatibility), I'm changing this PR to always extracting AppImages before running them. Quoting #dev-only on Discord:

On the cpack branch, I'm making an executive decision to just never trust AppImages will ever work without extraction. So far:

• :white_check_mark: Ubuntu 20.04 Desktop
• :no_entry_sign: Ubuntu 22.04 Desktop
• :no_entry_sign: Ubuntu 20.04 Server
• :no_entry_sign: Ubuntu 22.04 Server

This will be transient to devs but I just don't see a good reason to ever try to run the AppImage from FUSE when the majority of systems just seem to be missing it or have problems with the version.

Note, master does this already, but for different reasons.

At a glance, this may be a bit confusing, because the build system will create a symlink with the AppImage name. e.g. linuxdeploy-x86_64.Appimage --> .linuxdeploy-x86_64.AppImage/squashfs-root/AppRun, but this allows developers to call the binary by its expected name without having to know where the AppRun was placed. Hopefully this change is percevied as intuitive and not too confuscated™️ . 🍻

[messmerd]

On Linux, Carla Rack and Carla Patchbay crash when I try to use them, and this is not the case on master

EDIT: Here's the command-line output:

Carla appears to be installed on this system at /usr/lib[64]/carla so we'll use it. Jack appears to be installed on this system, so we'll use it. Lv2 plugin SUMMARY: 18 of 40 loaded in 80 msecs. For details about not loaded plugins, please set environment variable "LMMS_LV2_DEBUG" to nonempty. /tmp/.mount_lmms-1XSszgf/AppRun.wrapped: line 24: 800103 Segmentation fault (core dumped) QT_X11_NO_NATIVE_MENUBAR=1 "$DIR"/usr/bin/lmms "$@"

[tresf]

On Linux, Carla Rack and Carla Patchbay crash when I try to use them, and this is not the case on master

Thanks! Should be fixed via https://github.com/LMMS/lmms/pull/7252/commits/aab4cc5bf8ffe2149ea7f7e1774d3731a1dbea20.

[tresf]

On Linux, Carla Rack and Carla Patchbay crash when I try to use them, and this is not the case on master

Thanks! Should be fixed via aab4cc5.

Nevermind, still crashes. Investigating...

[messmerd]

@tresf There are 119 files and 2 directories under squashfs-root/usr/lib in this PR, but just 92 files and 1 directory in master.

I'm not sure why there are so many more, but of those extra files, there were libcarlabase.so and libcarla_native-plugin.so. After I removed them, Carla now longer crashed LMMS.

[tresf]

@tresf There are 119 files and 2 directories under squashfs-root/usr/lib in this PR, but just 92 files and 1 directory in master.

I'm not sure why there are so many more, but of those extra files, there were libcarlabase.so and libcarla_native-plugin.so. After I removed them, Carla now longer crashed LMMS.

@messmerd thank you. I'm fully aware of the cause of the crash. linuxdeploy is run twice. First using a qt plugin, which we first use to crawl dependencies and create the .AppDir structure, but then we call it a second time passing the --output appimage flag which we call only after manual cleanup of these carla libraries is performed. Instead of being a replacement for the appimagetool command, it re-runs the dependency tree, so any manually removed files are copied back in. Here's the offending line:

https://github.com/LMMS/lmms/blob/22f17e09535b9d05eb54af46c18daa25e0c356e2/cmake/linux/LinuxDeploy.cmake#L244-L250

There's a new --exclude-library flag which takes a globbing pattern that I'm experimenting with to see if it can replace the manual cleanup efforts.

[tresf]

Apologies for the whitespace changes in the latest commit 508f430, but spaces and tabs were inconsistent.

The relevant fix is the addition of ${EXCLUDES} variable which adds --exclude-library=pattern. It could be argued that the glob should be provided to both calls to linuxdeploy and the manual cleanup effort removed (rather than having two steps 1. cmake do the glob, 2. provide any found filenames to the --exclude-library flag) but at this point I'm afraid to change too much in fear of causing other issues.

Another big miss was never providing linuxdeploy the -executable= flags. We were building it but never passing it to linuxdeploy. This was would likely have caused some linking issues if not fixed prior to merging. (ignore the double dash change, that's just for readiblity).

There are 119 files and 2 directories under squashfs-root/usr/lib in this PR, but just 92 files and 1 directory in master.

I haven't done a comparison yet, but one thing that linuxdeploy does that linuxdeployqt did not is to attempt to fetch the licensing information for all dependencies. Most of these attempts fail when verbose logging is turned on, but it's one possible culprit. The second directory is a qt wrapper script exclusive to this tool.

I haven't tested the AppImage yet but I believe the Carla loading issue is at least resolved. I'm curious what the directory tree compare looks like. I used to use a commercial tool called "Beyond Compare" for comparing file trees, perhaps there's a FOSS equivalent.

[messmerd]

I've confirmed that Carla no longer crashes.

However, it looks like there are copies of all the LMMS plugins and some other .so files in both usr/bin and usr/lib/lmms instead of just the latter. And for at least the past couple commits, the LMMS icon hasn't been showing up on the AppImage file.

As a side note, I just realized xpressive is a whopping 71 MB. It's also always last to finish compiling in our CI builds, so maybe something should be done about that. (Though not in this PR of course)

[tresf]

Although undocumented, it appears appimagetool is available as part of the linuxdeploy tool, as can be observed when verbose logging WANT_DEBUG_CPACK is enabled. I may attempt to use this directly. I'll have to put a few more hours into it.