Verify in CI that app runs well on each OSes

[lucydodo]

Describe the new feature

Often, after a modification is made to the code, issues are reported that prevent the app from running on certain OSes.

• Related
  • #3582
  • #3625
  • #3651
  • #3654
  • #3659

So I try to see if we can make improvements so that the CI system can confirm this. I'm hoping it's as simple as checking the exit-code that occurs after launching the app.

Does this feature exist in another product or project? Please provide a link

Not applicable

Do you have a screenshot? Please add screenshots to help explain your idea.

Not applicable

[lucydodo]

FYI: I'm assigning a high priority to the next release coming next week. Therefore, this improvement will be attempted after the release the next stable version.

[justinclift]

Maybe some kind of automated "does the application start?" test after every build?

It'd need to be done in a separate container/vm/something than did the build though, to make sure having the development libraries installed on the same system doesn't give a misleading result.

Have seen that exact problem happen with Qt libraries, where testing on the build system itself can hide packaging problems.

[lucydodo]

Can you elaborate a little more? :)

[lucydodo]

Here's how I was originally thinking of doing it.

Run a GitHub-hosted runners with multiple OSes simultaneously as a matrix. And run the following (pseudo-code) script:

run sqlitebrowser
if (exit_code == 0)
    success
else
    fail

[justinclift]

Yeah, that approach sounds like a decent idea. :smile:

Can you elaborate a little more? :)

Yep. At least on macOS, if you compile an application with the Qt development pieces already installed then the various paths (eg PATH, LD_LIBRARY_PATH, (etc)) of the development libraries can give misleading results.

For example, when a person installs then runs DB4S on their standard macOS computer it'll try loading the Qt libraries from inside of the DB4S package installed under /Applications.

If the DB4S package was accidentally made badly (ie bug during packaging), then the Qt libraries in the package might not actually be present, or might be present in the wrong sub directory (etc). When the user tries to launch it, an error will be generated and the user will commonly let us know there's a problem to investigate.

---

Now, if that same broken package is instead launched on a computer that has the Qt development libraries installed, then it wouldn't immediately fail if the Qt libraries are missing from the package. Instead, it'll use the Qt libraries from the Qt development install then launch as per normal.

So, it'd look like DB4S was working as intended and the package was all good. Except it's not.

So doing testing of the DB4S packages on the dev system they're built on isn't a foolproof test, and your approach of doing the tests in a matrix (ie new VM per OS test) is a good one that sounds like it'd avoid that problem.

---

Hopefully the above makes sense. Only mentioned it due to hitting this exact problem when originally creating some of the macOS packaging scripts. :smile:

[lucydodo]

Ah, I understand perfectly. As you said, I will make that test it on a Runner image without Qt installed. :)