beeware
Packaged Standard Library missing modules
Fact
While exploring #381, trying to package the Mu Editor on Windows, I realized that the packaged Python environment is missing modules from the Standard Library.
In particular I realized that tkinter, pip, and venv are missing.
Then I tried the same on macOS and confirmed that they're missing too.
Question
While I do understand that by creating and using an alternate support package these limitations can be overcome, I wonder if, from a user/packager experiece, it wouldn't be desirable for the packaged environment to be as capable as the environment in which the original code is developed and tested and, thus, have it be that way by default?
Thought
(warning: crazy idea ahead) :)
I wonder if instead of using a pre-built support package briefcase could instead "copy and adjust" the current Python installation into the packaging src directory. Maybe some adjustments might be needed to ensure the "copied" environment is relocatable and works nicely, but how hard could that be? (grin)
Of course, for some target platforms that is not an option given that briefcase is packaging with a foreign target like iOS targets being packaged on macOS, for example.
Thoughts?
In the windows case, the interesting detail is that we're using the stock "Windows x86 embeddable zip file" provided by Python.org. I'm guessing tkinter is omitted because the extra DLLs needed; pip and venv are omitted because the "embedded" use case shouldn't (generally) need to install extra code or set up a new environment.
The macOS (and iOS, and Android) builds that Briefcase uses are all managed by BeeWare; we've mostly followed the lead of the Windows installer in terms of omissions. In fact, having pip in the iOS binary could actually constitute grounds for ejection from the iOS App Store, as they prohibit using any mechanism to get around App Store code review.
That said: I completely agree that customizing the standard library is one of the big issues facing Briefcase.
"Provide everything" is obviously a safe default. However, it's also not a good solution - it means you end up with support packages (and thus apps) that are a lot bigger than they need to be.
I have a very slightly selfish desire to avoid packaging tkinter because it's a competitor to Toga (and my problems with Tkinter are why Toga exists); but fundamentally, I don't have any good reason why Briefcase shouldn't support tkinter.
pip and venv are another matter; there's definitely a subset of use cases (and, admittedly, Mu is one of them) for which including pip and venv make sense; but for the default "user application" case, they're "overhead with security implications".
"Relocating" an existing system install isn't an easy option, because of dependencies on external DLLs.
So - we're left with the problem of how to make it easy for end users to build and use custom support packages. This is currently supported in Briefcase by allowing you to define the support package that you want to use; but that leaves the task of actually building the support package to the end user.
We could invest in building tooling to manage that process; but the better fix might actually be to go the other direction. There's a been a little bit of discussion on the Python core team about a "Minimum Viable Python" distribution - a Python distribution that contains literally nothing in the standard library, with all the standard library specified by as dependencies. You want httplib? tkinter? pip? venv? Include it in your requirements file. There's obviously detail to sort out here, but it has the benefit of (a) being explicit, and (b) ensuring that support packages are as small as they plausibly can be - which is a benefit for the application distribution use case.
@freakboy3742,
Thanks for sharing your detailed thoughts. They are pretty much inline with what I imagined could be your ideas -- having myself been in tune with your work on BeeWare for a while now -- and also very much aligned with the "BeeWare vision", if I may call it so. Consistency is important.
I'm coming from a different background with a (maybe?) different vision of a Briefcase with a "life of its own", without betraying the needs/purposes of BeeWare -- could it be(come) the holy grail of cross-platform (and cross-target?) Python application packaging? Will there be a day when a user can just say "briefcase, go!" and have their application automatically packaged and distributed to "all platforms", regardless of their development platform? We can dream... And then we could do something, like you have been doing. :-)
I have used many Python packaging tools in the past and I like the direction Briefcase is taking, more and more (not sure about pyproject.toml, but ultimately that's an implementation detail and a whole other conversation). Congratulations + Credits + Thanks a lot to you are in order! :-)
The ability to handle arbitrary support packages is very valuable and, like you point out, lets users adjust -- the challenge in building such support packages is a whole different story. Interestingly, I just found out the Python Standalone Builds project and took the macOS variant for a quick spin (a bit reckless, downloading a running an arbitrary binary, but I publicly assume that I did just that). The image size is reasonable (~100MB) for an all batteries included package (tkinter, ssl, venv, pip, all included -- fudamental Mu requirements) and, after trivial trimming down, I brought it down to ~35MB. Not bad! While I understand the need to have custom Python builds for iOS/Android, I wonder if the efforts associated to maintaining the common platform variants (say, win/lin/mac) under the BeeWare umbrella could be merged with or, maybe better, leverage that project's work... (were you aware of it?)
I have heard of the "Minimum Viable Python" yes, thanks. That may be the future and it may even be a good idea -- I have not given it much thought, yet. But people like you and me want to do things today-ish... :-)
PS: I think I'll kick off a mini-tkinter project (no objections to Toga at all, but wanting to work within the Standard Library) to serve as an "exerciser" to both briefcase and the ability to integrate Python Standalone Builds as a support package.
More PS: One important aspect I forgot mentioning, regarding the ability to "copy and relocate" the current Python installation is, more than non-system shared lib dependencies -- which can also be copied and relocated -- the matter of distribution. One may not be allowed to do that!
But the possiblity of running Briefcase fully offline from the start is something I myself find potentially valuable, even if not easy to achieve.
On the subject of handling non-shared system libs: The AppImage target on Linux actually already does this. Part of the packaging process is inspecting binaries to determine libraries that need to be linked, copying them into the bundle, and rewriting them to reflect their new location. That approach seems to work reasonably well for Linux (well enough that an AppImage built on Ubuntu 16.04 will run under Fedora 29). So - it can be done on some platforms, and it may be possible on others (see #371 and #372 for some possible options); it's just complex.
As for running Briefcase completely offline; I completely agree that is desirable. I spend enough time on planes that this would be a useful feature to have. I know it's not currently possible because the support package download needs to be online to determine the release, but I'd definitely consider that a bug (#388).
I'll leave this ticket open as a marker for the fact that we need a generic fix for customizing support requirements. In that context, I consider venv, pip and tkinter to be completely fair game for the packaging process. Python-Standalone-Builds might be a good project to collaborate with to that end. And +1 to any effort to package tkinter as a standalone - I'd love to be able to offer Tkinter as an option in the default Briefcase template.