klayout icon indicating copy to clipboard operation
klayout copied to clipboard
verified publisher icon KLayout

macbuild's heavyweight homebrew python build is not patching pip correctly

Open thomaslima opened this issue 5 years ago • 7 comments

First of all, sorry for being away all this long. It's been a very tough year for me.

I appreciate the new routines present for building klayout with homebrew's binaries for python. I was looking at what it takes to create a brew formula to build klayout's application against the most recent qt and python. It is surprisingly straightforward to create such a formula, actually, and it might be a more convenient way to deploy klayout for experimental users like me. I think it could also automate make the mac-brew-based CI builds more robust.

Anyway, the new versions of pip changed the internal location of the code for installing packaged. They really don't want pip to be used as a module. I.e. running pip.main(['install', 'numpy']) from within klayout. The reason is that they want to quit the python process after being done... I knew that but took the risks anyway, after all I can always reboot klayout after installing packages.

That aside, pip install is currently putting packages in ~/Library/Python/3.8/lib/python/site-packages instead of klayout.app/Contents/MacOS/python, which was the original intention. I think this is because the sys.path seen from within klayout contains the home Library before the klayout.app's location. For example:

>>> sys.path
['/usr/local/opt/[email protected]/Frameworks/Python.framework/Versions/3.8/lib/python38.zip', '/usr/local/opt/[email protected]/Frameworks/Python.framework/Versions/3.8/lib/python3.8', '/usr/local/opt/[email protected]/Frameworks/Python.framework/Versions/3.8/lib/python3.8/lib-dynload', '/private/tmp/klayout-test-20201022-16428-1yr8l4j/klayout-homebrew-formula/.brew_home/Library/Python/3.8/lib/python/site-packages', '/usr/local/opt/[email protected]/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages', '/private/tmp/klayout-test-20201022-16428-1yr8l4j/klayout-homebrew-formula/HW-qt5Brew.pkg.macos-Catalina-release-RsysPhb38/klayout.app/Contents/MacOS/python', '/private/tmp/klayout-test-20201022-16428-1yr8l4j/klayout-homebrew-formula/HW-qt5Brew.pkg.macos-Catalina-release-RsysPhb38']

I don't know how to resolve this yet and which option is better for the user. I like the idea of containerizing klayout's python environment as much as possible and not let it install packages in the home user folder. But some users might want to get their system-wide or user-installed packages and use within klayout, especially those who are installing klayout from homebrew. I, for example, use some scipy's optimization algorithms to compute the best curve paths before laying out a trace. So I would need scipy available if I was running from within the macros.

Given the above, I think I would like to improve the "heavyweight" build in two ways:

  1. nail down how the sys.path, and site.getsitepackages() is behaving from within klayout.
  2. add an option during build to select the python package locations available (default to isolated packaging).
  3. right now sys.executable from klayout returns klayout.app/Contents/MacOS/klayout which prevents me from instantiating a subprocess like subprocess.check_call([sys.executable, '-m', 'pip', 'install', 'my_package']). I would like to: a. ship a python executable within the klayout.app application, which loads with the same environment as klayout b. point sys.executable from within klayout to that new python binary.

I like the idea of the "lightweight" build as well, but I haven't managed to try it yet. I would like to use this build option for the brew install klayout, which will install the two other prereqs formulas qt and python3, and build klayout from source or download a prebuilt "bottle". For this, we need to customize the build4mac.py so that it can take some configuration variables that include file paths, instead of global "HB38" or "Qt5Brew" strings followed by hardcoded filepaths. I would like to pursue this in partnership with @Kazzz-S because I would like to respect the current coding style.

thomaslima avatar Oct 23 '20 16:10 thomaslima

@thomaslima,

Welcome back, the strong reinforcement :-) Thank you for your proactive suggestion. I believe your idea, once available, will be beneficial for Homebrew users.

Kazzz-S

Kazzz-S avatar Oct 25 '20 02:10 Kazzz-S

Dear @Kazzz-S

I have made some progress in refactoring build4mac.py and makeDMG4mac.py. I would like to invite you to take a look at my changes at this stage (you can find them in the "work in progress" #680 pull request). I am hoping that the changes did not impact the default behavior of build4mac, but I would like you to double check.

My goal is to be able to craft a custom main() function that intercepts and changes a few parameters between build steps. In particular, I would like to directly specify the locations of the qt, ruby, and python libraries with paths provided by brew formulas. This way, when the user installs with brew install we can use the build4mac script to create a bottle that links to the user's current brew-installed libraries.

I am planning to finalize the PR #680 after your review, and start another one looking at the other items in this issue.

thomaslima avatar Nov 16 '20 19:11 thomaslima

Dear @thomaslima,

Thank you very much for your effort in making the build system simpler and more versatile. I've already merged your first pull request. However, I faced an issue with deploying the binaries (for LW-Homebrew / Catalina). Just now, I have merged the second one. Sure. I'll check, test, and modify if needed, the updated build system. Regrettably, I cannot promise a due date since I'm busy these days. Please wait for some time patiently.

BTW, what do you think of supporting Big Sur? I'm not brave enough to update my main PC spontaneously. You have already added if release == 20: for BigSur, which is fine:-) Since I purchased my Intel Mac just one year ago, I will have no chance to buy a new machine for years. So, I've set up a virtual machine with VMware Fusion for BigSur and tested some DMGs built for Catalina. The results are as below.

  1. HW- Homebrew (with Qt 5.15.1) works
  2. ST- (with Qt 5.14.2 from MacPorts) does not start.
  3. ST- (with Qt 5.15.1 from Homebrew) works (OS-bundled Python is still 2.7!!!)

Regards, Kazzz-S

Kazzz-S avatar Nov 16 '20 22:11 Kazzz-S

@Kazzz-S Take your time. This is opensource! :)

I have changed the hardcode directory for Catalina SDK (changed from 10.15 version of xcode to 11.0). This might have caused your issue, perhaps.

I have added the line for BigSur but have not tried it yet. I do not have any VM and don't plan to install it in my computer because of hardware constraints. However, Github Actions and Microsoft Azure both offer Big Sur preview images, so we can build from there.

thomaslima avatar Nov 17 '20 03:11 thomaslima

@kazzz and @thomaslima ... I need to admit I do not fully understand what's going on (lacking MacOS experience), but I love the way you do this together! :)

Thanks a lot!

I'll prepare 0.26.9 for Windows and Linux now. I have added two more (small) bug fixes in between, so it's worth releasing.

Best regards,

Matthias

klayoutmatthias avatar Nov 27 '20 17:11 klayoutmatthias

I have figured out item number 1 and 3 in the tasklist. Bringing python's documentation https://docs.python.org/3.6/library/site.html to the context of this issue, the installed packages location is based on sys.prefix / lib / pythonX.Y / site-packages, which in the HW distribution resolves to klayout.app/Contents/MacOS/../Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages. If we need to install packages, it needs to go to that folder. So I have decided to include a python binary in klayout.app/Contents/MacOS/python that has the same sys.prefix as the one above. This binary can be used with python -m pip to install pypi packages within site-packages.

Note: Of course we can install the packages to alternative locations, like '.klayout/python', or anywhere else included in sys.path, but the issue there is that when klayout is upgraded or reinstalled with a different python minor version, the installed packages will potentially fail or crash, and will need to be reinstalled by hand.

sys.prefix is based on the linked library path within the klayout binary. The only other way to manipulate it is to include a pyvenv.cfg file one folder above the binary, i.e. klayout.app/Contents/, which becomes the new sys.prefix. This is not particularly desirable because it messes with the klayout.app structure.

In order to do that from within klayout, we need to tell klayout where is that python binary. I chose to replace the variable within python that usually tells the interpreter which binary initiated it: sys.executable. This is how many scripts on github calls a python subprocess to do other routines, like pip install.

At this point, I'd like to ask @klayoutmatthias what can be done in windows and linux's builds to replicate this behavior. How can we install python packages that would be available to klayout's macro environment. I believe it is possible to place a copy of the python binary with which klayout was built in a predictable location and patch sys.executable somehow, for windows, linux and mac.

thomaslima avatar Nov 28 '20 18:11 thomaslima

Some useful information, specifically how sys.executable is derived from CPython's source code. My recommendation is to manipulate sys.executable within klayout's source code during startup, pointing to the correct location in windows, mac, or linux. We can set _base_executable to the python binary with which klayout was built, and executable to the modified location based on whether python was shipped with klayout or not. https://stackoverflow.com/questions/22236727/mismatch-between-sys-executable-and-sys-version-in-python

thomaslima avatar Nov 30 '20 17:11 thomaslima