openvdb Publish Python library on PyPi?

Hello! I'm very excited about using OpenVDB, but I'm looking for a cross-platform, easy-to-install way of interacting with it. I have successfully compiled but it would be great to be able to install the python library like pip install pyopenvdb. I'm not sure of the challenges to do that, but I thought I would bring it up because having python packages available via pip decreases the startup cost by an order of magnitude, easily. Thoughts?

Jul 05 '17 23:07 ataber

Hey all, sorry to dig up such an old issue, but I'm interested in being able to install openvdb's python bindings as a package from PYPI as well. I'd be happy to submit a PR with the packaging necessary to publish on PYPI, if the maintainers would accept it. I can also write the github workflows to automate building and publishing the releases. Would this be something the maintainers be okay with?

Apr 13 '22 02:04 peytondmurray

Hi @peytondmurray - I think this would be great to see! I'm just not sure what exactly this looks like as I haven't worked too much with PYPI. My understanding is that you would be contributing a bunch of scripts to allow us to automate and test the publishing process of a PYPI package from within the OpenVDB repo? I see a number of pyopenvdb packages already published by the community - do you see any issues with us publishing our own with the same name?

I'll speak to the other maintainers to confirm we'd be happy to support this.

Apr 13 '22 09:04 Idclip

My understanding is that you would be contributing a bunch of scripts to allow us to automate and test the publishing process of a PYPI package from within the OpenVDB repo?

Yeah, that's right. Actually, I think there's a way to set up Github actions to automatically publish (not just test) a new version of pyopenvdb on PYPI whenever you make a new release. If we can get that working then the process of publishing to PYPI won't require any extra effort at all when it comes time to make new releases of OpenVDB.

I see a number of pyopenvdb packages already published by the community - do you see any issues with us publishing our own with the same name?

There are indeed issues with publishing our own with the same name, but I think the effort in overcoming them will be worth it. For reference, there are currently two separate projects on PYPI which show up if you search "pyopenvdb", but they're both really the same:

pyopenvdb 0.1.4 is a project for building and packaging pyopenvdb in a docker container. There's a github page which has commits from January 2021, but no PYPI release has been made in 2 years.
pyopenvdb-3.8 0.1.5 has a link to the same github page as the first result.

I'm guessing the author made both projects in the course of figuring out how to publish packages to PYPI (definitely done this myself before...). In any case, the rules governing PYPI package name transfers are clear: the first thing to do is just try to talk to the owner of the packages. There haven't been any releases on either of these projects in two years; if we email the author three times, and we don't get a response within six weeks, then the projects already meet all the requirements for being considered "abandoned." At that point I think we have a good case for transferring ownership of those PYPI packages to the official OpenVDB project maintainers.

Of course, all of that won't be necessary if we can get a hold of the owner of the projects. I hope they'll be willing to give up those projects, especially if we publish the officially supported python packages in their place.

I'll speak to the other maintainers to confirm we'd be happy to support this.

Thank you! I'd also be happy to contact the owner of the existing packages that we'd need to remove on behalf of the OpenVDB project to get the process started, but I'll leave the decision to do that up to you. Let me know how you'd like me to proceed.

Apr 13 '22 18:04 peytondmurray

Hey @peytondmurray - sorry for the delayed reply. Thanks a lot for outlining the process, this all sounds good to me! I confirmed today with the other maintainers that it would be great to pursue this and we'd happily support your proposal of an automated/CI based system for building and publishing pyopenvdb.

If you're still interested in looking at this please feel free to start the process and e-mail the author of those other published packages on our behalf. We have a public e-mail/lists [email protected] which you can include us on (all messages available here https://lists.aswf.io/g/openvdb-dev/topics) but please let me know if you need anything else!

Apr 26 '22 21:04 Idclip

Awesome, I'll get started on this. Thank you! :tada:

Apr 26 '22 21:04 peytondmurray

If you need any help I have lots of experience setting this type of thing up, hit me up! I'm also keen on getting my hands on openvdb in the python ecosystem. :)

May 03 '22 17:05 Korijn

An update on progress so far, since I know some other people have expressed interest here:

[x] Contacted the current owner of the pyopenvdb package on PyPI (Alex). Alex was super nice, very responsive, and excited that there are other people interested in python packaging for OpenVDB. He agreed to transfer ownership of the pyopenvdb package name to the OpenVDB maintainers, on the proviso that he is credited with writing the original packaging for the python bindings. I said that I thought that this would be fine - we can credit him in the README.md for the python bindings or some other appropriate place (CONTRIBUTORS.md or something, maybe?), as long as the maintainers agree
[x] Fork the repo and started a new branch for python packaging development
[x] Add README.md explaining installation, dependencies, and build toolchain
[x] Add pyproject.toml: specifies python build-time dependencies and options for builds triggered by CI
[x] Add setup.py, which makes openvdb/openvdb/python a python package that can be installed with pip. I've configured it to use scikit-build as a way of running cmake automatically during the build process, and setuptools_scm to set the appropriate package version string based on the current state of the git index
[x] Set up github CI to build and publish an sdist for the package. I'm still in the process of testing this by uploading a package to https://test.pypi.org/legacy, but I think this is basically complete. I'm still trying to get wheels to build as part of the same CI pipeline, so I've left this unchecked for now
[ ] Set up cibuildwheel to automatically build wheels for python 3.8, 3.9, and 3.10 with manylinux images. This is in progress, and a bit fiddly to get right. Unfortunately I think I'll have to build Boost 1.70 inside the container (which will be slow; there is a github action to do this, but it doesn't support Boost 1.70, our minimum version) before building OpenVDB. Only then will I be able to build pyopenvdb. If wheels are too much work to generate, we can always just distribute an sdist which should cover pretty much everyone's use case - the only advantage of distributing wheels is that they're smaller and faster to install.

Currently the github workflow on the fork is configured to trigger in two ways:

By a release, in which case it will build wheels and a source distribution for the released tag before uploading to PyPI
Manually, in which case the person who triggers the build can specify a git ref for the build

As it stands, you can go to my fork, go to the python directory and run pip install . and it should build as long as you have the correct dependencies - including Boost, OpenVDB, CMake, and the python build dependencies specified in pyproject.toml.

May 07 '22 22:05 peytondmurray