google
Adding a Python API
Hi @mmoffitt,
Thanks for your excellent work on this allocator. Reading your paper was a lot of fun!
I wanted to experiment with your allocator, particularly to embed it in a larger compiler flow and benchmark it against other allocators I've been using. For this, I needed a Python interface (as many ML compiler workflows are Python-based), so I went ahead and created one using pybind11. You can find the initial version here. While it's still somewhat rough, I've ported all the tests to pytest and added examples and a Python CLI.
With this, one can now:
pip install minimalloc
and use it like this:
import minimalloc as mm
problem = mm.from_csv_file("benchmarks/examples/input.12.csv")
problem.capacity = 12
solver = mm.Solver()
solution = solver.solve(problem)
print(solution) # Solution(offsets=[8, 8, 4, 4, 0])
Since the Python interface is just a wrapper, the performance is identical to the C++ interface. I've added a small script that uses both the C++ and Python CLIs to verify identical results and comparable performance:
(venv) minimalloc$ bash scripts/run-challenging-compare.sh
benchmarks/challenging/A.1048576.csv [Python] Elapsed time: 4.011s [C++] Elapsed time: 4.046s PASS
benchmarks/challenging/B.1048576.csv [Python] Elapsed time: 2.791s [C++] Elapsed time: 2.776s PASS
benchmarks/challenging/C.1048576.csv [Python] Elapsed time: 0.528s [C++] Elapsed time: 0.540s PASS
benchmarks/challenging/D.1048576.csv [Python] Elapsed time: 2.769s [C++] Elapsed time: 2.757s PASS
benchmarks/challenging/E.1048576.csv [Python] Elapsed time: 2.849s [C++] Elapsed time: 2.859s PASS
benchmarks/challenging/F.1048576.csv [Python] Elapsed time: 4.609s [C++] Elapsed time: 4.641s PASS
benchmarks/challenging/G.1048576.csv [Python] Elapsed time: 0.977s [C++] Elapsed time: 0.975s PASS
benchmarks/challenging/H.1048576.csv [Python] Elapsed time: 1.407s [C++] Elapsed time: 1.421s PASS
benchmarks/challenging/I.1048576.csv [Python] Elapsed time: 3.026s [C++] Elapsed time: 3.031s PASS
benchmarks/challenging/J.1048576.csv [Python] Elapsed time: 1.087s [C++] Elapsed time: 1.093s PASS
benchmarks/challenging/K.1048576.csv [Python] Elapsed time: 0.425s [C++] Elapsed time: 0.430s PASS
All tests passed - Python and C++ APIs produced identical results
I wanted to ask whether you'd be interested in upstreaming this to your project. If so, I'd be happy to clean up the PR, split my changes into smaller, more reviewable PRs, and help maintain the Python bindings going forward.
This would enable many opportunities down the line, such as publishing to PyPI, making your work easily accessible to a much wider audience for integration into their compilers and experiments. For example, an ETH Zurich project called Deeploy has integrated your allocator in their Python code. But they currently have to go through CSV files.
Thanks again for your work and for making it available to the community.
Best, Fabian
This sounds fantastic, I love it. Let me know if there's anything I can do to help get this idea off the ground!
Hey @mmoffitt,
Thanks a lot for merging the changes and for being so welcoming to contributions!
In regard to your comments here https://github.com/google/minimalloc/pull/16#issuecomment-3349624102: With these changes in, we can now build and use the Python package locally. This enables people to pip install minimalloc/, but only if they have previously downloaded/cloned the repo git clone --recursive [email protected]:google/minimalloc.git. If we wanted to make this a true 'hands off' experience, i.e. enable people to "just" pip install minimalloc, we'd need to publish the package on PyPI. If you are okay with that, I am happy to look in to it. There are some minor additions needed to the pyproject.toml, but the rest should be pretty straightforward.
I also wanted to ask you if you'd be interested in setting up some github actions, such that the tests and some formatting checks can be run automatically on new PRs. If we set this up, we could also automate the PyPI publishing, similar to how they have done here https://github.com/google/tunix/blob/main/.github/workflows/pypi_release.yml, for example.
I went ahead and did a test upload to TestPyPI https://test.pypi.org/project/minimalloc/0.1.0/. Let me know what you think. With this, you can now do a truly "hands-off" pip install without having to clone this repo. Just do pip install -i https://test.pypi.org/simple/ minimalloc (the -i ... is still needed because we have only published to the test index, not the real one). If you are okay with this, then we can do the same to the official PyPI. If not, I will delete the TestPyPI entry again.