rustworkx icon indicating copy to clipboard operation
rustworkx copied to clipboard

Make `rustworkx` build and run with pyiodide

Open IvanIsCoding opened this issue 9 months ago • 11 comments

Closes #703

This adds support for Pyodide. We list Pyodie in a brand new tier that is Tier Experimental. Currently, we build with pyoide-build==0.30.2 using the following flags:

export "RUSTFLAGS="-C target-feature=+atomics,+bulk-memory,+mutable-globals -C link-arg=-sSIDE_MODULE=2 -C link-arg=-sWASM_BIGINT -Z emscripten-wasm-eh"
pyodide build

This requires a Rust with nightly-2025-02-01 toolchain and emsdk 4.0.8. I tested with Pyodide 0.28.1a which maps to Python 3.13. Because we depend on so much unstable stuff, this is not ready yet for our CI. I need to figure out a way of adding it.

Related: https://github.com/pyodide/pyodide-recipes/pull/90

IvanIsCoding avatar May 10 '25 15:05 IvanIsCoding

Pull Request Test Coverage Report for Build 16921538859

Details

  • 0 of 0 changed or added relevant lines in 0 files are covered.
  • No unchanged relevant lines lost coverage.
  • Overall coverage remained the same at 94.642%

Totals Coverage Status
Change from base Build 16916606073: 0.0%
Covered Lines: 17805
Relevant Lines: 18813

💛 - Coveralls

coveralls avatar May 10 '25 15:05 coveralls

So I tried following: https://pyodide.org/en/stable/usage/building-and-testing-packages.html

I had to install emsdk 4.0.8., pyodide-build==0.30.1, rustc 1.88.0-nightly (00095b3da 2025-04-03)

I got this error: https://pastebin.com/TaT892sU. So yeah, I'd like to keep this build outside of our repository but at least add the code such that when emscripten-forge builds we run.

IvanIsCoding avatar May 10 '25 15:05 IvanIsCoding

@sorin-bolos how did you setup the SDK to build the version from your PR? I might need some help here

IvanIsCoding avatar May 10 '25 15:05 IvanIsCoding

Maybe I should try using the version pinned here: https://github.com/pyodide/pyodide-build/blob/7126c5f1bd56ed7f40ce29cbcbe588267a42434c/pyodide_build/config.py#L240C24-L240C42

IvanIsCoding avatar May 10 '25 21:05 IvanIsCoding

Somehow I got Successfully built /home/ivan/Projects/rustworkx-dev/dist/rustworkx-0.17.0-cp39-abi3-emscripten_4_0_6_wasm32.whl, but only when running with RUSTFLAGS="-v" pyodide build. I will test the wheel.

IvanIsCoding avatar May 10 '25 21:05 IvanIsCoding

Well, I can pip install using the pyodide environment but a simple import rustworkx crashes:

(.venv-pyodide) ➜  tests git:(more-uv-features) python
which: no grealpath in (/home/ivan/Projects/.venv-pyodide/bin:/home/ivan/.local/bin:/usr/share/Modules/bin:/home/ivan/.cargo/bin:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/var/lib/snapd/snap/bin:/home/ivan/.dotnet/tools:/home/ivan/.dotnet/tools)
Python 3.13.2 (main, Apr  3 2025, 07:34:46) [Clang 21.0.0git (https:/github.com/llvm/llvm-project 4775e6d9099467df9363e1a3cd on emscripten
Type "help", "copyright", "credits" or "license" for more information.
warning: can't use pyrepl: No module named 'msvcrt'
>>> import rustworkx
Pyodide has suffered a fatal error. Please report this to the Pyodide maintainers.
The cause of the fatal error was:
Error: Dynamic linking error: cannot resolve symbol invoke_ii
    at stubs.<computed> (/home/ivan/.cache/.pyodide-xbuildenv-0.30.2/0.28.0a1/xbuildenv/pyodide-root/dist/pyodide.asm.js:9:19341)
    at wasm://wasm/0122615e:wasm-function[3874]:0x3ca8c6
    at wasm://wasm/020b656e:wasm-function[4078]:0x2a30bc
    at wasm://wasm/020b656e:wasm-function[4054]:0x2a0f12
    at wasm://wasm/020b656e:wasm-function[4070]:0x2a28c5
    at wasm://wasm/020b656e:wasm-function[2181]:0x1ad753
    at wasm://wasm/020b656e:wasm-function[1119]:0x16345c
    at wasm://wasm/020b656e:wasm-function[1121]:0x163668
    at wasm://wasm/020b656e:wasm-function[1122]:0x1636e6
    at wasm://wasm/020b656e:wasm-function[3519]:0x25a95b {
  pyodide_fatal_error: true
}
Stack (most recent call first):
  File "<frozen importlib._bootstrap>", line 488 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 1320 in create_module
  File "<frozen importlib._bootstrap>", line 813 in module_from_spec
  File "<frozen importlib._bootstrap>", line 921 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1331 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1360 in _find_and_load
  File "/home/ivan/Projects/.venv-pyodide/lib/python3.13/site-packages/rustworkx/__init__.py", line 14 in <module>
  File "<frozen importlib._bootstrap>", line 488 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 1026 in exec_module
  File "<frozen importlib._bootstrap>", line 935 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1331 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1360 in _find_and_load
  File "<stdin>-0", line 1 in <module>
  File "/lib/python313.zip/_pyrepl/main.py", line 30 in interactive_console
  File "/lib/python313.zip/_pyrepl/__main__.py", line 6 in <module>
  File "<frozen runpy>", line 88 in _run_code
  File "<frozen runpy>", line 198 in _run_module_as_main
Error: Dynamic linking error: cannot resolve symbol invoke_ii
    at stubs.<computed> (/home/ivan/.cache/.pyodide-xbuildenv-0.30.2/0.28.0a1/xbuildenv/pyodide-root/dist/pyodide.asm.js:9:19341)
    at wasm://wasm/0122615e:wasm-function[3874]:0x3ca8c6
    at wasm://wasm/020b656e:wasm-function[4078]:0x2a30bc
    at wasm://wasm/020b656e:wasm-function[4054]:0x2a0f12
    at wasm://wasm/020b656e:wasm-function[4070]:0x2a28c5
    at wasm://wasm/020b656e:wasm-function[2181]:0x1ad753
    at wasm://wasm/020b656e:wasm-function[1119]:0x16345c
    at wasm://wasm/020b656e:wasm-function[1121]:0x163668
    at wasm://wasm/020b656e:wasm-function[1122]:0x1636e6
    at wasm://wasm/020b656e:wasm-function[3519]:0x25a95b {
  pyodide_fatal_error: true
}

IvanIsCoding avatar May 10 '25 21:05 IvanIsCoding

I think setting verbose on the previous comment erased the flags from pyodide build so we are back at step one.

IvanIsCoding avatar May 10 '25 22:05 IvanIsCoding

Ok, we definetely build. Any code that calls Rayon fails, but:

  1. https://gist.github.com/IvanIsCoding/8bb97d428169e073af4afa598e1d2932 has the code of a index.html page that calls Floyd-Warshall
  2. Download index.html from the gist and https://github.com/IvanIsCoding/rustworkx/releases/download/v0.17.0-alpha-pyodide/rustworkx-0.17.0-cp39-abi3-pyodide_2025_0_wasm32.whl, put the files in the same folder
  3. Run python -m http.server --directory . in the same folder to serve a very simple HTTP server
  4. Open http://localhost:8080

It works! The issues inside the pyodide env seem to only affect Node.js, inside a browser it works like a charm: works_inside_the_browser

IvanIsCoding avatar May 11 '25 03:05 IvanIsCoding

After telling the nightly compiler to rebuild std, I built https://github.com/IvanIsCoding/rustworkx/releases/download/v0.17.0-alpha2-pyodide/rustworkx-0.17.0-cp39-abi3-pyodide_2025_0_wasm32.whl which seems to be working.

I will try to see if I figure out how to run our unit tests to confirm things work. But this is shockingly good.

IvanIsCoding avatar May 11 '25 04:05 IvanIsCoding

I have no clue how to include this on CI or even if we should, but there more I test the more things seem to work. I will open an issue on the pyodide-build repository and make a recipe on the pyodide-recipe repository.

IvanIsCoding avatar May 11 '25 16:05 IvanIsCoding

Overall this looks fine to me, Just one nit inline. I'd like to try to find time soon to pull this locally and test out wasm and pyodide before approving though just to validate all the new options.

Some minor notes about Rust WASM. There are three Rust targets! We could support more, but here are the caveats:

  • wasm32-unknown-unknown: easiest target to build for rustworkx-core. impossible to build rustworkx because of Python. cargo doesn't have a default runner for this so it's hard to test.
  • wasm32-unknown-emscripten: used by Pyodide. emscripten goes from LLVM -> WASM from what I understood, so there needs to be some coordination between emscripten and rustc versions. I higly recommend using Pixi from #1450 to build it
  • wasm32-wasip1: in theory Python supports this, but I have yet to see someone building and publishing for WASI

After #1450 lands, I think we could test wasm32-unknown-unknown for rustworkx-core with https://rustwasm.github.io/wasm-pack/

IvanIsCoding avatar May 29 '25 12:05 IvanIsCoding