python-blosc2 icon indicating copy to clipboard operation
python-blosc2 copied to clipboard

Wheels for WASM

Open FrancescAlted opened this issue 2 years ago • 3 comments

I did some attempt at creating wheels for WASM, and had some (mixed) experiences with Pyodide and scikit-build-core:

  1. Pyodide needs version 0.23.4 (latest 0.24.1 does not work: https://github.com/pyodide/pyodide/issues/4287)

  2. CMake config in python-blosc2 needs a bit of modification in order to run (see attached patch).

  3. The included zlib-ng codec has issues with WASM compiler, so it has been removed from the list of codecs here.

  4. Some packages (notably py-cpuinfo) are not available in Pyodide repos.

  5. Some tests are passing, but others do not. E.g.:

$ python -m pytest tests/test_compress2.py
=============================================================== test session starts ================================================================
platform emscripten -- Python 3.11.2, pytest-7.4.4, pluggy-1.3.0
rootdir: /home/faltet/blosc/python-blosc2
configfile: pytest.ini
collected 24 items

tests/test_compress2.py ..............FF........                                                                                             [100%]

===================================================================== FAILURES =====================================================================
__________________________________________________ test_compress2[641091-cparams1-dparams1-True] ___________________________________________________

nbytes = 641091, cparams = {'typesize': 1}, dparams = {'nthreads': 4}, gil = True

    @pytest.mark.parametrize("gil", [True, False])
    @pytest.mark.parametrize(
        "nbytes, cparams, dparams",
        [
            (7, {"codec": blosc2.Codec.LZ4, "clevel": 6, "typesize": 1}, {}),
            (641091, {"typesize": 1}, {"nthreads": 4}),
            (136, {"typesize": 1}, {}),
            (1231, {"typesize": 4}, blosc2.dparams_dflts),
        ],
    )
    def test_compress2(nbytes, cparams, dparams, gil):
        blosc2.set_releasegil(gil)
        bytes_obj = b" " * nbytes
        c = blosc2.compress2(bytes_obj, **cparams)

        dest = bytearray(bytes_obj)
        blosc2.decompress2(c, dst=dest, **dparams)
        assert dest == bytes_obj

        dest2 = blosc2.decompress2(c, **dparams)
>       assert dest2 == bytes_obj
E       AssertionError: assert b'\x00\x00\x0...0\x00\x00\x00' == b'           ...             '
E         At index 0 diff: b'\x00' != b' '
E         Use -v to get more diff

tests/test_compress2.py:91: AssertionError
__________________________________________________ test_compress2[641091-cparams1-dparams1-False] __________________________________________________

nbytes = 641091, cparams = {'typesize': 1}, dparams = {'nthreads': 4}, gil = False

    @pytest.mark.parametrize("gil", [True, False])
    @pytest.mark.parametrize(
        "nbytes, cparams, dparams",
        [
            (7, {"codec": blosc2.Codec.LZ4, "clevel": 6, "typesize": 1}, {}),
            (641091, {"typesize": 1}, {"nthreads": 4}),
            (136, {"typesize": 1}, {}),
            (1231, {"typesize": 4}, blosc2.dparams_dflts),
        ],
    )
    def test_compress2(nbytes, cparams, dparams, gil):
        blosc2.set_releasegil(gil)
        bytes_obj = b" " * nbytes
        c = blosc2.compress2(bytes_obj, **cparams)

        dest = bytearray(bytes_obj)
        blosc2.decompress2(c, dst=dest, **dparams)
        assert dest == bytes_obj

        dest2 = blosc2.decompress2(c, **dparams)
>       assert dest2 == bytes_obj
E       AssertionError: assert b'x00\\x00\\x...x00\\x00\\x00' == b'           ...             '
E         At index 0 diff: b'x' != b' '
E         Use -v to get more diff

tests/test_compress2.py:91: AssertionError
============================================================= short test summary info ==============================================================
FAILED tests/test_compress2.py::test_compress2[641091-cparams1-dparams1-True] - AssertionError: assert b'\x00\x00\x0...0\x00\x00\x00' == b'           ...             '
FAILED tests/test_compress2.py::test_compress2[641091-cparams1-dparams1-False] - AssertionError: assert b'x00\\x00\\x...x00\\x00\\x00' == b'           ...             '
=========================================================== 2 failed, 22 passed in 0.24s ===========================================================

I don't think fixing these should be terribly complicated, but I don't have more time now. I am attaching my modifications here in case someone else would want to continue this effort.

pyodide-python-blosc2.patch

FrancescAlted avatar Jan 08 '24 13:01 FrancescAlted