zarr-developers
Blosc fails to import using PyPy
Minimal, reproducible code sample, a copy-pastable example if possible
from numcodecs import Blosc # executed via PyPy
Problem description
The latest numcodecs package (0.6.4) installs successfully via pip in a pypy-based venv. I see no errors or warnings when installing zarr (or when installing numcodecs independently) using -vvv. However, importing Blosc throws the exception: ImportError: cannot import name 'Blosc' (similar errors are thrown from Zarr when opening a Blosc-compressed file).
I assume this is related to an incompatibility with PyPy, but I couldn't find any documentation about using PyPy with Zarr.
Version and installation information
Please provide the following:
- Value of
numcodecs.__version__: 0.6.4 - Version of Python interpreter:
Python 3.6.9 (1608da62bfc7, Dec 23 2019, 10:50:17)
[PyPy 7.3.0 with GCC 4.2.1 Compatible Apple LLVM 10.0.0 (clang-1000.11.45.5)]
- Operating system (Linux/Windows/Mac): macOS 10.15.5
- How NumCodecs was installed (e.g., "using pip into virtual environment", or "using conda"): pip into venv
I don't know that any explicit work has been done on PyPy support. So this might not work.
That said, components like Blosc depend on a working compiler to be available. On macOS this means installing Xcode and Xcode Command Line Tools.
It's unfortunate that pip lacks a way to communicate a warning (like noting some component won't be built) to the user more clearly. Have mentioned this to a few pip developers earlier this year.
Okay, thanks. After some digging it seems like this could be related to issues with the build in PyPy and/or c-blosc needs a separate CFFI-based wrapper (likely) to work with PyPy. I think it might be helpful to add a note about PyPy compatibility to the docs for anyone else scratching their heads about this.
Just to add we are disabling the C extensions for PyPy on these lines. This goes back to the first commit of Numcodecs ( https://github.com/zarr-developers/numcodecs/commit/4e9ee6b176dd343cc68697cc768bec512df5f848 ) and appears to have been originally disabled in Zarr with PR ( https://github.com/zarr-developers/zarr-python/pull/22 ).
https://github.com/zarr-developers/numcodecs/blob/b7c5a81407422b00a16b6eba3c63def335be51d5/setup.py#L348-L349
@alimanfoo, do you recall what issues you ran into when trying to build Numcodecs on PyPy?
FWIW I tried just dropping those lines and building with PyPy locally on macOS and it seemed to work. Submitted PR ( https://github.com/zarr-developers/numcodecs/pull/246 ) to make that change and add an item to the CI matrix.